HTTP over serial port? Experiment in Go

This is an experiment, nothing more than a proof of concept. It is not meant to be used in real projects (at least not in the form presented here).

Source code for this project is available on GitHub.

Problem statement

Provide extensible protocol for communication between two Linux devices over a serial port.

Introduction

HTTP

Most traffic between web browsers and backends is communicated over HTTP. It is a well-established standard, implemented in all major programming languages.

Go makes it very easy to write programs that use HTTP. Since it is such a nice protocol, let’s try to use it in something other than network communication.

Communication stack model

Communication stack is at the base of all network software. Operating systems provide TCP/IP stack, which can be used by applications, like web browsers, etc.

The OSI model defines seven layers of communication protocols. Usually, web browsers use the full protocol stack. They send HTTP requests over TLS over TCP over IP over Ethernet or WiFi.

In this experiment, we will explore what happens if we run HTTP not over TCP/IP, but over a serial port.

Serial port

In the basic form, a serial port is a point-to-point connection between two computers. There is no real network; one cable connects only two hosts. In Linux, serial ports are mounted as /dev/ttyXX, and on Windows as COM ports.

Because there are only two participants, there is no problem with addressing or routing.

A serial port, however, does not guarantee anything in terms of connection reliability. If some bits are flipped or lost, the serial port driver is not going to do anything about it. It will not even detect the problem (with some exceptions due to optional parity bits, but let’s not go into details).

Experiment

Prerequisites

Linux computer (or two). I used two ports on the same computer.
Physical serial ports connected with each other. They may be USB-UART converters or old-school RS-232 ports.
- Note down the serial port names on your system. You may need to modify them in the example code.
Code from examples repository.

The code

Source code for this project is available on GitHub.

We will be using the http package in Go.

First, let’s look at the server. We will use the http.Serve function. It needs a listener and a handler. A handler is the same thing that you would use in a regular HTTP server. In this example, I decided to use the standard mux.

Listener is the fun part. We will create one using the serial port. It needs to pretend that it can listen for incoming connections. On a serial port, we know that there is exactly one connection, and it is always connected. Our listener needs to return a Conn, which is something that can Read, Write, and provide some network-related information (we will mock this part). In the example, Port is both the listener and connection: port.go

With the port wrapped like this, the http.Serve will happily run, waiting for client requests.

Note that the Accept function needs to return only once, because there can be only one connection. Internally, http.Serve will do:

for {
	rw, err := l.Accept()
	...
	go c.serve(...)
}

After the connection is accepted, it will start a goroutine that will handle the client.

We cannot let it accept more than one connection, hence the endless time.Sleep in the Accept function.

Now it is time for the client. The interface is not that obvious this time, as it needs the RoundTripper interface, which is not trivial to implement. Luckily, there is a default implementation that we can tweak a little bit to make it use our Port object.

Client requests involve some error checking, which we wrap in the read function. Client methods need URLs to work, but since our Port cannot connect to anything other than the other side of the cable, the hostname part of the URL does not matter (but it must be set to something).

Running it

To run the example, you will need two terminals: one for the server and one for the client. Start the server first (it will enter its infinite loop) and then the client. The client will do its work and exit. The server will remain running.

It comes as no surprise that the setup is working. If you look at what is actually sent over the serial port, you will see HTTP frames containing control segments and actual data.

Breaking it

This part is not included in the example. You will need to modify the code yourself.

Serial ports are not reliable. Try to simulate the situation when some bytes are sent incorrectly (the easiest way to do it is to modify byte slices in the Write function of the Port).

If you run it like this, everything falls apart. HTTP reports errors or returns wrong results. It does not try to recover from that; it does not even retry the operation. Since it is supposed to run over TCP, it just assumes that the connection is reliable.

Summary

Good things

It is not that hard to use a serial port as a communication channel for HTTP. It is possible because Go internally uses pretty simple interfaces, and the language lets you do such crazy things as long as the types match.

This is a very good thing: you do not need to worry about marshaling messages and choosing which function to call on the server. You can use a well-tested standard library to handle the boring stuff for you.

Bad things

The example is too good to be true.

The HTTP communication is only as reliable as the underlying layer. HTTP does not add any more reliability because it does not really have to. Normally, it is the TCP layer that makes sure that bytes go where they are supposed to go. The TCP/IP stack checks for errors, verifies checksums, and retries if needed. HTTP just builds on top of that.

Conclusion

I wrote earlier that this example cannot be used in a real project. Not as it is now. It could be improved if some more work was put into the reliability of the connection.

Remember the problem that we are trying to solve. What alternatives do we have? I would explore these two approaches:

Implement your own protocol stack. Sure, you can do that. Define what is a frame, start and stop conditions, handle length, error checking, maybe even encryption. Then you need to decide how to handle different resources: probably by defining commands in the protocol. To do that, you need to write encoders, decoders, marshallers, and unmarshallers. Sounds like a lot of code to write. And even more testing.
Choose a lightweight protocol, made specifically to work over a serial port. From the top of my head, I can give MAVLink as an example. Protocols like this are not very popular, so this means that you would need to learn a little.

But there is a third way: use the HTTP approach, but provide reliability to the serial port. This would mean that you still would have to write a protocol, but only lower layers, mostly for integrity checking, retries, etc. Basically, you would have to provide reliability. I would look at how TCP does that and clone that to my solution. As a bonus, you would be only a few steps away from adding TLS to the stack.