Bodging a web server with number 8 wire and hope

Kyle Martineau-McFarlane
25 January 2023

Introduction

In 1996, we got our first PC---Windows 95, beige everything, and a 28.8kbps modem which blocked up out phone line so much my father probably wished he'd never bought the thing. It didn't take me long to create my first website, or my second, or my third, as was the way of bored lonely thirteen-year-olds in the mid nineties.

I turned this proto-webmastering into something almost resembling a job; first re-writing my high school's web site over one summer, then doing odds and ends for various businesses and organisations in South Auckland over the years. But in 2008, when I first discovered the Django framework, I fell in love with the idea of web development all over again. Its clean separation of models, views and templating were a balm to my early-2000s-PHP spaghetti, and it captured my imagination in ways that Ruby on Rails never quite managed.

While I've always had a basic understanding of what's going on under the hood (I was the sad sort of performative nerd who would telnet to an email server and run POP commands to read email), I've never really interacted with the world of socket development, parsing or the actual machine I'm using: my world has been too high-level and abstract, especially since I got into the world of cloud engineering.

That's changing since I acquired a Raspberry Pi Pico W - a microcontroller with a range of I/O pins and assorted useful things... like a WiFi chip! I want to make my own NFC-powered audio player so my children can listen to home-recorded stories whenever they want. Part of this solution involves a web server running on the player itself for management purposes (uploading songs and programming NFC tokens). As a Python guy, I'm using MicroPython as the runtime on my Pico. I tried using both the Flask and then lightweight Bottle.py frameworks, but neither would run on the MicroPython stack successfully.

But surely that's half the fun of such a project as this! We have access to a network, we can program sockets, and we can parse text. What more do we need to write a really basic web server? Let's get started after this short caveat.

Caveat: this is possibly the worst excuse for a web server. Don't use it for your business. Don't connect it to the internet. Don't do anything important with it. Take it as a learning exercise and please don't judge me too harshly.

Goals

My goals for this web server framework are fairly simple:

  1. It will just serve plain old HTTP traffic on port 80
  2. I want to be able to plug in functions to serve different methods and endpoints with decorators like all the cool frameworks
  3. I want it to run on MicroPython without any external dependencies beyond the Python standard library

A basic structure

Let's start with something simple. A class to represent our server, and the necessary commands to instantiate it:

class Server:
    def __init__(self, address, port):
        self.address = address
        self.port = port

    def serve(self):
        pass

server = Server("0.0.0.0", 8080)

if __name__ == "__main__":
    server.serve()

We've taken the first step: we have some running code. It doesn't do anything yet, but we are on the way. The next step is to interact with sockets and thereby connect to the machine's networking stack. Ooh!

Sockets

For the purposes of this project, sockets and ports are the same thing. We want to tell the OS that we want to open a socket for a particular network interface / port combination. Once we've done this, we want to listen for any traffic on this socket and respond with traffic of our own. In Python, we use the sockets library to do this.

Start by adding an import sockets line at the top of the file, then modify the serve() method:

def serve(self):
    self.s = socket.socket()
    self.s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    addr = socket.getaddrinfo(self.address, self.port)[0][-1]
    self.s.bind(addr)
    self.s.listen(5)
    print(f"Listening to {self.address}:{self.port}")

    message = (
        "HTTP/1.0 200\r\nContent-Type: text/plain\r\n\r\nHello, world.".encode(
            "utf-8"
        )
    )
    while True:
        try:
            self.cl, addr = self.s.accept()
            with self.cl as cl:
                request = cl.recv(1024)
                request = request.decode("utf-8")
                print(request)
                cl.send(message)
        except Exception as e:
            print(e)
            self.s.close()
            raise SystemExit(1)

If we run this, we can curl http://localhost:8080 and get a "Hello world." message back! We just served some content over HTTP.

So what's going on?

  • We start by creating a socket by calling socket.socket(). This gives us a socket we can use. We set some options on this socket (self.s.setsockopt()) which tell the OS that we don't want to reuse this socket in the future. Not including this option means that if we stop our application and try to re-open it, we'll see an "Address in use" error until the OS decides we aren't coming back after all.
  • We then get an address (socket.getaddrinfo()) which includes the network interface and port to which we wish to bind. Once we have done this, we bind the new socket to this address (self.s.bind()) and state that we want to be a server socket with a maximum of 5 connections waiting at any one time (self.s.listen(5)).
  • We define an HTTP-compatible string to return in the variable message. HTTP minimally includes the protocol and version (HTTP/1.0), the status (200), headers (in this case, just Content-Type) and a body. We'll make sure we send some other helpful headers later on.
  • We then enter a loop and wait for a connection to our socket (self.s.accept()). When a connection comes through, we are given a client socket (self.cl) and an address (addr) - the latter is the IP and port from which the request has been made, the former is a client socket which is connected to that other side. The server socket then returns to its previous job of listening for new connections, and does nothing further with this connection; it's all been handed off to the new client socket.
  • We receive up to 1024 bytes from the other side (cl.revc(1024)) and decode the bytes to give us the request, which we print.
  • We then send the pre-defined string (cl.send(message)) to the other end of the connection.
  • In the event of any errors, we close the server socket and exit the application.

Conclusion

We've written a very simple, non-compliant thing that looks like a web server. We've looked at how sockets work and how Python interacts with them, and we've successfully used curl to interact with our quasi-webserver.

Next time: extending our server to do horrid things with requests and returning dynamic data to our clients!