9P, the perfect protocol for IoT

When I say it's a disaster, I'm not talking from nowhere: ten years ago, I was a ‘cybercop’ at the National Cyber Unit in France. And in 2016, Mirai arrived. I don't know if you've heard of it?

Mirai was a big botnet that installed itself on printers, routers, cameras… and bombarded targets with non-legitimate packets. Just to give you an idea: Mirai's source code is available.

I wanted to show you this part of the Mirai source code that you can go and look at: there's a list of credentials, and it tries them one by one. There's ‘root:root’, ‘admin:admin’, ‘root:password’, ‘root:123456’, and so on. And just with 60 common login/password combinations, they've managed to create a botnet of several hundred thousand machines, capable of releasing 1 Tbit/s of DDoS power.

That's the logo of the group that found Mirai and brought it down a bit. But until 2018-2019, we still had Mirai descendants that were active. And if you think the situation has changed, think again. In fact, nothing has changed at all.

It's always possible, either through loopholes or default logins, to get into these machines. Well, it's a little better now: most connected objects arrive with a login that depends on the client, the serial number… but it's not much better either.

This is what a typical IoT architecture looks like.

As we have a very small machine, we're going to use a small kernel. So we start modifying the Linux kernel by removing certain components.

The problem is that from that point on, we're no longer in the mainline. So, generally, there are no automatic updates. When there's a kernel update, it doesn't apply to your thing. So you have an obsolete kernel.

The same goes for libc: glibc is several dozen or even hundreds of megabytes in size. So you don't use it. We use musl, which is smaller. But to get it to fit exactly, you mess with it. As a result, it can no longer be updated. So it's obsolete too.

You have default credentials. And the horrible thing about IoT is that you need a web interface at all costs. Since most people don't know how to use anything else, you need web, web, web.

So you have a web server. And this web server is generally preferred. And why is that? Because, since you like security, you've given it the right to flash the device's firmware. And to do that, you need to be root.

But there's only one possible web server, so it's privileged, and it contains application code often written with your feet, under pressure. The result: disasters like the ones we've seen.

The IoT is a pile of errors one on top of the other. Compromise one and a botnet can penetrate and turn your device into a zombie.

To help you understand how 9P works, there are three things you need to remember.

The first is that everything is a file. And when I say everything, I really mean everything. I'll show you this in a few moments during the demo, but first I'll try to explain it to you with my hands.

The second thing is that 9P is the protocol for transferring actions on files. Be it:

• locally - on a real file on a disk -,
• in IPC - between two processes, with ‘files’ that are actually interfaces -,
• or remotely - on another machine -, everything goes through 9P.

When you do an fopen, fread or fwrite, it generates 9P, which is sent to the process listening, or interpreted by the kernel, to manipulate the files.

The third thing is the perfect isolation of each process. The view that each process has of the file system is unique. If you start another process, it will have its own view of the file system.

And that's really important. Because by default, you have what is now called containerisation. But you've had it since the early 90s, on machines with very low resources.

Today, we deploy it on servers with Kubernetes, Docker and so on. But Docker doesn't run on a connected object! That's for big servers.

But here, it runs on machines with a few kilobytes of RAM, and it isolates the processes properly. It's really well done.

This is the 9P protocol.

The slide is ugly because it's unreadable, but my point is that you can fit the whole protocol on a single slide. You have what… a dozen messages. And each of these messages corresponds exactly to a file-related system call.

You want to open a file? That generates a Topen, and the return is Ropen. Want to read a file? It generates a Tread, and the return is Rread. And that's all there is to it. Once you know how to manipulate files, the protocol is relatively simple.

That's the reference. If you have this documentation in front of you and you've already used system calls for files, you can implement 9P in half a day. It's really not that difficult.

And there are implementations everywhere. In every language you can imagine.

• There are some in bare metal.
• They're in web browsers.
• They're all over the stack.

It's not some obscure protocol that has come out of nowhere. It is even well integrated into the Linux kernel.

For example, you have v9fs, a 9P driver. With a basic standard Linux, you can mount 9P servers. When I say ‘mount’, it's like a USB stick. A 9P server, because it's files, you do a mount, and your files appear as if you've just plugged in a USB stick.

But in reality, you're talking to a server on the other side of the world.

Windows Subsystem for Linux? It uses 9P to display Windows files in Linux. QEMU, the virtualiser? It also uses 9P to pass files between the host and the guest.

There's 9P everywhere. But you can't see it, because it's a systems thing.

But it can be used in the IoT.

The idea is not to have a big house that collapses as soon as you remove a brick.

It's about having a field of menhirs: each process is in its own namespace. If a hacker ever manages to get into a process because of a flaw - for example, remote code execution - well, he won't be able to do much. Because he's completely isolated.

Maybe he won't even have access to the network - that's a trick I'll show you later. It won't have access to the files. And above all, it won't have access to the other processes.

So the impact of a compromise is very limited, unlike a privileged web server where, as soon as there's a remote code execution, the whole machine is lost.

And once again, this system was developed in 1987. That's the year I was born. So yes, we're way ahead of the game. But that's Bell Labs.

This is the Linux version from 20 years ago. Back then, you could still cat a .wav file on /dev/dsp, and it would come out on the speakers. It wasn't bad.

Input/output was text: stdin, stdout. It didn't work so bad.

Joysticks? To be honest, on Linux, it worked pretty well. You can literally read a file, and you can see which buttons are being pressed. If you want to make a little game, it's a great interface.

You've got /proc, which has been imported directly from Plan 9, and which lets you see processes, debug them, and so on. You have access to memory, block devices…

Some things don't work so bad.

Sorry. I mean, it started drinking.

Something called ioctl has appeared. I don't know if it rings a bell… but basically, you can read files - ok. But before you can read them, you have to configure them. And ioctl is the garbage system call where you put everything.

Do you want to play a video? OK, then. But first you need to make an ioctl to specify the resolution, framerate, etc.

Want to access the sound? Well, /dev/dsp worked with OSS 20 years ago, but now it's ALSA, PulseAudio, I don't know what… and ioctl is everywhere, because nobody wants to make simple files any more.

The mouse? That was easy… except that there are 27 of them. Do an ls /dev/input on your Linux, and see what it looks like. You can't tell which is which.

So, to ‘fix’ the problem: → we read all the mice at the same time, → but in fact none at a time.

It's all nonsense.

And it doesn't stop there.

Now we're even iouring, because reading/writing files was too simple. So we create circular buffers like in the 70s. And we use mmap, because read is too rustic. And we end up having to map files in memory, and produce unreadable code.

In short, it's a mess.

You have to read man pages for five years. You have to go and look at the source code to understand. Developing under Linux with ‘everything is a file’ has become a huge joke.

Everything is a file. Absolutely everything. And it's really a file in the sense that you can read and write.

Here are a few examples:

/dev/pointer

you read it, you have the mouse position. Do you write in it? The mouse moves.

/dev/draw

you read it, you have the pixels. You write, you draw the pixels.

/dev/audio

you put a .wav file in, it plays the sound. Do you want to record? You read into it.

/dev/audioctl

replaces ioctl. There's sometimes a ctl next to it, for metadata (framerate, etc.), but it's text, documented for each device.

Not some obscure ioctl system call, with incomprehensible binary flags.

It's clean, simple, documented.

And among these files, if you want to do IoT, there are some you're going to like:

Serial ports

you can send bits directly over cable.

If you have an FPGA

you can talk to it live.

You have GPIOs

so on a Raspberry Pi, for example, you can control LEDs, relays, etc.

USB

is supported.

Touch screens too

you can tell whether it's a long press, a short press, a stylus, several fingers…

You've got it all here. This is the equivalent of stdin / stdout. If you want to know which user you are, you read /dev/user. If you want to change it (and you have privileges), you write to it.

In short, it works really well. It's clean, beautiful, clean.

(This demo is on my Inferno is Cool page) So, the demo.

I'm going to try something, hang on. I'm going to do an SSH demo.

I'll connect to my server to show you.

Well, I'm using my phone, so it's not guaranteed to work. Just give me a couple of minutes - thanks.

I'm attempting a demo on a 20-minute talk, which is a bit ambitious…

What's happening here: This is the client. This is the server.

The server is running at OVH. It runs Linux, which runs Inferno.

What I haven't told you yet is that Inferno isn't just an operating system. It's also an application.

So you can run it as an app. In fact, it emulates an OS. It's very light.

So right now, I have a small, cheap server at OVH, running Inferno.

On the screen, you see something like :

listen -v ‘tcp!*!5546’ { export /net& }

What does that mean?

It means: → we listen on port 5546, → any incoming connection is authenticated, → once authenticated, we run the export /net command.

What comes from the network goes to the standard input of this command. And what comes out of the standard output of this command goes back to the network.

So it's really easy to do network scripting with this.

What does export do?

It's used to export part of the file system via 9P.

Here, we're exporting /net, which is the network interface. When I tell you that everything is a file, that's really all: → There are no sockets on Inferno. → Want to open a connection to a site? → You write to /net/tcp/clone, and that's it.

So here I am on Inferno running on my local machine.

If I run a script I wrote, my-ip, I get the IP address of my phone (via 4G), something like 92.xxx.xxx.xxx.

If I do the same thing on the server, I get 51.178.xxx.xxx, the IP of OVH.

And now the cool thing: I mount the server's /net locally.

So I've taken the server's network file system and mounted it in /n/rdklein.fr on my machine.

There, in this directory, I have access to the remote server's /net.

I do :

sh -c { bind /n/rdklein.fr/net /net; myip }

And then, bam, I get the server's IP address: 51.178.xxx.xxx.

But if I restart myip without this trick, I come back to 92.xxx.xxx.xxx - my local IP.

Why do I do this? Because each process has its own view of the file system. Using sh -c, I started a new process. In this process, I binded /net to that of the server. Result: my script sees the remote IP.

So now, in two lines of code, I've created one VPN per process.

Just for comparison: WireGuard in the Linux kernel is 100,000 lines of code. With Inferno, we have a simple, elegant and powerful solution.

[I didn't have time to do the second demo] The point I wanted to make with Omegle :

The Web model is a big, powerful central server with stupid clients connecting to it.

The model proposed by 9P is completely different.

You can have a rendezvous point (like a DNS to find each other), but then Alice and Bob connect directly to each other.

And in Inferno, the notion of ‘server’ and ‘client’ becomes blurred. Anyone can be a server, anyone can be a client. You can create a distributed universe by assembling bits of files between machines.

And you can export part of your world to someone else, with :

• authentication
• encryption,
• access control.

So why am I talking to you about all this in the context of the IoT?

Because in IoT, we often use HTTP.

You either use a REST API or a web interface, but everything goes through HTTP. As a result, encryption becomes a transport problem, not an application problem. We use TLS (or SSL if we're late) at TCP level.

But authentication becomes an application problem. HTTP can't handle it properly.

Some people have said that we're going to do TLS authentication with client certificates… → In theory, it works. → In practice, it was never deployed.

So we deferred authentication in the app. And the security problems this poses are enormous.

In 9P, by contrast, authentication is built in from the outset.

When a client connects, it is already authenticated. As a result, the server launched for this client does not need to check anything. It is only launched if authentication has been successful.

So all the cryptography, login management, password management… is taken out of the application layer. And that strengthens security.

As for the rest, HTTP verbs (GET, POST, PUT, DELETE, etc.) are just file system calls.

• Read a file = GET.
• Write to a file = POST.
• Delete = DELETE.
• Go to an offset (seek), which is PATCH, difficult in HTTP but trivial in a file system.

There's no point in having hyper-powerful machines, all connected to the Internet, that hackers can take over.

On the contrary: make your machines very small, running Inferno, speaking 9P, on a serial port. If someone can hack into them, it's because they're in the room.

And at that point, if they want to open the window, they can just do it by hand. If they want to turn on the tap, they can turn it on. Physical access means game over.

So use small, short-distance protocols: → Bluetooth Low Energy, Zigbee, → serial, infrared…

Serial works great, and there's no reason to do anything else in 90% of cases. Here's a funny example: there's an experiment where they took Plan 9 and ran it on a LEGO Mindstorms. Literally, a plastic brick.

They managed to talk 9P in infrared between a computer and the LEGO. They set this up in an afternoon. And then, by writing files, you could control the LEGO robot's motors.

What would it look like in a kitchen?

You have temperature sensors and thermostats. You create a /dev/temp file. You read it → you get the temperature. You write to it → you define the target temperature.

You do this for each device. Then, you take a small Raspberry Pi, you mount these files: /mnt/frigo/temp, /mnt/four/temp.

And you build a graphical interface that talks to this.

The Raspberry Pi's Linux kernel takes care of access control. Once it's shielded, machines are dumb. They have no local access control. But since they're connected physically or via a very short radio network, that's good enough. → If someone is physically there, they can already turn the oven knob by hand. It's not a security issue.

So, you can compose any universe you want, with 9P. It's really not that difficult.