Dr Glendarme, or How I Learned to stop Kerberos and Love Factotum

I used to work at the French National Computer Crime Unit. The actual official logo is this one. It means something like "Digital Crime Fighting Center," and the rest is… the 9front style logo we used to do.

One of our friends spoke Russian, and he retro-engineered this acronym to mean something like "The Soviet of Whimsical Computer Scientists," or something like that, and it was quite nice.

And so what we had to do is manage a lot of forensics, digital forensics activity. So you had a few dozen investigators. And they would all be part of different investigations at the same time, filling in different roles. Some of them would be the lead investigator. Some of them would be like the experts in one precise piece of technology, for example, cryptography wallets on Windows. And they will fill different roles on different investigations, but at the same time, using sometimes the same machine, sometimes some servers, and so on.

So it was a huge mess, and we wanted to make sure that there could be no leak. What I mean by that is… It's about data privacy, but also about legal proceedings. We cannot have a piece of one case leak into another case. This would jeopardize both cases. So we have to be very careful.

And because we did not have a lot of money, we had a very small number of powerful servers, but every big computation was on those servers, so we needed to be very careful with isolation of processing.

And this is an example workload we have to deal with. Can you read at the back of the room? Is it big enough for you to read? Yeah? Thank you.

So you go to the suspect house, and you seize the hard drive, and you have, from this moment on, you have 24 hours to find solid evidence.

Because in 24 hours, you have to put him in front of a judge, and he will have an attorney, and you will have to provide actual evidence that this guy is a bad guy. And so the clock is running.

So what you do is you stream the hard drive. to a small binary that is going to, at the same time, dump the stream into another hard drive to make a copy, compute the hash of the whole hard drive. I will explain later why. And also search some specific things, like magic numbers for JPEG images, high entropy region, which are probably encrypted regions, and so on. And this is done in a single machine, in a single quite powerful machine, so that we can make sure that the link between the hard drive and the CPU is full all the time. And so we don't spend a lot of time copying the hard drive.

Nowadays it takes a few hours to copy a full hard drive sector by sector, and it's quite long when you only have 24 hours to find the evidence you need.

Then the copy is made. The images that have been carved are given to an image analyst, but once the copy is made you can mount it and you have access to the file system, which will also give you the images, not all of them because some of them have been deleted, which you would have found by carving, but anyway.

And the image analyst can work with that, and you can extract for example geolocation data, and your geographical information system analyst can use the data, along with for example data from the cell phone, to cross-reference it and whatever.

Once you find some encryption you can go to your big cluster, but the big cluster is also a host for very confidential information, so whatever you put in cannot go out without a form, like a real-life paper form.

And so we have that kind of complex workflow, and we have multiple workflows like this running simultaneously on our IT system.

And before we came in, what we used to do was literally pass the data along from office to office on the hard drive. It was very inefficient, quite costly, and frankly not very good for the chain of custody of digital evidence.

And what is good with 9P is that you can put the host and network boundary wherever you like, and you can just code your stuff as if, like you test it locally on actual files, and then you run it. And maybe the file is on the other side of the planet, maybe it's in this hard drive, you don't care, and it works the same. And this is very beautiful, it's a beautiful abstraction that we all like. And so we worked on putting all of this workflow through file system calls.

And for that, we needed a way to make sure that identity, like who is running what, was a somewhat consistent notion. And we ran into Unix problems like UID being numbers. So if you create Alice and Bob on one machine, and then Bob and Alice on the other, the UID will be switched. And that creates a whole lot of problems.

So you can have UID translations with NFS, but it was overly complex, and it was not good. So by that time, I had read the Plan9 papers, and I was like, okay, they solved the problem like years ago, it should be easier to do that now. Why are we dealing with this?

So let's talk about what we did. From the introduction tool that we did, I know that you are quite familiar with this. You are familiar with Plan9 and 9P. And again, this is the first time I talk to an audience that knows 9P. So this should not be new to you. This is what Plan9 from user space does right now, like without our work on top of that. So basically, you have Alice. She has a machine called Atlantis, and she's running a process called foo. And this process is going to access some files. So far, so good. It would be interesting if there were just files. They are located on Bob's machine. Bob's machine is called Bermuda, and the files are, in fact, here.

So the first thing that we want is that Alice on Atlantis and Alice on Bermuda are the same person. So we need a central authority to tell both machines who is who.

This is usually done with LDAP or on Windows with Active Directory, but both are hard to configure and are hard to secure. And you can tell Factotum about who is who. And so this is what we want to reproduce.

This Plan9 from user space tool called srv is going to create a Unix socket. So this is, I don't know if you can read, it's a folder in /tmp. It's the emulation on Unix of namespaces in Plan9. And so you will create a socket in it, and Mount is going to talk to this socket. The system call related to files is going to be translated into 9p. And this 9p is going to be sent to this socket. And srv is going to send this across the network to the other machine. And on the other machine, 9p serve, which is, again, a piece of code from Plan9 from user space, is going to listen to this 9p stream. And if you have on this picture, you have only one client. But if you have multiple clients, 9p serve, and we will see how. It's going to streamline, to multiplex them into one connection. And this stream is going to go into a 9p server.

We have put a placeholder here it's just u9fs which is a basic file system server but, as you remember our workflow: we had multiple servers but this is just a placeholder any 9P server would do.

And so there is a missing piece. The missing piece is over here is how do we make sure Alice is actually Alice and how do we launch u9fs as Alice ? The color of whatever you see is tied with the owner so Bob is in red and Alice is in blue and you can see that u9fs is in blue this means that from the point of view of Bermuda's kernel this process is owned by Alice.

Why do we want that ? Because we want this process to be what we call security agnostic. Thanks to 9p it's already network agnostic so you can program your your stuff without caring wether a piece of file is in your machine or somewhere else but we want also your piece of code not to deal with identity and access management and that kind of stuff the kernel should do that for you.

This piece of code is actually a process owned by Alice so we need Bermuda to be aware of who Alice is and to have a way to check Alice's identity and we didn't find that in plan 9 from user space so we created vrs.

The name is just a mirror of srv there is nothing to it. so basically what vrs does is listen to 9pserve and once it sees a Tauth message, a 9P authentication message, it's going to talk to an authoritative factotum and then Alice's factotum and the authoritative factotum are going to talk to one another until this one is satisfied with all the answers and they will give the green light to vrs and then only vrs is gonna fork and create this this server and vrs is gonna drop all administrative privileges before the server is launched so this this program should not have any elevated privileges and then it could even if it's pwned by an attacker it shouldn't be able to do much damage so this was a goal.

Do you have any question before I go further this is perfectly clear for all of you. This is wonderful I took months to explain to my interns.

[fumbling with the zoom for ages]

So one thing I forgot to mention is that the AFID and so on are capabilities on the 9P.

Once you have opened the file, for example, for writing, you can give this FID to whomever. And if they send the correctly crafted 9P message with this FID in it, they will be able to write to the file, which is very nice. We want to be able to do that. But if you do that on a style network without encryption, you're going to get burned. So we used STunnel, which is a nice little utility with which you can use TLS on any connection. It's a tunneling stuff.

It's quite good. And before we switched to 9P, we used SSH everywhere. But it was complicated. I wrote in the paper why we switched.

Yeah, so if you want to use that today, I have some bad news. So basically, you need the PAM module. And then you need to tell NSS where the PAM module is and how to configure it. And also you need a modified version of factotum because to let the kernel know about which users exist and what the UIDs are and so on.

What we did was modify factotum in such a way that when you mount it, you have like basically an ATC password file, whatever, something like that, or maybe a group file as well. And then you point NSS to those mounted files. The kernel, without any further coding, will know what you're talking about.

So we have a modified version of factotum. And then we have vrs proper. The problem is, this modified version of factotum is basically a fork of Plan9 from user space from 2019. And we have not kept up with the ongoing development. So it's basically obsolete.

And also, for the life of me, I couldn't compile the PAM module on my current distro. So it's a bit problematic.

But effort is ongoing. So yeah, last time I saw the system working was when I was at the computer crime unit. It was in late 2020 on a Debian stable.

Debian is not moving very fast, to say the least. So it should work now. But yeah, so the code is online, but it's a kind of bit of a mess. Basically, it's like a stripped up plan9port repo without even proper attribution. I'm sorry about that. I'm going to correct that soon. But yeah. But yeah, it's difficult to work with. So why am I doing all this? So let me back up.

So you all know about the trusting trust attack where you put a backdoor in the compiler and every time you try to compile the compiler, it will put the backdoor in the compiler and so on. And what this is, it's basically all the binary that you need in order to compile GCC. You can see that the version of GCC is quite ancient, but this is only 60 megabytes. It's a lot of binary to analyze, but it's quite low if you think about it.

[Question from IRC]. What has actually changed in Factotum?

Yeah, the only change we made is when you mount it, when you mount–

[IRC] So you said that your version was obsolete, so what's changed that's made it obsolete, I guess, is the question.

I just, if you try to compile it, if I just, I did some effort to compile it on a new version and I just copied old code to the new Factotum. I guess some patch along the two years just broke things and I didn't repair them. But it's not a big deal, yeah.

Yeah, anyway, so this is only 60 megabytes of binary code and their objective is to go down to 512 bytes only. There is a project called Stage Zero whose goal is to uncruft this binary blob and from that on bootstrap all the GNU tool chain.

So this is a crazy effort and they are actually quite close to making it. So this is very exciting to see and every, like in 2019 and 2020, they put up a blog post saying, okay, we've cut up from 250 megabytes to 60 and so on. So if you are interested in that kind of stuff, I will really encourage you to take a look. It's a, on the blog of the GNU Guix project, you will see that kind of stuff. And yeah, am I running out of time? What time is it? No? No? Okay. Cool, thanks. 20 minutes. Okay, perfect.

[Audience] That was in 95 and then I stopped using GCC

What year, what year was that?

[Audience] 95, and after that…

So this is a good point. So this effort is actually good for software quality. I know that the suckless community at large and the GNU project have antinomic, antithetic goals.

So the XZ compression system needs GCC but GCC needs sed and sed needs XZ. So they were using like sed 1, GNU sed, not the actual sed, from 1993 in their project. And so they complained to the maintainer and they said, okay, this is ridiculous. We need, please just stop giving the source as XZ. At least give us a source as GZ or whatever. And they actually did it.

So the upstream changed the way they worked to make it more boostratable. So they did like an almost 30-year leap forward. So they care about that and they are able to make other people care as well. So this is a worthwhile effort and I think it should speak to you because you strive for software simplicity and so on. So this is actually a step in the right direction.

So why did we – am I talking to you about GNU Guix? It's because what GNU Guix wants to do is reproducible science. You said you worked in supercomputers. The guy who started the project actually works in supercomputers as well. His job is to make sure the bioengineering, bioscience stuff runs the same every time it's run and so on. So he wants to do reproducible package management and so on. So he based his work on Nix. Maybe you know about Nix. And Guix is basically guile, the GNU Guile language plus Nix.

And what we wanted to do was computer forensics but reproducible as well. I told you that when we copy a hard drive, we make a hash of the content of the hard drive. That is because the defense attorney can go and say, "Okay, this is not the actual hard drive for my client." And so what we do is we take out the evidence bag and say, "Okay, this is your hard drive. This is the hash of the data we used. If you hash this hard drive, you will see that it's the same hash. So bit by bit, it's exactly the same data."

What we wanted to add was not only is this the same data, but it's also the same software. And GNU Guix this is basically a Merkle DAG, DAG is directly acyclic graph, it's a graph you cannot have a cycle because otherwise you depend on something that depends on you or whatever.

But Merkle means that, this is just a friendly name, but the real name of this is actually a huge hash digest, which depends on every single bit of the source code. And so everything that depends on this package, if you change one bit here, it will change the hash of this one, so its name will change, so the name of this one will change as well, you see what I mean?

There is a cascading change of hash, and so when you hold a package, you hold not only the package, but all the dependencies of the package, and the build dependencies as well on everything.

So if you change one bit anywhere, you change the package and you cannot, maybe you don't know what it does, but at least you know that you've been, something fishy is going on. So this is quite nice.

If you want to do reproducible forensics, because then you can give the judge and the defense attorney, this is the data, this is the software we used to analyze the data, this is the hashes, you can run it again, you will see that the output is exactly the same, and now we can talk about the source code of what we did, and we can see if what we did is actually conclusive evidence, but at least we can talk about the meat of the matter and not care about handling of evidence and technical bits of which version were you using. And usually when there is a trial, the analysis was done four years ago, and you don't even remember what you were doing that day, so this was important for us.

But the project was put on, the 9P project was put on hold because we switched our efforts to reproducible forensics, thanks to Guix. Also, I got out of there and switched jobs, and also we finally got the hardware we needed, and every investigator got a very powerful computer, and we didn't need any more to have a single forensics computer to share, so it kind of failed by the wayside, but still, good news: