As I mentioned in my first blog post, most of my career has been spent in kernel space. I’ve of course done a ton of development work on Linux as a platform, but I started on the Linux kernel team with essentially zero knowledge of how the Linux kernel itself works. In case you’re curious why a career kernel engineer never bothered to look at the Linux source code: I spent the first part of my systems career as a kernel engineer working at a company that already had legal issues with the upstream GPL community. Looking at the Linux kernel source code, therefore, seemed like a really bad idea, as I wasn’t trying to end up in a courtroom accused of stealing intellectual property from a community of altruistic upstream contributors. Of course, like any other red blooded systems hacker, I still trawled LKML looking for the juiciest Linus rants…but I never checked out Linus’ tree, and I made it a point to never look closely at any patches.

So when I joined the Linux kernel team at Meta, I did so with admittedly close to zero Linux kernel experience. Several weeks into my stint, I finally feel like I know about 1% of what there is to know. It’s been an interesting and rewarding experience even as a senior engineer with plenty of kernel XP prior in my career. Therefore, I’m going to be writing a series of blog posts on my experience onboarding as a Linux kernel contributor so you can have a fun afternoon laughing at my mistakes and hopefully we’ll both learn something useful in the process.

In the first post of this series, I’ll describe the process for getting plugged into the upstream community. That is, getting your email setup where it needs to be, subscribing to various Linux mailing lists, and viewing patches. If that doesn’t sound like it warrants its own post…just wait.

Sending Emails

Unlike most open source projects in the modern age which use something like github, gitlab, etc., the Linux kernel community does all communication via old fashioned, plaintext encoded email. Yes, you read that right. It’s all done over email. And not just any email, it’s done over plain-text emails. Sending anything other than plain-text emails is out of the question, as the Linux kernel mailing lists (more on this below) will silently just drop any emails that don’t honor this requirement.

How do we send plaintext emails? There are two solutions:

1. Use a UI-based email client such as Thunderbird that supports sending plaintext emails.
2. Go back to your roots, and use a terminal-based email client such as mutt that lets you control the encoding with config options, and send emails that you compose in your editor of choice (vim for yours truly).

I personally really like being able to use vim to compose and view emails, so I decided to go with mutt (well, NeoMutt actually). Let me take you through how to set it up.

Setting up mutt

There are all sorts of guides online describing how to set up and configure mutt. Take a quick scroll through their online manual to see what I mean. My suggestion is to get set up with the bare minimum to send and receive emails, and then trawl through the manual to configure mutt to your liking (this is seriously worth it and will pay big dividends on your productivity long term). So to that end, let me share my favorite parts of the mutt setup that will let you get going with the upstream community. I’ll leave it to you to have fun configuring your own environment. Note that for this section, I’m assuming you’re comfortable navigating a terminal, and can use some terminal-based text editor like vim. If you have no experience using a text editor, type vimtutor in a shell to learn how to use one.

Getting mutt and dependencies

mutt is almost certainly shipped as a package on any distro you might be using. I won’t go into detail about how to download it. If you’re not sure how to interact with a package manager, you’re probably not ready to be sending patches to the Linux kernel upstream community.

You’ll also need gpg to securely store your email password in a plaintext mutt configuration file. gpg should similarly be shipped with any package manager you could possibly be using, so go ahead and download it if you don’t already have it. Once you’ve downloaded gpg, you should create a gpg key. To do that, you can follow the instructions for creating a gpg key on github.

Configuring mutt

At this point we’ve downloaded mutt and gpg, and we have a gpg key. The next step is to create a configuration file that mutt can read to open IMAP connections for reading mail, and SMTP connections for sending mail. Let’s start by creating a .mutt directory in $HOME:

$ mkdir -p ~/.mutt

Now, we can write our email password to an encrypted file that we’ll later read from mutt. You’ll want to do something like this:

# Put a space as the first character of the command
# to avoid your password being stored in .bash_history.
#
# Also, replace -r with whatever ID corresponds to your key, probably whatever
# email you used when you created it.
$  echo -n 'mypassword' | gpg --encrypt -r void@manifault.com > ~/.mutt/password.gpg

Great, so we now have an encrypted file ~/.mutt/password.gpg that contains your email password. If you want to see it, something like the following incantation will do the trick:

$ gpg-q --decrypt -r david@bytelab.codes ~/.mutt/password.gpg

Awesome, let’s move onto configuring mutt. When mutt is started, it looks for a configuration file in a few places. One of those places is ~/.mutt/muttrc, so that’s where we can put our configuration file. Your configuration file will probably look something like this:

# vim: syntax=bash

# Setting up mutt identity
set realname = "David Vernet"
set from = "david@bytelab.codes"

# Login and Passwords
#
# Note:
# To create a file with an encrypted password, use the following
# incantation:
# echo -n 'mypassword' | gpg --encrypt -r david@bytelab.codes > ~/.personal_configs/mutt/password.gpg

# IMAP / SMTP password.
set my_password = "`gpg --batch -q --decrypt -r david@bytelab.codes ~/.mutt/password.gpg`
set imap_user = "david@bytelab.codes"
set imap_pass = $my_password

# Point mutt to whatever SMTP server you're using.
set smtp_user = "dcvernet@bytelab.codes"
set smtp_url = "smtp://$smtp_user@smtp.gmail.com:587"
set smtp_pass = $my_password

# Point mutt to whatever IMAP server you're using.
set folder = "imaps://imap.gmail.com:993"
set spoolfile = "+INBOX"

# TLS all the way
set ssl_force_tls = yes

Make sure you’ve enabled IMAP on the server you specified in the folder variable. I clearly use Gmail for my IMAP and SMTP servers, but you can use whatever you’d like. Login by invoking $ mutt on the command line, and you should see your emails pop up looking something like this:

If you see your emails, congratulations, you’ve set up an email client that will let you interact with the upstream community. If you weren’t able to connect, try running mutt with -d <level> to run it with debug output (which it writes to ~/.muttdebug0).

Also, don’t worry if your terminal doesn’t look exactly like mine. I had a few of my colors pre-configured for this screenshot. I’ll do a separate post where I talk about how to massage mutt (or in my case, neomutt) so it fits like a glove.

Subscribe to upstream lists

Now that we have an email client up and running, we can subscribe to get notified of when contributors in the upstream community email patches (or just send any kind of general correspondence).

The place where the magic happens is called vger. Navigate to the site, and you’ll see…a website that looks like it was designed and built by a bright young middle-schooler in the early ’90s. I wish I could say this was an anomaly, but it’s not. Get used to this. People in the systems world generally do not take the time to “pretty things up”. You’ll feel fortunate if and when you find documentation, and when you do, it’s typically either a website that looks like this, or a plain-text file that was written by someone in vim who hasn’t taken their hands off their keyboard since they were old enough to drive.

Moving on, subscribers to all of the various kernel lists are managed by Majordomo. Majordomo literally dates back to a time when many people had access to email, but not a web browser. It is what’s known as a Mailing List Manager, and is basically a simple, ancient perl script that will add or remove your name from whatever list you tell it to (as long as you tell it to in plain-text!). If you navigate to the vger-lists page, you’ll see a large set of named lists. These lists correspond to the various subsystems and communities of the Linux kernel. There are clearly a lot, because the Linux kernel has a very rich ecosystem!

Now, you may be wondering why we have so many lists related to development of the Linux kernel if it’s all in a single source tree. Fully answering that is outside the scope of this post, but the TL;DR is that the actual day-to-day development of the Linux kernel happens in many different subsystems. Each has a list of maintainers who are responsible for reviewing code and sending Linus some sets of patches to be merged into the mainline kernel on some cadence. I’ll discuss this in more detail in a follow-on post. For now, interested readers can check out https://www.kernel.org/doc/html/latest/process/development-process.html for more information.

Anyway, let’s pick a list to subscribe to. The linux-kernel list is an option, but it’s extremely high traffic. The list info says it receives 200–300 messages a day, but in reality it’s actually in the thousands. I’m interested in live-patching (and have myself contributed a bit to upstream), so I’ll subscribe to the live-patching list. To do that, I have to send a plaintext encoded email to majordomo@vger.kernel.org, with a body that just says, “subscribe live-patching”. Let’s see what that looks like in neomutt (press ‘?’ to see a list of keybindings – you probably want ‘m’ to compose a message):

Once we send that off, we’ll see the following reply from Majordomo:

Pretty straightforward. It just wants us to send another email from our email address to make sure that we actually wanted to subscribe and that some troll somewhere didn’t sign up for us. Once we send the reply as requested, we’re successfully enrolled:

And now we can lurk and see what kinds of discussions are taking place regarding live-patching the Linux kernel at the top of tree. Pretty neat!

Closing Thoughts

We didn’t discuss how to actually send patches upstream (you don’t use mutt directly when sending a patch for review, but rather git-send-email). Nor did we discuss the kernel itself. However, we did learn how to plug ourselves into the community in a way that closely reflects the “interface” of a real kernel contributor.

We’re almost ready to start sending out patches! In my next post, I’ll pick up where we left off, and describe my experience in sending out my first patch (which ended up being accepted and merged into the mainline repo). In that post I’ll also write up my suggestions for where to start as a brand new contributor and get used to the upstream process.

An aside on the whole upstream communication system

While I had a bit of fun being glib at the expense of Majordomo and the system that’s in place for upstream communication and collaboration, I actually want to point out that the system works great for what it is, and does have some advantages:

1. This is the way things have been done for over twenty-five years, and the results speak for themselves. Linux is ridiculously pervasive, it’s free, and it’s run by a global community of contributors who work on separate subsystems that are housed in separate repositories. It’s actually kind of a miracle that it all works, and I could see the argument for there being no need to mess with something that’s been working well for decades.
2. While I’m not at all an authority on how this system was put in place, I have a feeling that the slightly higher barrier to entry is somewhat by design. Kernel work can be very challenging, and I think that requiring people to be able to set up a plain-text encoding email system and self-sufficiently read and discover how to be contributors, is on some axes correlated with how likely they are to know what they’re doing and send out good quality code.
3. In my opinion, the whole communication system is a bit arcane, but the Linux kernel documentation is pretty damn good. There’s also a lot of great content on https://linux-kernel-labs.github.io/refs/heads/master/index.html as well. So while it’s easy to dunk on the vger website for being poor, etc., I think the theme is that the Linux kernel community strongly prefers content over showmanship — which isn’t to say that there’s not a ton of room for better documentation — more on that in a follow-on post.
4. Perhaps most importantly, the people who set up this system are probably the ones who have contributed the most to the kernel over the years, and continue to have the deepest breadth of knowledge. Who the hell am I, or anyone else, to tell them what system they should or shouldn’t be using to continue to iterate on the global powerhouse of a free and open-source operating system they’ve built?

Thanks, and see you all next time!