Adventures in Unix administration

I’ve been underground and off the blog for quite a while now. A lot of distractions have been going on, and the biggest project over my last couple of years has been under heavy non-disclosure, so there hasn’t been much to write about and even less time to write it in.

But one of the biggest distractions is on its way out. That’s right… the server is getting an upgrade! Yay!

Big Picture

For something like ten years, my old domain and after that the domain lived on a series of Intel-based towers running various Linux distributions. It worked pretty well, but it was a little resource-constrained–the original was a 33Mhz 486 with 4MB of RAM running on a nailed 28.8Kbps modem line–and it actually burned a lot of energy. I once calculated that simply keeping that server up was running about $10-15 a month in electricity. Huh.

Then my favorite person in the world got started on her doctoral dissertation. Her research required an online survey, so I registered the domain and looked for a web-based survey system. I didn’t feel like adding yet one more thing to my poor overloaded physical server, and I was looking for a chance to try out a virtual private server (VPS) instead.

I finally got smart and ordered a VPS from the nice people at John Companies and it worked GREAT from day zero. I quickly migrated my other domains and functions to the new server once proved out, and I’ve been thrilled with it.


My VPS is on an older physical host that doesn’t support recent versions of FreeBSD, and it’s basically out of disk space. My email has been running with about 20MB free for years now. It’s crazy.

So a few months ago I asked JohnCo tech support for a fresh VPS on a newer physical host. It runs the latest FreeBSD and has tons of disk space… for about half the price I had been paying. Brainless!

And now I’m finally migrating functionality from the old server to the new one.

First: email

Now my old setup was a continually-upgraded Sendmail (for SMTP) and imap-uw (for IMAP and POP). They worked fine, but sheesh. There’s a reason why the Sendmail “bat book” is among the physically largest books O’Reilly publishes. It is a crazy, overcomplicated, high-maintenance, Byzantine mess. It supports everything from UUCP to smoke signals. I’m not sure who really needs all that anymore.

So yay, it’s time for Postfix. It’s simple! It’s easy! It has a config file you can actually read! And I got that to work in essentially no time.

For an IMAP server, I searched around just a little and found Dovecot. Okay, don’t get me wrong, I like Dovecot’s security model and its sane configuration file syntax. And I finally understand Plugin Authentication Modules (PAM) well enough to regulate login security.

And we’re back to “But.”

But the documentation set for Dovecot 2 truly rots. Wait, no, I don’t mean that. I mean that it’s an extraordinarily obtuse problem-solving resource. There’s a nice spot on the Wiki on “What if it doesn’t work?” but it’s pretty close to empty, and it didn’t address the actual problem that I had.

Which was…

…the mail_uid and mail_gid variables that are absolutely required to be populated so Dovecot knows what user identity to assume when an IMAP or POP user comes to call.

It took me a late night of experimentation to focus down to this one observation: When a system user (i.e., someone in /etc/passwd) authenticates into IMAP using PAM, Dovecot doesn’t just automatically pull the uid and gid into its mail_uid and mail_gid variables.

Which led to the obvious question: By what mechanism does Dovecot get mail_uid and mail_gid from PAM?

This took another late night. There is really nothing in the documentation that spells it out. It talks about variables in the config files, but isn’t clear on whether you can use variables in mail_uid and mail_gid settings. It doesn’t have anything on PAM other than basically “hey, here are a couple of config lines that should work on Linux.” And the documentation doesn’t point to a resolution of the error message I actually got.

Rule of Thumb

“If I have to read your source code to solve a configuration problem, your documentation rots.”

So it’s a couple hours plowing through documentation to find where Dovecot pulls a uid and gid from the authentication source. It requires setting of an internal variable called MAIL_STORAGE_SERVICE_FLAG_USERDB_LOOKUP in a C function called mail_service_storage_lookup().

I couldn’t figure out how that flag got set, other than by tracing the call chain all the way up to main() and trying to set a -u flag on the command line. I never figured out why that didn’t work. I got so desperate I just hardcoded the MAIL_STORAGE_SERVICE_FLAG_USERDB_LOOKUP flag into the source and rebuilt.

Okay, that’s just completely wrong.

After that, it was easy to set up SpamAssassin (to flag incoming spammishness), Apache (regular web server), Subversion (to latch onto Apache as a version control engine), BIND9 (for DNS resolution), and MySQL (database).

I’m just about ready to cut all services over to the new host. Wish me luck.