Upgrading A Mail Server From Debian Bookworm to Trixie Link to heading

A few days ago I upgraded my mail server stack -— Postfix, Dovecot, Rspamd, OpenDKIM, PostfixAdmin, and Roundcube —- from Debian Bookworm to Debian Trixie. Overall, the upgrade was successful, but it wasn’t without a few bumps. I thought it might be useful to record the process, the issues I ran into, and some lessons learned.

Preparing the Test Environment Link to heading

The first thing I do when I think of upgrading a server is making an inventory of what is running on that server. Generally speaking, it’s one server per task. The authorative DNS server runs nothing but that. The web server runs nothing but websites. The mail server runs only mail. Well.. It runs a recursive DNS server to facilitate lookups for RBL, and for such things as rDNS, SPF, DKIM and DMARC checking. And – we’re not done yet – it runs a webserver for the webmail (Roundcube).

So, an inventory is needed, to find out what software is on the server, and which release notes need to be checked, besides the Debian Trixie release notes.

Having prepared all that, I moved on to create a snapshot of the production mail server, and provisioned a new test server from it. One does not simply upgrade a server, and so one does not start with simply upgrading a production server to see what will happen.

This approach is safer than attempting the upgrade directly on production and gives a reliable baseline for testing. Using a snapshot of the production server essentially gives me a live copy of that server to experiment on.

This strategy has several advantages: I can make mistakes without consequence, I can test different upgrade approaches, and I can see the impact of service restarts and configuration changes in a controlled setting. It’s actually rather fun to mess around a server in such a care-free way!

Despite this nice sounding intro on preparation, I did actually not review the Debian Trixie release notes thoroughly enough ahead of time. This caused a few surprises. It’s tempting to skim these documents, especially when you’re confident about your system, but small changes in a distribution upgrade can have big consequences, particularly on boot-critical services or heavily customized setups.

Encrypted Disk Unlocking Link to heading

The first hiccup appeared when, after the upgrade from Debian 12 to Debian 13, I booted the Trixie test server. An encrypted disk failed to unlock. A quick search revealed that systemd-cryptsetup is required since Debian Trixie (see the Trixie Release Notes at 5.1.10), and it was missing.

Fix it by running

apt install systemd-cryptsetup

after the upgrade to Trixie, but before rebooting.

Once installed, the disk could be unlocked without further issues.

Dovecot 2.3 → 2.4 Upgrade Link to heading

The Dovecot upgrade was expected to be the most involved part of the process. Moving from 2.3 to 2.4 requires rewriting parts of the configuration. It’s surprising that they didn’t change the major version number, since the changes most certainly break backwards compatibility.

One unpleasant surprise was that the Debian installer halted with an error because the dovecot.service failed to restart, due to errors in the configuration. I had selected the option to restart services automatically during the upgrade, and the halted upgrade left me unsure whether the upgrade had completed successfully.

Nevertheless, I decided to try to get Dovecot running, so that I would at least have the correct configuration.

A couple of the changes from Dovecot 2.3 to Dovecot 2.4 are:

  • The configuration syntax has been changed.

  • The plugin { ... } section no longer exists. Plugin settings are global the same as all other settings.

  • mail_location has been split into component directives.

  • SSL certificate directives have changed (ssl_cert → ssl_server_cert_file).

  • userdb and passdb must now be defined separately.

More details can be found on the Dovecot website.

The documentation is not quite perfect yet, and it took me a while to figure out all the new configuration settings. I eventually approached this incrementally, trying to start Dovecot by running systemctl restart dovecot.service, and checking the journal (journalctl -xeu dovecot.service) in case of error messages.

Another useful command is, of course, dovecot -n, to check my configuration and look for such things as missing braces (really? Yes!).

After having gotten the Dovecot 2.4 configuration right, I backed up the Dovecot files to my local PC (or is it: I backed the Dovecot files up to my local PC? English is not my native language.. Pray forgive me if I made a mistake).

I took note that there was some sort of PHP issue, as both PostfixAdmin and Roundcube had errors that looked like a PHP issue. However, that was going to be dealt with on a new attempt.

Next, I proceeded to re-provision the VPS from the snapshot and started all over again (remember the interrupted OS upgrade?). This time I stopped the dovecot.service and copied the new Dovecot configuration files to the test server before the upgrade to Trixie. Then I ran the OS upgrade – and remembered to install systemd-cryptsetup before the reboot. This time the encrypted drive came up flawlessly, and Dovecot was happy.

Hooray for test systems!

PHP 8.2 → 8.4 Transition Link to heading

PHP 8.4 was relatively straightforward. The upgrade process handled most modules automatically, but the web server was not automatically reconfigured to communicate with the PHP 8.4 FPM server. The upgrade output noted this, but in a long list of services and modules and “stuff” it’s easy to miss:

a2enconf php8.4-fpm

I am used from Debian that it enables and activates anything you install – the php-fpm.service was automatically started and enabled upon upgrading. However, this last step had to be done, and it is easy to miss it in the spew of output from a full distro upgrade.

Post-Upgrade Verification Link to heading

Once the major components were addressed, I verified that everything was working with systemctl and log files.

I also sent a couple of test mails to check deliverability and mail headers, as well as the logs on both the sending and receiving servers.

The rest of the stack – Postfix, Rspamd, PostfixAdmin, and Roundcube -— now came up without further issues. Logs were normal, and test messages were delivered correctly.

Monitoring will continue over the coming days and weeks, to catch any subtle issues that might arise still.

Lessons Learned Link to heading

  • Read release notes carefully. Even minor changes, like systemd-cryptsetup being required for encrypted disks, can interrupt a boot.

  • Test on a snapshot or clone of the production server first. This prevents accidental downtime.

  • Be cautious with automatic service restarts. They don’t always succeed, and failures can halt the upgrade process.

  • Verify critical modules and configurations manually. Automation is useful, but it is not infallible. Expectations are a bit of a trap.

  • Incremental validation is safer than wholesale changes. Step-by-step testing avoids cascading failures and makes troubleshooting easier.

In the end, the server is stable, all services are functioning as expected, and the mail stack is ready for another few years of service. Now, over to the next server!