The Big Upgrade
April 2023 (481 Words, 3 Minutes)
My VPS has been set up in 2018. Over these years it has accrued a couple dozens of different systems - my blog, web apps, databases, a handful of backends and so on - overall, in half a dozen different languages. Each one took some time to install, a lot of which was far from obvious (and most of which has been mostly forgotten).
Since Ubuntu 18.04 which the server was running has been out of active support for about a year, the time has come to upgrade. Upgrading the OS in place was out of the question - it has never worked great in the past, so trying it on a production box would be a risk. Which meant a new box would have to be set up with gradual migration of all systems. That process was going to take time and it would not be simple.
Since no forced upgrade should be wasted, I used it as an opportunity to try a couple of new things:
- Instead of Ubuntu family, I decided to switch to Rocky Linux. As much as I was looking forward to the upgrade, I don’t want to undergo this fun every 5 years, so with 10 years of support the choice was obvious. This was also exciting - my production Linux experience started with RedHat Linux 3.0.3 in 1995 and that distro was head and shoulders above the rest in terms of management and reproducibility. Since then, I used pretty much every RedHat release until they had stopped producing free distribution. So it felt good to try RedHat again.
- Everything on the current server was installed manually. I live by the principle principle, “if I have to do it more than twice - it has to be automated.” and I’ve been thinking about an excuse to try Ansible, so this was a great opportunity. So one of the goals was to automate the setup via Ansible as much as reasonably possible.
- Finally, I’ve been using WordPress for the blog for a long time but it just felt wrong. Instead of a rather opaque system, I wanted to have my content in cleartext and under version control. So, the third goal was to migrate the blog from WordPress to a static generator with Markdown support (like this or this or even this).
The whole process took several months but I am done now, with all 3 goals accomplished. Everything has been migrated successfully and is in production on the new box; the old box will be shut down in a couple of weeks.
In addition to the goals above, a number of valuable lessons have been learned:
- Automating everything from A to Z sounds noble but might not be practical. It would make the migration much longer. The decision was made to install the webapps manually but to automate everything else (OS setup, software installation and configuration, etc). Now that everything is running, I’ll look into creating Ansible playbooks for my software as well.
- Automating MySQL/MariaDB setup via Ansible is a real pain (I mean
mysql_secure_installation). I ended up effectively manually replicating all the steps of the script above as Ansible steps. - I genuinely hate SELinux. I really like and appreciate the purpose, but the subtle errors all around webapps with obscure errors and arcane lore required to deal with them is just not great. You never know where it will bite you: writing to website log - check, access to an innocuous file by Python code - check, ability to send mail by PHP - check! And so on and so forth. The latter problem was the last straw - after several futile attempts to allow PHP to send mail (some of which resulted in a hung Apache process) I’ve thrown the towel and turned SELinux off. I plan to revisit the topic in the future.
To be continued… In 10 years!