undertakingyou

I had a contract to update three linux servers at various locations for my client. Each server presented a different set of issues.

Ubuntu has really made these updates easy with the do-release-upgrade command, part of the update-manager-core package. Running the command does pretty well everything for you, and all you have to do is follow a couple of on-screen instructions.

The first server to update was a public webserver. It sits in the datacenter at Xmission, and houses several websites, a database server, and some dns services. This was on ubuntu 8.04, and I was migrating it to 10.04. I logged in from my house via SSH, ran the do-release-upgrade command and we were off to the races. I rebooted afterwards, and waited about 5 minutes. Sure enough I can even SSH back into the box. Then I realize that the upgrade removed my MTA and Database server as part of cleaning up unused packages. Really seems silly to me that is would do that. Thankfully just installing the database was enough, all the databases and users, etc. were still there. Then a fight with apache to actually read the virtualhosts and serve them right. I don’t know what I changed, but it worked.

The second was a backup server onsite. It was at ubuntu 9.04, but with two release upgrades I was running strong on the LTS. Everything else just seemed to work, so that was an easy done.

Last machine was an internal webserver, running some intercompany webapps. It also served as a MTA for the office, allowing copy machines to send mail. It has been the real pain. Upgraded remotely via ssh and it never came back after the reboot. When I physically got to the machine to check it out it was saying something I have never seen before. Grub was complaining that it couldn’t find /dev/mapper/root-device and would drop me to a busybox shell. But I could see /dev/mapper/root-device from the shell. I fought with this for a while. After something like 100 restarts I got the bright idea and just typed ‘exit’ at the busybox shell. Guess what!!! The machine started fine! I have no idea what the machines damage is, but from now on I just type exit when restarting the machine. Thankfully the server doesn’t restart very often, in fact, this update was the first time in 1 1/2 years that I have restarted it. But my trial was not over then. Last large issue was the /etc/network/interfaces file. It kept the old file, but then would not work right. Weirdest symptoms, I could get to the machine fine from 2 of our 5 subnets and the computer could not get to the outside world. I fought with it a while and finally just disabled one of the interfaces and renamed some of the alias’ and away we ran.

I guess the moral of the story is, no server update ever goes smooth. Thankfully I am done for two years.