As many of you may remember from my last upgrade, things usually don't go all that well for me, especially when I'm gambling with technology. A friend of mine at work has been encouraging me to upgrade to the latest flavor of Ubuntu, but given my past history, I've shyed away from the early adopter philosophy when it comes to Ubuntu server upgrades. Now that it has been over a month since the release of Hardy Heron 8.04 LTS, I decided to cross my fingers and dive head first into an upgrade on a Friday night.

On April 21, 2008, Ubuntu Linux 8.04 LTS (a.k.a. Hardy Heron) was released in both desktop and server editions. I'm not much of a desktop Linux kinda guy when it comes to servers, so all four of my servers run headless server versions. Prior to the upgrade, they were running Ubuntu Linux 7.10 (a.k.a. Gutsy Gibbon). That upgrade alone caused enough headaches, mostly due to the fact that I'm running RAID 1 on all my servers. You can read more about why in my nip tuck blog.

As I mentioned earlier, the "cmon man, upgrade" seed has been planted on several occasions by a friend at work. I felt particularly antsy tonight and figured why not just get it over with. My girlfriend was already out with her Sex and the City entourage, which left me all the time in the world to once again beat my head against the wall when things didn't work out. Without further delay, here we go.

The upgrade started at approximately 6:45PM on Friday, May 30, 2008. I had four servers to upgrade.

• Cerberus (router, firewall, jabber, external dns, proxy, vpn)
• Phobos (file storage, subversion, internal web services, internal dns)
• Arioch (mail services)
• Icepick (public web services)

I decided to start with Phobos, for no particular reason other than if I killed the box, I didn't care that much. I then planned to proceed in the order of Cerberus, Icepick, and Arioch Of all the configurations, Arioch and Icepick were the most complicated so I wanted to work out any issues/failures with the easy guys first. Each upgrade started with the following command that made my skin crawl:

sudo aptitude install update-manager-core 

Phobos Status:

• Started approximately 6:45PM, Friday, May 30, 2008
• Completed at 7:03PM, Friday, May 30, 2008
• Total upgrade time of 18 minutes
• Upgrade went smoothly and reboot was successful
• Slight problems with space on /boot, so I deleted some previous versions of kernvel files, etc.

Cerberus Status:

• Started approximately 8:35PM, Friday, May 30, 2008
• Completed at 9:00PM on Friday, May 30, 2008
• Total upgrade time of 25 minutes
• Upgrade went pretty smoothly and reboot was successful
• Slight problems with space on /boot, so I deleted some previous versions of kernel files, etc.
• There were some really wonky problems with my openswan setup after installing 8.04. I was previously using l2tpd, but 8.04 automatically replaces that xl2tpd. Fair enough, but the init script doesn't create the /var/run/xl2tpd directory. This causes xl2tpd to fail to start. Even after you create it, it gets deleted on reboot, so when you do reboot, it still fails to start. To resolve this, I went ahead and updated /etc/default/xl2tpd with DAEMON_OPTS="-C /var/run/l2tp-control". That took care of it.
• I started receiving the "Main Mode message received on my_ip_address:500 but no connection has been authorized" when connecting to my VPN. Strange, nothing really changed. After about 2 hours of debugging, I discovered that all the ipsec connections that I defined in ipsec.conf were not being loaded. It came down to having left=%defaultroute in the ipsec.conf. After changing it to left=my_ip_address, everything worked fine again. Another annoying upgrade problem.

Icepick Status:

• Started approximately 9:28PM, Friday, May 30, 2008
• Completed at 2:12AM, Friday, May 31, 2008
• Total upgrade time of 4 hours, 44 minutes
• Upgrade for the O/S went smooth, the web server components puked heavily
• Upon reboot, mongrel clusters were crashing. Turned out rubygems were all borked up. The gem list command would result in uninitialized constant Gem::GemRunner. Had to add the line require 'rubygems/gem_runner' to /usr/bin/gem.
• After fixing the gem issue, mongrel servers would load, but there were all kinds of rails issues. Since I'm using ferret, I have a drb server running. After the upgrade, the startup script I created was no longer working. You'll want to create a startup scrip tthat drops privileges, which can be kinda tricky. I'll include my script at the end of the blog.

Arioch Status:

• Started approximately 6:35PM, Monday, June 02, 2008
• Completed 7:00PMPM, Monday, June 02, 2008
• Total upgrade time of 25 minutes
• No real problems, just Zimbra 5.0.5. Let's just say I gambled with Zimbra 5.0.5 and lost. After some troubleshooting, I was able to upgrade to Zimbra 5.0.6 and everything was up and running on Ubuntu 8.04 LTS. One minor glitch I ran into was an artifact of the ldap logging level changes between 5.0.5 and 5.0.6. Logging in 5.0.6 had an ldap_log_level parameter of 32768 while it was set to 16640 in 5.0.6. The result was that I was seeing a lot more ldap logging that caused me to think something was wrong. I did a zmlocalconfig -e ldap_log_level=32768, restarted Zimbra, and all was fine again.

Ferret DRB Server Linux Startup Script