Due to a disk failure in an already degraded RAID array, the main melon.org server: ambrosia.melon.org went down at roughly 1914 EST (-0500) last night, and all the contents of /home were lost.

The server was brought back into operation at 0103 this morning, and the following websites, largely unaffected by the data loss, were running again: www.anarreshealth.ca, rolling.melon.org, kiwano.melon.org, winter.melon.org, and fleshy.melon.org (except that the images linked to from rolling.melon.org are actually on unripe.melon.org).
Mail service also resumed, and no mail was lost.
By 0230, unripe.melon.org was restored from backup (with the exception of the images which are still being restored from backup as this is being written).
The sudden interruption in service caused a (very) slight, but not automatically repairable error in the database server, which was repaired by 0304.
The gallery2 site, shiny.melon.org was storing all its images in /home, and they remain to be re-uploaded from their original sources (in at least one case, this is a blessing in disguise).

User directories (and outside sites) have typically been the user’s responsibility to back up. My own user directory is being restored from backup right now (and includes the images still missing from rolling.melon.org). Additionally, for the bulk of mp3s, videos, etc. in the affected filesystem, The Pirate Bay is being used as a backup, since it requires much less effort than heading out of town to get my CDs/DVDs out of storage, and re-ripping them.

In other news, bringing the server down to install a replacement disk also provided an opportunity to switch to a vserver-enabled kernel. I’ve got a few spare IP addresses, and I’m doing some contract work that involves vservers, so if you’ve got an excuse for me to hone my vserver skills a little, I’d love to hear about it.