If you’re a visitor to this site more than occasionally, you’ll probably note that (a) the system has been down for a few days and (b) that this blog has gone through a few changes. Well, there’s a good reason for that. I’ll keep the gory details behind a cut at the bottom of this entry… but what else is new?

Well, I almost hate to admit it, but not much. Life is busy as always; always hectic. My Mac is still working out as an incredible tool to use on a daily basis, in part because of it support for open standards like X, as well as having Windows running in a Parallels VM on my desktop. This basically means I have the power of three environments to play with on one screen; OSX, UNIX and Windows. All this with one machine, one screen, one mouse and one keyboard. Sure, it means sometimes my screen can get a little cluttered, but it’s more than worth it.

Hopefully I’ll get more time to write soon. As the weather warms I hope to be getting out and doing more interesting stuff, too. For now, well if you want to see the gory details of my last few days then click below.

Still with me? Oh goody! Yeah, OK… I like this new blog. It allows me to split up my rants so that you only have to read it if you want to. WordPress is a lot nicer than bBlog. I was happy with bBlog and had no intentions of changing… but what happened? Why did so much change? Why was I down for three days?

Let’s start with Wednesday night. Let’s be honest here; I don’t quite know what happened. Sometime during Wednesday night my system apparently decided it didn’t want to play any more. I suspect a memory failure, but… well, read on.

Thursday morning I awoke to a normal day. I don’t really know why but I SSH’d to my server that morning from my laptop before I left for work. At first things seemed normal, then I tried to copy something to a directory. Segmentation Fault. With arched eyebrow I tried MV. Same result. Hmm… I had seen this happen before with this system, and last time a reboot cleared everything up. OK, so let’s restart the system. Since I was at home I decided to do it from the console.

So down she goes. First go the VM’s I had running some services. No problem. Then went the system itself… and stopped about half way through throwing screens full of Segmentation Fault errors for each service it tried to stop. Not good. After 10 minutes of this I hit the power; maybe that was a bad idea. On the next boot, what did I spy but a 5 minute delay after a few segfaults loading UDEV and other necessary system components. Ach! I have a corrupted server on my hands, and I need to be at work soon. Then it started moving again… some of the services came online but seemed more than a little flaky. Finally I got a logon prompt, so I logged in as myself and tried that same copy. Same results. This did not exactly make my day.

With the system essentially unusable, I decided to hell with it; I’m off to work.

During lunch I took a break, thinking that maybe I could GOOGLE a way to fix the issues. With a few minutes research I downloading an upgraded OS CD and attempting an in-place upgrade might be the way to fix the corrupted and bad tools. The original OS had been Mandrake, so I downloaded the latest Mandriva (the distribution changed its name a while back)

While downloading the CD’s, I thought to myself that maybe this is an opportunity in disguise. For months I’ve had a nice system sitting around idle; a dual-processor Xeon 1.4 with 1Gb of RAM and a 36Gb SCSI Hard Drive. Now, the hard drive would have to go; I already had a nice RAID in the old server that gave me about 60Gb to play with and it was 95% full anyway. So, it was off to NewEgg. One short debit-card torturing shopping trip later, and I had two 320Gb SATA drives on the way with a nice hardware RAID card. Hardware RAID with Linux support isn’t cheap, but it’s by far the best way to do a RAIDed configuration. Software RAID is cool and all, but I wanted hardware!

OK, so Thursday evening I get home, I boil the kettle and make myself a cup of tea and I sit down and try to recover my installation. While the CD is running (it takes a while) I hook up that new server I’ve had sitting around and build out a new operating system. I decide to go with Gentoo so I can (a) keep on top of updates and stay on the leading (bleeding?) edge of Linux technology. Gentoo is tough to set up, but it’s by far the most well optimized installation, and allows you to install JUST what you want with no chaff. Basically, if you’re not familiar with Gentoo, it’s pretty much a “roll your own” Linux with some basic tools to help you get started. Everything’s compiled from source, nothing’s binary unless you want it. This means that so long as your config files are set up right, the entire system will be optimized entirely for your system. Sweet!

Anyway, I set up Gentoo and start setting up all the server components I want. I keep GUI tools to a minimum and install only a very basic X which I’ll probably rarely if ever use. I already know I’m planning to use the two 320Gb drives, but figure that having a running system I can transfer to those drives will be better than starting from scratch.

Two more CD changes later, I reboot the system. OK… it won’t even boot with the new kernel. So I reboot using the old kernel again… Segmentation Faults all over the place. I decide at this point to just give up on the box. Obviously the installation’s badly corrupted, and the attempted upgrade probably just added to the problem. Ah well, as I said before this is a good excuse to upgrade!

So I had a Gentoo box with some basic services. I had a trashed Mandrake / Mandriva installation with all my data. And I had nowhere to put it all. Ah well, guess that means another day down.

Friday night, I received the drives. Yup, NewEgg did their usual bit of shipping stuff fantastically quickly. I’m impressed. I’ve used NewEgg for a couple of years and they actually never cease to surprise me.

Anyway, I digress. Installation was a snap; slap in the controller card, mount the two drives, route a couple of cables (damn… SATA is so much nicer for routing that IDE or SCSI!) and bingo. First boot I get to configure the array, and I have two nice new 7200rpm drives for my server.

Driver installation was pretty easy all things considered. The driver’s released as open source. Well, actually a binary module with open source wrapper… but it all can be patched into the kernel with no muss, no fuss. One quick compilation later and I now have a working kernel with all the drivers I need to run off the new drives. Oh, and here’s where Linux excels; I moved the entire operating system using “cp -Rfpdx” (four instances, one for each filesystem), plus a copy from an NFS mounted copy of my home dirs from the old server (thankfully NFS was working!). Once that was done, a quick bit of magic with “grub” and I was able to shut down. Broke open the case, pulled the power connection from the SCSI drive and put it all back together. A quick flick of the power switch… and voila. One working server.

Well… almost.

It’s amazing; you don’t realize how much stuff you’ve installed over years until you move it to new hardware and you’ve got to re-do it all. Well, if I’d been able to recover the old OS it probably wouldn’t be a problem… but since I was migrating to Gentoo all of a sudden I find myself with most of my applications, my web site and everything else down. Basic web serving? Not a problem! My web mail and my blog? Toast. Still, it was a start.

So I get building. Building PHP, building postfix, building amavis, building assp… you get the idea. A lot of software goes into running a server, and it takes time to get all those services back.

So here I am; Sunday night. I’ve been working on this on-and-off since Friday, but the site’s back up. During the process of rebuilding I did discover that bBlog was flaky as hell on Gentoo. No worries though; I found a script online that could migrate everything to WordPress. Quite frankly. WordPress is a MUCH superior solution for blogging than bBlog. bBlog is nice, but kinda “amateurish” in terms of features. So WordPress is my new blog engine. Look for some more changes as I progress and learn more about this software.

Still some stuff to do; got to pull the tape drive from the old server and put it in the new one. Once I do that, I’ll pull the old server out entirely and probably list it on eBay. It’s a nice box, it really is; and the Mylex controller I used as a RAID controller was a rock solid piece of kit for a couple of years. Make an awesome little dev box for someone… and it’ll take dual CPU’s (though I never got around to upgrading it). Anyone reading this wants to make me an offer… email me and I’ll entertain offers on it. It’s an HP Proliant ML350.

The new server? A Compaq Deskpro W6000; dual 1.4Ghz Xeon CPU’s, 1Gb RAMBUS RAM, 2x 320Gb 7200 SATA drives on a Highpoint RocketRAID 1742. Darned nice little box… and a lot quieter than the server it replaces 😀

Anyway, enough waffling and boring. Sorry this has been a little dry; I’m tired and it’s been a long weekend building out the new server. Still some bugs to fix (gallery’s still down), but I’ll fix them as time progresses as well.

Peace out.