Donnie’s Blog

A few days ago my WordPress installation was hacked. This is not a big surprise, since I haven’t been keeping up with the releases as they came out, and there have been a lot of security bugfixes along the way.

The hack was also very subtle, or it didn’t really accomplish a whole lot. I am not entirely sure what to make of it. But the exploit basically involved using the WordPress upload functionality to drop a PHP script into my /tmp directory, and also to ensure that it would always be run when a WordPress page was rendered.

Fortunately, when this happened, it broke certain parts of my website, so it became obvious pretty quickly. Also, fortunately, I keep a week’s worth of backups around, so if I catch a problem right away then I can roll back my installation to the day before the problem.

So that’s what I did this time; I rolled back my website software to before the hack, then I applied the latest and greatest upgrades, and finally I went through some of the “site-hardening” guides for making it much harder to hack these pieces of software.

We shall see if it actually solves the problems! And I for one will be much more diligent about applying upgrades when they come out…

Finally I have an Uninterruptible Power Supply on my server! “Uninterruptible” is decidedly over the top, but at least I’ll get clean shutdowns now if the power carks it on me.

I went ahead and got one of the APC battery-backup systems, a Back-UPS ES 750VA model. (This model is discontinued now; APC has a new, more streamlined version with the same general features.) The UPS has a port that allows you to connect it to a USB port, so that the power supply can tell the computer that things are about to get ugly. I still need to set up the power-monitoring software on my Linux box, but after that, I should be all set.

A few days ago there was this Slashdot article about a serious Linux kernel security hole. Since I was running a version of Linux that had the issue, I thought I had better patch it on my server ASAP.

My machine basically has nobody on it, so I am really not worried about somebody using the exploit directly. But, I do have a number of network services, like my mailserver, webserver, SSH server - and if any of those has a vulnerability, I sure don’t want somebody using that to get root access on my box!

So, last night I manually patched my kernel source, rebuilt my kernel, and rebooted the machine. It was exciting and nerve-wracking; since it’s been so long since I built a kernel myself (2000? 2001? It was at DALi I am sure…), I didn’t know if I would get it right! Then there’s the fact that I am running RAID1 on my boot partition, and I wasn’t sure how the kernel update would go with that.

But, everything went fine. The machine rebooted fine, and when I tried the exploit on the patched system, nothing bad happened. All my services started back up without a hitch.

It was kind of exciting!

Wow, am I glad that’s over.

In the ongoing saga of getting my new server set up, I was right at the point where I was going to switch my hard disks over to a RAID1 setup. I have two 500GB SATA disks, and probably about 200GB of data on the one of them. So I went through the process outlined in many places online:

1. Create a number of RAID1 partitions on the empty disk, to match up what you have on the full disk. These RAID1 partitions are set up with two devices, the empty hard disk and “missing”, which tells the RAID controller that you haven’t gotten around to adding the other device yet.
2. Copy all the data from the “normal” disk to the new RAID1 disk.
3. Get as much of your system ready to boot under RAID as possible. This includes editing /etc/fstab, making an /etc/mdadm.conf file, and setting up lilo to boot off the RAID partition.

…except that lilo wasn’t cooperating. You see, lilo doesn’t like to install onto a degraded RAID1 array, with the versions of kernel/lilo/whatever that I have.

#$@*. Doomed.

Oh well, let’s carry on!

Next came booting the rescue OS that comes with the Linux installer and finishing up the installation. The idea is that you finish setting up the RAID1 partitions under the rescue OS so they have a chance to fully sync up before you bring the system back up. This is the first place where I thought my 200GB of data was lost (well only the few dozen megabytes I generated since my last full backup, but that still had me freaking out), because I just couldn’t get the RAID1 partitions loading.

“Oh wait, maybe I need to load some module.” Ah yes. Today’s episode, brought to you by the Linux command “modprobe raid1“. Now I can see my precious data again…

Once the RAID1 partitions were all synched up, it was a small matter of getting all the boot-related stuff taken care of. This would have been simple, except that the instructions I was following had me do something like this:

  mount /dev/md1 /mnt/raid
  mount /dev/md0 /mnt/raid/boot
  chroot /mnt/raid /bin/bash
  source /etc/profile

This was to switch the system into a state like the final running system, so that I could complete the boot setup. Only problem is, /proc, /dev, and /sys are all empy, and I kinda need those if I’m going to do any sweet-talking with my hard disks.

That was the second “#$@*. Doomed.” moment. But, thankfully, I got it figured out pretty quickly (/proc was easy, /sys was kludgey, and /dev was gross), and got my boot stuff taken care of.

And now, it all seems to work! You are reading my blog again, after all.

I never want to deal with that again. If I have to, well, I’ll know how. But the reason I went through all this rigamarole in the first place was that I didn’t want to buy an extra hard disk, and in retrospect, I think I would have rather just had an extra HDD sitting around. It would be my emergency device/paperweight, or something.

I need smarter wiki software. Mainly because I tend to be a bit dense when it comes to using wikis. I am too used to the niceties of document editors that autosave your temporary work, and that will ask you if you want to discard unsaved edits when you close the application. But wikis don’t always offer that, and it has bitten me at least half a dozen times in the last six months. Either I hit the preview button and forget that I haven’t actually saved my changes yet, or I accidentally mash Shift-Backspace and navigate away from the edit page and the web browser doesn’t remember the form contents. Every time it’s completely demoralizing too, because I just wrote something amazingly profound, then stupidly obliterated it, and I know there is no easy way to get back to what I just lost.

I normally use PmWiki for my personal wikis because it is just so lightweight and easy to set up. There are just a few PHP files, not a lot of options, the format is simple to learn, and all the pages are stored on the filesystem so you don’t have to set up a database. Easy to backup and migrate, too! But this also means that it doesn’t have all the nifty client-side scripting that modern web applications provide, and so sometimes I shoot myself in the foot.

Oh well. PmWiki does in fact offer a plugin component that does the whole “Do you want to save your edits?” option; I think it’s probably high time I installed that.

Whining over.

Well, it looks like I have my mailserver and webserver back up and running, all on my new machine. It seems a lot zippier too, probably because of all the extra memory and processors.

I still have to switch my hard disks over to RAID1, which I am sure will be quite an adventure. I’m really not worried about it, except for the /boot partition. If /boot gives me trouble then I’m going to be outta luck for a while…

Now that the term is over and summer is here, I am getting to catch up on a few maintenance tasks here and there. One of the things I did was to get my photo gallery software back up and running.

I also have a ton of backlogged photos that I need to post, so hopefully I will get everything onto my website soon. To kick it all off, here are some photos of praying mantises that were hanging out in my back yard. These photos are from last year. I just saw my first mantis for this year, so I expect to take more photos soon!

Phew, I think that is just about over…

On Sunday I threw the new hard disk and power supply into my server, and thankfully everything started right up. However, transferring the operating system to the new disk turned out to be significantly trickier. In the end I decided to go ahead and install Mandriva 2007 Spring since I was hoping to do that sometime soon anyway. It turned out to be a good choice (at least so far…), since some of the essential packages are much better integrated into the distribution. Setting up Postfix with TLS went surprisingly quickly, for example.

Of course, while I am juggling all this, I am also trying to get third term grades wrapped up, and consult, so I am stressed out! But I think it will get better now that this admin hassle is out of the way. Hopefully my server will survive the pwn3d-timeout and I will get to stop thinking about all this stuff for a bit.

You may have noticed that my website has changed a bit. I upgraded my installation of Wordpress. The biggest reason why is that I was just getting sick of all the spam comments. I had quite a list of blacklist words, and it was getting annoying, fighting that battle. The newest version of Wordpress is supposed to have some very slick features for dealing with spam, so we will see how it goes.

Hopefully I won’t have accidentally opened up any new security holes…

I still need to update my Gallery installation too, so that integration will be missing for a while.

Earlier yesterday, millions of bots, e-mail harvesters, and web-crawlers were disappointed to discover that donniepinkston.net was offline. (A few people might have cared too…?)

(more…)