WTF?

Yeah, I know. There used to be a Team Fortress 2 blog/forum here. It got hacked yesterday. This is just a place holder till I can get the site cleaned up and back online.

So what happened?

At about midnight on November 29, 2011, Papershadow tweeted me that something weird was going on in the forums, and it looked like someone had hacked them.

So I hopped online to have a look. There was a weird thread going on about corn and bending over or something, where everyone had copied and pasted the first response. Now, being ubercharged.net, that kind of thing isn't actually that unusual, so it took me a minute or two to figure out what was up. For starters everyone had had their post count reset to 0. And everyone had some number set as their signature.

So I log into the admin panel to see if anything weird was up, like a moderator gone rogue, or werewolf zombies attacking or something. Nothing in the moderation log (which would happen if a mod was posting as someone else, or changing posts). Someone reckoned Zogulon's account had been hacked (he's an admin). This was plausbile, so I banned him just to be safe temporarily.

Next, I run the upgrade of Simple Machines Forum to the latest security patch release version (yeah, I know, should have already been done - closing the gate after the horse is bolted and all that...), and post something in the thread to that effect. It's about now that our friendly attacker gets a bit more creative, and posts the database password into the forum (which I'm guessing he read from the database config file on the web server file system). At this point, I officially freak out.

The attacker doesn't just have admin access to the forum, they've got file system level access to the web server, and probably to the database directly as well. I figure that there is some mechanism for editing templates etc through SMF or wordpress that is being used to read/write files on the web server. So I log into the admin panel again, and put the forum into maintenance mode. That doesn't do much. The attacker promptly inserts a die() statement in the PHP code that renders the forum, killing every request, and rendering it useless.

Time to go nuclear... So I SSH onto the web server, move ALL THE THINGS out of the public_html directory (so the site effectively isn't there anymore), and start looking into it. Then I realise that because the attacker had file system access, they would have been able to change the other sites hosted under the same account: boomercharged; gamercharged; Pyro Rock Star; and a few other personal sites I've got. So I go nuclear on them as well - deleting the sites that are no longer in use from the web host control panel, and moving out and putting up placeholder pages on the active ones.

Which pretty much brings us to where we are now. I'm pretty sure the attacker can't get back in (there isn't anything there for the moment...), but I need to do a bit more digging to make sure of that - even if they did, nothing is even running right now on any of my sites.

How did they get in?

My guess is that there was an unpatched security vulnerability in either Wordpress, or SMF that allowed the attacker to upload a PHP exploit script - probably something to do with avatar uploads not being validated correctly.

Once the exploit script was uploaded and on the public web directory path, it could be loaded from a browser, and the PHP code contained in it could execute arbitrary code, access the file system, and access the database.

I found this nasty lurking around as a result.

What's the damage?

Mainly, I look stupid, and you can't read the blog/troll the forum.

I think blog/forum account passwords are safe. The passwords are salted and hashed, and the originals are not anywhere in the database - but in a situation like this, it's good to be cautious, and if you use your password from ubercharged anywhere else, best to change it there (if you use the same password for internet banking as for some random forum, you should really have a think about how you're doing it...).

The attacker did get the database password, but that was a one off password, and the only reason they got that is because it's stored in plain text in a configuration file on the server (so the PHP code knows how to connect to the database).

There are daily/weekly/monthly backups of the file system and database for ubercharged and the other affected sites. I keep a copy of these on my home file server. At worst, when everything is cleaned up, a few days of forum posts might be lost if I roll back to an older database backup.

I'm hosting ubercharged on a shared hosting provider. This means that I've really only got access to my home directory on the server (as would the attacker), and they shouldn't be able to change Linux system level executables or install deeper rootkits than just the PHP one - that's assuming the hosting provider has their Linux web server security up to scratch, if they don't we've got bigger problems...

Ultimately though, I feel a bit silly. I haven't been keeping up with security updates for the software that is running on here, and I supposedly do this stuff for a living (software development of web apps). I'm fully aware of the vulnerabilities, but figured no one would really bother with this little site, and seeing as this is a hobby thing, and not something I earn a living off, I let it slide when I should have kept my game up.

What now?

I'm going to spend a bit of time auditing and cleaning up what's on the server (might just nuke the whole home directory and set it up again from scratch to be safe). I've got a clean copy of all the code in a git repository, so I can deploy a known clean version that is fully upgraded and free of any exploits. Then I'll look through the database, and try to figure out what is and is not safe in there. Not sure how I'll do that.

Then I should be able to restore everything back to how it was.

Don't know how long any of that is going to take. I've got a 14 month old baby, so time is kind of scarce at the moment.

Cheers - madlep