The Crunchy Debian Centre

The statute of limitations is past on this one, so a tale of sysadminry.

Once upon a time there were two servers, which we shall call Robert and Dave. Between them they ran an important system that ran on a somewhat clunky framework. They were load balanced, but for database purposes one was a Master and one was a Slave, with a magic switch so that they could swap should it become necessary. They ran Gentoo, because when they were bought it was the only “released” distro that supported the RAID cards they had.

After a while a new system ran on these servers, which for operational reasons wasn’t load balanced, and that ran within PHP. Because these servers were too important to ever take down for maintainance, and compiling PHP on Gentoo was – at the time – a heartache akin to losing your first true love. So the sysadmin at the time installed a debian chroot on Robert and ran apache within that. Shortly after that, I’m hired, and eventually the old coder leaves, and I take over both those projects.

A while later, a drive in the RAID on David fails. Me and the lead dev wander down to London to fix it, meeting a courier with the new drive at the datacentre. In the process of replacing the drive, David’s raid controller entirely fails and takes the remaining drives with it. Now, because Gentoo compiles its own packages, system upgrades require a large amount of load, a great deal of planning and a significant amount of downtime while this goes on (No, you can’t recompile the system while it’s still running as a high loaded real server, don’t be silly), so updates have been confined to security updates, but that’s fine, because it’s only been about nine months since it was new.

We can’t install the same version of Gentoo anymore, because it’s so very far out of date that the package management server doesn’t support it. The idea of the mirrored servers running different versions of the core OS cannot be borne, but we can’t update the other server either, because there’s no upgrade route from there to here. Fortunately by this time Debian has updated and now supports the RAID drivers of the servers. So we install Debian on it, and it works. And we install the Big System on it, with it’s clunky framework. But the version of the clunky framework is a major revision after the one the system was written for, and we don’t support that version (because it doesn’t work). We don’t have time to rewrite the software. The old package for the framework doesn’t work, and the binary download of the old version doesn’t work either.

Can’t go over it, can’t go under it, have to go around it.

So we copy the contents of Robert onto the new debianised David, and set up a chroot on David running Robert’s gentoo root, and thus the old working version of the framework.

So now we have Robert, running Gentoo with a crunchy Debian centre; and we have David, running Debian with an orangey Gentoo centre. Except that because the Gentoo centre is a copy of Robert, that too has a toffee flavoured Debian install, although it’s not activated. Inception eat your brains out, six years early.

At this point, me and the other coder had spent eight hours in the air-conditioned white-noise of a hosting centre room, during which the number of things that had gone right was rapidly drowning in the sea of everything going wrong. Since it was working, we went home and fixed it later.

So, things to learn:

  • Raid is not a backup solution.
  • Factor in a method to test and install security and system upgrades, otherwise you’re going to be forced to find all the holes in sudden short order.
  • Note that this isn’t specifically “Gentoo sucks”, it’s that “Not having an upgrade path sucks”
  • Raid is not a backup solution.
  • A major problem was that Gentoo was the exception, all the rest of the servers ran Debian.
  • Raid is not a backup solution.
  • Just because you have made all the backups in the world before you start, does not stop everything going wrong in short order.
  • Bring a packed lunch to the datacentre.
  • Raid is not a backup solution.


Today, I have:

  • Unpacked six boxes.
  • Caused a very large bookcase to spontaniously and destructively re-flatpack itself…
  • … onto my ankles, which hurt.
  • … and my girlfriend’s hand, which also hurt.
  • Watched Sin City.
  • Regained my kitchen from the encroaching box horde.
  • Moved Aquarionics to a brand new server (Thus the lack of template, not that you’ve noticed from your RSS Reader / Livejournal Page / Facebook)
  • Caused a smoke alarm to go off at 1:15am on a Sunday morning.

If tomorrow is half as productive, I may accidentally demolish the house.

Enhanced by Zemanta

New Flat Adventures – Season 3, Episode 2: In Hot Water

From the middle of December 2010 until January 2011, our old flat’s boiler didn’t work very well. I’ve gone on about the problems with this at length elsewhere, but it culminated in a massive, epic Scott-of-the-antarctic journey in the run up to Christmas to try and buy a halogen heater in the middle of a snowstorm, which I completely and entirely failed to write up. I should do that.

During this time, I spent a great deal of time in the company of Vlad the Installer, plumber extraordinaire, and his dedication to spending as much time as possible – at fixed cost – making the damn thing work. The next year, when the boiler fell over again, Vlad was ill and so another company came to look at it. They had neither the wit or patience of Vlad, and so declared it dead and invoiced for a new one, fixing the entire problem in four hours flat.

There is an important fable in there about dedication, patience, and the knowledge of when to give up, but that is not what came to mind when I arrived at the new flat this afternoon for delivery of the white goods to find a plumber on my doorstep waiting for me.

He was here because the boiler doesn’t work. Because it’s a 4 day national holiday, it’s going to be Wednesday until we get heating and hot water.



New Flat Adventures – Season 3, Episode 1, The Key To A Successful Move

I bought a new kettle yesterday.

There is a tradition that when I get to a new house, the first thing I do when I have the keys is go inside and make a cup of tea. This has happened every move since Bedford (Which is five), but was actually started with my very first flat of my own in Sunderland. So I bought a box of tea-bags in my lunch break on Friday, preordered a kettle and some cheap mugs in Argos at the same time, and picked them and a bottle of milk up on the way to Palmers Green where the estate agent is.

The flat is lovely. In total it’s smaller than our current place, but the kitchen is nicer. Also, it’s twenty minutes by bus from work, which is an improvement on twenty minutes to the first of two trains to work. It’ll gain me about three hours a day of extra life, which is worthwhile.

So me and Fyr went to sign the contracts, and in return for a great deal of money got a small set of keys to borrow for a year. Quite a small set of keys, in fact, because instead of being two full sets of keys for the house, it was in fact three quarters of a set of house-keys, four duplicate window keys and three duplicate mailbox keys. This is mostly fine, as keys we have can be duplicated.

The problem is, of course, in the fourth of the three quarters. For we have a deadlock key for the flat door, and we have a key for the back door, and we even have a yale key for the flat door. We do not, however, have a key for the building door, and it looks like the landlord’s phone number is office hours only.

Of course, from the back door we can access the front door, so it’s not a massive problem, but still. Keys would, generally, be handy.