The Crunchy Debian Centre

The statute of limitations is past on this one, so a tale of sysadminry.

Once upon a time there were two servers, which we shall call Robert and Dave. Between them they ran an important system that ran on a somewhat clunky framework. They were load balanced, but for database purposes one was a Master and one was a Slave, with a magic switch so that they could swap should it become necessary. They ran Gentoo, because when they were bought it was the only “released” distro that supported the RAID cards they had.

After a while a new system ran on these servers, which for operational reasons wasn’t load balanced, and that ran within PHP. Because these servers were too important to ever take down for maintainance, and compiling PHP on Gentoo was – at the time – a heartache akin to losing your first true love. So the sysadmin at the time installed a debian chroot on Robert and ran apache within that. Shortly after that, I’m hired, and eventually the old coder leaves, and I take over both those projects.

A while later, a drive in the RAID on David fails. Me and the lead dev wander down to London to fix it, meeting a courier with the new drive at the datacentre. In the process of replacing the drive, David’s raid controller entirely fails and takes the remaining drives with it. Now, because Gentoo compiles its own packages, system upgrades require a large amount of load, a great deal of planning and a significant amount of downtime while this goes on (No, you can’t recompile the system while it’s still running as a high loaded real server, don’t be silly), so updates have been confined to security updates, but that’s fine, because it’s only been about nine months since it was new.

We can’t install the same version of Gentoo anymore, because it’s so very far out of date that the package management server doesn’t support it. The idea of the mirrored servers running different versions of the core OS cannot be borne, but we can’t update the other server either, because there’s no upgrade route from there to here. Fortunately by this time Debian has updated and now supports the RAID drivers of the servers. So we install Debian on it, and it works. And we install the Big System on it, with it’s clunky framework. But the version of the clunky framework is a major revision after the one the system was written for, and we don’t support that version (because it doesn’t work). We don’t have time to rewrite the software. The old package for the framework doesn’t work, and the binary download of the old version doesn’t work either.

Can’t go over it, can’t go under it, have to go around it.

So we copy the contents of Robert onto the new debianised David, and set up a chroot on David running Robert’s gentoo root, and thus the old working version of the framework.

So now we have Robert, running Gentoo with a crunchy Debian centre; and we have David, running Debian with an orangey Gentoo centre. Except that because the Gentoo centre is a copy of Robert, that too has a toffee flavoured Debian install, although it’s not activated. Inception eat your brains out, six years early.

At this point, me and the other coder had spent eight hours in the air-conditioned white-noise of a hosting centre room, during which the number of things that had gone right was rapidly drowning in the sea of everything going wrong. Since it was working, we went home and fixed it later.

So, things to learn:

  • Raid is not a backup solution.
  • Factor in a method to test and install security and system upgrades, otherwise you’re going to be forced to find all the holes in sudden short order.
  • Note that this isn’t specifically “Gentoo sucks”, it’s that “Not having an upgrade path sucks”
  • Raid is not a backup solution.
  • A major problem was that Gentoo was the exception, all the rest of the servers ran Debian.
  • Raid is not a backup solution.
  • Just because you have made all the backups in the world before you start, does not stop everything going wrong in short order.
  • Bring a packed lunch to the datacentre.
  • Raid is not a backup solution.

New Flat Adventures – Season 3, Episode 2: In Hot Water

From the middle of December 2010 until January 2011, our old flat’s boiler didn’t work very well. I’ve gone on about the problems with this at length elsewhere, but it culminated in a massive, epic Scott-of-the-antarctic journey in the run up to Christmas to try and buy a halogen heater in the middle of a snowstorm, which I completely and entirely failed to write up. I should do that.

During this time, I spent a great deal of time in the company of Vlad the Installer, plumber extraordinaire, and his dedication to spending as much time as possible – at fixed cost – making the damn thing work. The next year, when the boiler fell over again, Vlad was ill and so another company came to look at it. They had neither the wit or patience of Vlad, and so declared it dead and invoiced for a new one, fixing the entire problem in four hours flat.

There is an important fable in there about dedication, patience, and the knowledge of when to give up, but that is not what came to mind when I arrived at the new flat this afternoon for delivery of the white goods to find a plumber on my doorstep waiting for me.

He was here because the boiler doesn’t work. Because it’s a 4 day national holiday, it’s going to be Wednesday until we get heating and hot water.



Avenge: Best served cool

(No spoilers)

Batman Begins rebooted the bombastic batman movies into a darker, quieter, more serious and far better series. Superman attempted to redo that series into a high-contrast heroic fantasy, and failed.

Iron Man made a similar attempt to be a bright action movie, and (IMO) did it very well. The other Marvel movies so far have been mostly good to great, but the most important two of them I think are Iron Man 2, and Avengers Assemble.

The Spiderman series is interesting of itself, but never really got over its own origin story. Hero movies have this weird obsession with the creation of the hero, and the first two Spiderman stories seemed to focus on the idea of becoming a hero, and when they got to the the point where he was fighting the evil of the week more than his own head, the film collapsed like an over-observed soufflé. Fantastic Four 2 did the same thing, to an extent.

Iron Man 2 started with Stark being Iron Man already. Being known for that, and already having the suit and the confidence, he went though the Wheel of Plot (Setup, Defeat, Man-Up, Win, Return to the world, Having learnt something) without needing to define the character at the start.

Avengers has two advantages to being an Origin Story. The first is that it doesn’t have to set up or introduce most of the main characters, but the second – and I think unique in Hero movies – it doesn’t need to set up the villain either. Tom Higgleson’s Loki from Thor was a strong and complicated character, and survived the denouncement at the end of his own movie. He’s even more awesome here than he was in Thor, which is a neat trick, and it gives me hope that the shooting-gallery of comic villains can work in a movie context, and that Marvel Studios is actually capable of producing comic book movies without rebooting into a new Origin Arc because a new writer/director wants to put his/her handprints into that set of concrete.

The writing is Whedonesque (obviously) and the characters stay true to their original movie characterisation while sifting slightly into the more quickfire style. It does a good job of setting up the strengths and weaknesses of all the main players both as heroes and as people, and actually pulls off all these massive characters working well as a team without losing the self-reliance from their own movies.

The action sequences range from slightly by-the-numbers though to absolute awesome, and there are crowning moments of awesome for everyone, even the complicated villain. Samuel L Jackson is recursive, given that he’s playing a character based on him, so he’s quite good. It’s good to see Colby Smoulders doing something other than HIMYM.

You need to know about Iron Man, you should probably have seen Captain America, but you probably need to have seen Thor before you watch Avengers. It does stand alone, but the story follows on from Cap and a lot of the characters will make more sense if you’ve seen Thor.

There’s one post-credits sequence halfway though (After the animated credits, before the white-on-black roll) but nothing at the end.


BrewDogged Determination: The 2012 Brewdog AGM & Birthday

The event can be divided into four bits without much difficulty, those being the AGM, the Bars, the Shop and the Music.

Late into the night the stage was host to several bands, ranging from a somewhat lonely singer/songwriter though to the semifamous and quite good Kassidy. The building being a great concrete barn of a place, the acoustics left something to be desired, but the bands were quite good and the sound did the best they could in the space, I think. As it filled up, the echoing chasm effect wore off.

To some extent the music and sound-checks drove the shop situation worse. For whatever reason – I assume ease of stock-checking – the order-for-later-delivery bulk-beer shop was running orders by taking them down, finding all the stuff, filling and addressing the boxes there and then, and then sending them off for later postage. This meant that every order took ages, and with only five or six people taking the orders on thirty feet of counterspace, queues to place orders went though a joke into epic failure. The two hours it took to actually get my order though are the least fun I’ve ever had whilst at a festival celebrating decent beer.

The bars too suffered a little from having vast amount of queuing space to limited serving spots, and could have done with more pumps, but the token system (Since they weren’t licensed for a cash bar, you had to buy tokens worth 1 or 1/2 pint, depending on the beer) sped up the ordering process (because change is annoying and fiddly). It’s hard to complain too much when pints of BrewDog’s as yet unnamed Pale Ale are going for a couple of quid a pint, as are their more famous 5am Saint and Punk IPA brews. So good beer, served at a high quality. Queues got a little silly during some periods, but that’s not really avoidable completely when you have 1500 beer fans and two bars.

I missed the actual AGM bit because of queuing for the shop, but the reports I heard were all very positive. The second London Brewdog has a site (in Shoreditch), they hope to start issuing dividends soon, the new brewery provides an 8-fold increase in yield for them, although they’re going to need to spend a few months after it comes up doing quality and consistancy control on beers previously created at the old factory. 66% of their output goes overseas, they’re looking at international bars, Sweden likes BrewDog.

Then on to the aforementioned music and more beer. I managed to track most of what I was drinking on Untappd (Where I am “Aquarion”), and probably need to play with some of the Stone breweries beers, as the ones I’ve tried I’ve liked a lot.

Aberdeen is nice enough, though how the exhibition centre managed to get two different Holiday Inns built next to it beggers belief a bit (and nobody warned either of the massive influx for this weekend, something they didn’t cope with perfectly).

When I booked the train back, I knew I was going to consider 9:30 the next morning too early, but I can only imagine past-me was attempting to get some kind of vengence. It worked. I am quite tired.

Personal Projects Random Work

Updates, Invisible Content and the lifecycle of the feedback junkie

Updates are kind of hard to come by. The venerable Aquarius has celebrated his blog’s 10th birthday, which kind of made me think “Gosh, 11 and a bit years. I should update more often”. And Thus.

So, an update, then. At work, I’m working on the Next Big Thing for Work, which will have absolutely no impact on the tech industry at all, and because of the inward-facing nature of same will be worth not much.

Home-wise I’m working on The Book, which continues… well, crap, really. The story is there, and the writing is as not bad as is reasonable, but it turns out the actual story is far too slow to actually get moving, let alone get to anything that will hook the reader. The writing continues apace, and by this method the pace will be fixed, we hope.

Also, Lifestream, and more specifically the current incarnation of NicholasAvenell dot com. The actual display of which isn’t anything revolutionary, but I’m having fun writing the grid system it all runs under.

PiracyInc continues as well, with a new Combat system that I really should have worked out ages ago, but may end up being rewritten (again) into a language I know, because learning at the same time as doing does not appear to be producing the results I need it too. Right now the economics system works fine without any player input, so now all I need is added pirates.

So most of what I’m doing is entirely invisible to the public eye.

The only actual visible thing I’m producing right now is a series of parodies/satire on UK Larp systems, and the idea of the “perfect system”, which isn’t really interesting to my current readership who aren’t already involved, I think. It’s an interesting vector for creative writing. More interesting, to me, is the aspect of user feedback that centralising on Facebook provides. The frictionless UI that supplying a “Like” provides appears to increase user feedback, or at least enough positive feedback to avoid the “I’m shouting into a black hole” feeling that I’ve found posting anything on the internet tends to engender. This, I predict, will last until something I’ve invested actual mental effort into results in significantly less interaction than previous throw-away jokes, at which point the frictionless interaction will turn into a more friction-focused stick with which half my ego uses to beat the other half. The problem with being a feedback junkie is that the comedown is rough.

Also, the highs are hard to come by.

So, tomorrow work, and then on Thursday up to sunny Glasgow, Saturday the Brewdog AGM, Sunday Glasgow again, and Monday the long trek home. It’s entirely possible you’re in for a great deal more introspection over the next few days, for which I can only apologise.

Since I have to get up in seven hours, I should probably take my leave of this pub.

Projects Random

Under the warehouse

Once upon a time, the deepest dungeons of SJGames’ Warehouse 23‘s basement were free to rummage in by the forces of the internet. Alas, technical problems took down the basement, and it hasn’t really recovered.

This is an attempt to do the same thing. We don’t have the ten years of submissions the old one did, nor do we have the vast vault of imagination that the old one was seeded with. We do have vote buttons, though.


'Stromfront, Part One

Taken at Maelstrom's final 2006 event, Matrimony

It’s been nearly six years since I started playing Maelstrom, and it ends this year. Between the sunstroke and the admin problems, I think I was quite lucky in getting my Out of Character “worst event” over with quite so quickly, and in the last six years every single aspect of the character save his player has changed quite dramatically (For starters, I rewrote the background from the one posted here before I sent it to PD, adding a lot more angst and a number of key hooks for character traits, most of which have now triggered).

The character is now a fairly well respected priest of the goddess of free thinking, which is a pretty good thing considering he started with none of this. His list of friends has waxed and waned over the years, but every time I think his story arc has finished, another thing appears. I’ve been plotting various characters on the expectation of his death over the years, but now I doubt I’ll play another character seriously in this campaign.

This event waxed and waned also. The four day easter event is an interesting set of challanges, being the first event of the season as well as the longest and, usually, the coldest. By going up on Thursday for the traditional day-before social gathering, I ended up sleeping in a tent in -4 temperatures. My camping gear isn’t top-level – I camp six weekends a year, almost always during summer, up from four until last year – but it wasn’t pleasant, and by Saturday night I was entirely fed up of life under canvas.

In character wasn’t a lot better. Site layout rendered most of my usual game complicated by long walks and difficult communication lines, and the weather and bleak feel of the game was causing a lot of people to – entirely rationally – give up and go home. By Sunday morning I had a deadline. I had enough cash to get a taxi to the station or I could get lunch, so now I had a Deadline.

I don’t think I like four day events, in general. Three day events tend towards a structure of  Prepare, Build, Execute, Repair of major plotlines, and the extra full day in the middle ends up being a bit vague, lessoning the critical timing. Sunday dialled that directly back up again, with an attack by the current leading army of “bad guys” on the main encampments, a glorious withdrawal and retreat, and a wonderful moment in character where I realised that the gate was being manned by two of my cousins casting spells to check for undead taint and me checking for soul symbols on their foreheads. Between minor revolutions in things he is a part of, and the tense battle of the last night, it had picked up again.

I will say I hate fighting an off-field enemy. One of the great things about Maelstrom is that almost everything there is is driven by players and characters. Even if you are fighting the monster crew in a variety of silly hats, you can be sure that the hats are looted from the city you saw razed in downtime, the monsters are never more numerous than you would have seen if you had spent your downtime looking for them; and for a long time the “bad guys” of the system have been “over there” in another encampment, rather than lumbering on from off-site. The wall and the palisade turned an epic player campaign into a Tower Defence game where the only people who got to play were the ones who ran the gatehouse.

I’m used to having an enormous amount of fun at Maelstrom events, and while I still had a lot of enjoyment, the change of pace of the game is making it harder to play the bits I enjoy. Not enough to not plan to be at the next one, but enough there’s a lives counter in my head, and it’s just had a point taken off.

Enhanced by Zemanta

The Source

The Source by Aquarion
The Source, a photo by Aquarion on Flickr.

It’s okay guys, I’ve found the source! We can cut it off at the root!



And this is why we shouldn’t allow our girlfriend to borrow our iPhone to have something to listen to while she does the washing up.



Doomed to repeat it

Having sped past AqCom’s 13th birthday with the lack of updates your RSS reader has come to expect from this once proud site, I’ve done something I was going to do for that, but entirely failed to. With the use of my own archives and, an (almost) complete list of Designs though the Ages.

19992000 2000b 2003b 20032004200520062006b20082008b20102012