computing linux sysadmin

Emergency Server Moves, Automation,

So, I’m having a bad server day.

This is a Monday Morning email if ever there was one
This is a Monday Morning email if ever there was one

It’s my fault, to a large extent. Earlier this year I discovered that my server had been sending daily emails complaining about a problem with one of the mirrored hard-drives, but these were going directly into Spam. I asked my hosting provider (Hetzner) to swap a new drive in, which they did, but the process of copying the old hard-drive mirror to the new one failed because of an error on the old hard drive. Not having the time to devote to fixing it properly, I bodged a fix and moved on with my life.

This is not a good email to receive either, but in addition to the last one, it's worse
This is not a good email to receive either, but in addition to the last one, it’s worse

Earlier this week, that problem developed into a fatal one, it’s getting worse, and so now I have a limited time to move everything off.


So now I own a brand new server, continuing with my usual conventions, this is

One of the reasons I didn’t jump to a new server earlier is because I wanted to improve the setup of it. Atoll – the server which is failing – was set up three years ago in a hurry as I attempted to move stuff off of fjord – its predecessor – before the billing period was up. All of it was before I moved back into sysadmin/devops, and it’s a very traditionally set-up server – the kind I refer to at work as “Artisanal”; Hand crafted by craftsmen, unique and unrepeatable because it has no documentation as to *why* any config is as it was. My mission at work is to automate that stuff out of existence, and I wanted to apply some of the stuff I’ve learned in the last few years to my own systems.

Server herding over the last ten years has shifted away from big servers doing multiple jobs, and towards huge servers pretending to be many tiny servers doing single jobs. Initially I thought of setting up the new server as a virtual hosting server, setting up many small servers. I’ve come to the conclusion this is impractical for my use-case, which is that the server tends to host many dozens of tiny websites and services. Dedicating a slice of resource to each does none of them any favours, and increases the admin burden rather than decreases it. Instead, I’ve gone towards a halfway house solution of putting most of the separate servers on Docker.

docker-logo(Docker, for those who haven’t seen it, is basically chroot on steroids, an enclosed mini-OS that shares physical resources with the host, but only has access to the service and disk resources you specifically give it. For example, you can grant it a port to talk to MySQL on the host (or another Docker container), and a directory so it can maintain state between restarts).

Rather than manually set up the server for Docker, and have many automated boxes inside a grand artisanal crafted shelf, I’ve decided to use Ansible to manage the main server setup (and a lot of the DNS, since most of my sites use Amazon’s easily scriptable Route53 DNS server service). I’m still learning Docker, but I’m comfortable in Ansible, so I haven’t gone as far as to orchestrate the Docker setup itself with Ansible, just the install.

All of this is kind of like getting everyone off of the capital of Sokovia before the rockets hit the right altitude. In this metaphor, the unbeatable forces of entropy will be playing the part of Ultron
All of this is kind of like getting everyone off of the capital of Sokovia before the rockets hit the right altitude. In this metaphor, the unbeatable forces of entropy will be playing the part of Ultron

Docker’s given me some fantastic advantages just in the 12 hours or so I’ve spent using it. I’ve long been a user of things like Virtualenv for python to separate out my language and library versions on a per-application basis, and the ability to do that with whole OS packages is very useful. It enables one of the holy grails of PHP development: Hosting multiple sites with different versions of PHP at the same time. So WordPress can be on the brand new PHP7 (you’re using it now, in fact), while a PHPbb install can remain on 5.6 until they get their act together (or I switch it to Discourse or something). For this traditional hosted kind of server, which does many things for many different people, it’s really useful.

Taking Control
Taking Control (Photo by Ady Satria Herzegovina, used with permission)

All this is still an arse, though. Working through today, I’ve got WordPress and the forums moved over, some of the static sites, but still less than 20% of functionality’s been moved. Taking a break now (It’s midnight as I write this) for some gaming and then probably sleep. Maybe I can finish this tomorrow…




Spotify Killed The Something Something – On not owning my music collection

I’ve been a premium subscriber to Spotify for 7 years, and it changed how I listen to music, and recently changed a lot again. And since I’m looking for things to post about, it seems like a decent topic.

Because I’m a certain kind of process geek, there’s a rough process for how my music listening works. Pre-2009 this was set up with a set of automated playlists that would select random selections from stuff I rated highly, stuff I hadn’t rated yet, recent additions and stuff I hadn’t heard for a while, weighted towards things that were on more than one list.

In Spotify, when I find a track I like it gets added to my “Starred” list (recently removed as a UI feature and replaced with My Music pluses, which work less well for me), every so often I’ll copy stuff from the Starred list onto the Like [CURRENTYEAR] list, and stuff gets removed from starred when I get bored of listening to it a lot.

Day to day, I mostly listen to either the current starred list, Spotify’s Discovery personalised playlists (which do an occasionally terrifyingly good job of introducing me to new music I like, and are the source of most of the additions to Starred since it started)  the entire 7 year collection of Like playlists, or stuff I get recommended by friends, foes and twitter (I have a host of other playlists, some artist based, some mood based, a lot single-album or larp-character playlists). Current Like list, Starred and Discovery get synced to my phone for offline listening, so I’m not dependant on network connections.

I don’t really buy physical albums anymore, and I was always more a single track (though not always singles) person than a whole-album one. I’ll buy any new Divine Comedy album, a few other artists, but generally my music for the last 7 years has lived on Spotify, which makes me fucked if they go under or suddenly become complete bastards (and on the hook for a lot of MP3s if I want to keep my favourites), but I went into that straight-up and eyes open, entirely unlike stuff like the – in retrospect – ironically named “Plays4Sure” Zune store stuff, which pretended to sell you albums while actually selling offline-available streaming services.


Mimir and the tyranny of things that just about work

So this is about LARP, but also about systems and process. Bare with me, it might get a bit long.

I had pretty much no interest in Odyssey when it started. My first introduction to it was when a group of friends was invited to a test of the combat system a year or so before Event 1, and since LARP Combat really isn’t my thing, I passed. I knew some friends were involved in writing story, but I hadn’t seen anything that wasn’t caught up in the arena stuff, and I was kind of wrapped up in my head a bit anyway. After E1, everyone I knew who played was frothing about this wonderful game, and everyone I knew who crewed was drinking heavily to forget. After some discussions with the latter about ways the game might go better next time I sent an email to PD saying I’d like to join the story team, and eventually became a ref instead.

At the second event, I spent friday evening wandering around the field in my brand new boots, and Saturday morning through to sunday afternoon unable to walk. I became desk-ref by invalid status, and started being the point of contact for story related questions that came to the ref desk. Over the next few events I started to make suggestions as well as ask questions for players, became a member of story team proper, got given a Head Ref title, and generally did my best for Odyssey.

The most annoying thing about the first few Odysseys was the Kudos tracker, which is the system the game uses to keep track of money sacrificed to the Gods and other worthy personages. From Event 2 though to around Event 5 it was a spreadsheet on a shared drive, and because the front desk needed to add things to it, and Story needed to add things to it and make calculations based off it, we were forever shouting at each other, because having it open in Excel or Libreoffice on one computer meant nobody else could write to it. Eventually, I decided this problem wasn’t one that would fix itself, and basically wrote a really simple PHP/MySQL app that did what the spreadsheet did, only with multiple users. This was the first version of Mimir, the Odyssey game management system. Next event I added a blogging platform, so every NPC in the game has a little blog where they write down what happened to them, so the story team can see what’s going on. Also it doubles as a notes area to keep track of “These characters did this thing” in a way that doesn’t result in a thousand tiny doc files stored locally on one of a dozen PCs.

There it lay for a couple of years. It worked well enough, and adding the next big thing to it would take ages, and I never got around to it.

Oh, but I wish I had.

For the first event last year I finally added the promised feature of “Blessings” – always with the scare quotes – that tracks special effects on any given player. From being blessed by Jupiter to strike down foes with thunderbolts, to the secret abilities of the Zodiac Council, to the curse of Dionysus on the guy who stole his drink; Blessings in Odyssey are one of the key interactions and effects that Story will give out.

And they were an absolute arse to maintain.

The first version of a system was an Excel workbook. One sheet was a table of blessings, the second a designed form where if you put the primary key of the blessing in a special place it would generate a pretty form you could print. This was single user, broke if you opened it in Libreoffice, and just about worked. For a couple of years it was another spreadsheet that was used as a data source by an access database that generated PDFs of blessings you could print, and that just about worked… and it all worked enough that there wasn’t really an impetus to replace the system, but enough that every interaction with the system (which was the same for Greater Mysteries – big spells – and Artifacts – Item information sheets) caused a slight uptick in everyone’s stress levels.

At Event 9 it all went to fuck.

A perfect storm of a series of mistargeted blessings going out that absolutely fucked up someone’s event, a block of blessings going entirely missing because they were using the wrong version of the spreadsheet and a set of lower-impact minor bad blessings that were just not game improving; joined forces with a period of time where the only laptop that could generate the blessings couldn’t be accessed… it just all fell over. For the remainder of the event blessings were not allowed out unless one of Me (Head Ref), Ian Andrews (Head of Story) or Si Brind (Lead Si Brind) had physically signed them. The event itself went well, all the players were happy (save the ones who got screwed over), but this needed to be fixed.

This is the problem with things that Just About Work, be they inherited processes or systems, applications or relationships with people. They’re not, none of them, going to one day get better. They’re either going to slowly grind to a halt, or they’re going to go off like a firework in someone’s face; and in this case the face was a player who didn’t even _nearly_ deserve it.

You'd be surprised by how many Greek blessings are for reasons of caprice. Or maybe not.
You’d be surprised by how many Greek blessings are for reasons of caprice. Or maybe not.

Over autumn I wrote some entity relationship diagrams, over winter I wrote some user interface, over spring I wrote the code to make the interface work and regretted using a home-built framework for it. By event 10 – first event 2015 – I had a system that could import the old spreadsheet and turn it into a modernish web interface with a three-point approvals system, print queue and (the bit that hurt the most) PDF generation of any player’s blessings on demand.

In an exceedingly complimentary post, Ian credits this addition with making last event go so smoothly from back-end side. I’m less convinced it was just – or even mostly – that, this is a really good team and it was working like oiled clockwork last season, but it almost certainly did help, and reduced stress levels a great deal, as well as prep time for next event.

I suspect the moral of this story is something like “There’s always time to create better tools” (there isn’t), or “The right tool in the right place and you can move the world with your thumb” (Which is, but finding “Right” is almost impossible). Mostly, I think it’s “Make better things, and things will be better”.

It’s also “Come to Odyssey”, but you knew that.

Anyway, source for Mimir will be released after the last event (it has some spoilers in it), though data won’t. The data will be used to make some pretty graphs to show, though.