NetBSD upgrade (by alaric)

Bah!

I've upgraded my main home server from NetBSD 1.6.1 (ancient) to 3.0 (current), for two reasons.

1. There was a bug in the VIA Rhine network driver, causing it to drop packets and/or stop working (drop ALL packets until brought down and back up again) under load
2. I need to install Asterisk, and 1.6.1 wasn't up to it - Asterisk requires a native pthreads implementation.

So I bravely reinstalled. NetBSD's upgrade process is a bit ropey, so I had to hack around a bit to get it all up and running.

But now I've lost my CD-ROM drive!

Read more »

Nominet Lunch (by alaric)

Today, I attended a Nominet Members Lunch.

Nominet are the organisation in charge of the .uk domain name hierarchy. My company is a member of Nominet, giving us voting rights, and a "tag holder", meaning we can create .uk domain names and are charged "cost price" by Nominet for running the root .uk name servers and administrative procedures.

It was brilliant - my company are a very small member, yet I sat and chatted with the Chief Executive about issues facing Nominet. There are not many systems about where a "little guy" who wants to discuss how something is run can have lunch with the Chief Executive and other staff (in this case, Nominet's IT and Marketing/Communications directors) and chat about it.

Yet I worry that, in theory, the 3,000 odd Nominet members could all sign up for the Lunches, drink free wine, eat free food, and discuss football. It's one of those wonderful things that works beautifully because nobody's thought to try and spam it yet...

I went, not because I have a particular political axe to grind, but because one of the points that came up at the EGM is that a lot of the members aren't getting involved in Nominet governance; and so I felt I, as somebody who will never make much money from .uk domain name registrations so whose views on it are purely from a making-the-Internet-cool perspective, should set a good example to others by making my voice heard. Plus, I like getting into a good debate. We discussed EPP, IPv6, Nominet's voting rights, expanding the scope of Nominet to bid for ENUM and .eu, and other issues.

Silly Globix! (by alaric)

One of the companies I do work for hosts their equipment at Globix's Prospect House datacentre in New Oxford Street, London.

And it's become increasingly apparent over the past couple of years that Globix aren't doing a great job.

It all began to crumble on the day everything went down. Now, Globix pride themselves on the lovely backup power system. They have a giant generator, roomfulls of batteries, and all the usual stuff. They explained to us how they're better than many such places because they test their generator under a simulated load with a giant resistor on the roof.

However, still, the power to the entire floor went down for many hours. Apparently, the external power had gone down. The backup batteries had flawlessly switched in while the generator started, then the generator took over and started recharging the batteries. However, it turns out that the power switching system that connects generator, batteries, incoming power, and the load had a special cooling system, which was... connected to the external power feed. And so it wasn't running. The switch unit overheated, and shut itself down, killing the machine floor.

Whoever installed that cooling system was, therefore, somewhat incompetent.

But it gets worse. Power was restored to the floor. I went in to talk to the machines that didn't come back up cleanly, and couldn't help but notice that it was:

1. Hot
2. Dark

It became apparent that they were running on a reduced load, to keep their switch cool enough. To reduce the load, they'd killed the AIR CONDITIONING SYSTEM. And the lights.

In order to reduce their on-paper downtime, they'd brought the computers back up without aircon, for about a day.

In the year or so since that happened, we've had a lot of hardware failures - which we'd not been plagued with before. It's pretty likely that the roasting the machines had didn't do them much good.

Then about six months ago, we were going to install a new machine. We followed the usual process; on the day, we emailled in to say we were bringing in a machine. However, unlike the zillion past times we'd brought in machines, we got an email back saying we had to wait until there was power available.

Eh?

It turned out that they realised they were running short of power in the data centre. Now, they'd not told anyone this, perhaps because it's embarrassing. Instead, they waited until we tried to put a machine in, then revealed that they'd introduced new restrictions. Since we'd been planning this machine insertion for several weeks, had we known, we'd have gotten into the "power queue" sooner. Instead, we had to wait weeks until somebody else removed a machine, freeing up amps for ours.

We have space in our rack. We have spare IP addresses. We have spare sockets on the switch. Yet we are waiting on other customers to remove equipment before we can put our own in.

But it gets worse.

We receive an email telling us that our machines are in the rack the wrong way round. Apparently they want to line machines up so that alternate aisles are used to take air from, so the other aisles receive all the hot air for taking away. And they've just noticed that our machines don't fit their plan.

Now, the machines were initially installed long before my time at the company. Since I took over, I've just been putting machines in the same way as the original ones. And, in fact, putting them in the other way would be very hard, since Globix's power strips are down the mounting rails on the 'rear' of the rack. To put machines in facing the other way would involve rerouting the power supply to the rack.

Oh, they say, perhaps our engineers put the power strips down the wrong side of the rack when they set it up for you. Sorry. Please spend a day powering down all your machines, taking them out, moving the power strips, and putting them all back in. As soon as possible.

I understand your point of view. Globix had obviously placed the power strips in the wrong side of the cab leading your technicians to understandably put the kit the wrong way round. However in order to ensure that the data-centre is practicing best practice and ensure that all customers are having their cabs cooled on the correct side it is imperative that all customers are facing the right way round. We will offer you hands so all we really need is for someone to over see it but I'm afraid we have no other option other than to turn the kit around.

Grrrrr!!!!

MSA (by alaric)

I've just written an MSA daemon.

The role of an MSA (Mail Submission Agent) is defined in RFC2476.

Read more »

Generic indexed storage for TUNGSTEN (by alaric)

For TUNGSTEN, I plan to split the local storage manager into two layers; the physical manager and the logical managers. The physical manager takes a collection of block storage devices (such as hard disks), and a configuration giving advice on which disks to use for what, and provides an administrative interface for modifying said configuration and monitoring usage, and an interface to the logical managers. The latter interface provides access to a single unified associative data store; the local store is divided into regions for each volume mirrored on the local node (by a numeric volume ID), which are in turn divided into regions for each entity (by a numeric entity ID), which are in turn divided into regions for each object within the entity (by a numeric object ID), which may then be structured as the logical manager responsible for that object sees fit. The structure available to the logical manager within an object is a set of records, each with a key and a value. The physical manager provides the ability to create, abort, or commit transactions, and within a transaction, read, write, add, and remove records, including iterating through records by key range.

Read more »