Cool things I have worked on: Low-latency highly-available NoSQL data store (by alaric)

I've worked on a bunch of cool things in the past; and since I'm always explaining them to people, I realised it'd be good if I just blogged about them so I can just share the link.

I was asked to help design and then build a database to back a social networking system. The main requirement was to be able to satisfy lots of single-record primary-key fetches from a single thread as quickly as possible, because a PHP web app would be requesting lots of data to build the page; we had to do a few hundred of these "random point queries" in a small fraction of a second to get the page rendered in time. It had to be able to scale sideways, by adding more machines to the cluster. It also needed to be able to update this data, but at a much lower load - updates and creations of single records happened when a user went and did them, but people browsing around the site would hit several pages, each of which would request hundreds of records. Read more »

Programming with state machines (by alaric)

State machines are a really nice way of thinking about systems with state. As a notation for expressing imperative code, I much prefer this:

To this:

x = readchar();
while(x != EOF) {
   if(x == 'a') {
      doA1();
      x = readchar();
      while(x != EOF && x != 'x') {
        doA2(x);
        x = readchar();
      }
      doA3();
   } else if (x == 'b') {
      doB1();
      x = readchar();
      while(x != EOF && x != 'x') {
        doB2(x);
        x = readchar();
      }
      doB3();
   } else {
      doC();
      x = readchar();
   }
}

I've always wished I could write that sort of code as a state machine rather than a bunch of nested structured programming primitives. Although there is some charm to the idea of drawing a diagram on the computer and having that be compiled as executable code, though, that can also be fiddly and isn't well supported by conventional programming tools; so I'd rather actually describe my state machine in text. What I want is a syntax for writing state machines, rather than writing imperative code that implements the state machine.

For instance, the diagram I put above wasn't drawn by me; it was drawn by a tool called PlantUML from the following source code:

@startuml

state Processing {
      Ready --> A : a/doA1
      Ready --> B : b/doB1
      Ready --> Ready : not a or b/doC

      A --> A : not x/doA2
      A --> Ready : x/doA3

      B --> B : not x/doB2
      B --> Ready : x/doB3
}

[*] --> Ready

Processing --> [*] : EOF

@enduml

But the UML state diagram is meant for abstract modelling of systems, and lacks the precision and detail required for actual executable code. The state diagram doesn't make it clear that doA2 and doB2 need to be passed the character that was read, for instance. PlantUML's syntax, therefore, also lacks that kind of detail, so we can't just use that.

Also, in a programming situation, it would be nice to be able to express types of state machines: to be able to say that a group of state machines has some useful properties in common, defined by the "type". The type of a state machine defines some high-level states and what inputs or outputs are possible in each state; any state machine of that type has to follow that behaviour, even though it might break the states defined in the type into many substates. For instance, the example state machine above always accepts any character until it receives an EOF, then stops; that might be expressed as a type having a single Ready state with two transitions, one mapping an EOF to the end state and another mapping a character back to the Ready state, with no outputs. That type would be suitable for any state machine that accepts characters until they finish, so a generic "character input stream" might accept a state machine of that type to process them, without needing to know anything about As and Bs.

These thoughts have led me to the following model of a state machine that's suitable for use as part of a programming language (although I've not defined an actual syntax yet):

State machine type

A state machine type is a set of states. Each state has a name, and a list of message types which are acceptable in that state (I am assuming that the underlying type system is smart enough to include things like "character equal to a" as a type, rather than merely "character"). This list must be disjoint.

For each input message type, there's a list of possible next states and an output message type (which in our example above would be a void type, as we never output messages).

There is also an initial state. It's impossible for the initial state to be the target of a state transition on a message, so it only has transitions out to other states. Those transitions all represent possible constructors of this state machine, because the initial state represents nonexistance.

For instance, a type" for vending machine controller state machines might be (given a Result type which has two values, success and fail):

Initial state:
   input: (Create) output: (Void) next state: (Idle)

Idle state:
   input: (Coin(Value)) output: (Value) next state: (Credit)

Credit state:
   input: (Coin(Value)) output: (Value) next state: (Credit)
   input: (Refund) output: (Value) next state: (Idle)
   input: (Select(Product)) output: (success::Result) next state: (Idle)
   input: (Select(Product)) output: (fail::Result) next state: (Credit)

The Value objects output in response to Coin or Refund inputs happen to reflect the current credit in the machine, a fact which the state machine type alone can't represent (that would need to be explained in a design-by-contract invariant of some kind).

From this type can be derived the type of the state machine's transition functions, which in this case will be:

coin :: Idle -> Value -> (Credit,Value) |
        Credit -> Value -> (Credit,Value)
refund :: Credit -> (Idle,Value)
select :: Credit -> Product -> (success::Result,Idle)|(fail::Result,Credit)

Note that the Create transition isn't represented here, because this state machine can't actually be instantiated - it doesn't specify the behaviour enough. There's no way of saying which possible select result should happen, or of what values are returned as outputs. But the functions we have are automatically implemented by the state transition definition, and will cause the appropriate transitions to occur in any state machine that's a subtype of this one; every function accepts an instance of the state machine in a given state, and an input value, and returns a new instance of the state machine and the output value.

Subtyping

State machine types can be subtypes of other state machine types. A state in the parent type may be split into several states in the subtype; any transition to a state A in the parent type must be represented by a transition in the subtype to some substate of A, and all transitions possible from state A in the parent type must be represented by transitions from every substate of A in the child.

For instance, a subtype of the vending state machine might have an infinite number of substates of "Idle", one for each possible configuration of available things to vend inside the machine. If it vends three different products, then these substates might be called Idle(A,B,C) where A,B and C are nonnegative integers representing the stock of each item. The only transition out of "Idle" is when a coin is inserted, so every Idle(A,B,C) state must have a transition from a coin being inserted, returning a Value, and taking us into the Credit state.

However, our Credit state also tracks the stock, and also tracks the credit inserted so far - so we need an infinite set of states of the form Credit(A,B,C,X) where A,B,C are the stocks and X is another nonnegative integer for the number of pennies of credit. So the parent-type transition from Idle to Credit has to be replaced by a transition from every Idle(A,B,C) to Credit(A,B,C,X) where X is the Value of the coin inserted. All the transitions from Credit in the parent type need to be represented as transitions from every Credit(A,B,C,X) state to other Credit(A,B,C,X) or Idle(A,B,C) types as appropriate. But with that done, it can be shown that all the states and transitions of the parent type correspond to one or more in the subtype.

The one exception is the initial state - the creation transitions from that in the subtype need not correspond to those of the parent type.

Given an extension to the syntax to create "parameterised states" such as Idle(A,B,C), which are shorthand for a potentially infinite set of "concrete states" such as "Idle(4,0,1)", we might represent that subtype like so:

Initial state:
   input: (Create(a::NonNegativeInteger,b::NonNegativeInteger,c::NonNegativeInteger))
     output: (Void) next state: (Idle(A,B,C))

Idle(a::NonNegativeInteger,b::NonNegativeInteger,c::NonNegativeInteger)
   state implements Idle:
   input: (Coin(x::Value)) output: (x::Value) next state: (Credit(a,b,c,x))

Credit(a::NonNegativeInteger,b::NonNegativeInteger,c::NonNegativeInteger,x::NonNegativeInteger)
   state implements Credit:
   input: (Coin(x'::Value)) output: ((x+x')::Value) next state: (Credit(a,b,c,x+x'))
   input: (Refund) output: (x::Value) next state: (Idle(a,b,c))

   input: (Select(Product==A)) when: a>0 output: (Success) next state: (Idle(a-1,b,c))
   input: (Select(Product==A)) when: a==0 output: (Fail) next state: (Credit(a,b,c,x))

   input: (Select(Product==B)) when: b>0 output: (Success) next state: (Idle(a,b-1,c))
   input: (Select(Product==B)) when: b==0 output: (Fail) next state: (Credit(a,b,c,x))

   input: (Select(Product==C)) when: c>0 output: (Success) next state: (Idle(a,b,c-1))
   input: (Select(Product==C)) when: c==0 output: (Fail) next state: (Credit(a,b,c,x))

(We really should have represented the stock as a Map(Product,NonNegativeInteger) type and then not duplicated the select rules...)

Converting a parent state into a parameterised subtype state is only one way of splitting a state in a subtype, too. The syntax above would permit the creation of separate states that all say "implements Credit", too. A very simple vending machine that can only hold one instance of one product might split Idle into two states, Empty and Full, for instance; and probably split Credit into CreditFull(Value) and CreditEmpty(Value) to represent the Credit state while also parameterising CreditFull and CreditEmpty with the current credit.

The above syntax for parameterised states can get unweildy if there's a lot of parameters, especially if generally only a few of them are changed in any given transition (imagine the above example of there were ten different products to keep track of). Therefore, as syntactic sugar, it makes sense for it to be possible to define parameters shared implicitly by one or more states. Transitions into those states from states outside the group need to explicitly specify all the parameter values, but transitions within that group can use a different syntax to represent the next state, which only specifies what parameters have changed. All the rest are passed in unchanged, automatically. It might look like this:

Initial state:
   input: (Create(a::NonNegativeInteger,b::NonNegativeInteger,c::NonNegativeInteger))
    output: (Void)
    next state: (Idle(A,B,C))

parameters a::NonNegativeInteger,b::NonNegativeInteger,c::NonNegativeInteger {

Idle(a,b,c) state implements Idle:
   input: (Coin(x'::Value)) output: (x'::Value) next state: (Credit(x <- x'))

Credit(a,b,c,x::NonNegativeInteger) state implements Credit:
   input: (Coin(x'::Value)) output: ((x+x')::Value) next state: (Credit(x <- x+x'))
   input: (Refund) output: (x::Value) next state: (Idle())

   input: (Select(Product==A)) when: a>0 output: (Success) next state: (Idle(a <- a-1))
   input: (Select(Product==A)) when: a==0 output: (Fail) next state: (Credit())

   input: (Select(Product==B)) when: b>0 output: (Success) next state: (Idle(b <- b-1))
   input: (Select(Product==B)) when: b==0 output: (Fail) next state: (Credit())

   input: (Select(Product==C)) when: c>0 output: (Success) next state: (Idle(c <- c-1))
   input: (Select(Product==C)) when: c==0 output: (Fail) next state: (Credit())
}

Implementing state machine types

Note that we've suddenly started giving names to the Values flying around, and specifying an output value rather than just a type, and splitting transitions into different cases with disjoint boolean "when:" expressions. This, too, further constrains the type of the state machine; indeed, this machine can actually be implemented and will provide a simple model of a vending machine - although nothing will be vended.

This means that this new state machine's state transition functions will include a "create" function from the initial stock levels to an instance of Idle(NonNegativeInteger,NonNegativeInteger,NonNegativeInteger).

A real implementation could further subtype this, adding a uniquely-typed IO interface to the Idle and Credit states, passed into the create transition. The Success-returning select operations can then also use pure IO functions to mutate the IO context to one in which the physical mechanism of vending has happened.

But the implementation of a state machine exists in the expressions that return the output value, parameterise the next state, and the boolean expressions forming the "when:" guards. These are full expressions in the pure functional language I'm imagining this thing implemented on top of, so are fully Turing complete.

There is no syntactic distinction between a type and an implementation; a state machine type can be instantiated if all of its outputs have expressions giving them values (ok, singleton-typed outputs such as Void can be implied!). This is a bit like concrete versus abstract classes in an object-oriented programming language.

Chaining transitions

So far, we've only looked at transitions that accept an input value, return an output value and move to a new state. This means that state machine behaviour is defined entirely in terms of its reactions to external values. Since the values we generate for the output and the parameters of the new state are Turing-complete expressions, that is sufficient to describe any behaviour, but it's not always convenient to express complicated behaviour in an expression. Sometimes a complicated operation of a compound data structure such as a list would be better defined as a state machine in its own right.

In order to support that kind of thing, we can also support chaining transitions, which can drive state machine behaviour purely based on internal state, and so do not cause the generation of state transition functions. These come in three types.

The first kind is one that starts with an input value, but which then leads to a next state without producing a value. This next state must be a special "transient state", the rules for which are explained below.

The second kind is one from a transient state to another transient state. This has no input value and no output value.

The third kind is a transition from a transient state to a non-transient state, which produces an output value.

Transient states may not have any non-chaining transitions point to and from them. Every transient state must be referenced from at least one non-transient state through a chain of chaining transitions, starting with a kind-1 chaining transition then zero or more kind-2 chaining transitions. Every transient state must also lead to at least one non-transient state through a chain of zero or more kind-2 chaining transitions, then a kind-3 chaining transition.

Every such possible chain from a kind-1, through zero or more kind-2, to a final kind-3, is equivalent to a single transition from the initial to the final non-transient states; with the kind-1 transition defining the input value and the kind-3 transition defining the output value. They might be implemented as a set of state-transition functions that just tail-call each other, but the transient states are therefore "hidden" inside the state machine from external perspectives. Those state-transition functions are lexically local to the exposed state-transition functions implementing the equivalent non-chaining transition.

But although they are hidden from users of the state machine, a subtype of the state machine can indeed "see" them for purposes of splitting them into substates, adding transitions between them, etc.

Termination

The final thing I've not covered above is terminating state machines. There's an implicit "final" state available in every state machine, if at least one transition to it exists; no transitions from that state can exist. Any transition to the final state appears in the state transition functions as a function that returns the output value, but not paired with a new state machine instance, because there isn't one. It's gone.

It is entirely possible for a state machine to have no non-transient states apart from the initial and final states. For instance, a state machine that traverses a tree and sums up the labels on the nodes might be implemented in terms of transient parameterised states for the current position in the tree and the running total, with an kind-1 creation transition that accepts the tree as an input and a kind-3 finalisation transition that returns the count. This machine will therefore have a single state transition function - from Tree to Number. The actual state of the machine is entirely represented as state parameters passed between mutually recursive functions implementing the chaining transitions, and never appears as a value visible outside of that.

Conclusion

I think I've defined a workable semantic model for state machines that can be used as a way to define sets of pure functions between instances of state machines, that can handle both synchronous state machines driven by external events, and those that bumble around a data structure or iterate mathematical operations.

It needs a proper syntax; personally, I'd prefer one based on s-expressions or IRON, but suitably consistent forms could be defined for any language.

Towards the Family Mainframe (by alaric)

Last September, I posted progress on the construction of our domestic mainframe. To recap, the intent is to build a dedicated home server that's as awesome as possible - meaning it's reliable, safe, and easy to maintain. That rules out "desktop tower PC in a cupboard" (accumulates dust bunnies, gets too hot, easily stolen, prone to children poking it); "put a 19" rack somewhere in your house" is better, but consumes a lot of floor footprint and doesn't fix the dust bunny problem. So I've made my own custom steel chassis; fed cold air at pressure via a filter, incorporating a dedicated battery backup system, locked and anchored to the wall, and with lots of room inside for expansion and maintenance.

Since that blog post, I've finished the metalwork, painted it with automotive paint using a spray gun (which was a massive job in itself!), fixed it to the wall, and fitted nearly all of the electronics into it.

A significant delay was caused by the motherboard not working. I sent it back to the shop, and they said it was fine; so I sent the CPU back, and they said THAT was fine; so I sent both back together and it turned out that the two of them weren't compatible in some way that was solved by the motherboard manufacturer re-flashing my BIOS. That's now up and running; I was able to use the HDMI and USB ports on the outside of the chassis to connect up and install NetBSD from a USB stick, then connected it to the network and installed Xen so I can run all my services in virtual machines. It's now running fine and everything else can be done via SSH, but the HDMI and USB ports are there so I can do console administration in future without having to open the case (unless I need to press the reset button, which is inside).

The one thing it's lacking is the management microprocessor. I've prototype this thing on a breadboard and written the software, but need to finish off the PCB and cabling: but it will have an AVR controlling three 10mm RGB LEDs on the front panel, and three temperature/humidity sensors in the inlet and outlet air (and one spare for more advanced air management in future). But the idea is that the three LEDs on the front panel will display useful system status, and the environment sensor data will be logged.

Here's what it looks like from the outside; note the air inlet hose at the top left:

The socket panel on the left hand side worked out pretty well - 240v inlet at the bottom, then on the aluminium panel, three Ethernets, HDMI, and USB (my console cable is still plugged into the HDMI and USB in the photo, which won't usually be the case):

And here's the inside, with lots of space for more disks or other extra hardware; the big black box at the bottom is the battery backup system:

Now I have Xen installed, I'm working on a means of building VMs from scripts, so any VM's disk image can be rebuilt on demand. This will make it easy for me to upgrade; any data that needs keeping will be mounted from a separate disk partition, so the boot disk images of the VMs themselves are "disposable" and entirely created by the script (the one slightly tricky thing being the password file in /etc/). This will make upgrades safe and easy - I can tinker with a build script for a new version of a VM, testing it out and destroying the VMs when I'm done, and then when it's good, remount the live data partition onto it and then point the relevant IP address at it. If the upgrade goes bad, I can roll it back by resurrecting the old VM, which I'll only delete when I'm happy with its replacement. This is the kind of thing NixOS does; but that's for Linux rather than NetBSD, so I'm rolling my own that's a little more basic (in that it builds entire VM filesystems from a script, rather than individual packages, with all the complexities of coupling them together nicely).

I'm using NetBSD's excellent logical volume manager to make it easy to manage those partitions across the four disks. There are two volume groups, each containing two physical disks, so I can arrange for important data to be mirrored across different physical disks (not in the RAID sense, which the LVM can do for me, but in the sense of having a live nightly snapshot of things on separate disks, ready to be hot-swapped in if required). I still have SATA ports and physical bays free for more disks, and the LVM will allow me to add them to the volume groups as required, so I can expand the disk space without major downtime.

So for now it's just a matter of making VMs and migrating existing services onto them, then I can take down the noisy, struggling, cranky old servers in the lounge! This project has been a lot of work - but when I ssh into it from inside the house (over the cabling I put in between the house and the workshop) and see all that disk space free in the LVM and all the RAM waiting to be assigned to domU VMs that I can migrate my current services to, it's all worth it!

Learning things (by alaric)

I love learning new things. I'm usually struggling to find new things to learn; the last fun bit of computer science I learnt about was Bloom filters, and they didn't hold my attention for long. The last really fun bit of computer science I learnt about would be content-addressed storage, and I'm still having fun with that, but I can't find any more to learn about it. I'm having to make stuff up myself, which is rewarding in its own way, but much harder work.

Of course, this past week I've been learning TIG welding, which has been awesome. It's been a while since a whole new field of things to learn has opened up to me, and it's nice to work on a new class of physical skills. My routine physical learning is my weekly Krav Maga training, but I crave variety. My lust for learning benefits more from intensive two-week courses than an hour a week for years. I'd love to go and take a proper welding course at a college, but I can't spare the time; I have to practice when I can in the evenings and weekends. I'm getting good at horizontal/flat welds, but I'd like to master vertical and overhead (because if I work on anything large, such as the festival trolley, in my cramped workshop, I often can't rotate everything around to be nice and flat). Also, suspecting that the trouble I was hitting with stick welding is at least partly to do with the limitations of my old cheap AC welder, I want to use my new welder's capability to do nicely regulated DC stick welding and see if I can learn to do good stick welds. And I'd like to get some practice in welding aluminium and stainless steel, as I have applications for both of those.

So that's physical skills sorted, for now. Mentally, I've been learning how antennas work. There's no particular reason for this; it's something that's always puzzled me somewhat, but what's triggered the recent interest was a birthday gift from an old friend, of the ARRL handbook, which goes into some detail; and then meeting an interesting guy at a Cheltenham Hackspace Open Evening who turns out to design broadcast transmission antennas for a living, who answered a load of questions left open by the books. I still want to get a better intuitive grasp of the quantities involved - how many volts and amps appear on an antenna feeder line? What field strengths in volts per meter would you expect to find at what distance from an antenna? Does that relate to a corresponding magnetic field strength in tesla?

I've also been learning Morse code. This is quite interesting. I thought I'd memorise that table of dots and dashes and that'd be that, but it turns out that this technique tends to maroon people at slow morse speeds, as they mentally record a sequence of dots and dashes then mentally look it up in the table to find the letter. To get fast Morse skills, which tend to tend towards one or two characters per second, you need to learn to recognise the sound - "di di dah" - as a letter directly. So I've been using a combination of the Farnsworth and Koch methods to learn Morse; growing my "vocabulary" a letter at a time, and using an enlarged inter-character spacing. The latter is because I was tending to find that I'd hear a character, and write it down, but while I was writing it another would have been and gone, which was confusing. I want to reduce that inter-character gap, but I might wait until I've learnt the entire alphabet via the Koch method, so I can mentally be writing entire words rather than concentrating on a letter at a time - with a reduced alphabet, Morse training tends to involve writing down random gibberish (so far, I know M, K, R, U, S and O; at least the old Nokia SMS beep now makes sense to me... di-di-dit dah-dah di-di-dit!). Again, I have no particular reason to learn Morse - I learnt it as a child, but then forgot it through not using it, which had always faintly irritated me. I've often wondered about using it as a crude interface to tiny embedded computers, although it'd be frustratingly slow for most uses. The usual reason to learn Morse is to do CW amateur radio; that's an idea I've toyed with in the past, but being able to talk to random people over radio holds little appeal (I can talk to random people over the Internet much more cheaply and easily). However, I'd be interested in getting an amateur radio licence as a mental challenge, or as a means to some other project that requires radio communications capabilities, so I might go for a course if one comes up at a good time. I'd like to be able to operate a radio transmitter in an emergency situation, too.

I love to learn things, but I feel sad about not using the skills I pick up. Ok, I don't want to use my Krav skills - they tend to involve hurting people, and are only useful when already under danger of harm coming to me or people I'm protecting, which is nothing to be happy about. But I practice welding because want to make metal things. But by no means do I only learn things when I have a need for them; I learn stuff because it looks interesting and the opportunity arises, then I try and find applications. I've already been excitedly thinking about how aluminium welding will simplify the construction of one of my old back-burner projects - making a hiking staff out of aluminium tubing, that has a stack of lithium-ion batteries inside it at the base, a computer with a keyer in the middle so I can interact with it, and a high-brightness LED lantern on the top so I can have a variety of illumination options (white all around, white forwards only, red all around, etc). I had been working out complex systems of brackets and bolts to hold it all together, but TIG welding it would be much easier, neater, lighter, and stronger. Now, I had considered having a button that could be used to strobe the lights for Morse emergency signalling - and the logical next step was to include a co-axial semiconductor laser in the top that could shine a bright beam for signalling in Morse (and, at a pinch, be used to light fires; you can get 2W laser modules off the shelf these days, and all those lithium ion batteries are going to be able to source a lot of power...). So perhaps I should get a ham licence, and make the staff in sections joined together with insulators, and make it be a two-meter band dipole antenna (which is one meter long) with a CW transceiver inside, so it can also send and receive Morse by radio? That might be fun, and not much extra hardware as it'll already have a decent ARM microprocessor inside.

For now, though, I'd better focus on finishing off my server chassis (which I'm building my welding skills up towards), and make a new welding bench (mine is curved, and wobbles because the floor is uneven and the legs are too weak), and do some metalwork that's needed around the house... I'd like to do some more focussed Lojban study, too; right now I'm just picking up vocabulary by looking for words for things I don't know yet when I need them, but re-reading the reference grammar to remind myself of bits I rarely use would be good!

DNS issues today (by alaric)

Gah! This morning, my alerting system texted me to say that love (the primary server) can't talk to ihibehu (the backup server in the USA). A quick looked confirmed that we seemed to have some kind of routing loop in level3's network, which was therefore returning "TTL exceeded" to pings. I could connect to ihibehu OK from another network, confirming that it was just a local routing spat of some kind. I shrugged and moved on with life.

However, people started complaining they couldn't resolve DNS for stuff I host, so I had another look. love and ihibehu are both DNS servers (they go by the name of ns0.warhead.org.uk and ns1.warhead.org.uk in that role), and if one is unreachable, then the other should be contacted, so all should have been fine. However, it turned out that the IP address for ns0.warhead.org.uk was still pointing to its old location (and love don't live there anymore), so ns0.warhead.org.uk wasn't "working"; and so for the people whose route to ihibehu went via the routing problem, ns1.warhead.org.uk wasn't working as well.

Oops! One tricky aspect of distributed fault-tolerant systems is that sometimes part of them fails and you don't realise because all the user-visible stuff silently fails over. Therefore, you need to test things below the failover layer to make sure they work individually. Although I check both DNS servers are up, I wasn't checking that the "glue records" mapping the nameserver names to IPs pointed to the right place...

But I clearly remembered sending in the request to the registrar to change the glue record for ns0.warhead.org.uk when I moved it, didn't I? I checked my emails and, yes, I'd send that request, but with all the other stuff I was dealing with in the migration, I never chased it up. And lo, nestled among my spam emails was a response from the registrar, reminding me that I still had access to the interface to do it myself (The registrar used to be me, but I passed that mantle on to somebody else), and suggesting I do so. So it had never gotten done.

"No time like the present, then," I thought, and set out to send in the request, only to find that I don't still have access to the interface, because it also needs a password which I removed from my password databases when I passed control of the registry interface over. Doh!

So I've re-requested that the registrar does it for me. Thankfully, the routing loop has healed up and all is working again while I wait for that to happen. And I'm going to write a test for my glue records being correct into my monitoring system, because that was just sloppy!