10 Years Ago…. (by sarah)

Ten years ago today Alaric got to the train station and thought "you know what I don't want to go to work today, my pregnant wife is very sick and in that hospital just there, I'll go and see her instead". This was an unusual thought for him, as it was he mostly worked from home and only went in once a week for meetings.

It was bizar behaviour on his part but something I am so glad he did. He held my hand as I sat on machines that monitored my vitals and then he went to get my breakfast. I think I fell asleep, something was going on, nurses were running past the door, Alaric came and with a nurse helped get me to the breakfast room with it's TV.

He explained a bomb had gone off, we watched the news as it unfolded with a sickening sense of relief, Alaric could have, should have been on that train. Then the panic as we realised that it wasn't one attack but several - that it was hitting routes we knew. I tried to phone my friends and family who worked in London. Unsurprisingly the networks were jammed - in hindsight we should have been leaving the phones for emergency stuff but we weren't thinking we just wanted to check everyone was safe.

Our Drs started to disappear as they left to help or be medical stand by, my parents turned up thinking they were going to have to tell their very ill very pregnant daughter that her husband was missing on one of the blown up routes. They had been trying to phone him, none of the phones were working.

They were angry with him in that way you get angry when a child didn't come when you called, and you imagine the worst. Then he got hugged. And then the maternity ward began to break down. They say there is no stress induced pregnancy but woman after woman came in with blood pressure problems or in labour or both. The ward filled, there were women on trollies in the corridor - we were not on the labour ward but one woman ended up in the advance stages of having her baby in the maternity ward with me. They pulled the curtains round her bed, she was calling for her husband - her parents didn't know where he was, he had not done an Alaric, he was either dead, injuried or stranded in a motionaless London.

There was not enough beds or staff and bloody foot prints appeared and stayed on the floor. I was bewildered.

After much trying we got hold of as many friends and family as we could, checking they were all ok. More than one had had a near miss, were sitting still in London, sitting on steps crying or telling me how eeri it was with all the traffic stopped, with the hush, and with everyone being kind. London is normally a free for all, pushing, rushing, ignoring the press of humanity but that wasn't what was happening. Everyone was milling, quiet and in shock, everyone knew they were the lucky ones.

Everyone had been expecting the attack since 7/11 in the US, in truth London commuters had been being a bit nicer to each other since that point all fearing that this day was coming. If your train was delayed by more than ten minutes and you had no reception but someone else did - they would lend you their phone to phone and say you were alive. This affect multiplied on the day, I did not really register the stories at the time - I was too ill and mainly wanted to know that the people I cared about were fine.

That is not saying that I had no feelings for those who weren't, it was horrific but I needed to know my friends were fine.

When he got home Alaric found we'd been added to shout outs, it was before the days of social media but we did have blogs and mailing lists and everyone was checking that we were ok. People were worried.

My friends and family were lucky, but mum's friend son - not so much. He's alive due to the carriage he got in but bar the shock of the actual explosion and minor injuries he then had to be escorted past the carnage. Last I heard he still hadn't gone back to work, not all the scars were physical ones, not all of them could heal.

Much later on I realised that it had been even more of a close call for our little family, if I had not been ill and in hospital then we could all have been on the train. It's a strange twist of fate and one that wedges inside me, that me almost dying potentially saved all three of our lives. I say potentially because we might have been late or delayed or I might have weed myself on the way to the station or a million other things, but all of those things are nothing but grace as was me being so ill Alaric felt justified in not going to work that morning.

Terrorism is a horrendous thing, life taking for political gain, for power, religion, to make a point, to drive the wedge... murder is the only name for it.

It was muslims that time but it followed a tradition of London bombs. Someone asked me how I could still travel on the tube into London - the answer, "The RIA didn't stop us, oil disputes in the 70's did not stop my mother, hell she even had her bank hijacked, so why let this lot?". They weren't all muslims like the RIA were not all the Irish, the point of the bombings was to divide, to make an us and them, sadly with some people they succeeded and that is sad. Muslims were killed on the trains and buses, muslim doctors came and aided people straight away - before anyone really knew what they were dealing with, weather those drs were putting their own lives at risk or not.

Terrorists don't really care who they kill, who they injure or maim, that's kind of the point of the bombs. Ever wondered why we don't have metal bins anymore?

Anyway that is all besides the point. Today there are people remembering loved ones who should but aren't still here and no amount of photos shown on international TV is going to heal the wholes in those families. Later today I am going to light ten candles - one for each year, for the yawning chasm of pain, for those who were lost and those that still bare the scars.

DNS issues today (by alaric)

Gah! This morning, my alerting system texted me to say that love (the primary server) can't talk to ihibehu (the backup server in the USA). A quick looked confirmed that we seemed to have some kind of routing loop in level3's network, which was therefore returning "TTL exceeded" to pings. I could connect to ihibehu OK from another network, confirming that it was just a local routing spat of some kind. I shrugged and moved on with life.

However, people started complaining they couldn't resolve DNS for stuff I host, so I had another look. love and ihibehu are both DNS servers (they go by the name of ns0.warhead.org.uk and ns1.warhead.org.uk in that role), and if one is unreachable, then the other should be contacted, so all should have been fine. However, it turned out that the IP address for ns0.warhead.org.uk was still pointing to its old location (and love don't live there anymore), so ns0.warhead.org.uk wasn't "working"; and so for the people whose route to ihibehu went via the routing problem, ns1.warhead.org.uk wasn't working as well.

Oops! One tricky aspect of distributed fault-tolerant systems is that sometimes part of them fails and you don't realise because all the user-visible stuff silently fails over. Therefore, you need to test things below the failover layer to make sure they work individually. Although I check both DNS servers are up, I wasn't checking that the "glue records" mapping the nameserver names to IPs pointed to the right place...

But I clearly remembered sending in the request to the registrar to change the glue record for ns0.warhead.org.uk when I moved it, didn't I? I checked my emails and, yes, I'd send that request, but with all the other stuff I was dealing with in the migration, I never chased it up. And lo, nestled among my spam emails was a response from the registrar, reminding me that I still had access to the interface to do it myself (The registrar used to be me, but I passed that mantle on to somebody else), and suggesting I do so. So it had never gotten done.

"No time like the present, then," I thought, and set out to send in the request, only to find that I don't still have access to the interface, because it also needs a password which I removed from my password databases when I passed control of the registry interface over. Doh!

So I've re-requested that the registrar does it for me. Thankfully, the routing loop has healed up and all is working again while I wait for that to happen. And I'm going to write a test for my glue records being correct into my monitoring system, because that was just sloppy!

Ugarit 2.0 released (by alaric)

Unless I messed up the release process, Ugarit version 2.0 is now available from the Chicken Scheme egg repository.

What does this mean to you, dear reader? Not a huge amount; you can go and read the release notes at the bottom of the Ugarit documentation page for the fine detail. But, to summarise:

• The beginnings of archival mode! As well as storing chains of snapshots of a filesystem, as Ugarit has always done (generally to be used as a versioned backup system), Ugarit vaults can now also store "archives", which are groups of files or directory trees identified by arbitary metadata, such as "This is the song 'Ooofarno' by 'Bobby and the Beaters', which is track 11 out of 12 from the 'Fishes In The Sea' album", or "This is a photo of Aunt Mavis taken at 13:58 on the 3rd of August, 2020, at Uncle Bob's 100th birthday party", or "This is a PDF of a paper by Donald Knuth on ternary numbers, called 'Simplified Arithmetic in Base Three'". You can then find things by searching on this metadata, which is much, much, nicer than creating trees of directories to organise all your stuff into. The user interface for getting things in and out of archives is still a bit minimal - but I have plans to fix that.

• The way tags are stored has changed. Ugarit 2.0 will read vaults created by prior versions happily, but when it writes to a vault, it'll write new-format tags (which have type information as well as a pointer to something), which old versions of Ugarit won't be very pleased to find.

• We now store a "metadata block" in every vault, pointed to by a hidden tag (we didn't used to be able to hide tags, so old versions of Ugarit will show you a funny tag, and complain if you try and do anything with it, as it's a new-format tag). This stores a vault format version number, so we can better handle incompatible changes to the vault format going forward; and as it's hashed and encrypted like any other block, it means you'll get an instant error if you try and connect to a vault using the wrong hashing and encryption settings, rather than bizarre errors further down the line.

• We've made it possible to store large logs in the vault. When we do a snapshot or an archive import or something, we keep a log of warnings and recoverable errors that cropped up while doing so. This is stored as a file attached directly to the snapshot or import object, so it can be arbitrarily large.

• Added log.sexpr and properties.sexpr files in the explore-mode interface, inside every snapshot or archive import object, which let you access the log and the metadata. These are files you can extract, or you can look at them with the new cat command.

• Added a cat command in the explore-mode user interface to dump a file to the screen for viewing.

• Added a client-side cache of snapshots and imports, which significantly speeds up the exploration of backup histories and archive metadata. Optionally, you can make the cache persist between sessions, otherwise it's made afresh for every explore session.

• As well as the existing ability to fork a tag into two tags that share the same history (applicable both to snapshot tags and archive tags), added the ability to merge two tags into one, melding the two histories into one. This includes some exciting logic to combine those histories for display in explore mode!

• Added a new sqlite backend, which provides a storage inside a single file, managed by sqlite. I wrote it to make testing easier, but it's a useful storage backend in its own right!

• Tidied up the Ugarit internals, splitting the core up into a load of separate modules. This makes development easier for me, and means nothing to users.

So what's next? I want to improve the usability of archive mode - right now, it's quite easy to import a bunch of files, but you have to hand-edit a text file to provide metadata beyond what it can automatically extract (currently just basic file information, plus whatever it can extract from ID3 tags and Ogg metadata); and then you can explore the history of the archive (as a series of imports) through the explore interface, or use a command-line tool to search for files, and then extract them or stream them to standard output.

What I want is:

• A shiny (web-based?) UI for searching the archive, seeing thumbnails of images, and the ability to download files with a single click or to perform bulk editing operations on metadata with ease and panache.

• A music player, that lets me pick music from an archive to queue, or to be given an arbitrary search criteria to find music to random-play, playing direct from the archive.

• A way to pick photos and assemble them into galleries, which are then publicly visible through a Web interface. Sarah wants to be able to put sequences of photos together, as well as individual photos that don't form parts of sequences, into multiple albums, for her blog publishing stuff (which is often quite image-heavy). The current image publishing framework I threw together for her years ago is a bit limiting now, and quite clunky to use.

• A mountable filesystem that lets me access archives, either in a generic manner (with auto-generated directories for every property, and every value of that property, containing all files with that value of that property) or with customised directory layouts (such as presenting my music collection as /music/ARTIST/ALBUM/NUMBER:TITLE.EXTENSION, with all the capitalised bits generated from the metadata). I'd like to do this by adding this to the explore mode virtual filesystem, and then having that mountable.

Geo Bake Off – Geologist Despair (by sarah)

I mentioned the Geological Society's Bake Off to Jean - this is the result - she's been planning it for weeks!

The girls are seriously proud of this 7 cake monstrocity.

They have certainly enjoyed eating it 🙂

There is a lot of hidden stuff that went into this cake.

One of the themes was mud which is why there is chocolate orange mud flows 🙂

But there were all sorts of challenges and Jeany decided she wanted to try and complete as many as possible.

So within the river valley there is structure for a cross section.

And then she just got plan creative - with the structure of the cake and I believe some youtube research.

These are the marzi-bones fossil human ancestors or related species buried in a cash by volcanic ash - they may or may not have already been dead when this happened some more excavation will have to occur to find out!

The top layer of the Mud Tower is a chocolate gravel lens between a sandy mud and a volcanic ash.

You can see the colour difference really well in this photo.

Here is Jean cutting open mud tower to reveal whats within.

Spoiler... the chocolate gravel lense.

Here's the river valley with birds foot delta - at this stage the volcano is dormant or extinct.

This is the main part of the cake with Mud Tower and the ammonite loaf as zoomed in bits and the past hidden behind the lush "hill".

Of course there is a hidden volcano and... erm Jurassic Park toilet death scene...

The geologist hammer was another challenge - but being Jean it is a geo-thor hammer so is the wrong shape (to be honest she sneaked a time travel train into it so I was amazed there was no tardis). I did the writing.

Within there is an ammonite - this one was completely and utterly Jean's own idea and it worked and she is soooo happy she is taking it into school tomorrow 🙂

This was the tense moment of cutting in and finding out if the idea had worked. It's a bit flatter than intended but we agreed it's had metamorphic stuff happen to it thanks to the volcanos proximity.

The cake did kind of over flow but that's not surprising - here's how it was made...

Did I mention that she called this cake collective - Geologist Despair.

Volcano before lava.

She did try to put structure inside the volcano but it didn't work that well.

The volcano was fun to put together - she remembered Dino-Mountian I'd made her for her 5th? Birthday 🙂

How the river valley was put together...

One time travel train and it's in a tunnel - the tunnel was the challenge 🙂

And before the tunnel, infact she did a lot of icing moderling for this.

Of course Mary pulled her weight too 🙂 Mainly with rolling out icing and smearing chocolate everywhere!

She did most of the Mud Tower by herself 🙂

Mary put chocolate gravel leaking out of an erroded side and some other bits including sticking out marzi-bones 🙂

Mud cracks were a challenge - Jean went with the existing cake cracks and made the lonely tree which was another of the challenges.

Lonely tree... did I mention the lonely tree?

Other general cakey making pics...

Creating the Marzi-Bones...

I really love this idea 🙂

Creating T-Rex...

This has been EPIC - it took 3 days to make the cakes - Alaric is taking Mud Tower into work tomorrow etc... Both girls have enjoyed it so much and of course we used home grown eggs. The cakes themselves range from chocolate orange to mint to vanilla and strawberry in flavour. There are three icings and marzipan involved and some of the cake is me friendly ie gluten free (the volcano) and some is Mary friendly and so on.

Jean was a little sad as she had meant to put Mary Anning in and a geological map too but she forgot and just don't ask her about how atomically correct her loo death scene is ok.

Ugarit performance (by alaric)

Ugarit was once renowned for its poor import performance, and rightly so. However, it's a lot faster these days - not, sadly, due to amazing optimisation work on my part, but because Thomas Hintz made write-u8vector! faster. It's not released yet, but will be in Chicken 4.10.

There's still work to be done, though. In my experiments with archival mode, I imported 9GiB into an archive in:

real    24m14.822s
user    17m51.485s
sys     1m59.920s

Writing an uncompressed tarball of the same 9GiB took:

real    8m49.931s
user    0m1.076s
sys     1m1.315s

That's a factor of 3. Ugarit spends four minutes waiting for I/O while tar spent eight minutes, which is puzzling, but Ugarit spent seventeen minutes of CPU time while tar spent one second; this will be down to the fact that Ugarit still copies each byte of the file several times between reading it in and writing it out, and I know how to fix that!

There will be some unavoidable architectural cost in the fact that Ugarit will always use at least two processes - a frontend and a backend, with the data sent over a pipe between them - but I think there's a lot I can reduce first. Onwards and downwards!