DNS issues today (by alaric)

Gah! This morning, my alerting system texted me to say that love (the primary server) can't talk to ihibehu (the backup server in the USA). A quick looked confirmed that we seemed to have some kind of routing loop in level3's network, which was therefore returning "TTL exceeded" to pings. I could connect to ihibehu OK from another network, confirming that it was just a local routing spat of some kind. I shrugged and moved on with life.

However, people started complaining they couldn't resolve DNS for stuff I host, so I had another look. love and ihibehu are both DNS servers (they go by the name of ns0.warhead.org.uk and ns1.warhead.org.uk in that role), and if one is unreachable, then the other should be contacted, so all should have been fine. However, it turned out that the IP address for ns0.warhead.org.uk was still pointing to its old location (and love don't live there anymore), so ns0.warhead.org.uk wasn't "working"; and so for the people whose route to ihibehu went via the routing problem, ns1.warhead.org.uk wasn't working as well.

Oops! One tricky aspect of distributed fault-tolerant systems is that sometimes part of them fails and you don't realise because all the user-visible stuff silently fails over. Therefore, you need to test things below the failover layer to make sure they work individually. Although I check both DNS servers are up, I wasn't checking that the "glue records" mapping the nameserver names to IPs pointed to the right place...

But I clearly remembered sending in the request to the registrar to change the glue record for ns0.warhead.org.uk when I moved it, didn't I? I checked my emails and, yes, I'd send that request, but with all the other stuff I was dealing with in the migration, I never chased it up. And lo, nestled among my spam emails was a response from the registrar, reminding me that I still had access to the interface to do it myself (The registrar used to be me, but I passed that mantle on to somebody else), and suggesting I do so. So it had never gotten done.

"No time like the present, then," I thought, and set out to send in the request, only to find that I don't still have access to the interface, because it also needs a password which I removed from my password databases when I passed control of the registry interface over. Doh!

So I've re-requested that the registrar does it for me. Thankfully, the routing loop has healed up and all is working again while I wait for that to happen. And I'm going to write a test for my glue records being correct into my monitoring system, because that was just sloppy!

Ugarit 2.0 released (by alaric)

Unless I messed up the release process, Ugarit version 2.0 is now available from the Chicken Scheme egg repository.

What does this mean to you, dear reader? Not a huge amount; you can go and read the release notes at the bottom of the Ugarit documentation page for the fine detail. But, to summarise:

• The beginnings of archival mode! As well as storing chains of snapshots of a filesystem, as Ugarit has always done (generally to be used as a versioned backup system), Ugarit vaults can now also store "archives", which are groups of files or directory trees identified by arbitary metadata, such as "This is the song 'Ooofarno' by 'Bobby and the Beaters', which is track 11 out of 12 from the 'Fishes In The Sea' album", or "This is a photo of Aunt Mavis taken at 13:58 on the 3rd of August, 2020, at Uncle Bob's 100th birthday party", or "This is a PDF of a paper by Donald Knuth on ternary numbers, called 'Simplified Arithmetic in Base Three'". You can then find things by searching on this metadata, which is much, much, nicer than creating trees of directories to organise all your stuff into. The user interface for getting things in and out of archives is still a bit minimal - but I have plans to fix that.

• The way tags are stored has changed. Ugarit 2.0 will read vaults created by prior versions happily, but when it writes to a vault, it'll write new-format tags (which have type information as well as a pointer to something), which old versions of Ugarit won't be very pleased to find.

• We now store a "metadata block" in every vault, pointed to by a hidden tag (we didn't used to be able to hide tags, so old versions of Ugarit will show you a funny tag, and complain if you try and do anything with it, as it's a new-format tag). This stores a vault format version number, so we can better handle incompatible changes to the vault format going forward; and as it's hashed and encrypted like any other block, it means you'll get an instant error if you try and connect to a vault using the wrong hashing and encryption settings, rather than bizarre errors further down the line.

• We've made it possible to store large logs in the vault. When we do a snapshot or an archive import or something, we keep a log of warnings and recoverable errors that cropped up while doing so. This is stored as a file attached directly to the snapshot or import object, so it can be arbitrarily large.

• Added log.sexpr and properties.sexpr files in the explore-mode interface, inside every snapshot or archive import object, which let you access the log and the metadata. These are files you can extract, or you can look at them with the new cat command.

• Added a cat command in the explore-mode user interface to dump a file to the screen for viewing.

• Added a client-side cache of snapshots and imports, which significantly speeds up the exploration of backup histories and archive metadata. Optionally, you can make the cache persist between sessions, otherwise it's made afresh for every explore session.

• As well as the existing ability to fork a tag into two tags that share the same history (applicable both to snapshot tags and archive tags), added the ability to merge two tags into one, melding the two histories into one. This includes some exciting logic to combine those histories for display in explore mode!

• Added a new sqlite backend, which provides a storage inside a single file, managed by sqlite. I wrote it to make testing easier, but it's a useful storage backend in its own right!

• Tidied up the Ugarit internals, splitting the core up into a load of separate modules. This makes development easier for me, and means nothing to users.

So what's next? I want to improve the usability of archive mode - right now, it's quite easy to import a bunch of files, but you have to hand-edit a text file to provide metadata beyond what it can automatically extract (currently just basic file information, plus whatever it can extract from ID3 tags and Ogg metadata); and then you can explore the history of the archive (as a series of imports) through the explore interface, or use a command-line tool to search for files, and then extract them or stream them to standard output.

What I want is:

• A shiny (web-based?) UI for searching the archive, seeing thumbnails of images, and the ability to download files with a single click or to perform bulk editing operations on metadata with ease and panache.

• A music player, that lets me pick music from an archive to queue, or to be given an arbitrary search criteria to find music to random-play, playing direct from the archive.

• A way to pick photos and assemble them into galleries, which are then publicly visible through a Web interface. Sarah wants to be able to put sequences of photos together, as well as individual photos that don't form parts of sequences, into multiple albums, for her blog publishing stuff (which is often quite image-heavy). The current image publishing framework I threw together for her years ago is a bit limiting now, and quite clunky to use.

• A mountable filesystem that lets me access archives, either in a generic manner (with auto-generated directories for every property, and every value of that property, containing all files with that value of that property) or with customised directory layouts (such as presenting my music collection as /music/ARTIST/ALBUM/NUMBER:TITLE.EXTENSION, with all the capitalised bits generated from the metadata). I'd like to do this by adding this to the explore mode virtual filesystem, and then having that mountable.

Concussion (by alaric)

For all of the time I've known her, Sarah has been poorly. When we first met, she was being investigated for a bowel problem, and she also suffered shoulder pain from a bike accident when she was a teenager. When she was pregnant with our first child, she had an exciting array of complications, and took many years to recover from the experience; she had mobility issues and pain due to a separated pelvis, lacked energy, had reduced sensation in her legs, and so on. The second pregnancy was nowhere near as bad, but Sarah still bled for months afterwards due to problems with her womb lining.

However, some months ago, while trying to lose weight, she cut wheat from her diet; and a load of niggling health problems she had reduced significantly, while her energy levels rose. She suddenly found it easier to lose weight, and things were looking pretty rosy.

However, this meant that she wasn't certain of her limits any more. Last Sunday, she was dancing, and found that she could now dance hard, and didn't need to keep stopping and resting; until she was suddenly dizzy, and stumbled backwards off of the nice soft grass and onto a concrete path, where she fell over backwards and hit her head hard on the ground.

I didn't see it happen; I was looking the other way, but Jean called out to me, and I saw Sarah lying on her back. She was instantly surrounded by people, and a paramedic was at hand almost immediately, so other than providing my cloak as a blanket, my role was reduced to holding her hand and helping Jean to pack up our things so we could leave.

This led to a thrilling trip in an ambulance with her neck immobilised, and several hours in A&E being examined and X-rayed.

Thankfully, this revealed that she had no structural damage to her spine or skull, so she could be released from the collar and head restraints, and sent home.

Jean was a star throughout; she rode in the ambulance, with Sarah's glasses and phone and medication, while I followed in the car, and helped the ambulance staff by verifying the truth of questions they asked Sarah to check her memory function. She didn't panic, even though it was clearly stressful for her, and did all the right things.

On Monday, Sarah mainly slept. On Tuesday, my father and stepmother visited, and we had a day out; Sarah was still speaking slowly and having short-term memory problems (forgetting where she put things, mainly). But on Wednesday she was struggling, with her speech slurring and often struggling to remember words for things, so she spoke haltingly. So we took her back to A&E, where she was given a CT scan, which thankfully confirmed that she didn't have any bleeding inside her skull (which would have been bad news). So she was let out again, and told to come back if she wasn't better in another week and a half.

At the time of writing, a week after the accident, she still has difficulty concentrating for long conversations, and can't concentrate on reading or watching TV or anything like that for more than a few minutes at a time. If she tries, she starts to struggle to think clearly at all, and her speech worsens. I've not kept track, but she sleeps through the night and probably spends at least a quarter of the day napping, and has to restrict her waking hours to mentally non-taxing tasks. Loud noises cause her pain, and I've been having to work from home to keep an eye on her and to keep Mary from bothering her.

So, it looks like she has Post-concussion Syndrome; this means she may be better in a few weeks - or may well be like this for years. Apparently about one in ten people still have symptoms after a year.

Her personality is intact, and her memories all seem to still exist; she just has to take her time to find them, which makes conversation difficult at times. But she can't do any of the things she loves to do, because sustained concentration renders her zombie-like within minutes. Unless that clears up, her career in writing, crafts workshops, and science communication will be over. I feel reasonably confident I can leave her looking after herself at home now, but she won't cope on days when Mary is not at nursery, so if things don't improve my job may become awkward until Mary starts at school in September, too.

So, for the time being, all we can do is hope it gets better!

Ugarit performance (by alaric)

Ugarit was once renowned for its poor import performance, and rightly so. However, it's a lot faster these days - not, sadly, due to amazing optimisation work on my part, but because Thomas Hintz made write-u8vector! faster. It's not released yet, but will be in Chicken 4.10.

There's still work to be done, though. In my experiments with archival mode, I imported 9GiB into an archive in:

real    24m14.822s
user    17m51.485s
sys     1m59.920s

Writing an uncompressed tarball of the same 9GiB took:

real    8m49.931s
user    0m1.076s
sys     1m1.315s

That's a factor of 3. Ugarit spends four minutes waiting for I/O while tar spent eight minutes, which is puzzling, but Ugarit spent seventeen minutes of CPU time while tar spent one second; this will be down to the fact that Ugarit still copies each byte of the file several times between reading it in and writing it out, and I know how to fix that!

There will be some unavoidable architectural cost in the fact that Ugarit will always use at least two processes - a frontend and a backend, with the data sent over a pipe between them - but I think there's a lot I can reduce first. Onwards and downwards!

Ugarit archive mode manifest maker (by alaric)

When I last wrote about Ugarit progress, I had developed archive mode to the point where one could import a list of files with metadata from a "manifest file", and then search for files based on the metadata from the manifest and stream out chosen files. I gave an example of using this to play MP3s matching a search pattern:

[alaric@ahusai ugarit]$ for i in `ugarit search test.conf music '(= ($ artist) "UNKLE")' keys`;
do ugarit archive-stream test.conf music $i | mpg123 -;
done

Well, that was all based on hand-written manifest files, which are no fun to produce (our music collection is large). As such, I've been working on a "manifest maker" that takes a list of files and directories and makes a manifest file from them, recursing down through directories to list all the files. And for each, it automatically extracts metadata into the manifest file, which can then be hand-edited if required, and then used to import from.

The idea is that the manifest maker will have support for a number of file types it knows how to extract additional metadata from, and the first one I've implemented is ID3 tag extraction from MP3s. I've implemented the ID3 V2.2 and ID3 V2.3 specs, as those were the two that I found present in the subset of my MP3 collection I'm testing against!

For example, here's the output it produced for one of my MP3s:

(object "./test-data/THE HOLLIES - He Ain't Heavy, He's My Brother.mp3"
  (filename = "THE HOLLIES - He Ain't Heavy, He's My Brother.mp3")
  (mime-type = "audio/mpeg")

   ;; Unknown ID3 tag "COMM"="engiTunNORM\x00 00000402 00000000 00001B59 00000000 00004E65 00000000 000040EC 00000000 00015FD5 00000000"
  (keyword = "Pop")
  (name = "He Ain't Heavy, He's My Brother")
  (creator = "THE HOLLIES")
  (creation-date = "2002")
  #;(featuring = "")
  (collection-name = "Legends CD2")
  #;(collection-volume = "")
  #;(collection-volumes = "")
  (volume-index = 16)
  (volume-size = 18)

  (mtime = 1428948696.0)
  (ctime = 1428948696.0)
  (size = 4063360))

It prints out unknown ID3 tags as comments, in case a human can glean some useful information from them to put into the metadata, and it suggests the names of metadata tags I might be able to provide by hand that it hasn't found (in this case, a tag for other people featured in the music, and two for indicating that this album is part of a set. As it happens, it is, as the "CD2" in the name suggests, but it wasn't indicated in the ID3 so I'll have to hand-edit it; likewise, the date from the MP3 of 2002 is clearly for the production of the album, not that classic track... ID3 metadata is often a bit shabby!). Also included are file mtime, ctime, and size in bytes.

I hope to add Ogg Vorbis metadata next; I'd like to add EXIF support to parse information out of the JPEGs in our vast family photo library, but it looks much harder, and I'm not sure how useful it will actually be!