zmiku: An automation daemon (by alaric)

A few years ago, I wrote my own service monitoring system for my servers and networks; I did this because Nagios, the common choice, was just too complicated for my tastes and didn't cleanly fit my needs. And so, The Eye of Horus was born, and has been monitoring my servers ever since. I open-sourced it, but I've not migrated it to the new Kitten Technologies infrastructure yet, so I don't have a link.

A design goal for Horus was to limit what needed to be installed on the monitored servers; it's a Python script that you run from cron which sshes into the servers and runs shell commands directly; the results are sucked back from standard output. The configuration file format is easy to work with, and it's modular - the python script spits out a new status file listing the status of all the services that a set of CGIs uses to produce HTML reports on demand and to update rrdtool logs of measurables, and produces a list of changes to be fed to a notification system.

However, it has some rough edges - I decided to make the shell commands run on the remote servers all output a standard format of report, which means mangling the output of commands such as pidof with sed and awk in order to produce them, which is a pain to do portably. In general, support for generating different commands to get the same report on different platforms is poor, too. I never got around to implementing hysterisis in the change detectors to put a service that's rapidly failing and recovering into an "unstable" state. And it's written in Python, when I've migrated all of my new development into Scheme.

I was tinkering with the idea of a straight rewrite in Scheme, with the rough edges fixed up, when I noticed a convergence with some of my other projects beginning to form.

I've long wanted to have a system where some small lightweight computer (perhaps a Raspberry Pi), attached to the home LAN, drives speakers as a music player, streaming music from the home file server. There's off the shelf stuff to do that, but I wanted to go a little further and also provide a text-to-speech notification system; the box would also have a queue of messages. If the queue was not empty, it would pause the music (perhaps with a nice fade), emit an announcement ding sound, then play the messages in turn via a text-to-speech engine. I had previously had some success in helping my wife manage her adult ADHD by putting a cronjob on her Mac that used the "say" command to remind her when it was time to have lunch and the like, as she easily gets too absorbed in something on her laptop and forgets everything else; I thought it would be good to extend that so it worked if she wasn't near her laptop, by making it part of a house-wide music system composed of music streamers in many rooms. And it would be a good place to route notifications from systems like Horus, too. And as the house we lived in then had a very long driveway, we could have a sensor at the end of the drive speak a notification if a car entered the driveway (in the new house, we have a similar requirement for a doorbell that can be heard in distant rooms...). And so on.

But that started to lead to similar design issues as the notification system in Horus; sometimes a single event causes a lot of notifications to be generated, which spam the user when you really just want a single notification that tells them all they need to know. Horus has some domain-specific knowledge about what services depend on what listed in the configuration file, and it automatically suppresses failures that "are to be expected" given root failures, but it could be smarter (for instance, if the failure occurs after the root service has been checked and is fine but before the child services have been checked, then it will notify of the failure of all the child services, rather than noticing the suspicious trend).

And when multiple events occur in the same time span, yet are unrelated so such tricks can't be applied, some notion of priority and rate limiting need to be applied. If ten thousand notifications suddenly appear in the queue in a single second, what's the system to do? Clearly, it will start fading the music down the very instant a notificatoin arrives, but by the time it then gets to start talking a second later, it may have recevied a lot of messages; now it needs to decide what to do. Repeated messages of the same "type" should be summarised somehow. A single high-priority message should be able to cut through a slew of boring ones. And so on.

At the same time, I was looking into home automation and security systems. There you have a bunch of sensors, and actions you want to trigger (often involving yet more notifications...) in response to events. And similarly I wanted to try and automate failover actions; host failure notifications in Horus should trigger certain recovery activities - but only if the failure state lasts for more than a threshold period, to make sure expensive operations are not triggered based on transient failures.

Programming these complex "rules", be they for automation, analysing the root cause of failures from a wide range of inter-dependent service statuses, or deciding how best to summarise a slew of messages, is often complex as they deal with asynchronous inputs and the timing relationships between them, too; specialist programming models, generally based around state machines, help a great deal.

Also, having a common infrastructure for hosting such "reactive behaviour" would make it possible to build a distributed fault-tolerant implementation, which would be very useful for many of the above problems...

So, I have decided, it would be a good idea to design and build an automation daemon. It'll be a bit of software that is started (with reference to a configuration file specifying a bunch of state machines), and then sits there waiting for events. Events can be timers expiring, or external events that come from sensors; and the actions of the state machines might be to trigger events themselves, or to activate external actuators (such as the text-to-speech engine or a server reboot). And a bunch of daemons configured to cooperate would all synchronise to the same state in lock-step; if daemons drop out of the cluster, then all that will happen is that sensors and external actions attached to that daemon will become unavailable, and state machines which depend on them will be notified. In the event of a network partition separating groups of daemons, the states can diverge; a resolution mechanism will need to be specified for when they re-merge.

Having that in place would mean that building a service monitoring system would merely involve writing a sensor that ran check commands, and a standard state machine for monitoring a service (with reference to the state machines of services it depends on), generating suitable events for other consumers of service state machine to use - and human-level notification events in a standard format recognised by a general human notification handler running in the same automation daemon cluster.

The shared infrastructure would make it easy to integrated automation systems.

Now, this is a medium-term project as what I have is working OK for now and I'm focussing on Ugarit development at the moment, but I am now researching complex event processing systems to start designing a reliable distributed processing model for it. And I've chosen a name: "zmiku", the Lojban word for "automatic" or "automaton"; its goal is, in general, to automate complex systems. As I apply it to more problems, I'd like to bring in tools from the artificial intelligence toolbox to make it able to automate things in "smarter" ways; I feel that many of these techniques are currently difficult to employ in many cases where automation is required, so it would be good to make them more available.

Mary and Jean (by sarah)

I thought I'd just do a little catch up on the girls - Mary is walking, saying Mumma, hitting people because she is teething, keeps trying to dress herself, when that fails she brings the clothes to me to dress her then once dressed she gets her shoes and tries to push the pushchair over to me. If she doesn't want something she will grab it and throw it just to show she really does not want it!

She has a favourite toy - a little hamster who is her baby, she tries to play the recorder and she loves drums. She giggles and cries at the slightest things and loves snuggles.

Jean is obsessed with acronyms especially recursive ones which she spells out and says ALL THE TIME. She is having stress attacks over where to put apostrophes when something belongs to a plural and is correcting my spelling. She has been moved up a class with some of her friends for spelling and things. She is obsessed with Harry Potter and wants me to do her hair in Hermione styles and talks in parseltongue.

She is sad science club has finished but is happy she is getting to do her dancing at school. She sings her beavers theme all the time and wants to take part in Jujistu competitions. She lies about having washed both body and teeth so keeps being in trouble and is getting sneakier about it by the day!

She has declared she wants four jobs when she is older:

She wants to be in bands (three of so she can play the drums, guitar and sing), then she wants to be a Vet as she loves animals and wants to make them better, then she wants to be an artist (specifically she wants to be the colour artist to comic book/animation line drawings and her colouring is fantastic!) and lastly she wants to be a teacher.

I have to ration the amount of time she spends reading and doing maths so she actually goes and plays - when her friends come round she organises them into bands and theatre groups and puts on shows for me.

I think that is everything currently.

Oh apart fromt he fact that Mary escapes all and any strap you put her in and has taken to posting what ever she can into the bin if you don't watch her!

Jean is picking up Lojban even though Alaric is only really speaking it to the baby and she keeps coming home saying things in Spanish and French. She is also doing her own writing ranging from poetry to stories though most of them involve Daleks.

Cake Pops (by sarah)

This is our first batch of cake pops - made as a family for a delayed May Day celebration.

I had wanted to get this machine at christmas but with house moving and stuff I couldn't.

We all had fun making the cake pops 🙂

I think they looked good as the table centre piece 🙂

Here are Jean's cake pops

Here are Alaric's Goth Pops 🙂

Out of mine there were only two good designs - black icing with purple edible glitter which I could not get a good photograph of and my topary cake pop.

I wasn't sure about the thickness of icing etc... so we had made up different thicknesses to try and we decorated some then and there and others we left to dry and decorated with icing pens and things later on!

Alaric and Jean had fun making cake pop pals for me which I filmed and will be making wiggly pet e-comic type things with the still photos 🙂

There will be more cake pops and cake pop pals 🙂 My big project is a cake pop bucky ball 🙂

Science the Universe and Everything erm… including Cake Pops (by sarah)

So I have had my eye on a cake pop machine since before christmas and have bought one for basically £20 from lakeland as the amount of ideas I am having for them means I can't sleep and Jeany wants a darleck cake for her birthday so I would have to get one then!

But I have also been thinking on science and science communication and things right so two creative things that aren't cooking have come out of this already.

One is a bizar video of the cake pop characters that Jean and Alaric made called The Gape Mouthed One and is obviously part of a series not to mention the stills I tool will be finding their way onto the Wiggly Pets blog.

Then there is a poem. It is about the way I think basically and came out of Alaric's impression of me talking just after we got the cake pop machine. I have recorded as it was only written last night and I haven't had a chance to sort out the spelling issues yet 🙂

It is called ADHD The Polymath Dream.

On top of that I have found myself looking at Science Communication courses which is probably nuts but it occured to me that I now live in walking distance of a train station that can get me at least to either London or Bristol.

Me, Science, Geekery and Art (by sarah)

I have two thirds of the funding I need for phase on of Science-Art for all. It would be great to get that last bit of money in the pot so I start work on the project.

Also I am working hard on getting Ballads of the Scientifica in all of its forms ready. In case I haven't mentioned it before I have my Science-Art website and I also have Astronomy@Geology.

I am very happy about my science-art at the moment and where it is going. I feel that art is probably the best way in which to get the general public to engage with science and communication of the ideas and concepts to EVERYBODY is of upmost importance.

Science permeates every aspect of our lives and I do not just mean medicine and technology. Food, water, our clothing - all of it relies on science. But understanding that science is important - that it is the cure rather than the disease is still being lost.

Science is still seen as a dead boring subject - but it really isn't. It uses creativity as much as any art. The wonder of everything around us - this we need to show people. There has been some wonderful popular science programmes over the last few year which are helping to instill the awe of everything in people but we are a long way off.

I use art to try and communicate emotions and concepts. And am very pleased to have had some of my work likened to Carl Sagan.

I started the Ballads of the Scientifica to help me put all of my spoken and sung science-art in one place but have found it is linked to my visual art very much.

Here it is though it is still being worked on:

But my work is still infantile compared with the group I found yesterday called Symphony of Science. Mary and Jean both love the songs - that are made out of snippets of documentaries and the like.

And then just as I was thinking about it all I turned around to find the baby now 15 months had been selecting books off the bookshelf. She had a maths book, another on electrical experiments and the last was 40 More Gadgets For the Evil Genius - the geek in me found this halarious 🙂

I do still miss the lab - though and the deeper into art I sink the more keenly I feel the lose. I want to blast rocks with military grade lasers to see what they are made of - to make element maps and to piece together the evolution of systems be they life or rock.

But this isn't doable for me - not at the moment and I am over whelmed by the wonder of the naked ape infants I have in this house ie Jean and Mary. My main hope is that I am showing them the beauty of it all and how to think for themselves.

And I do actually feel very hopeful at the moment - I have a meeting about my science-art this afternoon then tomorrow I'm performing in Gloucester.