zmiku: An automation daemon (by alaric)

A few years ago, I wrote my own service monitoring system for my servers and networks; I did this because Nagios, the common choice, was just too complicated for my tastes and didn't cleanly fit my needs. And so, The Eye of Horus was born, and has been monitoring my servers ever since. I open-sourced it, but I've not migrated it to the new Kitten Technologies infrastructure yet, so I don't have a link.

A design goal for Horus was to limit what needed to be installed on the monitored servers; it's a Python script that you run from cron which sshes into the servers and runs shell commands directly; the results are sucked back from standard output. The configuration file format is easy to work with, and it's modular - the python script spits out a new status file listing the status of all the services that a set of CGIs uses to produce HTML reports on demand and to update rrdtool logs of measurables, and produces a list of changes to be fed to a notification system.

However, it has some rough edges - I decided to make the shell commands run on the remote servers all output a standard format of report, which means mangling the output of commands such as pidof with sed and awk in order to produce them, which is a pain to do portably. In general, support for generating different commands to get the same report on different platforms is poor, too. I never got around to implementing hysterisis in the change detectors to put a service that's rapidly failing and recovering into an "unstable" state. And it's written in Python, when I've migrated all of my new development into Scheme.

I was tinkering with the idea of a straight rewrite in Scheme, with the rough edges fixed up, when I noticed a convergence with some of my other projects beginning to form.

I've long wanted to have a system where some small lightweight computer (perhaps a Raspberry Pi), attached to the home LAN, drives speakers as a music player, streaming music from the home file server. There's off the shelf stuff to do that, but I wanted to go a little further and also provide a text-to-speech notification system; the box would also have a queue of messages. If the queue was not empty, it would pause the music (perhaps with a nice fade), emit an announcement ding sound, then play the messages in turn via a text-to-speech engine. I had previously had some success in helping my wife manage her adult ADHD by putting a cronjob on her Mac that used the "say" command to remind her when it was time to have lunch and the like, as she easily gets too absorbed in something on her laptop and forgets everything else; I thought it would be good to extend that so it worked if she wasn't near her laptop, by making it part of a house-wide music system composed of music streamers in many rooms. And it would be a good place to route notifications from systems like Horus, too. And as the house we lived in then had a very long driveway, we could have a sensor at the end of the drive speak a notification if a car entered the driveway (in the new house, we have a similar requirement for a doorbell that can be heard in distant rooms...). And so on.

But that started to lead to similar design issues as the notification system in Horus; sometimes a single event causes a lot of notifications to be generated, which spam the user when you really just want a single notification that tells them all they need to know. Horus has some domain-specific knowledge about what services depend on what listed in the configuration file, and it automatically suppresses failures that "are to be expected" given root failures, but it could be smarter (for instance, if the failure occurs after the root service has been checked and is fine but before the child services have been checked, then it will notify of the failure of all the child services, rather than noticing the suspicious trend).

And when multiple events occur in the same time span, yet are unrelated so such tricks can't be applied, some notion of priority and rate limiting need to be applied. If ten thousand notifications suddenly appear in the queue in a single second, what's the system to do? Clearly, it will start fading the music down the very instant a notificatoin arrives, but by the time it then gets to start talking a second later, it may have recevied a lot of messages; now it needs to decide what to do. Repeated messages of the same "type" should be summarised somehow. A single high-priority message should be able to cut through a slew of boring ones. And so on.

At the same time, I was looking into home automation and security systems. There you have a bunch of sensors, and actions you want to trigger (often involving yet more notifications...) in response to events. And similarly I wanted to try and automate failover actions; host failure notifications in Horus should trigger certain recovery activities - but only if the failure state lasts for more than a threshold period, to make sure expensive operations are not triggered based on transient failures.

Programming these complex "rules", be they for automation, analysing the root cause of failures from a wide range of inter-dependent service statuses, or deciding how best to summarise a slew of messages, is often complex as they deal with asynchronous inputs and the timing relationships between them, too; specialist programming models, generally based around state machines, help a great deal.

Also, having a common infrastructure for hosting such "reactive behaviour" would make it possible to build a distributed fault-tolerant implementation, which would be very useful for many of the above problems...

So, I have decided, it would be a good idea to design and build an automation daemon. It'll be a bit of software that is started (with reference to a configuration file specifying a bunch of state machines), and then sits there waiting for events. Events can be timers expiring, or external events that come from sensors; and the actions of the state machines might be to trigger events themselves, or to activate external actuators (such as the text-to-speech engine or a server reboot). And a bunch of daemons configured to cooperate would all synchronise to the same state in lock-step; if daemons drop out of the cluster, then all that will happen is that sensors and external actions attached to that daemon will become unavailable, and state machines which depend on them will be notified. In the event of a network partition separating groups of daemons, the states can diverge; a resolution mechanism will need to be specified for when they re-merge.

Having that in place would mean that building a service monitoring system would merely involve writing a sensor that ran check commands, and a standard state machine for monitoring a service (with reference to the state machines of services it depends on), generating suitable events for other consumers of service state machine to use - and human-level notification events in a standard format recognised by a general human notification handler running in the same automation daemon cluster.

The shared infrastructure would make it easy to integrated automation systems.

Now, this is a medium-term project as what I have is working OK for now and I'm focussing on Ugarit development at the moment, but I am now researching complex event processing systems to start designing a reliable distributed processing model for it. And I've chosen a name: "zmiku", the Lojban word for "automatic" or "automaton"; its goal is, in general, to automate complex systems. As I apply it to more problems, I'd like to bring in tools from the artificial intelligence toolbox to make it able to automate things in "smarter" ways; I feel that many of these techniques are currently difficult to employ in many cases where automation is required, so it would be good to make them more available.

Science the Universe and Everything erm… including Cake Pops (by sarah)

So I have had my eye on a cake pop machine since before christmas and have bought one for basically £20 from lakeland as the amount of ideas I am having for them means I can't sleep and Jeany wants a darleck cake for her birthday so I would have to get one then!

But I have also been thinking on science and science communication and things right so two creative things that aren't cooking have come out of this already.

One is a bizar video of the cake pop characters that Jean and Alaric made called The Gape Mouthed One and is obviously part of a series not to mention the stills I tool will be finding their way onto the Wiggly Pets blog.

Then there is a poem. It is about the way I think basically and came out of Alaric's impression of me talking just after we got the cake pop machine. I have recorded as it was only written last night and I haven't had a chance to sort out the spelling issues yet 🙂

It is called ADHD The Polymath Dream.

On top of that I have found myself looking at Science Communication courses which is probably nuts but it occured to me that I now live in walking distance of a train station that can get me at least to either London or Bristol.

Me, Science, Geekery and Art (by sarah)

I have two thirds of the funding I need for phase on of Science-Art for all. It would be great to get that last bit of money in the pot so I start work on the project.

Also I am working hard on getting Ballads of the Scientifica in all of its forms ready. In case I haven't mentioned it before I have my Science-Art website and I also have Astronomy@Geology.

I am very happy about my science-art at the moment and where it is going. I feel that art is probably the best way in which to get the general public to engage with science and communication of the ideas and concepts to EVERYBODY is of upmost importance.

Science permeates every aspect of our lives and I do not just mean medicine and technology. Food, water, our clothing - all of it relies on science. But understanding that science is important - that it is the cure rather than the disease is still being lost.

Science is still seen as a dead boring subject - but it really isn't. It uses creativity as much as any art. The wonder of everything around us - this we need to show people. There has been some wonderful popular science programmes over the last few year which are helping to instill the awe of everything in people but we are a long way off.

I use art to try and communicate emotions and concepts. And am very pleased to have had some of my work likened to Carl Sagan.

I started the Ballads of the Scientifica to help me put all of my spoken and sung science-art in one place but have found it is linked to my visual art very much.

Here it is though it is still being worked on:

But my work is still infantile compared with the group I found yesterday called Symphony of Science. Mary and Jean both love the songs - that are made out of snippets of documentaries and the like.

And then just as I was thinking about it all I turned around to find the baby now 15 months had been selecting books off the bookshelf. She had a maths book, another on electrical experiments and the last was 40 More Gadgets For the Evil Genius - the geek in me found this halarious 🙂

I do still miss the lab - though and the deeper into art I sink the more keenly I feel the lose. I want to blast rocks with military grade lasers to see what they are made of - to make element maps and to piece together the evolution of systems be they life or rock.

But this isn't doable for me - not at the moment and I am over whelmed by the wonder of the naked ape infants I have in this house ie Jean and Mary. My main hope is that I am showing them the beauty of it all and how to think for themselves.

And I do actually feel very hopeful at the moment - I have a meeting about my science-art this afternoon then tomorrow I'm performing in Gloucester.

The Annes and C J Cherryh (by sarah)

This post is about inspiration and following dreams and what I consider good fiction writing. However before I begin let me just say that I don't believe gender has anything to do with someone being a good writer - but this post is about females and their perceived place in literature.

Now I have always loved stories, loved hearing them, creating them and taking part in them. My dad used to read or make up stories for me pretty much every night until he had to work nights stacking shelves in Tescos, then the stories were restricted to holidays or sometime he would read out bits from what ever novel he was reading at the time.

One of my early favourites was Anne McCaffrey and her Dragons of Pern, though I actually started with the Doona books. What child head full of Thunder Cats could not like these stories?

I struggled with reading as many of you know but the burning wish to read these stories for myself grew until I was painstakingly stringing the sentences together myself.

Then I read Anne McCaffrey - everything the library had (though after I had read all of their point horror collection I will confess). You see I went from a can't read to an avid reader - I used to walk home from school reading a book and I walked home to do this as the bus was too noisy and crowded to read.

From the First Point Horror book that I managed to read at 12 years old - I was consuming the bigger of Anne McCaffrey's books in 2-3 days by the time I was 14.

This was also the point at which me and my friends became obsessed with horror movies and of course being teenage girls we fixated on VAMPIRES!

Which we avidly watched at our sleep overs - films like Lost Boys, and My Best Friends a Vampire and the new movie Interview With A Vampire - oh we were obsessed. Which was better Louis or Lestate? But for either you would need to kill them or become like them and surely they were dating each other? And so on (remember we were teenage girls).

This was another Anne having an affect on me - this time Anne Rice.

I mention this as it ment I suddenly started trying to write about vampires but due to the influence of Anne McCaffrey's books I felt I wanted to have my vampires explained with soft science fiction - essentially I had the idea of not just writing Scifi or even scifi fantasy but of melding the genres having a scifi fantasy horror cross breed and of course it all had to be mixed up with legends and myths ranging from norse to ancient greece and aztec. I had not yet touched on India or China or Japan and their myths.

At around this time I discovered another Anne - Anne Auel and her Clan of The Cave Bear series. Now it turned out my Mum was a fan of her books and I was only just allowed to read them!

The strength of the main character in at least the first two books (I start to have issues half way through Mammoth Hunters with how she treats men) led me even more onto the idea of including pre-history and the mysteries of our origin as a species.

It was then that my friends aunt having heard us go on and on about Interview with a Vampire bought my friend Taltos - a story by Anne Rice. We were like wow there are books by the vampire lady? And that was the next thing - starting with The Witching Hour.

These were along with Stephen King, Stephen Lawhead , Dean Koontz and Clive Barker were my big inspirations. Now imagine my shock to be told that a) these books and authors don't count as great works of literature and b) that girls don't write scifi or even read it?

During my A'levels I took a creative writing short course to help me as I was trying to write and side tracking myself with new stories and the like. The result was I got several poems published, ended up as part of a team doing a magazine which I lost heart with as the two main people had very different ideas of where it was going. And told by the Lady running it that my scifi vampire story had the makings of a best seller.

I doubt if this was true but I needed that boost - especially after being told by peers I couldn't have my female assassins and vampire warriors without it being all about the boobies. I didn't understand this as concept as I had been reading books with strong female characters doing all sorts, half of which were written by women. And of course though I wanted to punch her half the time there was Buffy the Vampire Slayer around as well.

I found the vampire story I was working on back then - the first three chapters printed on old yellowing tractor paper (yep I was retro as we had like fifth hand computing equipment). It was called The Forever Criminals and reads unsurprisingly like a point horror story as does The Crystal Singer (nothing to do with Anne McCaffrey's Crystal Singer - mine involved jade skulls and a female mulan-esk warrior) and The Warrior Monks (a story of post apocalyptic epic adventure). I can see I was aiming for the epic even then.

And the thought that I was supposed to write ROMANCE sickened me - it was something I found anathemous. I didn't like reading romance so why should I write it?

But I have found this repeatedly with people and I think I was also lucky to go to Imperial College were there was a whole Scifi Library. I had by this point also discovered Terry Pratchett who's female characters again were not the weak and willing and nor where they there just for the boobies.

However as I found more and more great scifi authors moving away from my beloved horror I started to feel the doom. I had proof that women could write horror and fantasy and scifi fantasy but what of scifi?

And then Alaric stepped in with CJ Cherryh and I knew I was right and that it shouldn't even be questioned that women like scifi or that they should even think of perhaps writing some of their own!

Anyway I what I have found is that two of these authors have continued to be an inspiration in me in my writing and creativity. This is CJ Cherryh who is on twitter and has partaken in some of the # scifichats I attend and the other is Anne Rice who has her Face Book page and actively encourages interaction and discussions - part of the reason this has been so valuable to me is that these two people who's work I love - I get to see they are people like me - they have cats and rooms stuffed with books etc... They are people who have struggled and suffered and not some mysterious force that churns out books.

Now it is important for me to see this sort of connection with people who have succeeded in an area I want to succeed in. Especially as I am still coming across people who are horrified that I wrote that horror story when they would have been fine with it if Alaric had written it. But I am no Mary Shelley - I do not need to hide behind a surname - women have the vote and have proven time and again that they can achieve if given a chance.

Our society has changed but there is still alot of prejudice out there - this is a minor thing to most people but even in rescent publications I have seen it suggested that you submit your scifi under a male pseudonym which is horrendous!

Gloucester Creative Olympics and The Launch of The Wind Vessel (by sarah)

Yesterday I went to The Battle For The Winds Episode One in Stroud, I had won a bronze medal for my song Windy Gloucestershire. This was part of the creative Olympics and also saw the launch of the Wind Vessel which will be in Cheltenham for the Torch and then will be going on to Wynmouth with the rest of the South West efforts.

It was in the Brunel Goodshed - there was a parade as well but with the weather an us all having been so ill recently we decided to miss that bit. The place was jam packed!

What we could see of the vessel being launched though Alaric and Jean seemed to really enjoy 🙂

There is unfortunatly no video or photos of me on stage - Alaric managed to take two one second videos at the beginning and end of my bit being presented my medal - it would have been a good angle too 🙁

My medal was made by the graphics students at the college. It was presented by Old Mother Gloucestershire who's outfit was brilliant 🙂

But here are the photos we did take of the general stage stuff before and after my bit 🙂

The we saw the launch of the Wind Vessel - which I actually really like - plus it doubles as a musical instrument and I always like things like that 🙂

After the main bit people started to disperse a bit and there was more space to see the martial art dancers and various other little shows. I went to the tent with the Creative Olympic stuff in but though people were asking to listen to my song - it was unfortunatly the only one not there - even though the organisers said it had been - so someone has borrowed it I assume or it's down the back of tent or something!

The place was full of little cafes and a giant junk igloo called the Mould Cafe - this led through to a section where kids could put their heads through and seem themselves as part of a video.

Mary was desperate to join in so once the crowds had thinned a bit we let her loose to toddle - but the main thing she wanted to do was be held up to the gymnasts ring!

And because I am me - I took arty shots of things 🙂