It Gets Worse (by )

Here is a news article about Alex that I recieved after having posted earlier

http://www.dailymail.co.uk/news/article-1053151/Six-dead-flood-chaos-thousands-evacuated-homes--weather-set-WORSE.html

I am feeling so sick right now with this.

edit: another news article

http://news.bbc.co.uk/1/hi/england/gloucestershire/7603417.stm

Another Friend Gone (by )

With the taste of one friends memorial still in my mouth from Tuesday I came home last night to find a message that another much younger friend had been killed in an accident at work.

Alaric had already left for London when I saw the message and I hadn't yet unpacked things from the weekend so I couldn't initially find my phone. I didn't and to a certain extent still don't beleive it. I became single minded in one thing - I had to phone Al I couldn't grip on to any other thing in reality.

I found the charger but then couldn't find the phone and then found the phone but couldn't find what I'd done with the charger - I was panicking as if I could change things.

I phoned Alaric and just got the answere phone so I just kept phoning I couldn't think of what else to do. I started doing random things to try and proove to myself that it was some horrible practicle joke - but deep down I knew it wouldn't be - this was the same reaction I had two years ago over Wilson's death but with out the anger.

I looked up his work website thinking that would help - he works up at Birdlip - basically in walking distance. I don't even know why I am writing this becuase it would just embarrasse him if he read it. Alex was at uni with me in my first first year and we had done stuff like caving. He also helped out alot when we where flooded and would meet me in Cheltenham for drinks.

We hadn't met up recently as things had been hectic and we just werent being free at the same time - we where going to do something probably a meal and I was looking forward to meeting his girlfriend that I had heared so much about.

I got through to Alaric and all I could say was please come back - I said I;d got an email saying Alex had been killed. Al said, 'Oh god I'm coming back' he hadnt been able to understand anything else I'd said.

The thing is mums going in for her second partial mastectamy today and normally I would be in getting texts from him asking how it was going but thats not going to happen now and stupid as it sounds I feel like everyone is dying on me.

Alex was so nice and helpful he was even going to lend me some stuff from his scout group for mine. I just can't face that this might possibly be true and at the same time I keep crying.

Filling the lake (by )

With all the rain recently, the place where I park the van has developed an annoying tendency to be a lake. So I have to paddle to get in and out of the van, and so does Sarah - Jean, being sensible, just refuses and asks to be carried.

The parking lake

So I invoked the power of Freecycle and obtained a van full of hardcore, then used a pickaxe to dig out the mixture of hard-packed gravel and mud that comprises the parking area, making a trench which I filled back with hardcore.

A trench full of rubble

After packing it down with a lot of treading, thumping, and the use of the big concrete roller, I dumped the gravel/mud mixture back in, to pack back together again and provide a smoothish surface.

The gravel laid back on top

I only had enough to do that one strip, but it's a start! When I've stopped aching from all this work, I'll bid for another load of hardcore on Freecycle and do the side where I get out - then another load for the strip down the middle...

UPDATE

It soon started to rain heavily later that day, so I took another picture. Sure enough, the lake is now constrained to one side of the parking area, and the bit I built up is kept out of the standing water. Yay!

Now when it rains, there's a non-lakey bit. Yay!

Managing lots of servers (by )

The way OSes work was rather designed around the notion of large centralised systems, which is increasingly out of date. There's too much per-node configuration splattered around all over the place; you have to manually set up processes to deal with it all, otherwise you start to lose the ability to replace machines since their configuration contains mysteries that you may or may not be able to recreate if you rebuild the OS install. You really need to be able to regenerate any machine from scratch in as little time as possible - not just restoring from a backup; you might need to recreate the function of a machine on new hardware, since you can't always get replacements for old setups.

Now, if you can get away with it, you can build your application to support distributed operation - then just set up a sea of identical OS installs by using a configuration script that sets up a server and runs an instance of your distributed app, but a lot of off-the-shelf software sadly fails to support that kind of operation. If you want to run standard apps that use filesystems and SQL databases to store their data, you need to be cunning.

How can we do better? Here's a few (rather brief) notes on an approach I'm experimenting with.

Boot your OS from a CD

Not quite a liveCD, though, since we do want to actually use the local disk for stuff.

  • modified RC chain that (union?) mounts /etc from the hard disk and then runs with that to get local network etc. configuration.
  • swap, /tmp, /var etc on the hard disk, obviously.
  • Makes rolling out new versions of the OS easy; forces you to prototype the system on a hard disk on a staging server or VM, then when it's ready, burn a CD, test the CD in a staging server, then if it works, burn a hundred copies and roll them out. USB sticks are another option, but a little more awkward in datacentre environments. Cost of having a human go and re-CD every server exists but is low and provides safety compared to automatic rollouts that could go disastrously wrong. The fact you can roll back by putting the old CD back, having a truly read-only root filesystem and OS (making it harder to hide rootkits) is great, though!

Use Xen

  • The actual loaded OS is just a Xen dom0 setup
  • Prebuilt domU root images exist on the CD-ROM, which are then spun up (based on settings in /etc on the hard disk). The root images get given a partition from the hard disk which contains their /etc and swap, and any local storage they need, in much the same way as dom0 boots directly from the CD.
  • Or your read-only domU root images could be stored on the hard disks of the servers and rolled out via the network; the advantages of distributing them on CD-ROM are a lot smaller than for the dom0 OS, as dom0 can enforce the read-only nature of the domU images, provide remote access to roll back to an earlier version and try again if an upgrade turns out to be bad, etc.

Virtualise storage

  • Use local storage on servers just for cached stuff and temporary storage. Eg, we have each server's configuration stored on local disk so it can boot, but that's just checked out from subversion. We put swap on local disks. But the contents of any server's disks should be recreatable by checking the configuration out from SVN again and/or rsyncing any shared mainly-read-only data (domU images etc) from authoritative copies.
  • For actual data that we care about, use network protocols (iSCSI, NFS, SQL, etc) to talk to special reliable storage services.
  • For domUs that have a criticial local filesystem, we use iSCSI. However, we use software RAID to mirror (or parity-protect) the filesystem over more than one physical iSCSI server, so that either can fail without losing data or causing downtime. Since the domU itself then stores nothing, should it fail (or the physical server hosting it fail), an exact duplicate can be brought up on another physical server and it will connect to the same iSCSI servers to provide access to the same data (and we hope that the filesystem used can recover from any corruption that arose during the failure, or else we're toast anyway).
  • Higher level storage protocols (NFS, SQL, etc) are served out from domUs that, as above, have stable block-level storage from software-RAIDed iSCSI backends. And, likewise, should the NFS server go down, we can resurrect an identical clone of it from the same iSCSI backend disks and it will carry on with the state the failed one left behind.
  • But where possible, use proper distributed/replicated databases!

Details

  • The dom0 ISO contains a bootloader, Xen, and NetBSD set up as a dom0 kernel, with /usr/pkg containing a bunch of useful core packages (sudo, subversion-base, screen, xentools, etc)
  • The dom0 ISO chain will:
    1. Mount the first partition on the first disk in the server that has a special marker file in as /config
    2. union-mount /config/local/etc/ over /etc
    3. now read /etc/rc.conf
    4. Run the normal /etc/rc tasks, including mounting /var and /tmp from the hard disk, mounting data partitions and setting up networking, ipnat, ipf, etc.
    5. Scan the list of Xen domUs to start from /config/local/domUs/* and start them, each with the correct disk images (from the data partitions), MAC address, and memory allocations.
  • /config/local and /config/global are svn checkouts
  • On all machines (dom0s and domUs), /etc/hosts is a symlink to /config/global/hosts, and any other such useful files.
  • domUs run pkg_chk, but don't have /usr/pkgsrc; they fetch compiled binary packages from a repository domU running the same base OS, which builds every package in pkg_chk.conf. This domU might need to be the NIS master, since that would be the only way to keep pkgsrc-created role user UIDs in synch.

How to bootstrap it

  • We need documented procedures for setting up a dom0 iso image, to make sure no important steps are missed...
    • Make a working directory
    • Install NetBSD source sets
    • Set up custom /etc/rc that finds a suitable filesystem to locate /etc from and mounts it as /config - or drops to a shell if none can be found.
    • Make a Xen3 dom0 kernel with "config netbsd root on cd0a type cd9660 dumps on none" and "options INCLUDE_CONFIG_FILE"
    • Put in the Xen 3 kernel
    • Configure grub menu.lst to load NetBSD on top of Xen.
    • Install core packages (xen tools) - /var/db/pkg and /usr/pkg will union mount over what we provide to allow for node-local extensions, although we shouldn't need too many in dom0.
    • Install grub and mkisofs as per http://www.gnu.org/software/grub/manual/html_node/Making-a-GRUB-bootable-CD-ROM.html
  • We need domU read-only root filesystem images created along a similar theme

Subversion layout:

  • /pservers/ - root of /config/local for each physical server
  • /vservers/ - root of /config/local for each virtual server
  • /global - /config/global
  • /docs - configuration checklists and scripts for dom0 and domU master images

Living in Groups (by )

Until relatively recently in human history, people tended to live in small but relatively intimate groups, sharing a lot of domestic arrangements; a party would go out hunting, another went out gathering, others looked after the children, others cooked the food, and so on.

This was, quite simply, more efficient. Economies of scale meant that a small team could cook for a large group in less total person-hours than each person cooking for themselves - especially when you compare the time consumed making and maintaining cooking equipment and the like.

These days, the same economies of scale have had the opposite effect - food is now produced in factories, and easy-to-cook ingredients and ready meals are cheaply available; this, combined with all sorts of other socio-economic factors, has lead to it now being quite practical to live entirely alone, spending your days working then coming home to a small meal you cook for yourself in minutes, cleaning your dishes and clothes in a machine, cleaning your floors with a machine, and so on.

And, thus, I suspect the loneliness of bachelor living is probably a modern phenomenon. Without ready meals and domestic appliances, moving away from home would be an unattractive prospect until you had a partner to team up with in order to form a breadwinning/homemaking duo (and the fact that sexist role models enforced a certain split of duties is, I think, entirely orthogonal to this issue) - and when you team up with a partner is precisely when you really start to want to be away from your parents...

Some of the best living arrangements I've had have been as a student, when the also-interesting economics of the cost of a place to live in London would force us to share houses (and sometimes rooms). Although we rarely actually cooked for each other, living in the same house as several other people was psychologically comforting for me. I really don't function well at all when living on my own; I've never officially done it, but in situations where all my housemates happen to be away for a few days, I've definitely started to slide into depression.

I currently live with my wife and daughter, so I'm basically OK, but even then, we still wish there were other similar couples we could share some resources with; if our house was larger we'd have lodgers. Personally I think my ideal would be having my own bedroom, office, bathroom, and kitchen (although I'd often cook for others), but sharing a big living room and garden, and being in the same physical building. There's increased security in a house that's rarely totally empty, and efficiencies in sharing resources (such a house would take up much less space than several individual ones, and consume much less energy), and increased convenience (you'd be quite likely to be able to find somebody to help you with something). And you'd have good times together.

This was a nice thing about what I did last weekend, which was to go on camp with my cub scouts; you might think that, with my legendarily complicated and busy life, the last thing I need is to donate my time to a voluntary organisation. But I work solving complex mental tasks (mainly on my own), face the difficult challenges of supporting my family under trying circumstances (shared with my wife, but we still feel quite 'alone' as a small family without much support from our extended family); after weeks of that, a weekend of hard work solving relatively simple problems (how to wash the puddle of sick away from outside the tent full of sleeping children, when it's raining heavily and the sick is slowly being washed downhill towards the tent? Answer: get digging equipment, dig a trench in the little gap between the tent and the puddle, scrape it all in there with the spade then wash it in with water, then close the trench) as part of a team is a delightfully refreshing change. Much more refreshing than a holiday spent just doing nothing; I'd be fretting too much about all the jobs I should be doing at home. Volunteering means I'm doing something somebody needs me to do, but working with others so it's a fun team activity rather than an ordeal.

But I wonder how many people would be happier living in 'communes'. A friend of mine is a Hare Krishna; I'm indifferent to the religion, but their culture is excellent - and part of it seems to be a high acceptance of living in groups sharing resources, which I think is very healthy.

Perhaps there's an opening for a property developer to set up some buildings with little apartments that then share living areas. Obviously they couldn't just be sold as independent units; perhaps they'd need to be owned by a limited company of some kind and the mortgage repayments, rent, or other expenses paid by all the residents paying a share, since the residents would need to be able to vet and veto potential new housemates, as rifts occurring in such a community would be fatal.

In the meantime, I wish our house had room for lodgers 😉

WordPress Themes

Creative Commons Attribution-NonCommercial-ShareAlike 2.0 UK: England & Wales
Creative Commons Attribution-NonCommercial-ShareAlike 2.0 UK: England & Wales