Fun in computer games (by alaric)

I've noticed a pattern in computer games which I find fun. Not all games I find fun; they can be fun in different ways. I'm just saying I've noticed a particular element which subtly contributes towards the funness.

Namely, having to make a tradeoff between two or more competing requirements.

Let's have an example - Desktop Tower Defence. It's a tower defence game, which means that you use your resources to build a set of defences that waves of attackers then flow into.

Firstly, clever placement of defences has a much greater effect than simply how much you spend on them. So the game requires some measure of thought, rather than repetitive accumulation of resources followed by spending them.

But the crux of the matter is that there are different kinds of attackers, which have different weaknesses. A defence set up in the way that would be the strongest against land-based attackers - a long winding zig-zag with turrets along it - would be weak against flying attackers, since they just fly over your layout in a straight line rather than being constrained to the paths. Against them, you want a solid block of turrets in a cross, under the two orthogonal lines they fly along. So you need to establish some tradeoff between the two challenges. Not to mention that there are turrets which only attack air targets, but have a high damage per cost ratio, and turrets which only attack ground targets, and turrets that attack both but have a worse damage per cost ratio. And turrets with long ranges, or high fire rates, or that do a lot of damage per shot, or damage neighbouring targets due to a splash effect, and so on.

Sometimes you can have a tradeoff that's too simple - it's amenable to mathematical analysis to find an optimal result. That's no good. It has to be too complex to work out on paper, but not too complex to grasp. The middle ground between the two is the area where experimentation is rewarding.

I probably ought to read A Theory of Fun for Game Design...

Plan B (by alaric)

I've hired a car to take up to the Lake District this weekend.

A Peugeot 308 with 350 miles on the clock that smells of new, picked up in London Euston, drop off in Gloucester, four days, for £166 from Avis at zero notice and with nothing more than a debit card and a clean licence with the same name on and a photo that matches my face - not bad at all. 1car1 was offering £200 for the same thing, but I'd need a utility bill, which by some miracle I don't happen to normally carry with me!

The van will stay here until I return next week and then I'll transport it to a place of fixing.

I broke the van! (by alaric)

While in London with the van, I had to deliver a server to a data centre - (InterXion London)[http://www.datacentermap.com/united-kingdom/london/interxion-london.html]. There's parking at the DC, but it's £20 to park a van there, so I decided to try and get into the car park marked just north of it on the map, in Quaker Street.

While I could find the car park, I couldn't find the way in - I could see cars all parked behind a fence, but no sign of the entrance. But while navigating the narrow little roads, I had to turn around in Grey Eagle Street:

View Larger Map

The road was narrow, only a single track, but with a wide pavement, so I opted for a three point turn. I turned hard right to bring the front of the van up onto the pavement, and the right wheel went up, but then I heard a grinding scraping noise. I was a bit surprised to be bottoming out on a not-particularly-high kerb, but I reversed back and found a place further down with a dropped kerb (but less pavement, since there was a car parked up there) to turn.

However, I noticed that my steering was suddenly a bit funny. I had to hold the wheel at about seventy degrees left to drive in a straight line. And the suspension felt odd - I could feel every little stone in the road, and if I went over even the slightest bump, I heard a funny creaking sound.

BAD NEWS!

After a quick check for visible signs of damage as soon as I had parked (none), I did the business of the evening, and then drove carefully back to Highgate for a closer look.

I parked with the wheel perfectly straight, then got out and looked at both wheels.

Here's the left one, pointing nice and straight ahead:

And here's the right one... pointing twenty or so degrees to the right:

Looks like the tracking's totally out, then.

And by the scientific method of placing my hand in the gap between top of tyre and wheel arch then holding it still while I walk around and try on the other side, the right hand side seems an inch or so lower than the left, too.

And all this when I'm supposed to be doing a five-hour drive up motorways tomorrow evening. Eeek.

I'm going to see if I can find a place to look at my tracking and suspension first thing tomorrow... but I'm dreading what it'll cost, and doubting I'll get it fixed the same day...

Filling the lake (by alaric)

With all the rain recently, the place where I park the van has developed an annoying tendency to be a lake. So I have to paddle to get in and out of the van, and so does Sarah - Jean, being sensible, just refuses and asks to be carried.

So I invoked the power of Freecycle and obtained a van full of hardcore, then used a pickaxe to dig out the mixture of hard-packed gravel and mud that comprises the parking area, making a trench which I filled back with hardcore.

After packing it down with a lot of treading, thumping, and the use of the big concrete roller, I dumped the gravel/mud mixture back in, to pack back together again and provide a smoothish surface.

I only had enough to do that one strip, but it's a start! When I've stopped aching from all this work, I'll bid for another load of hardcore on Freecycle and do the side where I get out - then another load for the strip down the middle...

UPDATE

It soon started to rain heavily later that day, so I took another picture. Sure enough, the lake is now constrained to one side of the parking area, and the bit I built up is kept out of the standing water. Yay!

Managing lots of servers (by alaric)

The way OSes work was rather designed around the notion of large centralised systems, which is increasingly out of date. There's too much per-node configuration splattered around all over the place; you have to manually set up processes to deal with it all, otherwise you start to lose the ability to replace machines since their configuration contains mysteries that you may or may not be able to recreate if you rebuild the OS install. You really need to be able to regenerate any machine from scratch in as little time as possible - not just restoring from a backup; you might need to recreate the function of a machine on new hardware, since you can't always get replacements for old setups.

Now, if you can get away with it, you can build your application to support distributed operation - then just set up a sea of identical OS installs by using a configuration script that sets up a server and runs an instance of your distributed app, but a lot of off-the-shelf software sadly fails to support that kind of operation. If you want to run standard apps that use filesystems and SQL databases to store their data, you need to be cunning.

How can we do better? Here's a few (rather brief) notes on an approach I'm experimenting with.

Boot your OS from a CD

Not quite a liveCD, though, since we do want to actually use the local disk for stuff.

• modified RC chain that (union?) mounts /etc from the hard disk and then runs with that to get local network etc. configuration.
• swap, /tmp, /var etc on the hard disk, obviously.
• Makes rolling out new versions of the OS easy; forces you to prototype the system on a hard disk on a staging server or VM, then when it's ready, burn a CD, test the CD in a staging server, then if it works, burn a hundred copies and roll them out. USB sticks are another option, but a little more awkward in datacentre environments. Cost of having a human go and re-CD every server exists but is low and provides safety compared to automatic rollouts that could go disastrously wrong. The fact you can roll back by putting the old CD back, having a truly read-only root filesystem and OS (making it harder to hide rootkits) is great, though!

Use Xen

• The actual loaded OS is just a Xen dom0 setup
• Prebuilt domU root images exist on the CD-ROM, which are then spun up (based on settings in /etc on the hard disk). The root images get given a partition from the hard disk which contains their /etc and swap, and any local storage they need, in much the same way as dom0 boots directly from the CD.
• Or your read-only domU root images could be stored on the hard disks of the servers and rolled out via the network; the advantages of distributing them on CD-ROM are a lot smaller than for the dom0 OS, as dom0 can enforce the read-only nature of the domU images, provide remote access to roll back to an earlier version and try again if an upgrade turns out to be bad, etc.

Virtualise storage

• Use local storage on servers just for cached stuff and temporary storage. Eg, we have each server's configuration stored on local disk so it can boot, but that's just checked out from subversion. We put swap on local disks. But the contents of any server's disks should be recreatable by checking the configuration out from SVN again and/or rsyncing any shared mainly-read-only data (domU images etc) from authoritative copies.
• For actual data that we care about, use network protocols (iSCSI, NFS, SQL, etc) to talk to special reliable storage services.
• For domUs that have a criticial local filesystem, we use iSCSI. However, we use software RAID to mirror (or parity-protect) the filesystem over more than one physical iSCSI server, so that either can fail without losing data or causing downtime. Since the domU itself then stores nothing, should it fail (or the physical server hosting it fail), an exact duplicate can be brought up on another physical server and it will connect to the same iSCSI servers to provide access to the same data (and we hope that the filesystem used can recover from any corruption that arose during the failure, or else we're toast anyway).
• Higher level storage protocols (NFS, SQL, etc) are served out from domUs that, as above, have stable block-level storage from software-RAIDed iSCSI backends. And, likewise, should the NFS server go down, we can resurrect an identical clone of it from the same iSCSI backend disks and it will carry on with the state the failed one left behind.
• But where possible, use proper distributed/replicated databases!

Details

• The dom0 ISO contains a bootloader, Xen, and NetBSD set up as a dom0 kernel, with /usr/pkg containing a bunch of useful core packages (sudo, subversion-base, screen, xentools, etc)
• The dom0 ISO chain will:
  1. Mount the first partition on the first disk in the server that has a special marker file in as /config
  2. union-mount /config/local/etc/ over /etc
  3. now read /etc/rc.conf
  4. Run the normal /etc/rc tasks, including mounting /var and /tmp from the hard disk, mounting data partitions and setting up networking, ipnat, ipf, etc.
  5. Scan the list of Xen domUs to start from /config/local/domUs/* and start them, each with the correct disk images (from the data partitions), MAC address, and memory allocations.
• /config/local and /config/global are svn checkouts
• On all machines (dom0s and domUs), /etc/hosts is a symlink to /config/global/hosts, and any other such useful files.
• domUs run pkg_chk, but don't have /usr/pkgsrc; they fetch compiled binary packages from a repository domU running the same base OS, which builds every package in pkg_chk.conf. This domU might need to be the NIS master, since that would be the only way to keep pkgsrc-created role user UIDs in synch.

How to bootstrap it

• We need documented procedures for setting up a dom0 iso image, to make sure no important steps are missed...
  • Make a working directory
  • Install NetBSD source sets
  • Set up custom /etc/rc that finds a suitable filesystem to locate /etc from and mounts it as /config - or drops to a shell if none can be found.
  • Make a Xen3 dom0 kernel with "config netbsd root on cd0a type cd9660 dumps on none" and "options INCLUDE_CONFIG_FILE"
  • Put in the Xen 3 kernel
  • Configure grub menu.lst to load NetBSD on top of Xen.
  • Install core packages (xen tools) - /var/db/pkg and /usr/pkg will union mount over what we provide to allow for node-local extensions, although we shouldn't need too many in dom0.
  • Install grub and mkisofs as per http://www.gnu.org/software/grub/manual/html_node/Making-a-GRUB-bootable-CD-ROM.html
• We need domU read-only root filesystem images created along a similar theme

Subversion layout:

• /pservers/ - root of /config/local for each physical server
• /vservers/ - root of /config/local for each virtual server
• /global - /config/global
• /docs - configuration checklists and scripts for dom0 and domU master images