The Art of Knowledge Unconference

This weekend 12-13 Dec 2009 in Bristol there is an Unconference part of the uncraftivism weekend and I am going to be reading some of my poems that I think appropriate! Like The Programmer's Lament and A Picture of Words.

Me and Alaric are hoping some of you will join us in Bristol

It looks like a really interesting event so I'm looking forward to it - lots!

BKU2009 on twitter and I think there's some online jabbery stuff occuring. There is also like a talk/culture cafe thing and a lot of free form stuff hence the Un- bit of conference!

On being oddly dressed

Last Friday, I stood up in front of a hundred or so people and gave a five-minute talk on some software I've spent two years of my life writing. However, I wasn't particularly self-conscious about the fact that I was oddly dressed.

For me, clothes are about:

1. Keeping me warm
2. Carrying my stuff

It's not that their appearance doesn't matter to me - I don't want to be wearing shabby or tatty clothes. I don't want to wear garish bright colours. I like my clothes to more or less match, so I tend to choose solid dark colours when I buy myself clothes, as they're easy to look smart in.

But every now and then I get a comment from somebody that I must be a bit weird to go around wearing a podbelt and an assault vest... and when the weather's bad, I got outside in a full-length heavy cloak. Luckily, saying that sort of thing disqualifies people from me being too interested in their opinions, so it doesn't particularly bother me.

I like carrying lots of stuff with me. I'm equipped for every eventuality. When people get things in their eyes, I'm there with a mirror and tweezers. I have the obligatory geek multi-tool, of course. My first-aid kit has brought comfort to many a cut finger. My little lengths of string have jerry-rigged many a repair. I always have a torch, a compass, a pen, a notepad, a monocular, and a laser pointer to hand; so I can navigate, find things in the dark, read small text on a projector from the back of the room (and then point to the thing I'm asking about with the laser). If a button comes off of something, I sit down, take out my sewing kit, and fix it. In my laptop bag is a pouch full of cables and adapters, which has saved the day on many a late-night data-centre emergency. When it's raining so hard that people are cowering in shop doorways, my cloak keeps me dry; at the conference on Friday, when there were no seats left, it folded up tightly and became a low stool so I could sit comfortably. A week or so ago, when I was driving home from London very late one night and became too tired to continue, I pulled into a dark lay-by and slept underneath it, warm and comfortable even when the temperature plummeted before dawn.

I'm not just hoarding gadgets for the sake of it - I do assess the trade-offs of every extra bit of weight to carry around. Weight in the podbelt isn't an issue as it carries very nicely on my hips, I barely notice the weight of it, but space there is at a premium. Weight in the assault vest is more of an issue, since it pulls at my shoulders. I've tried having just a podbelt, but it's not good to wear while sitting down, so I tended to take it off and sling it over the back of the chair, which makes things harder to get at; and I've tried just having an assault vest, but weight was a problem. The current combination means I can keep lightweight things I often need while sitting down (mobile phone, pens, pads, business cards, laser pointer, etc) on me all the time, while weightier things I tend to need more on the move (keys, wallet, tools, first aid gear) in the podbelt. I have optional extra things I add for specific "missions" that I wouldn't want to carry all the time, too - I have a special tool jacket with loops for screwdrivers and the like which I wear if I'm doing DIY in awkward locations, an extra assault vest with more specialist stuff for when I'm busy being a Cub leader, a black lightweight mesh one with large pockets for hiking (the large pockets accept good quantities of food, GPSes, and the like), a water flask that goes on the podbelt, and a spare podbelt pouch that I'm going to assemble a survival kit in: emergency rations, a survival blanket, that sort of thing.

When I've explained this to people who question the amount of stuff I carry, they say "But what are the odds of all these things happening?". But they happen all the time! So I'm happy being prepared for anything... it makes life a lot less stressful. My clothes and their pockets become an extension of my body; we are, after all, all cyborgs.

A new laptop

To my great displeasure, my shiny MacBook Pro was stolen from the office in London!

So, I grabbed our finance guy and we went down to the nearest laptop shop and picked up the cheapest thing they had in the shop that would meet my needs: a Hewlett Packard Pavilion dv7.

The first step was replacing Windows 7 with something. As I knew I might have some teething troubles with getting NetBSD installed, so might need to return to Windows to get online, I shrank the Win7 partition down so I could run it dual-boot rather than nuking the whole thing. However, it was worse than I feared - the NetBSD install CD wouldn't even boot - the boot loader came up, then complained it couldn't read the kernel.

Not good.

So I burnt an ISO of Arch Linux, which is the closest to BSD in the Linux world. No good, either - GRUB loaded, and couldn't load the Linux kernel. I downloaded the boot.kernel.org nano-ISO (which then boots over HTTP from a central server) and that booted OK; but many of the Linux installers I tried died of kernel panics during booting. I wondered if the difference was that boot.kernel.org had an ISOLINUIX-based installer rather than GRUB; googling for this, I found out that GRUB sometimes has trouble with some CD drives, and as such, Arch Linux came with the option of an ISOLINUX-based installer CD. So I burnt a copy of that, and pow, it worked!

I installed Arch Linux, but only in a small partition, still intending to try and install NetBSD via non-CDROM means at some point. I soon had X up, and all was quite well, apart from the fact that my wireless Ethernet module (a Broadcom 4315) didn't work. I found a driver and installed it, and then it was recognised, but it still refused to actually do any wifi. Bah humbug.

On a hunch, I re-downloaded the NetBSD installer ISO and burnt a new CD of it... and it worked, this time! Having wasted most of a week trying to get the first one booting then messing with Linux distros. So I left my Arch partition in place (in case I needed it for anything), and set up NetBSD. Got X working again, copied across the home directory I'd made under Arch rather than redoing my dotfiles, and was happy. Except that NetBSD also didn't like my wireless interface, and it had trouble with the ACPI too, so I can't read my battery status or do a suspend properly. I've found a NetBSD driver for Broadcom devices but I've yet to get it to compile (I think it was developed mainly for the macppc port, or I've just not applied the patch properly). When I get my kernel source tree compiling again I'll have a new kernel that also has verbose ACPI debug messages, which will probably help.

I get Xen working, though; it was painless now that NetBSD's boot loader can do multiboot kernels, meaning I didn't need to mess with GRUB; I just installed the xen tools, and a xen kernel, and a NetBSD/xen kernel, then added the following line to /boot.cfg:

  menu=Xen with 1GiB for dom0:load /netbsd-XEN3_DOM0 console=pc;multiboot /xen.gz dom0_mem=1024M

...and rebooted into a Xen dom0. Setting up some Linux domUs for my work has been more exciting: I downloaded Debian filesystem images and a kernel from stacklet.com, but when I bring it up, Debian complains it can't bring up the Xen virtual network interface - and that it's getting lots of disk errors on the Xen virtual block device (although it can actually read the filesystem without any trouble whatsoever). This may not be helped by the fact that the kernel I downloaded doesn't seem to match the /lib/modules directory in the filesystem image, alas. More work is required.

Still, I'm happy to be back on an X workstation. I liked my Mac, but I was feeling hankerings for open source software, and minimalism. I'm running dwm, dmenu, and remind, joined together with awesome shell scripts. I do miss the intergrated personal-information-management tools in Mac OS X; I want to brew up my own database of people/todos/events/etc in some Prolog dialect (eventually replacing remind), so I can express all the relationships between things that I want ("There is an all-day event on date X called 'Y's birthday' if there is a person P with name Y whose birthday has the same day and month as X", sort of thing) - it'd be nice if it could integrate with Thunderbird's address book, so I may have a look at the format of that (or look into a script to sync it to/from an LDAP directory).

Motivation

When I started working, long ago in 1998 at Internet Vision, motivation wasn't a problem: work was something I did to cheer myself up. This held when I moved over to Frontwire; but when the company abandoned its offices, sacked all of my department apart from me, and I had to work from home back in 2002 or 2003 or so (IIRC), I started to find it hard to get up and start working in the mornings; I realised that working on problems with other people was more of a motivation for me than the fear of being reprimanded for not getting enough work done!

Well, I left that company before long, and freelanced for a while, then got together with some others and formed a company, GenieDB. I can now combine the best of both worlds; I can work from home, in my own environment, while being in contact with my colleagues in our company IRC channel, and working together on problems. I find it hardest when we're all working on unrelated projects, so there's little daily sharing of issues and triumphs, but the level of de-motivation I feel then is small fry compared to how it was when I worked alone!

Nonetheless, since my Frontwire days, various other stresses have appeared in my life, so my base level of motivation is nowhere near what it once was. Carefully managing my morale in order to keep my head above water is an important concern.

Luckily, I made a breakthrough some months ago; for some reason or other I had to be up much earlier than usual, so was up at 8am one day. When I had dealt with the business that required the early morning, it was about 9am, and I didn't need to start work until 10am - so I used the extra hour to go and tinker with stuff in my workshop. It was good. Having had an early morning I was tired that night and fell asleep easily, and having had an hour of "me time", I didn't have my usual restless urge to go and do something fun rather than going to bed.

And I forgot to turn the alarm clock back to its usual time. So the next morning I awoke again at 8am. Except this time, having been to sleep earlier, I wasn't dog tired. So I got up and enjoyed two hours of me-time before starting work.

I was hooked.

Previously, I would wake up knowing I had to get out of bed, get Jean ready for preschool, deliver her there, then start working, spend my lunch break mowing the lawn or other domestic tasks, eat at my desk, work until it was time to go and collect Jean, bring her back, cook dinner, take Jean to bed, then try and catch up on domestic matters (while tired) before going to bed and having trouble sleeping. This not being a particularly delightful prospect, I would often lie in bed far too long, cherishing the ability to just lie there and think, knowing that getting up meant stepping onto a virtual treadmill.

But now I was waking up at eight in the morning, and positively leaping out of bed at the thought of going and doing something fun. I made a rule that, from 8am to 10am, I'd do whatever I wanted; I wouldn't accept requests. I'd get to my desk at 10am, lively and happy. I'd be more tired in the evenings (that extra two hours didn't come out of nowhere), but much less depressed, so I'd get the domestic stuff done sooner and end up spending more time with Sarah once Jean was in bed, then be off to bed in good time as I was getting tired.

My two hours in the morning even gave me time to do things like having showers, which I had previously had to try and fit elsewhere in the day, often ending up going several days overdue!

Even when I'm in London, I woke up at 8am and spent two hours pottering about on my laptop, or going for a walk.

Now that Jean's started school, it's not quite so good - I have to be up at 7:30am to start helping Sarah to get her up, and fed, and dressed, and leave the house at 8:15am to get Jean to the school for 8:45am, but then I'm back home at 9am for an hour of my own before starting work at 10am; I still find it hard to get out of bed knowing I have to do the school run before I can do fun things, and I don't fancy getting out of bed at 6:30am for an hour to myself before doing that When Sarah's healthier she might be able to cope with the school run on her own, though, so it might improve yet; she doesn't seem to benefit from starting the day with her own time as much as I do, so that might be a fair trade.

We'll see!

Too many projects

I have a curse: I'm too inventive.

Yes, this has many benefits; it's certainly useful in my career. But when my ability to come up with ideas far exceeds my available time to execute them, it's a bit sad.

I've met people who seem to think that ideas are rare things. They guard them jealously, seek patent protection, and hold onto them long after they're obsolete; if one of their ideas turns up somewhere out there in the wild, they are sure somebody has stolen it from them.

But I think that ideas aren't that precious. I'm glad when I see an idea I've had realised, since it saves me the effort of realising it. While thinking about Bayesian spam filtering I realised that it failed to take the ordering of words into account, and wondered if it would be possible to build a Markov chain model of spam and another of ham and then measure a message against those two models to see which they fitted best; so when I discovered CRM114 I was happy - and even happier when I found out that Markov-based spam filtering is in Alex Shinn's hato MTA. If you're having more ideas than you can implement, then it's great if somebody else implements them; it means you can focus your limited implementation efforts on one less idea, or use their implementation as a basis to build your idea (if it's slightly different) rather than starting from scratch.

So, without further ado, here's my current Idea List:

1. Improve my welding skills by making a "hexoctagon", a garden ornament about a metre across made of inch-wide strips of steel organised into three octagons joined about a common axis of intersection and then joined with further strips, so that it's a hexagon from above and an octagon from the side. Cheap material requirements, easy cutting to shape, and then lots of welds that should be within my ability, so good practice.
2. Rebuild my furnace, and get back into aluminium casting.
3. Work on my wearable-computer projects. I've been experimenting with casting electronic circuits into blocks of transparent resin, which makes them durable and waterproof while not being as bulky as a box, an also allows one to see LEDs and LCDs embedded in the resin with them. The interesting part is getting connectors and tactile inputs like pushbuttons to the surface, but I have some ideas I've been experimenting with in those areas. I want to develop these to build a head-mounted display with an LED torch built in, a wrist-mounted display with a chord keyer inside, and a modular central unit to go in an inside pocket with central processing, storage, power supply, and interfaces in. Wearable computers are fun.
4. Extend the structured cabling around my home into adjacent buildings. Challenges include thick stone walls and a driveway to get across. I'll buy a metre-long 2cm-thick drill bit when I have some money, as my 30cm one didn't make it, and I'm researching stringing a catenary over the driveway.
5. Finish implementing my SFTP client library for Chicken Scheme, then write an SFTP storage backend for Ugarit using it. Also, various other improvements to Ugarit, such as caching the hashes of files by their mtime, so we can easily tell if a file has not been changed without re-hashing it, for faster snapshots.
6. Tinker with hato, adding local delivery to virtual users defined in a database, and an IMAP server that can serve from both real and virtual users, so I don't need to create full UNIX users for every IMAP mailbox. Perhaps adding LMTP support, too.
7. Tinker with Chicken Scheme internals; in particular, I want to improve the blob support, to eliminate unnecessary copying between the heap and normal malloc()ed memory, and within the heap between blob and SRFI-4 interfaces to the same region of memory.
8. Ongoing programming language research, extending the work of my final year project at University on metaprogramming, with some refinements I have in mind to support hygienic macros.
9. Ongoing development of ARGON, which is a vast sea of subprojects in itself
10. Work out the economics of distribution and storage of liquid nitrogen. Is it practical to use LN2 as a means of cooling buildings, rather than refrigeration-based air conditioning units that suffer from inefficiencies due to their small scale, and the fact that they pump the heat just outside the building, where it can easily leak back in again? Heat pumps are particularly bad in built up areas where everyone's doing it. LN2-based aircons could even be portable pocket-sized devices, very welcome to motorcyclists in slow traffic on hot days, and commuters.
11. Make test strips for common dietary intolerances. As a vegetarian, I often wonder if a given dish at a buffet is "safe" or not, and it's worse for people with serious allergies. Is it possible to make cheap and reliable test strips that change colour in the presence of (eg) animal fats, or whatever it is in peanuts that kills people? The colouring in the food itself may prove an obstacle, but diluting it in water and having a control on the strip to compare the colour against should be able to mitigate that, I suspect.
12. Make a better satnav system. I don't really like the traditional "At the roundabout, turn left" systems. I much prefer looking at a map of my route and seeing what I'm driving past, and knowing where I am in relation to major landmarks. So I'd really like to just see a big map of my entire journey, and a small map of my local environment, both with a marker on showing where I am. But let's generalise this a bit. How about making a generic map display system, able to take a base map and then apply any number of layers, each provided by a software module, possibly over a network, with streaming updates. Then the GPS can just produce a layer with a single point that moves about, while a map server just produces a static map, or streams map data from OpenStreetMap on demand. A database of places is a layer that exposes a set of points. A route planning system can provide a layer that shows a fat blue line over your route. The UI should let you lock the display to a given map feature, such as the moving point that comes from the GPS unit. This is a bit complex for a simple satnav unit, but when you're managing a fleet of vehicles, you can pull in a map layer plus the GPS layer from every vehicle (via GPRS etc) and then see the all on the same map, without needing any special extra software. It's a construction kit for map-based information display. But why stop there? Generalise a little further. This map information is all just a source of data sets defining points, lines, and polygons in lat/long space, with streaming and real-time updates, and a viewer that knows how to display these things in lat/long space. Why not generalise the coordinate system to an arbitrary set of time, space, and other unit axes? The system can know about special types of axis - such as time and spherical coordinates - and then have generic mappings of the other ones into spacial axes, and allow analysis of arbitrary data sets, overlapping them if they have axes in common. So if you have a data set that's the spread of pollution with time, you can overlay the lat/long axes onto your map (or an augmented reality view), and have a slider to choose a position on the time axis; or pull it into 3D by adding the time axis and look at the spread of the pollution as a cone. Your email history can be a set of points aligned on a time axis, and your appointments can be points and regions on the time axis, sometimes with lat/long added (if you took a GPS reading of your position in the past, or if you know where you're supposed to be in future), that you can compare to the spread of pollution (did it coincide with you starting to get sick?), and plot on the same map you use for navigation (so you can work out the best route to get to all the places you need to get to in future), or just fold away the space axes and look at a timeline of your life that you can zoom into. Many applications can be fulfilled in one integrated "data viewer", and many network effect benefits reaped by having them integrated seamlessly.
13. Audio jukebox appliance. At its simplest, a small box with an Ethernet port, a power inlet, and an audio out port. It finds your music from one or more network servers, or has an internal hard disk you can put music on, or combines both. It will play songs from a queue, and if the queue is empty, will either be silent or will randomly pick songs that match a search query. But that's the boring part. The fun part is that it also has a message queue, and supports a simple network protocol to send it messages. A message causes the music to pause with a quick fade out, then a jingle (depending on the message type) played, then the message text to be read out via a text-to-speech engine such as Festival. When there's no messages in the queue, the music resumes. For extra coolness, add the ability to drive remote slave units over the network; the units are organised into a logical tree, and by default, every node inherits music and messages from its parent. So you set the global playlist at the root node, then on a child node, you can queue some songs and have that child (and its subtree, if there is one) then play different music, reverting to the global music (with a fade) when it ends. And you can send a message to any node in the tree, and that message will be text-to-speeched from that node and all its children. This is something I want about the house, and in my van (see next point), and would be great for secret underground bases, which definitely need computer voices making announcements.
14. Jazz up my van with a vehicle computer. An MP3 jukebox and navigation system. And a separate battery, so running them all (and the inverter) can't make the engine-starting battery go flat, but when the engine's running the auxiliary battery charges from the alternator.
15. Virtual networking. See blog post.
16. Write a "Practical Programming in Scheme" book. Base it on Chicken Scheme, as it's aimed at real-world tasks rather than education. Cover all the stuff that rocks in Scheme, but that basic introductions don't cover as they're trying to teach programming to new programmers: parameters, coroutines, fold, really making the best use of closures, that sort of thing.
17. Finish up some novels I have in various (early) stages of development.
18. Design CPUs. I think there's some fun to be had with unusual architectures. In particular, I want to develop tiny MISC-based microcontrollers, and I also have some interesting ideas on how to build a highly asynchronous move-based architecture that should make good use of instruction-level parallelism, while having a very simple control unit that lets you fit lots of them on a chip, sharing execution units between them.
19. Build a giant distributed (around the house and grounds) replicated storage array for storing my backups, as a single giant Ugarit archive, using a distributed-archive backend for Ugarit that I've designed, which will allow fine-grained control of a replication policy in a very fail-safe fault-tolerant way, yet being simple to implement.
20. Implement my old protocol agnostic middleware idea. "Middleware" is a bit of a dirty word these days, so perhaps "framework" would be better. I was excited that somebody seemed to have gone ahead and done this, but it's in Python 3 which is a bit limiting; I could put together an implementation in Scheme without too much fuss by layering existing interfaces to protocols such as HTTP, organising them into a common request-routing infrastructure.
21. Re-copyedit some role playing games (of the pencil-and-paper rather than computer variety) I wrote as a teenager, and set up http://www.point-defect.co.uk/ as my games publishing arm.
22. Write a Web-based MMORPG set in a procedurally generated universe I have designed, with certain interesting properties. Ideally turn it into a profitable online business, through advertising and subscriptions.

I think it's important not to try and have any illusion that I'll finish any of my projects. It's all too easy to sit and think about how cool each one would be, and to be sure I'm going to finish it because it would be awesome, and talk enthusiastically about how I'm going to do X, Y, or Z. When I was younger, I fell into this trap, and then felt bad about myself when I didn't.

Now I remind myself: these are my hobbies. I say I would like to do X, Y, and Z. But I make no promises, and set no deadlines. When I get some free time, I will work on whichever one I feel like, so progress is hard to predict.

I do feel sad, though, that I have interests in so many fields. I am jealous of people who have just one passion, and devote all their energy to it, and therefore take it a long way. Some days I'm not in the mood to mess with computers. Some days I'm not in the mood to mess with metal. It varies.

20091002: Updated with some more detail on some of the ideas.