Server upgrade (by alaric)

I host a heap of web sites (including this blog), email domains, source control repositories, mailing lists, and various other things (such as one of the official Chicken Scheme egg mirrors, a Jabber server, and an IRC server with bots). I do this with a combination of dedicated server hardware which I hire space, power, and connectivity for in London for the primary stuff, and a virtual private server in California for backup services and rapid DNS lookups from the USA.

This is a costly hobby, but it gives us a platform upon which to do interesting things, and lets me help other people out with free hosting; as I need to put in the time and money to run the infrastructure anyway, the spare capacity on it is essentially free.

The most demanding part is server upgrades. Periodically, I buy a new physical server, install it with all the software it will need, put it alongside the current hardware in the data centre, and transfer the data and settings across and configure everything that needs configuring on the new server until it works just like the old, then switch them over. I do this when the current hardware is getting full or overloaded or unreliable or just plain out of date, as I don't trust in-place updates of the core system software - it's too easy to end up with NOTHING working.

However, this has been overdue for several years. I bought the new hardware (this time, with a contribution from my biggest user of disk space!) nearly two years ago, and installed it in the rack nearly a year ago, but only yesterday did I get the chance to spend a day sitting next to it in London coaxing it into readiness then doing the final switch over...

It didn't go entirely to plan, of course. I'd previously written a script that used rsync to copy all the user data over; the first time I ran it it copied everything, then subsequent runs only had to copy the differences. The idea was that I would have less down time while I copied the data from the old server to the new (which has to happen with both servers offline, so that nothing can change during the copying process) if there was only the final changes to copy. However, I realised that the accounts of my biggest user of disk space weren't covered by my script as they had been slightly hacked to accomodate their growth.

And the whole process of moving the software configuration was made more complex by the fact that I had previously been running two servers in a kind of symbiotic cluster, in order to meet the load with the hardware of the time. Nowadays 64-bit multi-core behemoths with gigabytes of RAM are cheaply available and well supported by NetBSD, so everything can be done on one box. This is a much simpler setup, but it means that I had to undo the complexity of the previous setup when transferring everything across!

I ran into a few other unexpected problems, too; I noticed that the clock on the new server was terribly wrong, despite it running NTP. I did a manual ntpdate, and then just in case, another to check that it was now only a few millisecond out - but it was already half a second out again! It quickly became apparent that the clock was ticking about one second in every two seconds of real time...

Looking in the output of sysctl -a, it became apparent that I had a choice of time counter sources: it was using the TSC, but I also had an HPET, a clock interrupt, an APIC clock, and the good old 8254; my machine was brimming with alternate clocks. I tried switching to the HPET with sysctl -w kern.timecounter.hardware=hpet0 and suddenly time was running as expected. I popped that in /etc/sysctl.conf so it would come back on reboots, resynched the clocks, and everything's been fine since. I can only presume that the kernel was reading the CPU clock speed wrong, or some kind of dynamic clock scaling is happening, so that the (CPU-based) TSC wasn't having its ticks converted to seconds properly.

I had a big setback with the email setup, as NetBSD comes with Postfix as part of the base system but I wanted a more recent version from packages, but I ended up getting tangled with what version was being run in various situations and what configuration file was being used, which took a while to sort out. And then of course there's Mailman, the mailing list server software, which is complicated by needing write access to its filesystem-based state when run from the mail system (for incoming mail) or the Web server (for the web interface), so uses lots of setgid binaries and group-writable files and the like, and so always takes a lot of fiddling to get working properly.

But... I did it. And so, having completed my tax returns earlier this year (which is what freed up the time to prepare for and do this mission), I have now gotten rid of all the major obligations that have been hanging over me for the past few years.

I still need to visit London again - I've left the old servers running alongside the new in case I missed any files that need to be transferred; I'll give people a chance to check I've not missed any of their stuff before remotely powering them down (to save electricity, which I pay for) and coming in to take them (to free up the space). But that's relatively easy!

Ballads of the Scientifica (by sarah)

Todays science-art is an album of science and tech poetry, a possible life origins story and the song I Want To Know Everything. It is available as a free download for the remainder of the week 🙂

Enjoy

Computer Science (by alaric)

Is a Computer Science degree useful for people who want to have a career in software development? Many who work in the field come from physics, maths, or electrical engineering degrees, and do perfectly well. There's a widespread feeling that the concepts taught on computer science degrees, such as formal logic, proving the correctness of algorithms, functional programming, compiler theory, and so on are, at best, only vaguely useful in "real-world" software engineering, There's a sort of warm fuzzy feeling that knowing these things makes you a Better Programmer, even if you never use the knowledge directly, because you're more aware of the underpinnings of the tools you use. But I don't think anyone has ever shown a real benefit. With the obvious exception of people who go into niches such as compiler development, or writing tools for mathematicians...

Software development, in practice, is mainly engineering; often just following simple plans in obvious ways, like bricklaying. It takes skill to do it neatly and well, but not imagination or theoretical background. Familiarity with tools such as off-the-shelf libraries and standard system interfaces like POSIX are probably more useful than Prolog programming to most programmers. Debugging is, in practice, more valuable as a skill than using natural deduction to prove the correctness of algorithms.

But that's not to say that computer science is useless. Many modules in my computer science degree were engineering based, looking at practical topics such as building reliable distributed systems, dealing with concurrent access to resources, databases, networks, and operating systems. Those courses covered how things like TCP stacks are built, but that's necessary information to properly use them; information required by anyone who has to do a good job of writing network software. And the theoretical modules, on semantics, functional programming, logic, Prolog, and formal methods were useful to me as a special case of somebody interested in building new programming languages; a small minority of us nerds-among-nerds bury our heads in topics like continuation-based models of concurrency, and then emerge at the end with practical tools such as programming languages, threading libraries and distributed agreement protocols that the rest of the nerds can use to build applications with.

However, an electrical engineer will be taught programming, aimed at writing embedded software. It will be approached as an engineering activity, goal-oriented and pragmatic, emphasising requirements capture and verification of the result, and debugging. Issues such as working with the constraints of the hardware will be covered. It's no surprise that electrical engineers are widespread and successful in the software industry. But the electrical engineers who make it in software have had to do a lot of learning in their own time, and as such, it's harder to select them; they need to be individually interviewed in depth, rather than being rolled off the University assembly line pre-tested to a known standard.

So perhaps computer science degrees need to diversify further. Mathematics is often split into Applied and Theoretical sects; the distinction is sometimes arbitrary, with most topics straddling the divide in some way, but they are taught with different emphases. Theoretical mathematicians are better trained to go into mathematical research in academia or the more abstract R&D teams, while applied mathematicians are primed to dive into practical problems in statistics, simulation and optimisation. Perhaps we need something similar in computer science; I know that most degrees are modular, and mine let one end up with a degree title reflecting the specialisations one took, but I'm not talking about modules - I'm talking about a fundamental shift in emphasis in the degree, from day one. Everyone should start off with a year of practical software engineering, because even the most abstract theoretician needs to know how their work will be applied (and have the skills to build implementations of their theories, so they can be tested and then applied by others). Teach enough about compilers and computer architecture to give the student a head-start in optimising their code, without going into the detail required to build compilers or design CPUs. Give a nod to formal methods in showing how to design correct algorithms by informally argument.

Then in the second year and beyond, let it be down to modules; the software engineers can go into things like networking, databases, graphics, operating systems, high performance computing, distributed systems, and so on, depending on their desired specialisation. The theoreticians can go into abstract topics. And by all means, at the end, give them a Software Engineering degree if they did mainly software engineering modules, Computer Science if they did mainly theoretical modules, and something like "Applied Computer Science" if they did a mixture. Don't restrict student's choices, unless modules have an actual dependency on the knowledge from previous modules; but at the same time, give them guidance by explaining which modules will help them for different career paths. And don't force software engineers to spend their time learning abstract stuff they'll resent, in the vague hope that it will make them better programmers; it's no more useful than the electrical engineers working in software who had to sit through courses on filter design!

Alaric’s projects for this year (by alaric)

This year's going to be pretty busy with settling into the new home, but I have a few projects.

1. Finish the ring casting I nearly finished before the move. That's a priority.
2. Resurrect my aluminium foundry. In particular, it's our bronze wedding anniversary, so Sarah's going to design a pattern for a sundial, which I will cast in Aluminium bronze, a nice alloy that I can make myself from my scrap aluminium and bits of old plumbing...
3. Continue with minor stuff on Ugarit, but as a milestone, build the distributed storage backend, which will rock.
4. Work on my wearable computer project. No specific milestone for this, as it's currently a long drawn out research/prototyping phase as I sort out many details.

Wish me luck... I usually suffer from "all my weekends getting eaten up", but as my New Year's Resolution has been to spend at least one day every two weeks doing something fun with my children, I'm going to be booking weekend days in my calendar in advance through the year for that and my own projects. Before they get filled up!

Motivation (by alaric)