Opening up Open Source (by alaric)

One of the awesome things about free/open source software (FOSS) is that, as you have access to the source code, you have exactly as much power as the original authors of the software to modify and extend it.

When you are upset about something in closed-source software, or one hosted on the owner's servers on the Internet as a web app or API, all you can do is beg and plead with them, and threaten to take your future business elsewhere; they control the source code, so they have ultimate power. With FOSS software, in theory, the original authors have no special powers.

However, it often doesn't quite work out like that in practice.

Most FOSS software comes onto people's computers as a precompiled binary; they download it and run it. This is convenient and efficient. But if they then want to use their theoretical right to get in there and modify it, they need to do the following things:

1. Learn the programming language(s) it's written in, and the libraries/frameworks/APIs it builds on top of. (This is particularly tricky if they have no previous programming experience).
2. Find and download the sources (which shouldn't be too hard, but can still be tricky).
3. Set up a build environment that can compile the thing. This may involve installing compilers, and development versions of libraries in order to get the header files, and so on.
4. Actually make it compile. A lot of the time, a source package as shipped by the authors won't compile perfectly outright, as you need to apply patches written by the people who maintain the package for your platform, as each platform has their own conventions as to where things are placed in the filesystem, and their interaction with less-standardised bits of infrastructure such as service management frameworks, management of network interfaces, low-level hardware access, and so on.
5. Learn the workings of the codebase. For a large project, this can be VERY daunting, even to a seasoned programmer.
6. Actually design, implement, and debug the change. If the codebase is poorly architected, this can be made unnecessarily difficult, and require refactoring of existing code elsewhere within the codebase.
7. Install the software once it's building. As it's been custom-built, it may not interact nicely with your platform's package manager, so you may end up running it out of /usr/local/bin or ~/bin or similar, and have trouble making managed packages that depend on the one you're tinkering with correctly linking to it, interaction with system configuration tools, and so on.

I think this is harder than it needs to be. It puts a lower bound on the effort required to make even a trivial change that, in many cases, means it's not worth making it; we're as beholden to the whims of the developers of the package as we would be to a closed-source software company, and the supposed benefits of open source are denied to you.

We can break down the barriers into a number of categories, and look at what can be done to solve each.

Learning the programming environment

Whether you're a seasoned programmer or not, any given piece of software you didn't write will involve a set of languages, tools, libraries, and other things, that you may not be familiar with, and these will need to be learnt.

What might help is better standards for automatically-generated API documentation, so you can find the documentation of API functions as they are used in the software you're learning how to edit by just pressing a button in your editor.

But if it became easier, commonplace, and expected for people to dig around in other people's source code, for curiosity or to make their own changes, developers of infrastructure components would feel more compelled to document their interfaces in ways that casual programmers can quickly pick them up, and to make their interfaces simpler and easier to learn, because the expected audience would be less dominated by seasoned programmers.

And, conversely, if more people were casually getting involved in simple programming tasks in order to improve the software they use (if it became easier to do so), then the general programming ability of the population would also rise, giving more people the grounding in basic conventions and concepts required to understand programming tools...

Migrating from a normal installation of the software to a hackable one

This is perhaps the biggest hurdle, and yet, the most amenable to being overcome with better technology. It covers the whole spectrum from finding and downloading the source code, setting up a build environment, getting it building on your platform, and getting it installed as a first-class citizen in the eyes of your package manager.

I can think of two technical fixes to this problem, and the best bit is, they're both things that already exist out there rather than my usual kinds of crazy new reinvent-the-wheel thinking!

Firstly, package managers like Nix make it easy to establish build environments on your own hardware, as the build environment of any package can be requested, and automatically set up for you. Also, they are built around installing software from sources in the first place, and offer downloading pre-built binaries as an optimisation. It's quite easy to adapt a nix expression that builds a software package from downloaded sources to one that builds from a source tarball you've made yourself, and install it into an isolate "profile" in such a way that it's easily kept isolated from other software you have running and might not want to risk being broken by your experimental changes yet, and to roll the change back if it doesn't work out.

I suspect that many traditional package managers are written with users in mind, and not developers, which sounds laudable; but in practice it forces the distinction between user and developer, not allowing the former to easily migrate into the latter. Nix feels written for developers, of course acknowledging that developers are also users and still want to be able to install off-the-shelf prebuilt binaries easily. The inbuilt package manager for Chicken Scheme is likewise developer-friendly, letting you directly build from arbitrary checked-out source trees into a properly installed package; my development process for most Chicken software is to run chicken-install in my in-progress sources as the first port of call to compile and run it, rather than the usual idiom in most languages of compiling and running from the source checkout then "installing" the binaries as an optional, later, step. And yet a "mere" end-user of my software can type chicken-install ugarit, and Chicken will download the latest public Ugarit release from the Internet and install it for them. If they want to join me in hacking Ugarit, they can check out the latest sources from my web site and get stuck in pretty quickly.

Secondly, the move away from compiling software ahead-of-time into distributable binaries, towards on-demand compilation of source code at run time (with caching of compiled forms, of course), means that the normal installation of some software is the source code, there is no need for an external "build environment" to convert your changes into a runnable package, and changes to the source code can be immediately used without going through any kind of build/install phase. Because this makes tinkering with the software so much easier, it can make it become a routine part of using the software, rather than something special done only by special people. The typical Emacs user will have overridden various internal functions of Emacs from within their personal configuration; although many configurations can be done without doing so, customisation through function overriding is so accessible that people routinely customise their Emacs installations in ways that the original developers didn't think (or didn't have time/energy) to add as a configurable option. I wish all open-source projects were written in such an open manner, but it will require a lot of migration away from the batch-compilation model of C, C++, and Java.

Poorly-architected existing code

This is a thorny issue; even if you can easily get into the code of your application and make the changes you want, and you understand all the tools it's built with, it can still be hard to make the changes you want because of one of a number of kinds of inherent "fragility" in the way the software is constructed.

Usually, this boils down to some variant on the idea of some information being repeated all over the code, rather than kept in one place. If your code relies on communicating between its components by using a special file, for instance, and every place where this file is read or written contains its own code to read and write this file directly, then changes such as storing the file in a different place, or adding some extra information to it, or replacing it with access to a database or something, will be difficult. You'll need to find all the places where the file is used, and individually re-write them to reflect your changes. This is laborious, and you might miss some, leading to bugs when those bits of code are run but don't reflect the changes.

However, if the mechanism of accessing this shared state (reading and writing the file) was isolated into one place in the software, with an interface that is used wherever required and reflects only the essentials of the access to the shared state, then that mechanism can easily be changed to another, as long as it still preserves those essentials, relatively painlessly and safely.

Software developers, before they even write a line of code, be thinking about how their software might be changed in future, and make sure that they split it into modules with clean interfaces, to make that easier. As side effects, it also makes their code easier to test and debug, as the interfaces serve to define and clarify the responsibilities and expectations of each module, which makes it easy to write comprehensive tests for the modules.

If this seems like hard work, then you're doing it wrong. We've all heard of code (usually in Java, for some reason) that seems to have taken the Design Patterns book as a checklist of things to do, and features pages and pages of AbstractFactoryWrappers that just indirect everything; the actual code that does the task at hand seems to be scattered thinly amongst all this framework. That's not what I mean by designing your software to be extensible. I just mean splitting it into bits with an explicit interface between each, and makings those interfaces reveal as little as possible about the workings behind them, and putting duplicated code into modules behind interfaces rather than writing the same logic more than once, rather than making it into one big ball of inter-related mud. If you're not saving time by writing software like this in the first place, or if it seems like a burden, then you need to re-think how you write code.

I think programming languages can do a lot to help us write cleaner code, too. I find that when I'm writing C and C++, it's often hard to cleanly pull bits of functionality out into other functions due to the manual memory management which complicates interfaces, and the lack of lexically-scoped first-class functions. Code written in Lispy languages tends to be a lot more easy to refactor as it grows, leading to cleaner interfaces (on average), and the automatic memory management tends to make the interfaces simpler as well.

Also, a culture of open extensibility in software means that extensibility of your code is high in the programmer's mind at all times. Developers of Emacs packages seem to expect bits of their software to be overridden, and write it accordingly.

Conclusion

I think that making programming more accessible has very many good consequences. It gives people more power to get more out of their computers. It gives people more reason to trust computers (and as we move to a more online society, people are forced to place their trust in computers in order to take part; but being forced to place your trust in something you don't trust is a harrowing experience), as they can peer inside the software to see how it works, and fix it if it doesn't. It also means that everyday users of computers have an easy, and natural, transition into learning programming, which is a very rewarding pastime; and more people contributing to open-source projects means we all get to have a better quality of life.

So, open-source software developers, I implore you to consider these points. Try to make your software open and welcoming to newcomers!

Insomnia (by alaric)

There's something about the combination of having spent many weeks in a row without more than the odd half-hour here and there to myself (time when I get to do whatever I like, rather than merely choosing which of the list of things I need to get done urgently I will do next, or just having no choice at all), and knowing I need to get up even earlier the next morning than usual (to dive straight into a long day of scheduled activities), that makes it very, very, hard for me to sleep.

So, although I got to bed in good time for somebody who has to wake up at six o'clock, I have given up laying there staring at the ceiling, and come down to eat some more food (I get the munchies past midnight), read my book without disturbing Sarah with my bedside light, and potter on my laptop. I need to be up in five hours, so hopefully emptying my brain of whirling thoughts will enable me to sleep.

There's lots of things I want to do. Even though it's something I need to get done by a deadline, I'm actually enthusiastic about continuing the project I was working on today; making an enclosure for our chickens. This is necessary for us to be able to go away from the house for more than one night, which is something we want to do over Christmas; thus the deadline.

Three of the edges of the enclosure will be built onto existing walls or woodwork, but one of them needs to cut across some ground, so I've dug a trench across said bit of ground, laid an old concrete lintel and some concrete blocks in the trench after levelling the base with ballast, and then mixed and rammed concrete around them. When I next get to work on it, I'll mix up a large batch of concrete and use it to level the surface neatly (and then ram any left-overs into remaining gaps) to just below the level of the soil, then lay a row of engineering bricks (frog down) on a mortar bed on top of that in order to make a foundation that I can screw a wooden batten to. With that done, and some battens screwed into the tops of existing walls that don't already have woodwork on, I'll be able to build the frame of the enclosure (including a door), then attach fox-proof mesh to it, and our chickens will have a new home they can run around in safely.

Thinking about how I'm going to lay the next batch of concrete in a nice level run, working around the fact that I only have a short spirit level by placing a long piece of wood in there and levelling it with wedges and then using it as a reference to level the concrete to, has been one of the things running around in my head this evening.

Another has been the next steps from last Friday, when I had a fascinating meeting with a bunch of interesting people in the information security world. You see, I've always been interested in the foundation technologies upon which we build software, such as storage management, distributed computing, parallel computing, programming languages, operating systems, standard libraries, fault tolerance, and security. I was lucky enough to find a way into the world of database development a few years ago, which (with a move to a company that produces software to run SQL queries across a cluster) has broadened to cover storage management, distribution, parallelism, AND programming languages. So imagine my delight when said company starts to develop the security features in the product, and I can get involved in that; and even more when (through old contacts) I'm invited to the inaugural meeting of a prestigious group of peopled interested in security. That landed me an invite to the second meeting (chaired by an actual Lord, and held in the House of Lords!), the highlight of which was of course getting to talk to the participants after the presentations. I found out about the Global Identity Foundation, who are working pn standardising the kind of pseudonymous identity framework I have previous pined for; I'm going to see if I can find a way to get more involved in that. But I need to do a lot of reading-up on the organisations and people involved in this stuff, and figuring out how I can contribute to it with my time and money restrictions.

I'd really like to have some quiet time to work on my secret fiction project, too. And I want to investigate Ugarit bugs. Some bugs in the Chicken Scheme system have been found and fixed lately, so I need to re-test all these bugs to see if any of the more mysterious ones were artefacts of that. I'm in a bit of a vicious circle with that; the longer it is since I've been tinkering with the Ugarit internals, the longer it'll take me to get back into it, and the more nervous I feel about doing so. I think I might need to pick off some lighter bit of work with good rewards (adding a new feature, say) and handle that first, to get back into the swing of things. Either way, I'll need a good solid day to dig into it all again; trying to assemble that from sporadic hours just won't cut it.

I'm still mulling over issues in the design of ARGON. Right now I'm reading a book on handling updates to logical databases - adding new facts to them, and handling the conflicts when the new facts contradict older ones, in order to produce a new state of the database where the new fact is now true, but no contradictions remain. I need to work this out to settle on a final semantics for CARBON, which will be required to implement distributed storage of knowledge within TUNGSTEN. I need a semantics that can converge towards a consensus on the final state of the system, despite interruptions in internal network connectivity within the cluster causing updates to arrive in different orders in different places; doing that efficiently is, well, easier said than done.

I really want to finish rebuilding my furnace, which I hoped to get done this Summer, but I'm still assembling the structural supports for it. I've made a mould to cast shaped refractory bricks for the lining of the furnace, but I've yet to mix up the heatproof insulating material the bricks need to be made out of and start casting the bricks, as I still need to work out how I'll form the tuyere.

I want to get Ethernet cabled to my workshop, because currently I don't have a proper place for working on my laptop; I have to do it on the sofa in the lounge to be within range of the wifi, which isn't very ergonomic, doesn't give me access to my external screens, and is prone to interruption by children. I find it very motivating to be in "my space", too; the computer desk in the workshop is all set up the way I like it. And just for fun, I'd like to rig the workshop with computer-controlled sensors and gizmos (that kind of thing is a childhood dream of mine...).

This past year, I've tried booking two weekend days a month for my projects, in our shared calendar. This worked well at the start of the year, with projects such as the workshop ladder and eaves proceeding well, but it started to falter around the Summer when we got really busy with festivals and the like. I started having to fit half-days in around other things, which meant spending too much time getting started and clearing up compared to actually getting things done, so my morale faltered; and with so much other stuff on, I've been increasingly inclined to spend my free time just relaxing rather than getting anything done. On a couple of occasions I've tried taking a week off work to pursue my projects, but I then feel guilty about it and start allocating days to spending more time with the children or tidying the house, and before I know it, five days off becomes one day of actual project work. I need to stop feeling guilty about taking time to do the things I enjoy, because if I don't, I'll be too tired and miserable to do a good job of the things I should be doing! And rather than booking my monthly project days around other stuff that's going on, next year I'm going to mark out my two days each month in advance, and then move them elsewhere in the month if Sarah needs me to do something on that particular day, to decrease the chance of ending up having to scrape together half-days around the month (or to skip project days entirely, as I ended up doing last month). I feel awful about saying I'm going to spend days doing what I feel like doing rather than the things the rest of my family need me to drive them to, but if I don't, I think I'm going to fall apart!

Now... off and on I've spent forty minutes writing this blog post. So with my whirling thoughts dumped out, I'm going to go back to bed and see if I can sleep this time around. Wish me luck!

Is information security good? (by alaric)

One of the interesting things to have come from Edward Snowden's leaks of classified documents is that the American National Security Agency has been working to introduce flaws into the design and implementation of security technologies, in order to make it easier for them to break said security for their own ends.

There's been a lot of outrage about that. The argument for it is that the ready availability of strong security technology makes it easier for bad folks to conceal their crimes (and, worse, conceal the fact that they are planning such crimes, so they cannot be stopped in advance), so the NSA is right in acting to make sure people don't have strong security technology. However, even if we can trust the NSA (and that is far from certain) such vulnerabilities can be found by people we certainly can't trust: "cyber-criminals" intent on stealing our credit card details in order to rob us of our money, commercial competitors looking for strategic advantage, and so on.

There are also deeper issues that have been raised; this means that the NSA is covertly working to sabotage the products of US companies. Should they be allowed to do that? Can those companies now sue them for damages?

But I think that, at the heart of the debate over this, is an even deeper issue.

We have the NSA - the part of the US government officially responsible for information security - acting to subvert the information security available to US individuals and companies, on the grounds that it is harmful to the public if they have strong security. While on the other hand, we have individuals and companies striving for better security; working to make more secure products, choosing products that claim to provide security benefits, and so on.

This shows, to me, that there's a big unresolved question that US society as a whole - government and non-government together - needs to ask themselves: Is information security good? The government's official position seems to be that information security is harmful, as it makes it harder to provide a more general notion of security that is threatened by criminals, foreign governments, and terrorists; while everyone else's position seems to be that information security is good because they don't want information criminals and foreign governments stealing their secrets (terrorists don't seem to have cottoned to this trick yet) - and, maybe, because they don't want the government knowing ("stealing" is a contentious term here, as the government gets to define what "stealing" is) their secrets, too.

So before they can really debate whether the NSA's actions are justified or not, I think the US needs to step back and look at the bigger question: Should information security be a right, or not? If not, then they should just use legislation to stop companies and people from wasting resources trying to achieve it while other resources are being spent subverting it so they only receive an illusion thereof. That's just plain inefficient. And if information security is deemed good, then the NSA should be prevented from subverting it, and refocus its efforts on ways of doing its job without being able to break encryption; traffic analysis, meta-data analysis, exploiting specific installations of security systems where a threat is suspected, and so on are all time-honoured mechanisms that work even against well-educated adversaries that use encryption systems that the NSA hasn't been able to subvert.

Creativiti Tea – Summer (by sarah)

This weekend we hosted the second ever Creativiti Tea which was quiet frankly epic - though the actual event was on the Sunday the first person arrived Friday evening all the way from London! Two more arrived on Saturday when the BBQ ended up being cooked under the grill whilst Alaric did various bits of his programming projects and began work once again on the furnace.

We played board games and Jeany got an early birthday present which was Dr Who Monopoly!

I baked cakes (they are never as good as Al's ones and I was a bit out of practice with the icing!) - including mini vegan banana bread with red currents (these are from my parents garden I planted a red current stem when I was 13 and now they have taken over the garden - I am attempting to find more things to do with them than make jelly!).

Then at 10 am Creativi Tea began properly. Mainly we were writing - I personally wrote about 3000 words, we had people working on novels, poetry, short stories and more!

In the kitchen glueing and sticking were a foot.

Along with bead work - ankle bracelets seemed to be the thing.

Then there was sewing - mainly cat nip mice from old cloths - the cat nip was home grown by our friend Charlee.

Tom puss at least loves the cat nip mouse!

And I mean he really loves it!

A teddy bear was cut out and partially sewn together and the beginnings of a ball were started.

Music crept in with the odd bit of laptop composing and the Hammond Organ getting switched on 🙂

I sold £15 worth of books for Shelter (the books will be there until Christmas if anybody else is interested).

And we showed Charlee how to make sushi.

We geeked and created and played games and laughed and drank lots of tea 🙂 I am looking forward to organising the Autumn Creativi Tea which will also be my book launch! And also my friend Claire gave me the most beautiful roses 🙂

Oh and Jean's butterfly garden produced a butterfly 🙂

We had lots of fun and Alaric seems to have gotten one of our friends into learning Lojban - which had us all thinking about actual meanings of our names - turns out there is an older meaning for Sarah than princess - essence or core, pure centre... etc. I love finding out new things 🙂

Privacy (by alaric)

I have a looser attitude towards privacy than most people, but I have began to reconsider that lately.

Generally, I believed (and still do) that anything I do in public is pretty much exempt from privacy. I have no privacy objection to pervasive CCTV, because if I do anything in a public place, somebody could be watching me anyway. The fact that my enemies can now just consult massive archives of CCTV to find me rather than having to get somebody to follow me around isn't, in my view, a huge deal. Indeed, I quite like the idea of sousveillance, having my own recording of what happens around me. It might be inappropriate to be doing that in circumstances that the people around me consider "private", so I'd turn it off for their comfort when it seemed right to do so, but I would still assume that anything I do in the presence of other people is basically recorded to some extent - after all, it's in their memory, at least!

Likewise with monitoring my network traffic at my ISP; I have never had any illusion of privacy there. I encrypt traffic that matters, and accept that the existence and destination/origin of encrypted traffic might be used by my enemies for traffic analysis.

So, I didn't really have any objections to mass surveillance; I had far more objection to the facts that encryption is far from ubiquitous and that information security is not taught in schools. My feeling was that if I can't stop an enemy that doesn't abide by the law (eg, organised criminals) from performing traffic analysis on me, then I can't assume it's private; I can stop them reading my stuff or impersonating me by using public key cryptography, so as long as the law doesn't hinder that, I'm content.

As such, I always wished that Web browsers would just include some kind of unique user ID in the headers, ideally backing it up with a public-key signature of the entire HTTP request. Then we could dispense with session cookies, logins, and even things like OpenID; we'd just authenticate to our browser by supplying the keypair in some browser-dependent way, and then head out onto the secure-single-sign-on Web. There's no loss in privacy compared to the current status quo that people are happy to identify themselves to web sites with email addresses, but it'd be a whole lot simpler for users and for developers. And so that, basically, is the security model I developed for ARGON.

However, I am starting to change my mind.

I've always felt that the "hole" in my approach to privacy was that it depended on my own knowledge of security and my enlightened use of encryption; I wanted sufficient education to bring everyone to that level. Encryption tools are generally a bit clunky, but if more people wanted to use them, that would create demand for better tools (or, more pertinently, better integration into the tools they already use). I felt that if we could just get people to encrypt and sign their communications, and encrypt their storage, and use Tor for things where the cost is worth the protection against traffic analysis, everything would be fine.

However, what has made me start to change my mind is the move towards storing one's data on third-party servers. By which I mean, living your life through Facebook, or letting Google store your email and your documents. People are moving away from having a computer full of their stuff, and communicating semi-directly with their peer's computers, towards letting third parties hold all their stuff. Often third parties they don't pay money to and are in no contract with, so they have little or no leverage over.

It's easy to say that educating people in computer security would make them realise that's a bad idea, but I use many of these services despite not trusting them one bit; I do it because network effects force me to. I could run my own StatusNet server on my own hardware, but instead I use Twitter in order to make it easy for people to communicate with me. I use Facebook because it's the easiest way to keep up with my many peers that do, and sometimes because I am forced to; an organisation I am a member of uses a Facebook group for important announcements. Many people do not publish an email address, but instead require me to contact them through various third-party services.

In effect, we are being forced to hand our information to third parties, and to trust them with it. Variations on these services that store your information on hardware you control exist; variations on those services where you actually pay a service provider to store it on their hardware (in exchange for them looking after maintenance, amortizing up-front costs, and so on for you, and where they are more incentivised to keep your stuff secure so you trust them than to try and find ways to make money out of it) also exist.

But they are not popular, as the big "free" providers have the vast majority of the users, and the value of these services is in all your peers already being on them. Now that worries me.

I'd really like to see more push-back against this. If enough people used decentralised software like Diaspora or ran their own mail systems, then the network effects would benefit those, rather than centralised commercial outfits. Clearly, some large incentive needs to be found to push people over, and an unpleasant transition period where everyone needs to be on both. Eventually, organisations like Facebook, Twitter and Google would find themselves forced to interoperate with the decentralised protocol or lose their place in the market, and then would find themselves having to compete on points such as "privacy" when the same ease-of-use and functionality can be had elsewhere for little cost.

But, we need technical measures as well. Build sensible public-key infrastructure into the core of applications (including Web browsers). Ditch cookies, and replace them with explicit authentication: provide a system of public-key-signing HTTP requests as I suggest, but turn it off by default, and force web servers to request it with a status code, as is already done for HTTP authentication (not that that is used for web applications, alas). Let browsers seamlessly support multiple identities, and when a web site requests identification, let the user choose which identity to use; and then colour the border of the Web page according to the identity in use so they don't forget. And while providing identity management through that (controlled) mechanism, try as hard as possible to remove all other means of identification - don't send headers leaking lots of information about the user-agent and its capabilities and settings, and disallow Javascript from querying that sort of thing. Bundle Tor with browsers, so it can be turned on and off with the click of a button, as part of the "private browsing mode" found in many browsers.

I still don't think there's much point in trying to fix this with making information gathering and retention illegal (the recent PRISM scandals suggest that legitimate authorities will find ways to work around limitations on their information gathering, and organised criminals simply won't give a damn anyway); we need better technology that makes us anonymous by default and pseudonymous when we want to be. But there may be some value in legislation helping to break the stranglehold on the social software market held by big centralised organisations!

I'm updating the ARGON security model to work like this (not that that makes a difference to the Real World, mind...)