Category: Sci/Tech

Designing software (by )

One thing that has often annoyed me in my career as a software designer is coming into conflict with people who want me to do the worst possible job that is "just good enough".

This often comes out in statements like:

  • "There isn't time to design it properly, just hack something so we can start demoing it"
  • "We'll have to rewrite it anyway, requirements always change"
  • "Supporting more features than we need right now is a waste, add them when we need them"
  • "Can't we just do something quick and simple?"

Reading between the lines, they seem to think that "more designing" will mean more complicated software with more features, that will take more time to build.

I think the problem comes from how product management thinks of software - they want a feature, they request the engineers add it (ideally specifying the feature well enough that they actually describe what they want), they get that feature. And there's often some discussion about reducing the scope of the feature by removing some criteria to get it implemented sooner. It seems very much like "more features with more acceptance criteria equals more work".

I'd like to dispell that myth.

I assert that better-designed software.will take less time to write, and will be better due to being more flexible rather than through having more features, and through being easier to extend in future when new requirements come up.

There's a paragraph in the Scheme language standard known as the "Prime Clingerism", and it reads thus:

Programming languages should be designed not by piling feature on top of feature, but by removing the weaknesses and restrictions that make additional features appear necessary. Scheme demonstrates that a very small number of rules for forming expressions, with no restrictions on how they are composed, suffice to form a practical and efficient programming language that is flexible enough to support most of the major programming paradigms in use today.

It has been my experience that this approach applies to all forms of software design, not just programming languages. As a software architect, I see designing a feature as more like Michelangelo making the statue of David (famously, he said that the process of making a statue is just to get a big enough block of stone, and remove all the bits of it that aren't the statue).

Rather than thinking in terms of each acceptance criterion in the specification for the feature as a thing that will have to be done separately (so more ACs means more work), I like to work the other way: what's the simplest piece of software that can do all of these things? Most problems have a "core" that it can be boiled down to, plus some "details". I try to design software that has to meet specific requirements by making the "core" with the fewest assumptions and then adding "details" to tailor it to the exact requirements - which tends to mean I can add more details later to fulfill more requirements.

An example

For instance, many years ago, I worked at a CRM SaaS company. A big part of the app was sending out HTML emails to subsets of a user's customers to offer them some special offer, serving up the images in the email from our HTTP server (and tracking who saw the email by including the email message ID and the recipient ID in the image URL), and tracking what links they clicked by replacing URLs in the email body with URLs hosted by our HTTP server that would log the email message ID and recipient ID then redirect the user.

I was given a feature request: people were occasionally forwarding the emails to their friends, and when they did so, their viewing of the email images and their clicking of links would be logged against the original recipient, as their ID was in all the URLs. Our server would see the same person viewing the email lots of times and clicking the links lots of times. Learning how users responded to these messages was a big priority for our customers (we had extensive reporting systems to analyse this data!), so they were tetchy about this, and desired the ability to put "forward to a friend" forms in the email HTML. The recipient would put their friend's email address into this form and hit a button, and our system would send a fresh copy of the email to their friend, with their own recipient ID - so our user would know the forward had happened, and could see the friends' response separately.

Now, the "simplest" solution would be to add an extra feature to our HTTP server, accepting a form submission to a URL containing the recipient and message IDs, extracting an email address from the form submission in the usual manner, creating a new recipient in the DB with the email address (or finding an existing one), recording a forward from the original recipient ID to the "friend"'s ID, and sending a copy of the email to the "friend" - then responding to the form submission with either a hardcoded thankyou page or responding with custom thankyou-page HTML uploaded by our user, or a redirect to a URL set by the user.

A quick chat with the boss revealed that being able to host custom thankyou pages was a very desirable feature, and as that would involved embedded CSS and images (and maybe javascript), I clearly needed some kind of generic user-uploaded-content HTTP serving infrastructure. So I threw that together, letting customers created websites in the application and upload files, and an HTTP server that would server resources from them with a URL containing the website ID and the path of the file "within" that website - and with the ability to tag URLs with email message and recipient IDs for tracking purposes. We already had an engine to "personalise" HTML message content for a specific recipient, which also handled modifying all the URLs to embed email message and recipient IDs, so I used that again for hosting textual content such as HTML, meaning that if a link came in from an email message with message and recipient IDs, they would be fed into all further URLs returned by the system. To prevent spoofing of the IDs in the URLs, I reused the method we used on the image and redirect server URLs: the URL itself was "signed" with a short cryptographic signature when the software generated it.

But rather than hardcoding the forward-to-a-friend feature into that, I did something that took perhaps ten minutes more programming: I allowed an arbitrary number of named "commands" to be included in a URL. The only named command I implemented was "forward" that would do the forward-to-a-friend logic (finding all form submission fields called "forward-to" and using them as recipient email addresses, so you could make forms that accepted lots of addresses; making it loop over all fields with that name rather than just looking for one was the first assumption I removed to simplify the software while making it more flexible) then proceed with handling the HTTP request in the normal way.

But when a requirement came in to support unsubscribe forms, I just added an "unsubscribe" command, which took all of five minutes. Adding an "update" command handler to let people update their details in the master DB took a little longer, just because I had to list all the supported field names in the documentation and include a more lengthy example. And then, because I'd implemented the recipient and email message IDs in the URLs as optional things from the start, adding a "subscribe" command handler that created a new recipient in the DB from scratch, along with a tweak to the UI to let the user get an anonymous URL for any file in their website took an hour or so - and meant that users could now create subscription forms and generate a URL they could publish. I think I also added a "send" command handler to send a copy of a specific email message to the recipient ID in the tracking context; as the "subscribe" command put the new recipient's ID in the tracking context, a URL with a "subscribe" command followed by a "send-" command would handle a subscription and then send a welcome email to the new subscriber...

I added a few other command types that did various other things in the system, and all was good.

Now, I didn't "set out" to design an "overcomplicated super-flexible" HTTP server system that could do all these things when I was asked to add the forward-to-a-friend system. I just spotted that the following assumptions would be really easy to NOT bake into the core of the implementation:

  • Forward to a friend is always to a single friend; it's easy to create multiple form fields with the same name in an HTML form, and easy to make my server loop over any it finds.
  • Every visit to the new generic HTTP server will have a recipient ID and an email message ID; although forward-to-friends will as they will be linked from an email, it's easy to imagine that we might want to host subscription forms in future. So making the tracking IDs in the email optional (or allowing for other kinds of tracking IDs in future) by making the URL structure by a series of tagged fields (it looked something like http://domain/sss/Rxxx/Eyyy/Wzzz/foo.html for a request for file foo.html in website ID zzz for recipient xxx from email yyy, with sss being the anti-tamper signature) was worth the few minutes' work.
  • Requests to the web server will be forward-to-friend requests or just static resources in support of a response page; it was easy (given we already had an ordered list of tagged components in the URL) to added an ordered list of arbitrary commands (with URL path components like forward, subscribe, etc; any component before the Wzzz/filename part was considered a command or tag, and tags started with an uppercase letter).

The "extra work" required to do things that way rather than the "simple" way was tiny. But it meant that the core of the HTTP server was simple to read and didn't need to change as we extended it with more and more commands (by adding them to a lookup table); it made a powerful system that was easy to extend, easy to understand, and capable of things we didn't have to "add as features" (some or our users did quite creative things by stacking commands!). As the HTTP server core and the commands were small separate modules with a clearly-defined interface, they could be understood individually and were well isolated from each others' implementation details, the benefits of which are well documented.

So please help me fight the assumption that putting thought into designing software means it'll be complicated, more effort to implement, and that "it'll need rewriting anyway so there's no point"! Let me do my job well 🙂

Our radio adventures (by )

My last post was about amateur radio in general, but for those of you who are interested, here's a summary of where we're at and where we're heading.

Jean and I did the Foundation Licence course together and passed, so we have UK Foundation licences. That means we can transmit in nearly every amateur frequency allocation, covering the HF, VHF, UHF and microwave bands discussed in my blog post. We're limited to ten watts of output power (details vary slightly in some bands), which is enough to do most things: at that power level you can communicate over intercontinental distances on HF, especially using Morse or data modes. I'd like to progress to the higher levels of licence, not so much for the increased power levels (although those might be handy), but because at Intermediate level you can use transmitters you have built yourself; and at the Full level, you can operate outside of the UK (as that level of licence meets the requirements of international radio law).

What we've done so far:

  • Obtained FM VHF/UHF handhelds - the ubiquitous Baofeng UV-5Rs - and experimented with external accessories for them; we have a magmount antenna on the car and a UV-5R hooked up to the 12v socket in the car and a fist make, as a cheap mobile rig, and various different headsets, antennas, and pouches to find comfy ways to wear the things out and about.
  • I've tinkered with antenna construction; I built a quarter-wavelength vertical with a radials, hinged so it collapses down flat, for the VHF amateur band. It seems to work pretty well.
  • Chatted to each other on the handhelds. We've done some experiments for the sake of it around town, and used them to usefully keep in touch at a couple of outdoor events.
  • I've played with APRS a bit. I didn't explain that in my last blog post: it's a protocol for announcing position and basic status information in digital data packets over the radio, meaning you can track the positions of other stations near you.
  • Chatted to each other on a local repeater, as an experiment. (Repeaters are set up by volunteers on tall masts so they have a good view of a lot of the local area, and re-broadcast your signal for you, so that two people down on the ground amongst buildings and obstructions can still talk to each other).
  • I've joined in some conversations with strangers on the local repeaters, but it's still a bit nerve-wracking doing so!
  • I've done some experiments with building simple HF antennas and hooking them up to a wide-band receiver I already have, and picked up a few signals - but mainly noise at home.

Things we're interested in doing include:

  • Volunteering with the local RAYNET group, to get some practice in doing useful things with short-range comms (VHF and UHF). In particular, we both want to gain experience with radio etiquette and build up confidence.
  • Talking to the International Space Station, which our licences allow - we have all the required equipment apart from a more directional antenna! We'll probably start with relaying digital data from the ISS digipeater as it doesn't rely on an astronaut being around to say hi.
  • Talking to amateur satellites; requires the same equipment as the ISS, but involves talking to other people on the ground. This will be fun because (a) SPACESHIP!!!! and (b) you get to talk to people much further away.
  • Getting onto HF! Our home isn't great for an HF antenna installation, from my experiments, so I'm interested in "portable operation". This looks fun - there are contests such as SOTA based around making contacts from hilltops and mountaintops, or IOTA for making contacts from islands - and I like the flexibility of being able to set up anywhere.
  • Getting a VHF/UHF antenna on the roof at home, with a base station transceiver. I'd like to be able to sit at my desk and monitor the local repeater, for instance, so that Jean can get in touch with me while she's out and about. Access to our chimney is tricky and my desk is in the workshop which is at the other end of the garden to the house (requiring a long cable or an expensive remote-controlled transceiver), so I might need to build a small mast on top of the workshop: the signal might be good enough for the repeater, at least, and making a truss mast out of aluminium would be a fun welding project.
  • Maybe build a "magnetic loop" HF antenna on the workshop roof; those are supposed to get better results in circumstances such as mine. Experiments must be done!
  • We both want to learn Morse; this opens up the ability to make longer-distance contacts with smaller, cheaper, transmitters, and is kinda fun in its own right.

Amateur radio (by )

The radio spectrum is a complicated and varied thing.

Although some people seem to separate "radio" from "microwaves", I tend to lump them together, so I'm going to consider "the radio spectrum" to be the section of the electromagnetic spectrum from the very lowest frequencies we can usefully use (a few Hertz) up to frequencies that we'd start to count as "infrared light" (several terahertz).

Those bounds are based on practicality - for very low frequencies, the antennas required to efficiently transmit become impractically large and the bandwidth available to communicate gets ridiculously low; at very high frequencies, our electronics cannot process the signals and we have to switch to optical methods, which is a whole different ballgame.

I'm going to give you a whirlwind tour of the radio spectrum - but with an agenda: I'm just setting the scene so I can talk to you about amateur radio... Read more »

Protected: Moon Mania PechKucha Talk (by )

This content is password protected. To view it please enter your password below:

Gaia – Llanthony 2019 (by )

This weekend starting tomorrow there is going to be a giant Earth floating in the grounds of the historical site Llanthony Secunda Priory and Moon Mania Mega Make my community textiles project will be there with it 🙂

Scout Sewing MoonMania

It will be there from 10 am till 5 pm for three days:

Friday 28th of June Saturday 29th of June Sunday 30th of June

I will be there from 1 pm on the Friday, 2 pm on the Saturday and all day Sunday. If the weathers nice I will be outside by the coffee van if it's bad I'll be inside. This will also be the first proper outing of Moon Memories where I hope to snag people and get them to tell me their tales of the moon landings and space race or what their aspirations for the future of space exploration is. This is for a legacy piece which will result in a book and website and my hope is to archive these stories properly so that they are not lost to the sands of time (or moon dust!).

Gaia is part of Gloucester's amazing Earth and Moon festival which will be going on through out the summer and autumn and includes events at the Libraries including their annual reading challenge which is space themed, cinema at the GuildHall, the accompanying Moon to Gaia will be in the Cathedral later in the Year, and of course last but not least there is the Museum of Gloucester which has a Moon Exhibition which is opening this Saturday (29th of June 2019) starting with a kids craft corner focused on International Asteroid Day. I myself will be doing talks, creative writing that's moon themed as well as rock handling at the Museum.

MoonMania Talks and Rock Handling

Some of the events can be found in the SoMac brochure, some in the museums and libraries and Guildhalls own brochures 🙂

I also have various poetry, arts and science talks planned starting on Tuesday with a little PechaKucha talk at Liquor and Chow.

Moon Mania Creative Writing and Spoken Word

Now back to Gaia - this is an amazing installation that is going on a tour around the UK and is made by Luke Gerram with imagery from NASA. I am very excited to have the chance to be involved for Moon Mania!

Gaia is the name of the Greek Goddess who made/is the Earth, that which gave birth to everything, in Earth and Planetary Sciences we have the Gaia Hypothesis which takes the planet as a kind of mega organism with each part - trees, animals, sea, weather - all being part of the whole and what affects one affects the other. Computer Gamers will also be well aware of these concepts!

I have a mini obsession with Gaia so here is one of my favourite art making and concept creating videos about another fabulous art inspired by Mother Earth:

Rebirth of Gaea

Oh and I am looking for as many people as possible to be involved with the Mega Make!

Moon Mania Mega Make community textiles project

WordPress Themes

Creative Commons Attribution-NonCommercial-ShareAlike 2.0 UK: England & Wales
Creative Commons Attribution-NonCommercial-ShareAlike 2.0 UK: England & Wales