Cloud Storage

Currently, you can go to various providers and buy online storage capacity (IMHO, rsync.net is best, after research I did to find an offsite backup host for work). It's more expensive than a hard disk in your computer, and miles slower, but it has one brilliant advantage: it's remote. So it's perfect for backups.

And that's the heart of a free market - storage is cheap to the cloud providers (they just buy disks, and in bulk at that), but their storage has more value to you than your own storage because of it's remoteness. So they can rent it to you at a markup, and you get a benefit, and everyone is happy. Money flows, the economy grows, and one day we'll get to have affordable space tourism et cetera.

But large, centralised, cloud storage providers are attractive targets for people who want to steal data. They become centralised points of failure; if they go bankrupt, lots of people lose their backups. Therefore, it's smart to do your backups to more than one of them, just in case. But that means setting up your systems to talk to each one's interfaces, arranging payment and agreeing to terms and conditions with them all individually, and so on.

Surely this state of affairs can be improved? With ADVANCED TECHNOLOGY?

Well, I think it can, and here's how.

Imagine a marketplace for cloud storage. This might be a centralised trading server, or it might be a peer-to-peer protocol... greater minds than I are working on decentralised P2P marketplaces, I hope. But however it's implemented, imagine that I can run a daemon on my server that measures my free disk space, subtracts some amount (10GiB?) for my short-term growth, and rents the rest out on the marketplace. By looking at the depth of market (how many unfulfilled bids for how much storage are out there, ordered by bidding price, highest first), it can choose the best price it can rent my storage for that will use up my available storage. My offer will include a price to upload a block (base price + price per byte), the price to keep a block (base price + price per byte, and the billing period) and the price to download a block (base price + price per byte).

It's an interesting question whether periodic storage fees, or just having a "successful download bounty", will win out. Charging storage fees encourages the buyers to notify you if they don't want a block any more, but just charging for successful downloads (and just deleting blocks that aren't referenced on an LRU basis to free up space) is beautifully simple.

The trust model is rather different to normal cloud providers. If a provider loses their data, I can't sue them; I just don't get to pay them the download bounty for getting my block back. So I'll have to store my data widely across several providers, and prices will lower to take account of that, and I'll need to do trial downloads to check my blocks are still available from time to time, and if not, hire a new storage provider to take a new copy of that block from a surviving copy.

But all of this can be done in software. A storage manager app would present a simple get/store block interface to, eg, Ugarit or Tahoe-LAFS, but behind the scenes, it would manage relationships with providers, checking blocks are available, ensuring there's a sufficient number of copies of each, shifting between providers when rates go up or if a provider's reliability score drops too low, etc.

But all of this depends on it being easy for computers to send money between themselves, which is where Bitcion comes in. Storage providers and consumers can just run bitcoin wallets and arrange transfers between themselves.

The end result? I can run a daemon to rent out spare storage space on my system, and money would slowly accrue in a Bitcoin wallet. The daemon would rent out all but a safety margin of my space, and as I used up my safety margin, it would shed blocks (notifying the owner) to make more room, and increase its offer price in the market to reduce demand so that the lower-paying blocks move willingly and can be replaced with higher-paying blocks.

And I can run another daemon as part of my backup system, that would spend from the same bitcoin wallet to get backup space on other machines. When I have mostly empty filesystems, I will be spending little on backups, and earning lots on renting that space out, so money will accumulate... when I start to fill the filesystems up, the trickle will slowly reverse, and then perhaps I should spend my profits on a new hard disk before they all go and I have to top it up from my own Bitcoin wallet!

Details

The devil's in the details, as always. The marketplace will depend on being able to place bids in a standard format. Potential buyers will need to be able to introduce themselves, perhaps via an HTTP-based protocol served by the storage-for-hire daemon on my server; sign up for an account by registering a public key, and then access upload/download/delete block interfaces. The daemon would quote a price in the market, but each block upload would have to be annotated with the rates the buyer is offering, to avoid race conditions when rates change during a transaction. Blocks with unattractive rates can be rejected by the server. There would need to be a back channel for the server to asynchronously notify buyers that it needs to get rid of a block - I'd hate to force buyers to have public IPs (many will be behind NAT) by giving them an HTTP endpoint, but perhaps a choice of that or polling the server to ask for blocks that need to be shifted within a time limit would suffice. It would also be polite for the server to inform the buyer of any blocks it had to delete without notice, rather than waiting for them to check them.

But how to address blocks? On the one hand, I want content-addressed storage, as it prevents cheating. There's no way a bad server can claim to have blocks it's deleted by sending back random junk and saying "But that's what you gave me! PROVE I'M LYING!" if they are identified by hashes. But on the other hand, existing systems have their own addressing schemes (Ugarit identifies block by a keyed hash of their uncompressed plaintext contents, so that the hash doesn't give away the content (it's a keyed hash), but it will also remain unchanged if the compression or encryption algorithms are upgraded - old blocks can still be read while new blocks are written with the new algorithms, and old blocks can be re-compressed and re-encrypted without breaking the references to them). So enforcing that blocks are identified by the SHA256 of their ciphertext would exclude various uses.

The best scheme I can think of is this: each block is identified by a client-supplied ID string combined with a hash based on an agreed algorithm. So the server would say "I support SHA1, SHA256, and Tiger", and the client would say "Ok, here's a block I want to call Boris, and I like SHA256", and the server would reply with "Ok, that block's called Boris:<256-bit hash>". The client should check the returned hash matches the hash it computed itself. A client that's happy with server-assigned IDs would give all their blocks the same name (the empty string), as the hash in the resulting identifier keeps it unique. The server will store the block by hash (deduplicating blocks with the same hash), but keep a per-customer table mapping names to hashes. If the client hasn't provided distinct names, then the LAST mapping for the name provided is kept.

Meanwhile, on retrieval, a block can be requested by name, or by hash. The client should remember the hashes, even if it uses names, so that it can check that the server isn't sending it a garbage block.

As a Ugarit backend, this would work fine; the Ugarit keyed hash can be used as the name, and the server's hash stored for cross-checking on retrieval. If the local store is lost due to disaster, it could either be restored from another backup somehow, or it could just be skipped and we hope that the servers don't lie to us (the latter would be better than refusing to try to restore at all!). Ugarit tags (which are the roots of the hash tree) can be stored by using the tag name as a block name, and using the fact that multiple uploads with the same block name just overwrite the name->hash mapping.

Needless to say, clients should encrypt ALL their data! You can't trust random providers.

Have I missed any other scams? Servers might try to accept lots of blocks and keep the upload fees and never keep them. That provides an incentive to servers to not charge upload fees at all, and just hope to make money on download fees and/or storage. It'll be interesting to see how the market ends up structuring itself! Also, as it's a low risk to accept data from somebody but a high risk to send money, I think the protocol should be based around periodic billing at the end of the period, rather than per-operation micropayments (that makes more efficient use of Bitcoin's transaction charge and hour transaction confirmation latency, too). Billing periods could be anything from a day upwards.

But this is a real cloud, in a sense far beyond the current definition of cloud computing. Millions of tiny providers, all competing in a marketplace, with the clients automatically spreading their risk across them in a fine-grained way. I think that'd work for storage, as it's easy to define and commoditise; doing it for computation might be possible, but it'd require much more standardisation of execution models and sandboxes and the like...

(Thanks to the folks in #bitcoin on Freenode IRC for inspiration for all this!)

UPDATE: A friend suggests an improvement over periodic downloads to check the data is still there. Have a "check" operation where the client supplies a random key and a block name or hash, and the server has to hash the block along with the key and return the result. That allows the client to check the block is still there if it has a way to get a local copy of the block. Otherwise, it would still have to rely on downloading the block and checking the hash matches.

3 Months and Counting

Mary is coming up to 4 months but was three when the picture was taken - she is belly laughing, attempting to sing along with me and Jean and looking round at the world in interest. She will remain sitting up if you put her in that position and if place on her front she makes slow progression towards what ever it is she wants!

She loves fabric and cloth and has unfortunatly sampled the delights of cat tail :/

Her favourite toys are the Minni Mouse Jean bought her and Gaia (the planet earth soft baby toy with mirror in it etc..

She will stop crying at the sight of her sister or me with the glittery guitar. She loves me singing the alphabet but only if I sign the letters at her whilst singing. She moves her hands in a sort of poo sign whilst going red and still appears to do the milk sign though this is probably parental wishfulness.

She loves sucking her hands and is the drooliest baby I have ever met! She even acts excited when Alaric comes home from London!

Health wise she has had the vaccinations, she appears to have an issue with cow milk (formula) and right from word go has had more skin issues than Jean.

Her sister is far too huggy with her and still gets annoyed if people try to 'take her sister away' ie hug the baby. She also tells us off if we say Mary is daft!

And this time an ambulance…

So I had stopped bleeding for about 2 weeks with just a little bit of 'marking' every few days and then I started what I again thought was a period and it was ok for a couple of days and then Mary had a hungry night and fed and fed and fed (breast). But I could feel the flow getting heavier and heavier and I started having to get up to change pads etc.. again and again and again and then it was time to get Jean up for school so I got up the last time and felt the bleeding become very heavy and was on everything - I made a right mess of everything.

And not quiet being with it I got into the bathroom and started to feel strange. I remember thinking this isn't good and Jean burst in her normal fashion and apparently I asked her to help me and tried to follow her out of the bathroom as she went to fetch my dad. He came up the stairs just in time to try and catch me as I passed out which was fortunate as I landed on the almost full length mirror I had propped up on the landing.

This mean I only busted the frame rather than shredding myself to pieces. But it also ment I was slumped on my knees laying sideways on a mirror. Jean apparently called my mum on my mobile phone (I think she just pressed the button she knows calls people and mum was the last person I'd phoned). Dad was trying to get me up to put me back in bed when mum arrived, I was trying to talk but unsuccessfully.

She called an ambulance, by the time they arrived the bleeding had slowed and I was trying to get up off the floor because as I kept repeatedly telling everyone I had to get Jean to school :/ I cried cos I was bleeding on the carpet and had ruined my favourite things and the baby was hungry and Jean had to go to school :/

It took both ambulance people to get me up off the floor and I screamed with pain Mainly the pelvis. I was starting to feel more with it and embarrassed that I was causing a huge fuss and appeared to be half naked.

My sugar level was low but I hadn't had any breakfast so they spent time trying to get me to eat and drink. After three pints of water I started to feel a lot better - food had been hard as I felt so sick. As the bleeding had slowed down they called the Drs surgery and I went for an emergancy appointment.

And after an examination and a phone call to the hospital I have more antibiotics and hormone tablets to stop the bleeding. More bloods have been taken and I have a hospital appointment for Monday - by the time I got to the Dr I was just in pain and so very tired. I had lost lots of black jelly again though.

My womb is enlarged and the wound on the outside is even acting up again. I was barely able to walk with the crutches at the Drs surgery the pelvis was acted up so much. I think it extrememly lucky that I didn't pass out whilst using the crutches.

And then I should have just been going home to rest and take tablets but it didn't quiet work like that. Leaving the Drs we went to the Chemists to get the prescription or rather we stayed in the car whilst Dad went - none of us realised how late it was as I'd passed out at around 7 in the morning.

This meant that it was actually past lunchtime and Dad had not had his morning snack and had had lots of stress and so went hypo at the shop :/ They told him to stay put whilst his chocolate bar kicked in but he walked out and collapsed. He then got up and did it again. By this point people had stopped to help and he ended up sitting on someones steps with the pharmacists trying to find out where he lives. At this point me and mum were talking about how we should have made dad eat the banana he'd refused when we arrived as when there is stressfull stuff he tends to go hypo and then I saw the time and phoned him.

I ended up talking to the girl from the shop and they walked him down to the car - initially they'd asked if someone could come and get him not realising that we were in the car park. I had to explain that I'd been passing out, mum was a wheelchair user and we had the baby with us :/

They were concerned that he would try and drive but I assured them we'd wait and feed him the emergancy food.

So we got home just in time for the Jean pick up from school. I went to bed and slept as did everyone else except Jean who watched a film.

It was an adventure I hate living in Interesting Times :/

Weaning

Mary is only 3 1/2 months old and I am weaning her - I started by letting her just taste things at about 3 months because she would squeak angrily when we were eating - screwing her little face up and looking like a red Donald Duck. But because World Health Organisation recommends just breast for the first 6 months I was trying to hold off full blown weaning as in giving her mushed up foods to eat.

But she just would not sleep and began grabbing at food we were eating. So I chewed up banana and fed it too her and she slept like a log!

We had to wean Jean at three months as well after talking to the Midwife/Health Visitor who was coming round at the time - Jean was a hungrier baby though so I thought Mary would last out a bit longer.

Anyway the last few days have been great - she loves banana, carrot and parsnip, semolina (without sugar!), peas and sprouts. She is really enthusiastic about this and grins when she sees she's being put in the highchair. She even grabs the spoon and to my amazement tries to ram it in her mouth - though I think this is many due to the fact she tries to ram everything into her mouth at the moment!

Nerves and Guitar

I live near Cheltenham and that town has some very good poetry nights etc... so Friday saw be going along to Slak Bar to an open mic at Earshot (probably my favourite poetry night). But this time it was opened up so that the open mic included acoustic so I thought I'd take my guitar and play one of the songs I worked on last month.

But though I was fine if a little upset as Jean refused to talk to me as I said she couldn't come in and watch, things got a little delayed and my anxiety and fretfullness mounted.

For a start I have a glittery nail vanish covered guitar, it is not one of those you can add pick ups too and I knew I wasn't getting the vocal range to be heard well over it in any case. (not an excuess but my glands where up due to hay fever and because of breast feeding I couldn't take anything for it - I find this heavily affects my singing)

Then one of the real acts got up (Horston Longsail) then one of my friends got up and using my guitar and his string clamp thing played a lovely haunting song.

Then I got up - for a start I hadn't wanted to use the crutches but there was no way I was going to be able to get up onto the stage without them. So I was feeling a little flustered as well as nervous. I sat down and begain to play and stuffed it up! I was shaking which didn't help and also I was so nervous my hands were sweating and my fingers slipped of the strings - I'd never encountered that before. Stressing out also makes my voice even tighter and smaller and I couldn't check if I was in tune becuase of the ringing in my ears.

The neverous twitch I used to get where my lip curls up on the left hand side started. I was still attempting to play. I closed my eyes and sang the second verse of my song instead of the first but then managed to look up a bit and get more in the flow - though I changed to the wrong cord and had to alter the sequence slightly so that it would all still fit.

I got off the stage feeling I had really stuffed up - I was so annoyed at myself - how comes I can play the song through perfectly whilst Jean is doing things like licking the guitar to get my attention becuase its dinner time but I get infront of people and it all goes wrong?

It's nerves.

People were very nice about it and I got given tips in the fact the microphone needed to be nearer my mouth and told that it started out a bit hesitant but once I got into it it was good. I even got told the song was like my poetry full of the sweet and dark and that it sounded very PJ Harvey.

I also know that I wasn't really ready to 'perform' in public but that if I didn't start trying I never will be. The issue I have is playing infront of other people and like getting rid of the stutter there is only one way to cure this and that is - if I mess up an open mic I simply need to play more open mics. Now I haven't got a clue weather my songs are any good or even if I can still sing in tune I just have to hope that people aren't going to be too mean!

I am still annoyed with myself over getting so flustered though - and with being annoyed - it meant I didn't really click back into gear until latter in the evening so basically missed Avril Staples poetry as I was waiting for the ringing in my ears to stop!

But I did have a fantastic night - I really enjoyed the two headline poets Bohdan Piasecki and Michael Wilson.