@ndy: This scheme will happily let you fully replicate, by just making sure the sum of the trust levels of all your backends is 100% - so the only way to get full trust is to put every block everywhere. And what I think you're proposing - picking one backend as the target for each snapshot - won't work unless you also restrict reads to just that one backend for the duration of the snapshot - or the snapshot sent to that backend will refer to 'shared' blocks on other backends, so you won't be able to restore from just that backend. If you want that kind of semantics, just have a lot of backends and pick a different one for each snapshot; Ugarit doesn't need a clustered backend manager for that! The idea here is to manage a pool of storage, so you can just add extra backends in with more disk space when you start to run low, while having redundancy due to replication.

But for every uploaded block, it would pick the backends randomly, with the randomness weighted by the load weightings of the backends:

This worries me from a restore point of view. Surely you want all the data needed to restore a single snapshot in at least one discreet place so that restores are easy: either because you only require one location to be reliably connected to the net at the time or because you can just go to the remote location and fetch the disk.

Make sure you can use FS snapshots, where supported. And later versions of ZFS can tell you what changed between snapshots. Making plugins for this kind of task would be quite useful.