As David points out, the "Congruent Infrastructure" stuff at http://www.infrastructures.org/ is a good read.

The idea of having several basic -- a VM hosting layer, a distributed, reliable storage layer, and a higher layer of auto-installed VMs that provide all your core functionality -- is very attractive. (It's what DoC's been doing for years, for various degrees of storage distribution and virtualization.)

The idea that individual chunks of server hardware are expendable, however, is a useful one -- when building on unreliable commodity hardware, its vital! Key to this is the ability to either run up a clone -- or regenerate from first principles -- the machine image that's providing a particular service, and keeping its persistent datasets on a separate reliable storage infrastructure.

Cluster filesystems -- GFS, GFS2, Lustre, etc -- are starting to look exciting and usable, though I haven't had a chance to play with them yet. They look particularly important for IO-intensive workloads.

Control layers for your VMs start to become important. Most of the time, you want scripts on the work machines to pull changes from central repositories -- see DoC's Sys::Maint infrastructure, triggered by Cron. Their function is to take a (possibly broken, or minimally bootstrapped) machine image, and to make it good and ready for normal operations. This can include everything from setting up NTP to installing packages to making sure the correct system configuration files have been installed.

The other important control layer is the ad-hoc "push" mechanism -- for when you need to execute an important change to a (total) subset of machines right now. Batch tools that connect over SSH using some means of automated authentication is good for this; for example, see my remote script.

It sounds like you're definitely thinking along the right lines, though how much of this you need to implement may vary depending on how far up you need to scale (and with what resources).