Virsto and Deduplication (first in a series)

by Alex Miroshnichenko, CTO

Posted Thursday, May 20, 2010 in Technology          4 comments

Tags: dedupe, false assumptions, no dupe

Since our Virsto One announcement we have received a stream of questions regarding our data deduplication solution. This is understandable given the dedupe hype that media teams are pushing. Even the least sophisticated of hardware distributors feel compelled to put a "we support dedupe" message on the front pages of their data sheets; some even add yellow ribbons.

Virsto one does dramatically reduce the amount of storage consumed by virtual disks (VHDs in Microsoft parlance). However, Virsto One does not perform data de-duplication as the industry has coined the term. While dedupe is a useful technique in many environments including virtualization, the specific nature of virtualization workflows calls for what we at Virsto call no dupe. More on that later.

Deduplication 101

This post is not the best forum to provide a comprehensive guide on the diverse universe of data deduplication technology. For this discussion, I will call data dedupe any technology that analyses data content for the purpose of finding redundancies for subsequent reduction of required space in storage or bandwidth for transmission. Readers old enough to remember their high school years before MySpace may also remember a similar data reduction technique called compression. There are substantial differences between compression and deduplication but both techniques rely on analyzing data content.

Virsto One does not use data deduplication in the above sense. We do not look inside data blocks as they pass from virtual machines to I/O channels. (We don't even touch the data, but that's a topic for another post.)

Introduction to Virsto's thinking on VM duplication

Virsto One is built on the realization that in a virtual environment the overwhelming majority of virtual disks are not independent objects. Rather, they are derivatives of each other.

This is an important insight, so let's think about it for a few minutes.

Virtual machine images are fundamentally different from disk images in the physical server world. The magic of virtualization is that now a server (or desktop) is just a big file that is formatted to look like a disk (VHD, VMDK, or other format). This paradigm shift is enormously powerful, and enables all kinds of cool things like live migration of virtual machines across physical servers. It also completely changes the workflow of provisioning servers or desktops.

A new virtual machine is normally – in fact, almost always – created (or provisioned) as a copy of another reference VM. Such reference images are referred to as templates or golden images. As an example, a golden image may contain a reference installation of an operating system with certain applications.

Want a new VM that is similar to one that you created earlier? Just copy the image, simple. Remember, since that VM is just a big file that looks like a disk, making that new VM image means making a copy of the original golden image. Lots of people call that cloning, although you should beware that not every vendor means exactly the same thing when they say "clone".

What is a clone anyway?

In everyday non-technical conversations, when we hear the word "clone", we think of a separate, independent copy of something. The copy is its own entity with a life of its own, but at the moment of creation, the clone is indistinguishable from its antecedent.

Cloning implies repeatability. If you can make one clone, why not two? Ten? A thousand?

An important concept is that each instance is an individual and can change over time. So a year later, those thousand clones may look very similar to each other, but they are no longer identical. Some of the clones may differ in only very small ways, while other may have more noticeable variations.

This commonsense connotation of a clone is exactly the right way to think of VM image clones. Keep this imagery in mind.

The germ of an idea

For fun, you might want to think about how this notion might suggest a vastly different and better storage architecture for VM environments. If you do, you might start to understand why we started Virsto Software a couple years ago.

In the next post, we'll discuss how this notion of cloning relates to the way virtual machines are provisioned and stored, and how Virsto One optimizes the process.  In part three of this triptych, we'll compare and contrast dedupe against no dupe.