ESXi 4, Moving VMs to new Storage

As previously discussed, we recently purchased an MD3000i iSCSI SAN from Dell. We are running the “free” VSphere ESXi 4 on 3 hosts. That’s right, no VCenter, no VCB, no VMotion, no SVM. So, how does one best approach moving these VM guests from local storage to our new lovely SAN? Well, we looked into the following possibilities:

1. Upgrade and purchase licenses for VSphere, VCenter, etc…
2. Use the VI client data store browse “Move” functionality
3. Try out a third party product like Veeam FastSCP
4. Try standard TAR/GZ combined with cp command
5. Try vmkfstools disk cloning

One important note. While testing some of these options, we became aware of a complicating factor. If you have created your disks as Thin provisioned and are moving to storage that has a different block size, the vmdk will become thick provisioned. This means that the “entire” contents of the fully thick provisioned disk will be moved, not just the used space.

Upgrade and Purchase VSphere Infrastructure Licenses

This option is one we continue to re-evaluate every 6-8 months. Working in a non-profit, we are always evaluating how to be the best stewards of the resources we’ve been provided. One of those resources is obviously financial resources. This seems like the only real option for moving running VMs.

While having VSphere Enterprise in place would make this process of migrating to new storage easier, we just cannot justify the long term expense. Our small environment just does not need what you pay for over the long haul.

As such, we ruled this out. Some downtime is perfectly acceptable in this organization.

Use VI Client Datastore Browse/Move Functionality

When using the VI Client, you can browse the VMFS datastores similar to a Windows Explorer window. This is accessed by viewing the storage configuration of the host in the VI Client. Right-clicking the particular datastore and choosing browse.

Obviously, this function does not work on running VMs. As a matter of fact, in one of our tests, we actually caused a problem doing so. This is a “move” not a copy. If the move fails mid-way through, some files will have been moved while others are still in their original location. Now you’ve got a bigger mess on your hands.

So, this option means the VM has to be powered off. Ok, we can deal with that. The other downside was that, as explained above, the new vmdk is thick provisioned and therefore, much more data is transferring that is really necessary.

Test 3rd Party Product Like Veeam FastSCP

I really do like the folks over at Veeam. I have used the monitoring /reporting product and like there licensing models. We thought we’d check out their free FastSCP product. Why? Well because is says “Fast” in the name.

The tool worked great. It did everything is said it would do. Like the other options, the VMs need to be powered down. We even tested copying between different ESXi hosts. Worked great. The only downside was the lack of speed. Yeah really! We did some reading and it appears that this may be a throttling of the ESXi product, not Veeam FastSCP. Bummer.

Try standard TAR/GZ combined with cp command

“Wow”, I though as I was researching. Why don’t we just tar/gzip everything and then copy to the new storage location and then untar. Well, not so fast our ESXi said to m as it spewed out an error that “tar: cannot store file ‘xxxxxxxxxxx.vmdk’ of size #########, aborting”. Yuck~!

It appears that BusyBox, the underlying unsupported ESXi console has a limitation of 4GB in ESXi3.5 and 8GB in ESXi 4. Therefore, trying to tar a 100GB vmdk file went no where fast. Check this option off the list.

Try vmkfstools disk cloning

This, we thought, is an interesting idea. Yes, the VMs still need to be powered off but perhaps this will shorten the downtime by lowering the amount of data that gets copied. See, as we talk about earlier, going to storage of a different block size will mean yout thin provisioned disks will begin thick. As such, the total size of the thick disk has to be transferred. This could siginificantly increase the copy time if you have lots of unused space on a disk.

The vmkfstools utility has a feature known as cloning. This allows you to make a clone copy of a vmdk disk to anyVMFS datastore. Now that’s all good but where this got us jazzed was the ability to force that newly cloned disk to be created as a thin provisioned disk. Eureka!

Now, first, make sure you remove all snapshots in the vm. Those are just going to cause us a headache.

So a command like this does the deed:
vmkfstools -i /vmfs/volumes/source/vmname/vmname.vmdk -d thin /vmfs/volumes/dest/vmname/vmname.vmdk


Note: you need to do this against the vmname.vmdk NOT the vmname-flat.vmdk. Be forwarned.

Some small scale, non-scientific testing gave us around 1.3GB per minute in transfer speed. Again, this is not true i/o but rather therate used for calculating down time for our migration project by taking the amount of used space on the source drive/time to create cloned drive = transfer rate.

Now, when the vmdks are moved, don’t forget to cp all the other VM files in the VM directories!

This looks to be the path we will be taking and thought we’d share how we arrived here.

Popularity: 44% [?]