1. Upgrade and purchase licenses for VSphere, VCenter, etc…
2. Use the VI client data store browse “Move” functionality
3. Try out a third party product like Veeam FastSCP
4. Try standard TAR/GZ combined with cp command
5. Try vmkfstools disk cloning

One important note. While testing some of these options, we became aware of a complicating factor. If you have created your disks as Thin provisioned and are moving to storage that has a different block size, the vmdk will become thick provisioned. This means that the “entire” contents of the fully thick provisioned disk will be moved, not just the used space.

Upgrade and Purchase VSphere Infrastructure Licenses

This option is one we continue to re-evaluate every 6-8 months. Working in a non-profit, we are always evaluating how to be the best stewards of the resources we’ve been provided. One of those resources is obviously financial resources. This seems like the only real option for moving running VMs.

While having VSphere Enterprise in place would make this process of migrating to new storage easier, we just cannot justify the long term expense. Our small environment just does not need what you pay for over the long haul.

As such, we ruled this out. Some downtime is perfectly acceptable in this organization.

Use VI Client Datastore Browse/Move Functionality

When using the VI Client, you can browse the VMFS datastores similar to a Windows Explorer window. This is accessed by viewing the storage configuration of the host in the VI Client. Right-clicking the particular datastore and choosing browse.

Obviously, this function does not work on running VMs. As a matter of fact, in one of our tests, we actually caused a problem doing so. This is a “move” not a copy. If the move fails mid-way through, some files will have been moved while others are still in their original location. Now you’ve got a bigger mess on your hands.

So, this option means the VM has to be powered off. Ok, we can deal with that. The other downside was that, as explained above, the new vmdk is thick provisioned and therefore, much more data is transferring that is really necessary.

Test 3rd Party Product Like Veeam FastSCP

I really do like the folks over at Veeam. I have used the monitoring /reporting product and like there licensing models. We thought we’d check out their free FastSCP product. Why? Well because is says “Fast” in the name.

The tool worked great. It did everything is said it would do. Like the other options, the VMs need to be powered down. We even tested copying between different ESXi hosts. Worked great. The only downside was the lack of speed. Yeah really! We did some reading and it appears that this may be a throttling of the ESXi product, not Veeam FastSCP. Bummer.

Try standard TAR/GZ combined with cp command

“Wow”, I though as I was researching. Why don’t we just tar/gzip everything and then copy to the new storage location and then untar. Well, not so fast our ESXi said to m as it spewed out an error that “tar: cannot store file ‘xxxxxxxxxxx.vmdk’ of size #########, aborting”. Yuck~!

It appears that BusyBox, the underlying unsupported ESXi console has a limitation of 4GB in ESXi3.5 and 8GB in ESXi 4. Therefore, trying to tar a 100GB vmdk file went no where fast. Check this option off the list.

Try vmkfstools disk cloning

This, we thought, is an interesting idea. Yes, the VMs still need to be powered off but perhaps this will shorten the downtime by lowering the amount of data that gets copied. See, as we talk about earlier, going to storage of a different block size will mean yout thin provisioned disks will begin thick. As such, the total size of the thick disk has to be transferred. This could siginificantly increase the copy time if you have lots of unused space on a disk.

The vmkfstools utility has a feature known as cloning. This allows you to make a clone copy of a vmdk disk to anyVMFS datastore. Now that’s all good but where this got us jazzed was the ability to force that newly cloned disk to be created as a thin provisioned disk. Eureka!

Now, first, make sure you remove all snapshots in the vm. Those are just going to cause us a headache.

So a command like this does the deed:
vmkfstools -i /vmfs/volumes/source/vmname/vmname.vmdk -d thin /vmfs/volumes/dest/vmname/vmname.vmdk


Note: you need to do this against the vmname.vmdk NOT the vmname-flat.vmdk. Be forwarned.

Some small scale, non-scientific testing gave us around 1.3GB per minute in transfer speed. Again, this is not true i/o but rather therate used for calculating down time for our migration project by taking the amount of used space on the source drive/time to create cloned drive = transfer rate.

Now, when the vmdks are moved, don’t forget to cp all the other VM files in the VM directories!

This looks to be the path we will be taking and thought we’d share how we arrived here.

So you can see the MD3000i with the faceplate off directly below the new R710. I’m tempted to leave the faceplate off so that the executives and visitors see more lights when they look in. For some reason, non-technical people tend to have a first reaction to hardware such that more lights is mo’ better.

Along with these two pieces of hardware, we went ahead and got a Dell PowerConnect 5424 iSCSI optimized switch. This will be dedicated to the storage segment for the time being. Configuration of the switch meant actually going back to the rack. Initial configuration must be done via serial cable (included). Once you configure the management IP address and administrator credentials, the rest of the configuration can be done via the web based management tool.

Now the R710 was purchased with the nice iDRAC. This web based, out of band, management tool is wonderful. No client to load. You can configure, manage and review just about anything at the hardware level here.

One of the things I really liked about the new iDRAC is the console redirection or virtual KVM functionality. Of course, this allows you to run a java application the is the server console, however, it also allows you to map client resources as virtual media for the server.

In our case, we mapped the VMWare Vsphere ESXi 4 ISO image to the R710 server. We booted to it and installed the operating system without so much as a single walk back to the serer rack. This is much akin to the VMWare ability to mount client CD/DVD drives or ISO images as media for the guest operating systems. The difference here is that this is for the physical hardware. Lights out operations, indeed!

Finally, the MD3000i is configured with the Dell MD3000i Configuration Utility. Downloading this zip file was easy enough, extracted the contents and ran the SAConfig.bat file. The wizard walks you through the process. We had already wired the management ports on the MD3000i to our LAN while leaving the controllers on the storage network. The Wizard easily identified the unit, asked us for information on how we wanted to configure ip addresses and the like and then applied the configuration to the MD3000i. 5 minutes later we used the Modular Disk Storage Manager to begin the configuration process.

Dell will be helping with the final configuration next week but already setting up iSCSI to the ESXi hosts has been fairly painless following the documented processes outlined by Dell and VMWare.

This series of articles present our findings and processes as we performed them. These processes may not be optimal nor preferred in other installations, but they represent the best case for our particular assignment. As such, we certainly welcome feedback.

Part I – Building the Test Environment

Before performing the upgrade we wanted to ensure that our processes would work. Additionally, we have a deference for knowing what roadblocks we might run into ahead of time. When dealing with databases, most times I am going to bet that people or system will be impacted by the DB services being unavailable. Therefore, time is important. Testing before hand can shrink the outage windows and help keep customers happier.

As such, we decided to replicate the key pieces of the infrastructure into a virtual environment. We did this using the VMWare ESXi platform which we are familiar with from our own server infrastructure environment. If you are unfamiliar with ESXi, it is a free version of the VMWare virtual infrastructure product line. You do have to register with the VMWare.com website to download and obtain a license key, but you probably want to be able to review the knowledge base and community forums anyhow. For an SMB like us, it is a great virtualization platform.

Testing Environment ESXi Server Setup

First, we needed the VMWare ESXi server. So we downloaded the latest ESXi bootable ISO from VMWare.com, burned the disc and booted our test server up. We won’t go into all the setup screens, but you can find ESXi installation instructions in a variety of places including this one. Essentially we just took the default installation, rebooted and then configured the management interface with a static IP, mask and gateway. Finally we setup a root password.

Converting or Moving Production to Test

Now whether your production servers are already virtual machines on VMWare or are real, physical servers you are going to need to create an exact copy of them for your test environment. VMWare has a great little, free, tool that allows you to do this. Although they have bigger, better (not free) tools to do this, our needs are such that the free tools work. The VMWare Converter tool is the key. Download it from VMWare.com. Then, we simply install the converter application on the servers that we need to replicate to test. [Note: Just found out that version 4.0 of VMWare converter was released on 2/12/2009. We were using version 3.0]

The conversion process takes the current server and converts it to a compatible VM (virtual machine) on the test server. This include all O/S, network, disk, CPU and memory settings. One note, you’ll want to ensure you have enough disk space on the test ESXi server to accommodate all the disks you are converting for all the VM’s.

[Note: We learned the hard way, having to rebuild the array on the test ESXi server from RAID 5 to RAID 0 in order to get the last bit of disk space we needed. Lesson:  sum all your disk space needs for all the servers you will be converting, ahead of time.]

The process is quite simple but may take numerous hours depending on the data needing to be migrated. So let’s walk through one example of converting one of our production machines in the following video.

1) This is the VMWare converter. Start the converter and choose to start a new conversion.

2) This is the new conversion project wizard splash screen. Click Next.

3) We will be stepping through setting up the source for the conversion. Click Next.

4) Source type will be physical machine regardless if source is physical or another VMWare VM.

5) Choose "local machine" and the converter will convert the current machine using the current users credentials.

6) Choose the disks to be converted. Be sure you include the O/S drive and any data drives you might need.

7) We will define the destination VMWare ESXi server.

8) Our destination type will be VMWare Infrastructure Virtual Machine since we want to convert his server to a VM that will run on VMWare ESXi.

9) This is the destination VMWare ESXi server name or IP and the VMWare login credentials.

10) Type the name you wish to give the new VM when it is converted. This is not the O/S name but the VM name that shows up in the VM inventory.

11) The datastore where you want the converted VM files to be stored on the VMWare ESXi server.

12) The number of NICs that will be created in the destination VM.

13) Place checkmarks on these two options to ignore hibernation and page files and install the VMare tools on the destination VM.

14) Do you want to VM to automatically power on after the conversion is complete? Click finish to start the conversion.

When the conversion completes, pat yourself on the back as you just converted your first server. Now just repeat this process for each of the servers you need to complete your test environment.

Just a quick observation and statement of the obvious. The length of time that a conversion takes is primarily dependent upon two factors, network and disk space. On the network side, I would highly recommend using Gigabit networks for converting larger servers. We found a many fold reduction in conversion time simply by moving from 100 megabit to Gigabit networks. Additionally, the amount of used disk space on the source server will also play a role in the conversion time. The more disk space in use on the source server that is needing to be converted, the more time the conversion will take.

Keep in mind I said “used” space. Unusued space on the disks being converted seems to play very little into the conversion times. As such, clean out your temp files and such before doing a conversion. It might save you some time when converting.

Configuring the Isolated Test Network

Doing this allows us to connect to the management network on the ESXi. Once installed, point your browser to the IP address of the ESXi server via https and you can download the VMWare Virtual Infrastructure client. (https://192.168.1.134 or whatever IP you gave your ESXi server). This client application currently only runs on Windows, sorry Linux folks. This is what you will use to manage your virtual clients.

In our case,we wanted the virtual servers to be able to connect to themselves but not the “real” LAN so as not to cause duplicate IP and Active Directory issues. To do this, we created an isolated virtual network. From the client application, we created a second virtual switch. When we created the virtual switch, we did not link a physical network card to the virtual switch definition. This essential makes a virtual switch that can talk only to other devices on the same virtual switch.This is a plus for testing of an isolated network environment where you do not want the test servers to talk to your LAN.  On the downside, your servers on this new virtual switch can’t get to the Internet either. (at least not without some other fancy networking tricks).

Here’s some screenshots of the process.

1) Create new virtual switch.

2) Select the Create a Virtual Switch option and click Next.

3) Give your new network a name and click Next.

Again, congratulations. You just finished creating you own host only virtual network switch for isolated server testing. Now you need to move your VM’s to use this new network. You’ll need to edit each VM’s setting and change the network that the NIC connects to be your newly create host only network.

1) Edit the VM settings.

2) Change the network to your host only network you created above. Now Ok to save.

That’s the majority of it. Again, we used this setup to convert 5 of our production servers including SQL 2000, SQL 2005, Terminal Services, File Services and Domain Controller all to this isolated test environment. Then we made a snapshot of each converted server which allowed us to go back to the fresh conversion state as many times as we wanted.

From here, we embarked upon our SQL Server 2008 upgrade testing.

Blessings.

The Problem

So here is the scenario. I am removing an old VMWare snapshot through the VI Client on a large file and print server. Of course, it is a Virtual server running Windows Server 2003. The Snapshot removal was being done in preparation of taking a new snapshot prior to extending on of the virtual drives on this virtual server. The Snapshot delete gets to 95% and hangs. After 45 minutes I restart the management agents on the ESXi server per a number of similar posts.

Upon reconnecting to the server via the VI client, I see suddenly that this server is now listed in the inventory as “Unknown (invalid)”. What? This can’t be happening! Can it?

The long and short of it is that I ended up created a new Virtual Machine and used the existing virtual disks (.vmdk files.) Started the new VM up and there it was. My server was back, or was it. A quick look at the event viewer told me the horrible news, there were events from the last few minutes and then nothing until about three months ago when we originally migrated the physical server to a virtual.

Oh my, the snapshots! I had the new VM pointed at the original parent disks not the latest snapshot files. I shut the server back down and say a quick prayer. I manually download and modify the .vmx definition to point to the various server-000003.vmdk files. (These were the last in the snapshot numbering scheme) Uploaded the VMX back and started the VM up.

No luck, I get an error message stating that there was a problem with the disks and the snapshot, “parent virtual disk has been modified”. Oh crud! I killed the snapshot linkages by starting the VM with the parent disks the first go round instead of the snapshots.

Solution

Thanks to my good friend Google, I quickly found my way to the Drive:Activated blog and this posting about the disk file linkage issue.

This boils down to ensuring that the you properly chain the parentCID in the largest numbered .vmdk disk file to the CID of the next largest numbered .vmdk file. Continue this until you get to the root .vmdk. (It is the one that doesn’t have the -00000x numbering after the disk name.

Since we are running on ESXi I needed a way to read and write the .vmdk files. There are a couple of ways to do it.

1. Use the VI Client Storage browser, download the .vmdk files, edit them and then upload them.
2. Use shell utilities to do the same directly on the ESXi server.

ESXi is missing some things from its big brother, namely a console. Thanks to numerous posts including this one, I entered the unsupported ESXi console.

From here, I navigated to the storage area and followed the concepts in the Drive:Activated article. Specifically I first used the grep command to quickly ascertain the various CID values for each vmdk file in each virtual drive series.


> grep CID= SERVER.vmdk
> grep CID= SERVER-000001.vmdk
> grep CID= SERVER-000002.vmdk
> grep CID= SERVER-000003.vmdk


You get the idea. I had a number of virtual drives in this server so I made myself a table to keep track of the CID and ParentCID.

As you can see above, the SERVER and SERVER2 disks had two parentCIDs that I needed to fix (RED). Best I can tell, the snapshot problem hung on the SERVER_1 disks as the 000001, 000002, 000003 disks all had the same CID. Kinda hard to link them together that way! By the grace of God, this was a small archive drive and I could restore it quickly from backups.

Finally I restarted the VM and after an anxious 10 minutes for ESXi to boot the server, the lovely Windows logon screen greeted me. I logged in and what do you know, we were good to go.

The good news is, the server is back up and running. Everything was either good to go or was restored from backups (4GB of files out of over 500GB).

Kudos to VMWare for this as
it is a great way to help folks like my organization to get started
with virtualization at low cost (free) for the Software and as we grow
we can grow in to the additional features of VM Infrastructure.

Straight from the site, some of the new features include:

• Web-based management interface:
  A new Web-based user interface provides a simple, flexible, intuitive
  and productive way for you to manage your virtual machines.
• Expanded operating system support:
  VMware Server now supports Windows Vista Business Edition and Ultimate
  Edition (guest only), Windows Server 2008 (Longhorn Server Beta 3), Red
  Hat Enterprise Linux 5 and Ubuntu 7.1, among others.
• Greater scalability:
  Take full advantage of high-end hardware with support for up to 8GB of
  RAM per virtual machine, up to two virtual SMP (vSMP) processors and up
  to 64 virtual machines per host.
• 64-bit guest operating system support:
  Run high-performance operating systems in virtual machines with support
  for Intel EM64T VT-enabled processors and AMD64 processors with
  segmentation support.
• Support for VIX API 1.2: This feature provides a programming interface for automating virtual machine and guest operations.
• Support for Virtual Machine Interface (VMI):
  This feature enables transparent paravirtualization, in which a single
  binary version of the operating system can run either on native
  hardware or in paravirtualized mode.
• Support for USB 2.0 devices: Transfer data at faster data rates from USB 2.0 devices.

Exciting stuff….

Well, working with a non-profit, we are always looking for ways to effectively use
the resources were are given. Those resources come as assets,
partnerships, financial investments and personnel. Since we want to
make the best use of the resources, we get the opportunity to look at
things from a different persective. Now that does not always mean
finding free or "cheap", despite what anyone thinks. When it comes to
technology, nothing, I repeat, nothing is free!

So, one of the
projects we are working on involves making a decision for the
organization on a reporting and eventually, an ETL platform.(More on
the details of that project in another post) As such, we really are
going to need to have another server for development/testing and for
production use. Interestingly, one of the existing servers, a Dell
Poweredge 6600, has room for another processor and lots more memory.
Hmmmm….that got us thinking. What about virtualization.

Now, I
have used virtualization technology in test environments a bit over the
last few years but have not kept up with developments in the
marketplace. Well, I was certainly pleased with what I found.

I
spent the last 9 years in a Microsoft shop and that was after a
VAX/VMS, HP-UX, AIX shop. So, I am not the biggest fan of the Microsoft
Server platform but also not a big opponent either. I think the
different O/Ses have their places. However, I can’t help but go back to
a Unix or Linux base as a workhorse. I was pleased as punch to learn
that the VMWare Server is now offered as a free virtualization platform.

So, I’ll spare you the gory details on not using Microsoft Virtual Server 2005
and move on to where we are at. Being a small IT shop, we don’t have a
data center chock full of servers that would need to be centrally
managed so the free VMWare Server product is fine. If you need this
kind of data center management for virtual environments, take a look at
the not-so-free Virtualization management platform from VMWare.

As part of my testing, er… playing, I loaded VMware Server on my Dell D520 Laptop
with 1GB RAM. My base O/S on the laptop is usually Windows XP Pro SP 2.
Install was a snap of the server. I quickly created a new virtual
machine for a test Windows XP Pro SP2 machine since I am constantly
messing my base O/S up by loading and testing lots of software. After
the first reboot of the VM, I logged into the machine and installed the
VMWare Tools. This is an important steps as it significantly improves
display and mouse performance as well as other under the hood
improvements. (Don’t forget this step.)

I typically use a bridged
network setup as it allows the VM direct network access via the host’s
NIC. Quick, easy and maintainable. Really, everything works the same in
the VM. The only quickly noticed difference is that instead of the
three finger salute to login (Ctl-Alt-Del) you use Ctl-Alt-Ins. Now, I
wanted to have a way to always get back to this clean install of the VM
operating system for new tests. That is where the snapshot feature
shines.

First thing to remember is that before you take the
snapshot of the guest O/S, you MUST defragment the guest O/S virtual
drive. You can do this either with the defrag tools in the guest O/S or
from the VMWare Server console. You do this by shutting down the guest
O/S. Then edit the VM setting and choose the Defragment option under
the hard disk options.

Once done with the defragmentation, take your
snapshot. This snapshot can be used to move the VM back to the exact
state it was in at the time of the snapshot. According to the
documentation, if you take the snapshot without defragmenting the
virtual disk first, you lose the ability to effectively defragment the
virtual drive. Keep in mind that the virtual disk is actually just a
file or files on the host operating system. Therefore, you will also
want to ensure the host operating system keeps it’s physical drives
free from excessive fragmentation.

Now, I certainly could not stop at Windows XP Pro. Hehe…. So, next on the list was a virtual Ubuntu Server 6.06 LTS and an installation of Ubuntu Ultimate.
Again, no problems. Worked like a champ. By the way, I loaded the
Ubuntu Ultimate by telling the virtual machine definition that the
virtual CD-Rom drive should actually point to and .ISO image. Again,
worked a treat.

Fair enough, but since Windows operating system can
be a bit tempermental and subject to security issues, what about using
a Linux operating system as the host to then run the Windows servers as
virtual machines. So, that’s what we tried. We loaded Ubuntu Server
6.06 LTS onto an older Dell desktop. So, we had a nice Linux server
running which we quickly loaded the VMWare Server onto. I chose not to
install web base MUI and instead to use the existing VMWare Server
Console on my laptop to remotely connect and manage the VMWare Server
on the Ubuntu server.

Sure enough, I slapped a Windows 2003
Server R2 installation CD into the drive on the Ubuntu server and went
back to my office. I fired up the VMWare Server console and create the
VM definition. Then I booted the VM and the CD-ROM on the Ubuntu server
came to life runing the Windows 2003 Server setup. Less than an hour
later, I have a fully functioning Windows server running as a VM on a
machine with Ubuntu server as the host operating system. Sweeeeet!

Now
I am testing the Linux client for Reptrospect which is what we use for
backups. I need to get comfortable with disaster recovery functionality
for the host and most importantly, the VM’s. One of the nice things
about the VM’s is that if I have all the files for each VM, I can move
then to another server, even with a different host O/S and start them
up. They’ll keep chuggin along. Nice!