1. Upgrade and purchase licenses for VSphere, VCenter, etc…
2. Use the VI client data store browse “Move” functionality
3. Try out a third party product like Veeam FastSCP
4. Try standard TAR/GZ combined with cp command
5. Try vmkfstools disk cloning

One important note. While testing some of these options, we became aware of a complicating factor. If you have created your disks as Thin provisioned and are moving to storage that has a different block size, the vmdk will become thick provisioned. This means that the “entire” contents of the fully thick provisioned disk will be moved, not just the used space.

Upgrade and Purchase VSphere Infrastructure Licenses

This option is one we continue to re-evaluate every 6-8 months. Working in a non-profit, we are always evaluating how to be the best stewards of the resources we’ve been provided. One of those resources is obviously financial resources. This seems like the only real option for moving running VMs.

While having VSphere Enterprise in place would make this process of migrating to new storage easier, we just cannot justify the long term expense. Our small environment just does not need what you pay for over the long haul.

As such, we ruled this out. Some downtime is perfectly acceptable in this organization.

Use VI Client Datastore Browse/Move Functionality

When using the VI Client, you can browse the VMFS datastores similar to a Windows Explorer window. This is accessed by viewing the storage configuration of the host in the VI Client. Right-clicking the particular datastore and choosing browse.

Obviously, this function does not work on running VMs. As a matter of fact, in one of our tests, we actually caused a problem doing so. This is a “move” not a copy. If the move fails mid-way through, some files will have been moved while others are still in their original location. Now you’ve got a bigger mess on your hands.

So, this option means the VM has to be powered off. Ok, we can deal with that. The other downside was that, as explained above, the new vmdk is thick provisioned and therefore, much more data is transferring that is really necessary.

Test 3rd Party Product Like Veeam FastSCP

I really do like the folks over at Veeam. I have used the monitoring /reporting product and like there licensing models. We thought we’d check out their free FastSCP product. Why? Well because is says “Fast” in the name.

The tool worked great. It did everything is said it would do. Like the other options, the VMs need to be powered down. We even tested copying between different ESXi hosts. Worked great. The only downside was the lack of speed. Yeah really! We did some reading and it appears that this may be a throttling of the ESXi product, not Veeam FastSCP. Bummer.

Try standard TAR/GZ combined with cp command

“Wow”, I though as I was researching. Why don’t we just tar/gzip everything and then copy to the new storage location and then untar. Well, not so fast our ESXi said to m as it spewed out an error that “tar: cannot store file ‘xxxxxxxxxxx.vmdk’ of size #########, aborting”. Yuck~!

It appears that BusyBox, the underlying unsupported ESXi console has a limitation of 4GB in ESXi3.5 and 8GB in ESXi 4. Therefore, trying to tar a 100GB vmdk file went no where fast. Check this option off the list.

Try vmkfstools disk cloning

This, we thought, is an interesting idea. Yes, the VMs still need to be powered off but perhaps this will shorten the downtime by lowering the amount of data that gets copied. See, as we talk about earlier, going to storage of a different block size will mean yout thin provisioned disks will begin thick. As such, the total size of the thick disk has to be transferred. This could siginificantly increase the copy time if you have lots of unused space on a disk.

The vmkfstools utility has a feature known as cloning. This allows you to make a clone copy of a vmdk disk to anyVMFS datastore. Now that’s all good but where this got us jazzed was the ability to force that newly cloned disk to be created as a thin provisioned disk. Eureka!

Now, first, make sure you remove all snapshots in the vm. Those are just going to cause us a headache.

So a command like this does the deed:
vmkfstools -i /vmfs/volumes/source/vmname/vmname.vmdk -d thin /vmfs/volumes/dest/vmname/vmname.vmdk


Note: you need to do this against the vmname.vmdk NOT the vmname-flat.vmdk. Be forwarned.

Some small scale, non-scientific testing gave us around 1.3GB per minute in transfer speed. Again, this is not true i/o but rather therate used for calculating down time for our migration project by taking the amount of used space on the source drive/time to create cloned drive = transfer rate.

Now, when the vmdks are moved, don’t forget to cp all the other VM files in the VM directories!

This looks to be the path we will be taking and thought we’d share how we arrived here.

So you can see the MD3000i with the faceplate off directly below the new R710. I’m tempted to leave the faceplate off so that the executives and visitors see more lights when they look in. For some reason, non-technical people tend to have a first reaction to hardware such that more lights is mo’ better.

Along with these two pieces of hardware, we went ahead and got a Dell PowerConnect 5424 iSCSI optimized switch. This will be dedicated to the storage segment for the time being. Configuration of the switch meant actually going back to the rack. Initial configuration must be done via serial cable (included). Once you configure the management IP address and administrator credentials, the rest of the configuration can be done via the web based management tool.

Now the R710 was purchased with the nice iDRAC. This web based, out of band, management tool is wonderful. No client to load. You can configure, manage and review just about anything at the hardware level here.

One of the things I really liked about the new iDRAC is the console redirection or virtual KVM functionality. Of course, this allows you to run a java application the is the server console, however, it also allows you to map client resources as virtual media for the server.

In our case, we mapped the VMWare Vsphere ESXi 4 ISO image to the R710 server. We booted to it and installed the operating system without so much as a single walk back to the serer rack. This is much akin to the VMWare ability to mount client CD/DVD drives or ISO images as media for the guest operating systems. The difference here is that this is for the physical hardware. Lights out operations, indeed!

Finally, the MD3000i is configured with the Dell MD3000i Configuration Utility. Downloading this zip file was easy enough, extracted the contents and ran the SAConfig.bat file. The wizard walks you through the process. We had already wired the management ports on the MD3000i to our LAN while leaving the controllers on the storage network. The Wizard easily identified the unit, asked us for information on how we wanted to configure ip addresses and the like and then applied the configuration to the MD3000i. 5 minutes later we used the Modular Disk Storage Manager to begin the configuration process.

Dell will be helping with the final configuration next week but already setting up iSCSI to the ESXi hosts has been fairly painless following the documented processes outlined by Dell and VMWare.

Now, don’t forget that it your SSH into your LISH console, you can see everything going on even with your Linode powered off. Click the Image Options link for the image you wish to resize.

Now, simply type the new disk image size. If you enter a larger number, the image will be expanded. If smaller, the disk image will be shrunk. Additionally, there is no need to worry if you try to shrink or expand beyond what is possible. The resize process will simply fail in such cases and return your disk image to its previous size.

This ability to resize disk images provides a very useful utility to squeeze the full potential of a linode as it evolves over time.

Call me a Linode fan boy!

Now, one of the biggest challenges is that there are multiple instances of the “image_attach” sections on a single content edit form, one for each category. The Drupal 5 version handled this by naming these by appending the category name to the fields. This allows the submitted form value for “iids” to be an array of categories. Each category itself being an array of image iid values.

For the life of me, my Drupal 6 version of this functionality failed to maintain the category arrays. After some frustrated hair pulling (hmmm…. no wonder my hair is so short), I remembered to take a look at the FAPI docs.

http://api.drupal.org/api/drupal/developer–topics–forms_api.html/6

An important thing to note: notice that $form['access'] has a '#tree' => TRUE attribute. this setting retains the full tree structure for all elements under it when it is passed to $form_state['values']. you must explicitly declare this anywhere you wish to retain an array’s full hierarchy when it is passed.

So, Drupal 6 collapses form elements where possible by default. By adding the #tree attribute, the hierarchy of form elements is maintained in the form_state submitted. Eureka!

Here’s an example of what I was getting without the #tree attribute:

And now with the #tree attribute set to true, the form hierarchy is retain and not collapsed:

I’m sure most of you doing Drupal development are saying “Duh!”. Well, let’s just say I really didn’t know. Thought I’d throw it on the blog so I don’t forget.

Blessings.

StackScripts™ provide a flexible way to customize our distribution templates. They’re very easy to use — find a StackScript, answer its questions, and click deploy. When the deployment is first booted, the script is executed and does its thing. You can even watch its progress by viewing the console.

As so many things with Linode.com and its community there is a public library of the Stackscripts available for a variety of purposes. Anything a from a standard LAMP setup to WordPress, Drupal and TRAC/SVN setups.

Already having a Linode up and running, I have also been thinking about re-doing the setup of ourLAMP server. Seeing this as an opportunity to both rebuild our LAMP server and test StackScripts, I set out to give it a try.

We already had a Linode configuration and a disk image actively running. The good news was that when you choose to deploy a new distribution via a stackscript, it will create both a new Linode configuration as well as a new disk image. Doing so keeps you from having to remove the currently running setup.

One of the standard Linode LAMP StackScripts provide a form like the image to the left when executed to deploy a new distribution. MySQL user and password, creation of a default database, as well as root O/S password and such. dec

Once the configuration is created, you now only need to boot into the new configuration. If you are logged into your LISH console, you will see the distribution go through the automated process of the StackScript. In our test case, an Ubuntu 8.04 LTS kernel was booted and installed, system updates installed and applied, postfix installed for local loopback only, MySQL installed and tuned, php installed and tuned and apache2 installed and tuned.

Pretty cool so far. Now, there is plenty left to do to get a fairly decent LAMP going that in our case would run Drupal.

Create a new non-root admin  user account called myadminuser and add to the admin group:

adduser myadminuser
usermod -G admin myadminuser

NOTE: Looks like the admin group is already part of sudoers so your new account has sudo privileges.

Update the timezone:

sudo dpkg-reconfigure tzdata

Secure MySQL services in a quick fashion:

sudo mysql_secure_installation

Enable a few Apache2 modules:

sudo a2enmod deflate
sudo a2enmod expires
sudo a2enmod cache
sudo /etc/init.d/apache2 force-reload

Add a few PHP tools:

sudo apt-get install php5-gd
sudo apt-get install php5-curl
sudo apt-get install php5-cli

Secure SSH daemon by changing the port for SSH and disallow root login:

sudo vim /etc/ssh/sshd_config
-- # Protocol 1
-- Protocol 2
-- ListenAddress 10.54.55.104 #your IP
-- Port 5502 # New port different than 22 and < 65535
-- PermitRootLogin no
-- X11Forwarding no

sudo /etc/init.d/ssh restart

Install the firewall and configure:

sudo apt-get install ufw
sudo ufw default deny
sudo ufw allow http/tcp
sudo ufw allow 5502/tcp
sudo ufw enable
sudo ufw status verbose

Configured PostFix by following pieces of our previous post on PostFix and domain forwarding.

sudo vim main.cf
-- See article referenced
sudo vim virtual
-- See article referenced
sudo postmap virtual
sudo ufw allow 25/tcp
sudo postconf -e "inet_interfaces = all"
sudo /etc/init.d/postfix restart

Lastly upon reboot we noticed that mysqld was failing. The problem was that mysqld was not binding correctly. We editing the my.cnf file and added BindAddress 127.0.0.1 above our existing BindAddress 10.54.55.104. This allows mysqld to bind and listen to both local loopback and the external IP (if needed)

At this point, we had not installed APC or MemCached yet. Performance was quite good with this setup and the minimal memory available to this test Linode. We ran six(6) Drupal , one (1) WordPress and one (1) CMS Made simple site with sub-second response times under fairly good load.

Kudos to Linode.com on their StackScript concept. I can certainly see this being very appealing for providers needing to crank out Linodes with a standard configuration for customers.

Blessings.

mysqlcheck -u root -p --auto-repair --check --optimize --all-databases

mysqlcheck is available in MySQL 3.23.38 and later.

The official documentation for mysqlcheck says: “The mysqlcheck client performs table maintenance: It checks, repairs, optimizes, or analyzes tables.” Now, this must be run on a MySQL instance that is currently running, unlike the myisamchk tool. So, you may want to stop all access to your instance. If you are using it for websites, perhaps kill HTTPD for a few minutes some other means.

What we have done is to install APC on the MediaTemple DV in order to improve performance to some degree by taking advantage of op_code caching.  Now, this is not a silver bullet for performance folks. This is just one piece of a total tuning exercise. Be forewarned.

[Note: You are on your own as with anything we present. We do not warranty nor represent explicitly nor implied that this will actually work for you. In fact, you might just hose things up badly. Don't call us, we warned you.  ]

1. Install Developer Tools and Enable Root Access

Log into your MediaTemple Account Center and go to the “Root Access & Developer Tools” page. If Root access is not enabled, you’ll need to enable it. Be sure to set a massively secure password on this!

Also, if the developer tools have not yet been installed, then click the button to install them. Wait until the installation has finished before proceeding.

2. Install APC

Use your favorite SSH client to log into your server via SSH. Once you have logged in, you will perform the following commands:

• cd /usr/local/src
• wget http://pecl.php.net/get/APC-3.0.19.tgz
• gunzip -c APC-3.0.19.tgz | tar xf -
• cd APC-3.0.19
• /usr/bin/phpize
• ./configure –enable-apc –enable-apc-mmap –with-apxs2=/usr/sbin/apxs –with-php-config=/usr/bin/php-config
• make
• make install

3. Let PHP know about APC

Now perform the following actions to modify your APC installation. (Thanks to TimLinden for the details)

• vi +/extension_dir /etc/php.ini
• press i
• press enter to start a new line
• type extension = “apc.so”
• press esc and then type :wq to exit and save (if you mess up type :q! to quit without saving)

4. Drupal APC customization

Thanks to an article from 2bits, we made the following modifications to the php.ini to set some specific APC configuration options.

• vi +/extension_dir /etc/php.ini
• press down arrow
• press down arrow
• press i
• press enter to start a new line
• type apc.apc.stat = 0
• press enter
• type apc.include_once_override = 1
• press enter
• type apc.shm_size = 64
• press enter
• press esc and then type :wq to exit and save (if you mess up type :q! to quit without saving)
• service httpd stop
• service httpd start

For those that don’t need key press by key press instructions for the php.ini, here is what we added to the php.ini

extension = “apc.so”
apc.apc.stat = 0
apc.include_once_override = 1
apc.shm_size = 64

Hopefully this will help you get things started. Obviously, we are still testing and tweaking but this has provided some good improvements for a few sites that are located on a MediaTemple DV 3.5 server.

Admin menu completely hosed.

Note that we have no “Content” menu in administration. Also, notice that in the screenshot, we are in the Configuration menu but only have a single configuration item. That certainly doesn’t look right.

So after a bit of Googling  we first tried truncating the cache_menu table without success. Finally we decided to try a simple thought. We disabled the Admin menu. Then we re-enabled the Admin menu and Voila, our admin menus have been fixed. Go Figure.

Admin menu all fixed.

Just goes to show that sometimes the basic troubleshooting steps can fix something that looks worse than it really is.

Let’s jump right in. The scenario is that we have a content type of article. The node-article.tpl.php displays the Drupal node author as part of the by line of the article. In our case, we wanted to change this.

Instead, we will have articles created by staff, as well as articles automatically generated from RSS feeds. It is important for us to honor our partners by not showing the article as being authored by us, but instead to change the authored by line to give credit to the originating feed source where appropriate.

Enough talk, let’s see details.

First, we need to make the parent feed’s title and link available as a variable for the templates. We did this in Drupal 6 by adding to the existing template.php file. Using the phptemplate_preprocess_node hook, we can add variables that can be accessed in node templates. (Note: If you are following along in an OpenPublish distribution, this function does not exist in the standard template.php file. You will need to add it.)

Keep in mind that when you use feedapi/feedapi node module to turn a feed item into a node, the $node object does not have all the details about the parent feed that generated the feed-item. It does however contain the nid (node id) of the feed that generated it. Soooo… in our custom code, we need to get this parent feed nid, load the feed title and url and then stuff the link into a variable to use in our node template.

Here is the function as added to the template.php:

/**
* Override or insert PHPTemplate variables into the node templates.
*/
function phptemplate_preprocess_node(&$vars) {

  // 20091029 Added for Parent Feed Title Link
 if ($vars['node']->feedapi_node->feed_nids) {
    $parent_feed_node_id = array_values($vars['node']->feedapi_node->feed_nids);
    $parent_feed_node = node_load($parent_feed_node_id[0]);
    $vars['parent_feed_link'] = l($parent_feed_node->title, $parent_feed_node->feed->url);
  }

}

Just a note that the array index of feed_nids is not zero based but instead, the array index is the same number as the nid. That makes doing the typical $vars['node']->feedapi_node->feed_nids[0] impossible. That is where the php array_values() function comes to the rescue. It takes all the array values from feed_nids and created a zero base indexed array.

Now, we have a variable called $parent_feed_link that is available in our template files. So, our next step is to use it. Since the content type we want to impact is the article type, we are going to modify our node-article.tpl.php. Below is the portion of the file that outputs the article by line:

<div>Article |
<?php print date('m.d.y', $node->created); ?>
 | By  
<?php  
 $author = user_load(array('uid' => $node->uid));
 if ($parent_feed_link) {
   print $parent_feed_link;
 }
 else {
   print l($author->profile_full_name, 'user/'.$node->uid);  
 }
?>    
</div>

As you can see above, we now test to see if the $parent_feed_link actually has a value. If so, we want to output that link. Otherwise, the article was not created from a feed-item and we want to show the actual author of the node.

I know this is simple for many of you, however it took me a little while to figure out where in Drupal 6 to generate these custom variables. In Drupal 5, you may have typically done this in the _phptemplate_variables hook in your template.php.

Well, did we do it wrong? Do you know a better way? Let us know with a comment below.

Blessings.