DR Recovery of a Hyper-V guest using Virtual Server 2005

Here's the scenario: You've convinced your managers to try out virtualization, and you've decided to buy a single server along with some shared storage, leveraging your current server as a second host for your environment. You've made the choice to try Microsoft Hyper-V (after all, it's included with Windows 2008 R2), and you get everything going on the new host. But when it comes to putting Hyper-V on the current server, it turns out that the CPUs, chipset and BIOS are incompatible. After some research, you learn that it will support Virtual Server 2005 (VS2005).

Microsoft is using the same virtual hard drive format (VHD) that they inherited from Connectix, so on the surface, one would hope that it would be possible to use a guest's VHD in either environment: simply create a new VM with the VHD as the primary disk and boot. You won't be able to use any of the really powerful features of virtualization like Live Migration, but at least you'll have an opportunity for short DR RTO/RPO if you could readily move the guest between hosts—even if it's a manual process.

Unfortunately, the two Microsoft hypervisors have very different VM (ie, virtual hardware) specifications (and limitations), and essentially share only the VHD format in common: attempting to boot a Hyper-V guest in VS2005 will fail miserably, and may do so without any indication of why it's failing: you get stuck at a black screen, and that's it. No BSOD, nothing.

Fortunately, there exists many references to porting Hyper-V guests into other environments that are just a Google search away. All those resources tell you that it's possible to run the guest in Virtual PC (the desktop hypervisor, not the server hypervisor) if you remove the integration tools. Some of them also add the step of performing a hal.dll swap. Of course, one thing is absolute: the version of the OS that you're trying to migrate must be a 32-bit OS: while Virtual Server 2005 itself may run fine on a 64-bit OS, it cannot support any 64-bit guest.

Fine. Caveat emptor. We didn't virtualize a 64-bit guest with Hyper-V, so this just might work...

Here's the first problem with the porting technique: if you could get the guest running on a recovery host in order to remove the integration tools, why would you fool around with doing anything else? It's running. Stop fooling with it and move on to get your production hypervisor working again!

Luckily, the real trick isn't related to removing the integration tools, it's getting the right HAL.DLL on the guest. So here's how that is done.

1. Prepare your Hyper-V guest
1. On your VS2005 host, create a VM running the same OS as the Hyper-V guest that you wish to recover. This is only temporary, as you're after one special file from the guest.
2. Get a copy of its hal.dll (%systemdrive%\WINDOWS\SYSTEM32\hal.dll)
3. Put a renamed copy of the VS2005 HAL (eg, hal.dll.vs05)  on the production Hyper-V system drive, in the same folder as the original HAL.
2. Test your disaster
1. Copy the Hyper-V guest VHD to your recovery host. If you do it while your guest is running, it will reliably reproduce the "improper shutdown" that will occur if your Hyper-V host dies unexpectedly.
2. Mount the VHD as a local disk on your VS2005 host machine; if you're running the latest version, this capability is part of the package.
3. Rename the original hal.dll to something else (eg, hal.dll.hyperv) and rename your VS2005 HAL to hal.dll. By renaming—instead of replacing—you should be able to reverse the process when you have a production Hyper-V host available again.
4. Dismount the VHD
5. Create a new VS2005 guest, using the copied, modified VHD as the primary disk. Do not connect the network to the guest, or you'll have all sorts of problems, the least of which being errors about duplicate names on the network.
6. Boot the guest. If you get all the way to the logon prompt, you've succeeded.
3. Test recovery
1. Shut down your test VS2005 VM.
2. Mount the VHD as a local disk.
3. Reverse the hal.dll renaming operation completed in 2c.
4. Dismount the VHD
5. Move the VHD back to your Hyper-V host
6. Create a new Hyper-V guest using the VHD from the VS2005 system. Again do not connect the network to the guest, or problems will ensue.
7. Boot the guest. Again, if you get a logon prompt, you're golden.

What to do if this doesn't work: go back to management and get a second, new server. Your company's livelihood shouldn't rely on a "bailing wire & duck tape" solution like this. You've already got the shared storage; that's the most expensive part of a highly-available virtualization environment.

What to do if this does work: see above. Don't rely on this solution. This is, at best, a complete kludge, a pig in a prom dress. I'm not even sure that you'd get support from Microsoft if you had an issue, even if it was totally unrelated to the environment.

And in my (biased) opinion, use VMware instead of Hyper-V. While both products have Type-II hypervisor options if that's a requirement (you need a full Windows OS on the "bare metal" host), the VMware guest is a much more mature, portable virtual hardware platform (it really can be as easy as copying a file when you're ready to move to the latest versions of VMware); Microsoft (and all the competition, for that matter) are still working to reach the sophistication & reliability of what VMware introduced years ago.

iomega ix2-200 NAS non-cloud to cloud upgrade

UPDATE: DO NOT attempt to adapt this process to update an ix4 to the cloud-OS version!
If you do so, the upgrade will apparently attempt to update the BIOS of the ix4, and there appears to be a sufficient difference between the two products (cloud, non-cloud) that you will end up "bricking" your ix4. This happened to me (yes, I know I was taking a big chance) and someone else I know. Having a complete set of pristine, factory-reset drives will not give you the ability to revert to the non-cloud OS if you have a problem with the upgrade.
For the record, the system I started with had a sticker on the flash chip indicating that it was running the "V1.0.7" code.
The only known fix is a "low-level format" back at iomega support.

---

Standing on the shoulders of others...

So you've got a "non-cloud" version of the iomega ix2-200, and you're miffed that iomega doesn't officially support an upgrade path to convert it to the newer "cloud" editions. Maybe you want some of those cloud features, or maybe you have a Mac with Lion, which apparently isn't supported on the non-cloud models.
The upgrade can be accomplished; however it is destructive. You will not be able to perform an in-place upgrade with this technique, although you will be able to "get back" to having the original equipment—if that's what you desire.

What follows is a recipe that worked for me; your mileage may vary.

Required Hardware:

• iomega ix2-200 (duh!)
• Blank, FAT32-formatted USB thumbdrive with a drive activity light, 2GB or larger.
• pin, toothpick or other small, stiff wire that can be used to press and hold the reset button on the ix2 for ~90 seconds.

Optional Hardware:

• New hard drives. This is a prime opportunity to upgrade a 1TB or 2TB (raw) model to a 4TB or 6TB unit; it also gives you a fallback position in the event you brick the thing while working with the new drives. If you go this route, however, do not try to improve performance by switching to 7200RPM drives: The heat output of those drives is too high for the tiny fan in the chassis to handle. Stick with "green", variable RPM drives, and you should be okay.
• USB-to-SATA adapter. This eliminates the need to crack the case of a desktop machine (for access to the SATA ports) to clear a drive that will be re-used for the conversion.

Required Software:

• Linux distribution with openssh, tar, md5sum, gunzip and the ability to mount files to a loopback device. Believe it or not, your ix2 itself will fit the bill if you don't otherwise have a linux distro handy; all you need is SSH access, and you're golden. This recipe assumes the use of the ix2.
• Latest/greatest upgrade of the ix2-200 "cloud edition" firmware from iomega. As of this writing, the most-current version was 3.1.14.995. The firmware comes in an encrypted tarball, but your linux distro will make short work of that little problem.

Preparation:

1. Prepare the ix2 for conversion: Copy all your data to another storage medium. If you brick your ix2, you don't want to be left hanging in the cold.
2. Create the conversion tarball.
1. Create a new share on the ix2. Copy the firmware image from iomega to the share.
2. Save the following code as a shell script in the same place:

   ifw=./ix2-200-3.1.14.995.tgz
   ofw=${ifw%.tgz}-decrypted.tar.gz
   ix2=${ifw%.tgz}-files
   
   mkdir -p $ix2/images
   mkdir -p $ix2/update
   mkdir -p $ix2/apps
   
   openssl enc -d -aes-128-cbc -in $ifw -k "EMCNTGSOHO" -out $ofw
   tar xzvf $ofw -C $ix2/update/
   
   imgs=$(find $ix2/update/)
   for img in ${imgs} ; do
     if [ -f $img.md5 ] ; then
       mv $img $ix2/images/
       mv $img.md5 $ix2/images/
     fi
   done
   
   mount -o loop,ro $ix2/images/apps $ix2/apps
   cp -p $ix2/apps/usr/local/cfg/config.gz $ix2/images/
   umount $ix2/apps
   gunzip $ix2/images/config.gz
   
   img=$ix2/images/config
   md5=$(md5sum $img)
   md5=${md5% *}
   md5=${md5% }
   echo "$md5" > $img.md5
   
   cd $ix2/images/
   tar czvf ../ix2-boot.tgz *
   

   You will need to edit the first line of the script to match the filename of the firmware tarball you placed in the folder with the script.
3. Log on to the ix2 using SSH. (the first part of this post will walk you through gaining SSH access to your ix2)
4. Verify you have an available loopback device to use as a mountpoint: losetup -f
• if losetup indicates that there are no free loop devices, you need to create the next one in sequence.
• Determine the next-needed loop device with: losetup -a
  The output will probably look a bit like this:

  root@ix2:/# losetup -a
  /dev/loop0: [0900]:15970 (/sysroot/boot/images/apps)
  /dev/loop1: [0900]:47905 (sysroot/boot/images/config)
  /dev/loop2: [0900]:15972 (/sysroot/boot/images/oem)
  

• As indicated above, the first 3 loops are in use (loop0 through loop2), so the next one needed is loop3.
• Use the following command to make loop3 (the 4th loop device):
  mknod -m 0600 /dev/loop3 b 7 3
5. Change to the directory of the share you created:

   cd /mnt/soho_storage/samba/shares/sharename

6. Make the script executable:

   chmod 700 scriptname

7. Execute the script:

   ./scriptname

   The script will take a while to run: the original tarball must be unpacked, several files get copied, a new tarball is built. It's time-consuming, especially on low-powered hardware like the ix2. But nothing you're doing is affecting the existing system; you're simply using a handy linux environment to get some work done.
8. The final result will be a new file in the *-files folder called ix2-boot.tgz. Back on your main workstation, copy this new tarball to your USB drive in the following directory tree:

   \emctools\ix2-200d_images\

9. Exit the SSH shell
3. The drives that will be targeted for the conversion must be clean/clear drives; preferably, they will be new drives that have never been used, with no existing partition table. If you're going to re-use the existing ix2 drives, or re-use drives that you've had in another system, you first need to wipe them out: delete any existing partitions. The conversion won't work if you have data on the drives. This is where your mettle as a gadget geek gets tested: do you throw caution to the wind and wipe the ix2 drives, or do you get new drives and "play it safe?"
4. Place the new/wiped drives into the ix2.
5. Plug in the USB drive to any of the three ports.
6. Press and continue to hold the reset button on the back of the unit while plugging in the power supply. The ix2 will automatically boot when power is applied, and if you don't keep the reset button depressed for the entire boot sequence, you will not activate the installation. Watch your USB activity light; although it will come on when the unit activates the drive, you need to keep the reset button depressed until regular activity (i.e., blinking) is observed. In my experience, this takes approximately 90 seconds from the moment you plug in the power supply.
7. Wait. The update will occur, and you'll see plenty of activity for the drive indicator on both the USB drive and the ix2. After less than 10 minutes, however, the ix2 will automatically power off.
8. Remove the USB drive. Reconnect the network. Power up the ix2.

If you successfully updated the image, the indicator lamps on the front of the unit will blink furiously for a few minutes. Once the main power indicator (bottom lamp) is solid, you should be able to reach the management console via browser (Hopefully, you have some way to tell what the IP address is; your router or other DHCP server should have a way to indicate the address—I use a DHCP reservation, so I always get the same address, regardless of configuration state for the unit).

If you don't get that far—the bottom lamp blinks, the drive lamp blinked a couple of times, but otherwise, no activity—don't despair: I've seen other postings that indicate success when performing the procedure twice; I never needed it (I've repeated the update three times so far). Worst-case: you pull the drives and replace your original ix2 drives. Nothing is stored locally on the device, so a clean drive swap is possible.

The second alert lamp will blink until data protection is complete (either because you switched to a "none" option or it finished setting up the mirror); you can expect the drive activity lamp to be lit solid during the data protection phase.

By default, the firmware wants the ix2 to be using mirrored protection. You can alter that through the setup, and with the cloud version, you have the option of using "linear" or "raid0" if you want to get optimal performance at the expense of data protection.

Once you have the device configured the way you prefer, recreate your shares and iscsi volumes and put your data back on the unit: you're now Cloud-Ready!

---

The science and tech fields have historically worked by advancing based on prior works; we don't reinvent the wheel, we take the various works of our predecessors and advance "the art" through mash-ups that fail more often than they succeed.


This is one of those mash-ups, and thanks go to the following...

• Christopher Kusek (@cxi), who's blog post helped expose SSH access for the ix2
• Scpnas, who's posting to nas-central.com documented the process for creating an initialization tarball for a USB drive from the encrypted firmware download from iomega
• Luciano Rocha, who gave me the clues needed to be able to mount the apps file to a loop device on the ix2, a required step in making the initialization tarball.