Upgrading vSphere 5.1.0 to 5.1.0a

VMware released the 'a' update to their vSphere 5.1 binaries (both vCenter & Hypervisor) on 25-Oct-2012. I downloaded the ISOs for both ESXi (VMware-VMvisor-Installer-201210001-838463.x86_64.iso), vCenter (VMware-VIMSetup-all-5.1.0-880471.iso)  as well as the offline update (ESXi510-201210001.zip) because VMware vSphere Update Manager (VUM) doesn't perceive these as patches.

Update: Since first posting this, I've been informed that VUM is able to patch the ESXi hosts, whether you're running 5.1.0 or 5.1.0a versions of vCenter. I infer one of two things from this: I went after the updates too soon (before VMware had published the update for VUM to use), or my VUM install isn't getting the update info correctly. This change only affects the way you (can) go about updating the host; the vCenter server upgrade doesn't change.

Note: The offline update package for 5.1.0a is not for use with VUM; you'll have to either install from the ISO or use command-line methods to perform an upgrade of ESXi. The latter will be covered in this post.

Reminder: If you run vCenter in a VM, not on a physical host, use out-of-band console access that bypasses the vCenter Server! As soon as that vCenter service stops—which it must—your connectivity to the VM goes away. You can use the VIC if you connect directly to the ESXi host that's running the vCenter VM; that's the way I do mine. Windows Remote Console should only be used with the "/admin" switch, and even then, your mileage may vary. Any other remote access technique that mimics the physical console access is fine. Just don't use the VIC or Web Client remote access via the vCenter services that you're trying to upgrade. "Duh" might be your first response to this, but the first time you forget and connect out of habit, you'll hopefully remember this post and smile.

As with all upgrades, vCenter is first on the list, and in the new model introduced with 5.1, that starts with the SSO Service. That was recognized as an upgrade, and proceeded and succeeded without any additional user entry beyond the obligatory EULA acceptance.

Just to be sure, after SSO finished updating, I tried logging in using the "old style" client (VIC) from the shortcut on the vCenter desktop: no problem. Then I tried it with the Web Client: failure. On a hunch, I restarted the Web Client Service, but with no luck: "Provided credentials are not valid."

Oopsie.

One more test: I'd used the "Use Windows session authentication" option in the Web Client. This time, I tried using the same domain credentials, but manually enter them instead of using pass-through: Pass.

That may be a bug; it may be a problem with unmatched versions of SSO and Web Client. Moving on with the rest of the upgrade...

The next step is to upgrade the Inventory Service service; like SSO, it can upgrade without specific user input. However, when the service is replaced with the newer version, vCenter Server (and other services dependent on vCenter Service) is stopped and not restarted. Manually restarting the services will allow you back at your system again, just in case you get interrupted while working and need to get back on before updating the vCenter Server service to the new version...

Like the previous services, vCenter Server recognizes that it's an upgrade and click, click, click to complete. Naturally, the services are stopped in order to replace them, but the installer does restart them when it's done. Upgrading the VIC is another click,click,click operation, as is the Web Client.

It did not, however, fix the pass-through authentication issue in the Web Client.

I spent a while in conversation with Chris Wahl and Doug Baer on Twitter, trying to get it straightened out. Both are VCDX and super sharp, and they gave me lots of solid advice for improving bits of my vCenter setup, but this one wasn't budging. At this point, I've given up on it: there's a solid workaround, so it's not a dealbreaker. Watch this space, however: if/when I figure it out, I'll pass along my findings.

VUM isn't updated in this release, so that bit doesn't need to be upgraded or reinstalled. However, the offline package isn't going to work with it (as mentioned above), so the upgrade is done using one of the alternate methods. My preferred choice is to use the busybox shell via SSH.

To use this method, I first used the VIC to upload the offline update to a datastore visible to all my hosts. Next, I put the first host into Maintenance Mode. Because I took advantage of the sweet "ssh AutoConnect" plugin, the console of the host is a right-click away. Once at the console, the following command is executed:

esxcli software vib update -d /vmfs/volumes/[DATASTORE]/[PATCH_FILE].zip

After a short wait, the prompt was back informing me that the update was complete, and a reboot was required to implement. You can use your favorite method of restarting the host, and once it returns to connectivity with vCenter, you have to manually exit Maintenance Mode. Repeat on each host until you're fully updated.

This update didn't replace Tools with a new version; the tools installed as part of the GA version are recognized as current, so I didn't get to see if the promise of no-reboot Tools updates would come to fruition.

iomega ix2-200 NAS non-cloud to cloud upgrade

UPDATE: DO NOT attempt to adapt this process to update an ix4 to the cloud-OS version!
If you do so, the upgrade will apparently attempt to update the BIOS of the ix4, and there appears to be a sufficient difference between the two products (cloud, non-cloud) that you will end up "bricking" your ix4. This happened to me (yes, I know I was taking a big chance) and someone else I know. Having a complete set of pristine, factory-reset drives will not give you the ability to revert to the non-cloud OS if you have a problem with the upgrade.
For the record, the system I started with had a sticker on the flash chip indicating that it was running the "V1.0.7" code.
The only known fix is a "low-level format" back at iomega support.

---

Standing on the shoulders of others...

So you've got a "non-cloud" version of the iomega ix2-200, and you're miffed that iomega doesn't officially support an upgrade path to convert it to the newer "cloud" editions. Maybe you want some of those cloud features, or maybe you have a Mac with Lion, which apparently isn't supported on the non-cloud models.
The upgrade can be accomplished; however it is destructive. You will not be able to perform an in-place upgrade with this technique, although you will be able to "get back" to having the original equipment—if that's what you desire.

What follows is a recipe that worked for me; your mileage may vary.

Required Hardware:

• iomega ix2-200 (duh!)
• Blank, FAT32-formatted USB thumbdrive with a drive activity light, 2GB or larger.
• pin, toothpick or other small, stiff wire that can be used to press and hold the reset button on the ix2 for ~90 seconds.

Optional Hardware:

• New hard drives. This is a prime opportunity to upgrade a 1TB or 2TB (raw) model to a 4TB or 6TB unit; it also gives you a fallback position in the event you brick the thing while working with the new drives. If you go this route, however, do not try to improve performance by switching to 7200RPM drives: The heat output of those drives is too high for the tiny fan in the chassis to handle. Stick with "green", variable RPM drives, and you should be okay.
• USB-to-SATA adapter. This eliminates the need to crack the case of a desktop machine (for access to the SATA ports) to clear a drive that will be re-used for the conversion.

Required Software:

• Linux distribution with openssh, tar, md5sum, gunzip and the ability to mount files to a loopback device. Believe it or not, your ix2 itself will fit the bill if you don't otherwise have a linux distro handy; all you need is SSH access, and you're golden. This recipe assumes the use of the ix2.
• Latest/greatest upgrade of the ix2-200 "cloud edition" firmware from iomega. As of this writing, the most-current version was 3.1.14.995. The firmware comes in an encrypted tarball, but your linux distro will make short work of that little problem.

Preparation:

1. Prepare the ix2 for conversion: Copy all your data to another storage medium. If you brick your ix2, you don't want to be left hanging in the cold.
2. Create the conversion tarball.
1. Create a new share on the ix2. Copy the firmware image from iomega to the share.
2. Save the following code as a shell script in the same place:

   ifw=./ix2-200-3.1.14.995.tgz
   ofw=${ifw%.tgz}-decrypted.tar.gz
   ix2=${ifw%.tgz}-files
   
   mkdir -p $ix2/images
   mkdir -p $ix2/update
   mkdir -p $ix2/apps
   
   openssl enc -d -aes-128-cbc -in $ifw -k "EMCNTGSOHO" -out $ofw
   tar xzvf $ofw -C $ix2/update/
   
   imgs=$(find $ix2/update/)
   for img in ${imgs} ; do
     if [ -f $img.md5 ] ; then
       mv $img $ix2/images/
       mv $img.md5 $ix2/images/
     fi
   done
   
   mount -o loop,ro $ix2/images/apps $ix2/apps
   cp -p $ix2/apps/usr/local/cfg/config.gz $ix2/images/
   umount $ix2/apps
   gunzip $ix2/images/config.gz
   
   img=$ix2/images/config
   md5=$(md5sum $img)
   md5=${md5% *}
   md5=${md5% }
   echo "$md5" > $img.md5
   
   cd $ix2/images/
   tar czvf ../ix2-boot.tgz *
   

   You will need to edit the first line of the script to match the filename of the firmware tarball you placed in the folder with the script.
3. Log on to the ix2 using SSH. (the first part of this post will walk you through gaining SSH access to your ix2)
4. Verify you have an available loopback device to use as a mountpoint: losetup -f
• if losetup indicates that there are no free loop devices, you need to create the next one in sequence.
• Determine the next-needed loop device with: losetup -a
  The output will probably look a bit like this:

  root@ix2:/# losetup -a
  /dev/loop0: [0900]:15970 (/sysroot/boot/images/apps)
  /dev/loop1: [0900]:47905 (sysroot/boot/images/config)
  /dev/loop2: [0900]:15972 (/sysroot/boot/images/oem)
  

• As indicated above, the first 3 loops are in use (loop0 through loop2), so the next one needed is loop3.
• Use the following command to make loop3 (the 4th loop device):
  mknod -m 0600 /dev/loop3 b 7 3
5. Change to the directory of the share you created:

   cd /mnt/soho_storage/samba/shares/sharename

6. Make the script executable:

   chmod 700 scriptname

7. Execute the script:

   ./scriptname

   The script will take a while to run: the original tarball must be unpacked, several files get copied, a new tarball is built. It's time-consuming, especially on low-powered hardware like the ix2. But nothing you're doing is affecting the existing system; you're simply using a handy linux environment to get some work done.
8. The final result will be a new file in the *-files folder called ix2-boot.tgz. Back on your main workstation, copy this new tarball to your USB drive in the following directory tree:

   \emctools\ix2-200d_images\

9. Exit the SSH shell
3. The drives that will be targeted for the conversion must be clean/clear drives; preferably, they will be new drives that have never been used, with no existing partition table. If you're going to re-use the existing ix2 drives, or re-use drives that you've had in another system, you first need to wipe them out: delete any existing partitions. The conversion won't work if you have data on the drives. This is where your mettle as a gadget geek gets tested: do you throw caution to the wind and wipe the ix2 drives, or do you get new drives and "play it safe?"
4. Place the new/wiped drives into the ix2.
5. Plug in the USB drive to any of the three ports.
6. Press and continue to hold the reset button on the back of the unit while plugging in the power supply. The ix2 will automatically boot when power is applied, and if you don't keep the reset button depressed for the entire boot sequence, you will not activate the installation. Watch your USB activity light; although it will come on when the unit activates the drive, you need to keep the reset button depressed until regular activity (i.e., blinking) is observed. In my experience, this takes approximately 90 seconds from the moment you plug in the power supply.
7. Wait. The update will occur, and you'll see plenty of activity for the drive indicator on both the USB drive and the ix2. After less than 10 minutes, however, the ix2 will automatically power off.
8. Remove the USB drive. Reconnect the network. Power up the ix2.

If you successfully updated the image, the indicator lamps on the front of the unit will blink furiously for a few minutes. Once the main power indicator (bottom lamp) is solid, you should be able to reach the management console via browser (Hopefully, you have some way to tell what the IP address is; your router or other DHCP server should have a way to indicate the address—I use a DHCP reservation, so I always get the same address, regardless of configuration state for the unit).

If you don't get that far—the bottom lamp blinks, the drive lamp blinked a couple of times, but otherwise, no activity—don't despair: I've seen other postings that indicate success when performing the procedure twice; I never needed it (I've repeated the update three times so far). Worst-case: you pull the drives and replace your original ix2 drives. Nothing is stored locally on the device, so a clean drive swap is possible.

The second alert lamp will blink until data protection is complete (either because you switched to a "none" option or it finished setting up the mirror); you can expect the drive activity lamp to be lit solid during the data protection phase.

By default, the firmware wants the ix2 to be using mirrored protection. You can alter that through the setup, and with the cloud version, you have the option of using "linear" or "raid0" if you want to get optimal performance at the expense of data protection.

Once you have the device configured the way you prefer, recreate your shares and iscsi volumes and put your data back on the unit: you're now Cloud-Ready!

---

The science and tech fields have historically worked by advancing based on prior works; we don't reinvent the wheel, we take the various works of our predecessors and advance "the art" through mash-ups that fail more often than they succeed.


This is one of those mash-ups, and thanks go to the following...

• Christopher Kusek (@cxi), who's blog post helped expose SSH access for the ix2
• Scpnas, who's posting to nas-central.com documented the process for creating an initialization tarball for a USB drive from the encrypted firmware download from iomega
• Luciano Rocha, who gave me the clues needed to be able to mount the apps file to a loop device on the ix2, a required step in making the initialization tarball.

Non-destructive Drive Expansion in the StorCenter

If you can't tell from the series of posts I've already published, I'm having some fun playing with the iomega ix2-200 that I received from Chad Sakac of EMC. In reviewing those posts, I realized that I didn't publish anything on the trick to expanding the storage on the ix2 (which should also apply to all the models in the StorCenter series) without destroying your data.

This technique is fairly straightforward, and while it takes time and a bit of work at the command line via "Support mode", you will also be best served if all your data is still backed up. Note: to get at the support page on a "Cloud Edition" unit, the URL is /diagnostics.html

Preparation...

1. Upgrade your drives with bigger models.
   
   1. It's not strictly required, but I suggest you upgrade starting at the highest labelled drive, working your way to the lowest labelled drive
   2. In order to make full use of each drive's capacity, they should all be identical.
   3. Shut down your unit each time you swap a drive (unless you're using a model that is explicitly hot-swappable).
   4. Allow the unit to fully redistribute the protected data before swapping the next spindle
2. Enable SSH access to your unit.
   
   1. There's an unlinked page on the system called support.html; access it by putting the address directly into your browser's address bar.

      

      support.html

   2. Check Allow remote access for support (SSH and SFTP) and click Apply
      
3. Use an SSH client to logon to your unit
   
   • username: root
   • default password: soho
   • If you have security enabled, you will need to append the primary administrator's password to the default password. For example, if your primary user's password is ducksauce, the SSH password is sohoducksauce.

Magic Time!

The devices can be expanded because of the storage subsystem being used in the Linux kernel. The process is straightforward: expand the "outer" container before expanding the inner container(s).

1. Dismount the user storage volume:

   root@ix2-200d:/# umount -l /mnt/soho_storage

2. Expand the RAID pseudo-device:

   root@ix2-200d:/# mdadm --grow /dev/md1 --size=max

3. Expand the LVM physical volume:

   root@ix2-200d:/# pvresize /dev/md1

4. Determine the free space in the LVM volume group:

   root@ix2-200d:/# vgdisplay
   

   --- Volume group ---
     VG Name               md1_vg
       .
       .
       .
     Free  PE / Size       476930 / 931.50 GB
       .
   

5. Expand the LVM logical volume by the amount of free blocks:

   root@ix2-200d:/# lvextend -l +476930 /dev/md1_vg/md1vol1

6. Mount the expanded volume:

   root@ix2-200d:/# mount -t xfs -o noatime /dev/mapper/md1_vg-md1vol1 /mnt/soho_storage

7. Expand the xfs file system:

   root@ix2-200d:/# xfs_growfs /dev/md1_vg/md1vol1

8. Reboot the system (so the web management tools will recognize the expansion):

   root@ix2-200d:/# telinit 6

Your device will reboot and (if you have email notifications correctly configured) it will notify you that "data protection is being reconstructed", a side effect of expanding the outermost RAID container.

Turbocharge your ix2-200 with SSD

The iomega ix2-200 is a nice little device for home or small-business NAS use, but it really suffers in the performance department: it only has a pair of 5900 RPM SATA drives under the hood, not to mention a slow (200MHz) CPU a single-core 1GHz CPU and only 256MB RAM. It's key benefit is the small package, which in turn equates to nice portability for a shared-storage device. However, if you're willing to break your warranty and get your hands dirty with a screwdriver and SSH client, you can turn it into something of a speed demon with some aftermarket upgrades.

Disclaimer: I don't advocate doing this; it's a bad idea for several reasons: you will sacrifice a ton of capacity (and cash) to gain performance improvements in a device that is really too underpowered to truly take advantage of the upgrade.

That said: I am also a gadget and hardware guy. I love playing with this stuff—especially storage and networking—and this was too much of an opportunity to miss. I did this more as an experiment to prove that a) it could be done and b) to see what kind of improvements were the result.

The ix2-200 itself will set you back at least $200 to start with, and 128GB SSDs (as used in the example) will also set you back $200 each plus the $20 each for the drive converters. For most users, you could find a much better way to spend $650-700 on shared storage. But if you need really, really fast, shared storage in a small, portable package—and capacity isn't as important as speed—this little Frankenstein's monster may be just the ticket.

Here's what you need, and how to get it done...

Preparation...

1. Get your hands on an iomega ix2-200. If you can find a used/broken model, so much the better; and because we're gutting the factory drives, pick up the lowest-capacity model (1TB raw in 2 x 500GB) you can get your hands on. It must be able to boot into the management console from at least one drive; I don't have an image that you can lay down on a new replacement drive, and I don't advocate going onto the Internet in the hope you can find one that is truly "clean".
2. Get a pair of the biggest SSD that you can afford. Don't worry about getting a "peak performance" model: even the slowest MLC-type units will still out-perform spinning disk, and still have more throughput than the ix2's SATA bus can handle. A matched pair is best because you can then use RAID0 for maximum performance.
3. Assuming that your SSDs will be the 2.5" form-factor, get a pair of Icy Dock 2.5"-to-3.5" HDD Converters from your favorite reseller; I bought mine from Newegg.
   Note: Icy Dock has recently released a Dual 2.5"-to-3.5" Converter with built-in RAID-aggregation capability that would potentially allow you to put four SSDs into the unit. Depending on the way the ix2 recognized the drives, you could do JBOD (so the ix2 would "see" all 4 drives and permit you to run RAID5 in the GUI software), RAID1 (and use manually-built RAID0 in the ix2) or RAID0 (and use GUI-built Mirroring). I haven't tried it, so your mileage may vary...
4. Assemble your new drives: put the SSDs in their Icy Dock converters
5. If you're using an ix2 that you already own, back up all your data currently on it. What you're doing is decidedly destructive, and you will end up with less capacity. You don't have to worry about the system partitions, just the user data.

Replace the original drives with your SSDs...

1. Change the protection method to "none". The only way the ix2 will allow you to do this is to remove all your shares and/or iSCSI volumes, and the only way to do that is to delete all your data (aren't you glad you backed everything up?).
2. Shut down the ix2. The device infrastructure wasn't meant to support hot-swap.
3. Remove one drive from the device (I always start with HDD2, but the system is designed so you could replace either one). Replace it with an SSD (in its converter, if needed).
4. Restart the ix2. When it finishes booting and you can see the management dashboard, the unit will have replicated the system partition (/dev/md0) to the new drive, and should notify you that it recognizes the drive substitution. By removing the protection in Step 1, there's no wait while the system replicates the data partition, and the available data (as shown in the dashboard) should be roughly equivalent to the sum of the two drive sizes.
5. Navigate to Disk Management in the control panel and verify that both disks are recognized by the system. If the new drive isn't recognized—the system displays no drive for the disk position you replaced, or the size is incorrect—you should stop and verify that the new drive is "good" in a regular PC. Even if the drive works, it is possible that the SATA implementation used by the SSD is not compatible with the ix2 hardware.
6. Shut down the ix2. Replace the other drive.
7. Restart the ix2. When it finishes booting, the appliance will have again replicated the system partition for reliability, and the remaining space on the drive will have been added to the pool of available storage.
8. Navigate to Disk Management and verify that both disks are recognized and that you have the full storage capacity you were expecting.

Assuming everything went correctly for you, your ix2 will show the storage capacity as a single, unprotected pool of space. What it doesn't tell you is that the drives are assembled into a "linear" array: only after the first drive gets full will the other drive be used. You now have three choices:

1. Live with it.
   You gain lots of speed by having the SSDs in there, but until you "fill up" the first disk's user data partition, that second one is doing nothing for you (although it is protecting the boot partition). If this is your choice, you're done. Enjoy using your device.
2. Switch the protection mode to "Mirrored".
   You use both drives equally, and if one "dies", the other one carries on because it has an exact copy of the data on the failed drive. There is a write penalty for this when compared to "no protection"—but it's definitely worth it if your only backup is tedious and annoying to restore—and reads should be slightly faster than a single drive, as the appliance can perform simultaneous I/O on the drives. The major downside, of course, is that the second expensive disk you put in the ix2 isn't "doing" anything but keeping your data safe. Personally, I only worry about this sort of thing for the boot/system/app partitions on these sort of appliance systems, and that's covered automatically for the ix2. If you choose this option, you can make the change through the device GUI, and when it's finished applying protection, you're done and can start using your device.
3. Convert the storage type to RAID0.
   On the negative side, RAID0 doesn't give any data protection, but then neither does linear: in either case, if one of the elements dies, so does all your data. On the positive side, however, all I/O will be distributed—in parallel—between both drives; there's no write penalty; you still get 100% of the available space for storage (no loss for protection). Too many people trust RAID "data protection" features to safeguard their data without having good backups. When you intentionally use RAID0, you know you have to keep good backups, so you force yourself to do so before trusting your data to the device. If you decide to choose this path, continue on...