Saturday, December 17, 2011

drobo's top-notch customer support

Although I'm a professional storage administrator, I really like the dead-simple maintenance that you get with the storage appliances from the folks at Drobo. It's true that they seem a bit pricey, and it's very true that their consumer-directed units are "performance-challenged," but when it comes to an easy way to create a huge, self-managing, readily growing storage pool for everything from home video and digital stills to a dumping-ground for backup images, it's really hard to beat.
I own four units:

  • 2 "first gen" 4-spindle, USB-2
  • 8-spindle DroboPro (USB-2, FireWire, iSCSI)
  • 5-spindle Drobo FS (CIFS)
The reason for this posting is because of a problem I had today with my Pro: although it's presenting a 16TB volume to the connected host via Ethernet/iSCSI, it's filled with a mix of 1TB and 1.5TB SATA-2 drives. The usable capacity is in the neighborhood of 8TB, and I had it roughly 70% full with backups and staging files when it decided to die.
I pulled the power, then restarted it, and it started a boot cycle.
After checking online and finding a troubleshooting post from Drobo, I went through the steps to address the problem. No joy.
I'd purchased the unit in November, 2009, so it was clearly out of warranty, but I took a chance and sent a help request to Drobo Support. It didn't take long before Chad was answering back through their support system, and by giving him good, descriptive information about the problem, he quickly responded with "please call me at..." to get some final information on my symptoms.
Unfortunately, they see this happen enough that they've actually coined a name for it: the Yellow Brick Road.
Luckily, the folks at Drobo recognized some time ago that this particular boot-loop cannot be solved by the customer, and even when the device is out of warranty, they'll RMA the thing and get you back in business.
By any measure, that's awesome customer support.

Thursday, November 10, 2011

"Nailed Up" SSH connection for Cisco devices

To all my fellow Cisco switch & router admins: have you ever been frustrated by the built-in, non-modifiable timeout for SSH support on the devices?

Well, there's a work-around that doesn't require tapping on the keyboard at least every 9:59...

The exec-timeout control for vty lines has no effect on SSH sessions; officially, you can't disable the timeout. Leave an SSH connection idle for 10:00 and it'll terminate from the remote (device) side. The alternative is to remote into your device using telnet; with that protocol, you can greatly extend or even disable the timeout.

But we don't like to use telnet, right? All data is sent in the clear, both keystrokes and screen paints, which includes authentication and logon credentials.

So how do we get the best of both worlds: secure, private connectivity of SSH with the extended, configurable (or disabled) timeout of telnet?

Answer: use both.

Seriously. Use both.

If you use SSH to remotely access a device, you'll get stuck with the 10:00 timer. But if you immediately telnet (or rlogin) back into the same device, you'll "mask" the SSH timeout with the telnet timeout (if one exists).

From the outside, this appears totally safe: all traffic going to and from the device is effectively encrypted by SSH. The telnet/rlogin session is happening "inside" the device where no man-in-the-middle or sniffer can see the traffic.

How does this work? Essentially, the timeout only measures idle time at an EXEC prompt; when you're using telnet (or SSH or rlogin, for that matter), the originating session is no longer in EXEC, it's running an operation.

One obvious negative to this is that you must keep telnet enabled as a valid transport for your device. A second negative is the practice of disabling timeouts on your vty lines. The fix for the first issue is to create a loopback interface for the express purpose of being a telnet endpoint; you then apply ACLs so that it is the only interface on your device that is permitted to accept connections on tcp/23. Assuming you build your ACLs so that telnet connections are only accepted when originated from the device itself, you will have effectively boxed in the insecure protocol inside the device. There's no real solution to the poor timeout choice, except making it long enough for most work without disabling it.

Thursday, November 3, 2011

Hypocrisy, thy name is Microsoft

So back in July 2011, when VMware announced vSphere 5 to the world along with its controversial new licensing model, VMware's competition was quick to join the hue and cry from customers that felt VMware was taking away something that, until then, was free.

You can Google it yourself; just put "vTax" in your search bar and you can get lost in all the write-ups,  venom and FUD.

One of my favorites is a TechNet article from the Microsoft virtualization team that attempts a side-by-side cost analysis between vSphere 5 and Hyper-V, showing how the increasing RAM and core density of a single server causes rapidly-escalating prices for licensing. While the author presents reasonable math on the cost to migrate from low(er) density hosts to high(er) density hosts, I have yet to see real-world Hyper-V support the consolidation ratio of VMware. In my personal experience, I've been unable to start up three 2GB VMs on an 8GB Hyper-V system for a classroom environment, while I've run five 2GB VMs on an 8GB ESXi server. In both cases, the host was strained to the point that the running VMs (2 vs 5) were sluggish and slow to respond, but I use the example to suggest that at scale it could take 50% less VMware infrastructure compared to Hyper-V, making a 10-to-10 cluster comparison no more appropriate than comparing apples to oranges for Vitamin C nutrition. And while both products have a limited set of approved/supported platforms for hardware compatibility, the VMware set is much broader than Hyper-V, giving the purchaser more options for savings that can offset the raw costs of licensing.

And on top of all that, Microsoft marketing money also paid for a spoof of VMware with the "VMlimited" microsite.

But today, Microsoft exposed its hypocrisy by unveiling a new licensing model when they announced the next version of SQL Server, SQL Server 2012.

In the same way that VMware is accused of "taking back" free features and benefits, Microsoft has done the same by eliminating the per-processor (socket) licensing option in favor of a per-core (physical thread of execution) model.

Like with VMware's old model, the old model for SQL Server allowed organizations to provision high(er)-density servers (in terms of both RAM and cores-per-socket) in place of low(er)-density servers without running into the need to acquire new licenses. The new model takes that away, and also puts the burden of reoccurring subscription expenses (Software Assurance, or "SA") into the mix in order to best-leverage the combination of SQL Server and virtualization.

Under the old model, an organization could pay $164,970 (USD, list) for six sockets of SQL Server Enterprise Edition; in a 6-socket VMware cluster, a DBA could then implement an unlimited number of SQL instances on up to 8 VMs per host. If the organization paid an additional $41,244/year for SA, the VMware cluster would be able to run unlimited instances on an unlimited number of VMs (subject to OS licensing limits—if any—and resource constraints of the virtual environment). Further, the old model had no limitations for the migration of SQL-hosting VMs from one server to another (i.e., vMotion). With modern, high-density servers that provide 512GB RAM and 24 cores of execution per 2-socket box, it's conceivable that dozens of multi-vCPU VMs could be hosted for costs in the neighborhood of $365,000 over a 5-year period.

Now compare that to the new model. At a (list) price of $6874 per core, that same six-socket, 12-core-per-socket cluster would cost $494,928 plus an additional 25% annually for SA, making the 5-year cost for the same hardware environment in excess of $1.1 million.

That's a 300% increase in costs for the new licensing model.

Hmmm.... Shall we now call per-core licensing the SQL Tax?

Thursday, October 20, 2011

auto-elevate in win7

I've had to deal with a poorly-written application for quite a few years: it misbehaves by requiring Administrative rights when running. I've spent many hours pouring over the capture from (SysInternalsprocmon without success: the application just won't reveal what trigger is needed to let it run in a low-security context.

Many sysadmins might question how this app ever got accepted into our environment with such a huge security flaw. The answer is simple: corporate policy didn't include running all desktops in limited-user mode. We still had a large selection of systems running on Windows 98 (for which "basic user" was not an option), and allowed users to run on Windows 2000 and Windows XP as local admins. We never caught the limitation during initial acceptance testing.

We were finally standardized on Windows XP when the policy change came that allowed us to switch end users from administrative to limited-user, but because the application was in limited distribution through our user base, it wasn't in the original test cases for switching to limited users. After the handful of users started submitting the same issue, we spent a while troubleshooting the app before we realized the common thread: it misbehaved only when running in the context of a limited user. While looking for a solution, we left the handful that required the app to continue to run their sessions in administrative mode. Remaining an administrative user was not an acceptable solution.

Luckily, Windows XP provides the "runas" utility, which permitted us to create a dedicated domain user that could have local admin rights as well as network access (a requirement of the application). With some limited, additional configuration, the special user account credentials could be used in an obfuscated batch file, and the primary user account could remain "limited". It worked beautifully.

Enter Windows 7.

Actually, the problem started with Windows Vista and the introduction of UAC (user account control), but because we never accepted Vista into the corporate environment, it wasn't an issue until Windows 7.
Being security conscious, we never considered disabling UAC for systems; however, that nicety became a burden on our use of our misbehaving application and the XP-type workaround: We could still use the alternate user, but UAC remained a barrier to the application, even when logged on as a local administrator.
So we experimented with ways to get an elevated session:
  • We tried 3rd-party tools, but they didn't work unless you were already in an administrative session
  • We tried "XP Mode", which was fine as long as we didn't try to access network resources; the standard user in the virtual environment is a local administrator for the system—something that should bother everyone using it.
After much trial-and-error, we discovered that a "nested" execution would work:
  1. we use "runas" to create the execution environment for an administrative user
  2. inside the session "bubble" that runas creates, we use the 3rd-party tool "NirCMD" to launch the pesky application in an elevated session.
The shortcut looks a bit like this:

runas /user:domain\special /savecred "c:\windows\system32\nircmd.exe elevate c:\progra~2\pesky\pesky.exe"
You can get nircmd.exe at; they have a number of nice, free utilities that can make your life easier.

The solution isn't perfect—a basic UAC prompt is still displayed—but it requires no user to remember alternate credentials. There are also setup requirements for preparing the administrative user's local profile (for network printers) and saving the administrative user credentials (prompted at first run), but until we can retire the application, we now have a working solution.

UPDATE: Changed link for NirSoft; the original post was incorrect.

Wednesday, September 28, 2011

iPhone Battery Drain, revisited

It happened again. I earlier blogged about the situation where my iPhone started eating the battery so fast that it was actually hot to the touch. Again, the issue was near-continuous network access to my home Exchange server (over ActiveSync), but this time I was able to resolve the issue by disabling calendar sync (which deleted all the calendar entries for that connection) instead of deleting the whole profile.
As soon as the calendar stopped being sync'd, the high network utilization went away; re-enabling the sync brought back the calendar items without causing the utilization problem.

Friday, September 16, 2011

iomega ix2-200 NAS non-cloud to cloud upgrade

UPDATE: DO NOT attempt to adapt this process to update an ix4 to the cloud-OS version!
If you do so, the upgrade will apparently attempt to update the BIOS of the ix4, and there appears to be a sufficient difference between the two products (cloud, non-cloud) that you will end up "bricking" your ix4. This happened to me (yes, I know I was taking a big chance) and someone else I know. Having a complete set of pristine, factory-reset drives will not give you the ability to revert to the non-cloud OS if you have a problem with the upgrade.
For the record, the system I started with had a sticker on the flash chip indicating that it was running the "V1.0.7" code.
The only known fix is a "low-level format" back at iomega support.

Standing on the shoulders of others...

So you've got a "non-cloud" version of the iomega ix2-200, and you're miffed that iomega doesn't officially support an upgrade path to convert it to the newer "cloud" editions. Maybe you want some of those cloud features, or maybe you have a Mac with Lion, which apparently isn't supported on the non-cloud models.
The upgrade can be accomplished; however it is destructive. You will not be able to perform an in-place upgrade with this technique, although you will be able to "get back" to having the original equipment—if that's what you desire.

What follows is a recipe that worked for me; your mileage may vary.

Required Hardware:
  • iomega ix2-200 (duh!)
  • Blank, FAT32-formatted USB thumbdrive with a drive activity light, 2GB or larger.
  • pin, toothpick or other small, stiff wire that can be used to press and hold the reset button on the ix2 for ~90 seconds.
Optional Hardware:
  • New hard drives. This is a prime opportunity to upgrade a 1TB or 2TB (raw) model to a 4TB or 6TB unit; it also gives you a fallback position in the event you brick the thing while working with the new drives. If you go this route, however, do not try to improve performance by switching to 7200RPM drives: The heat output of those drives is too high for the tiny fan in the chassis to handle. Stick with "green", variable RPM drives, and you should be okay.
  • USB-to-SATA adapter. This eliminates the need to crack the case of a desktop machine (for access to the SATA ports) to clear a drive that will be re-used for the conversion.
Required Software:
  • Linux distribution with openssh, tar, md5sum, gunzip and the ability to mount files to a loopback device. Believe it or not, your ix2 itself will fit the bill if you don't otherwise have a linux distro handy; all you need is SSH access, and you're golden. This recipe assumes the use of the ix2.
  • Latest/greatest upgrade of the ix2-200 "cloud edition" firmware from iomega. As of this writing, the most-current version was The firmware comes in an encrypted tarball, but your linux distro will make short work of that little problem.
  1. Prepare the ix2 for conversion: Copy all your data to another storage medium. If you brick your ix2, you don't want to be left hanging in the cold.
  2. Create the conversion tarball.
    1. Create a new share on the ix2. Copy the firmware image from iomega to the share.
    2. Save the following code as a shell script in the same place:
      mkdir -p $ix2/images
      mkdir -p $ix2/update
      mkdir -p $ix2/apps
      openssl enc -d -aes-128-cbc -in $ifw -k "EMCNTGSOHO" -out $ofw
      tar xzvf $ofw -C $ix2/update/
      imgs=$(find $ix2/update/)
      for img in ${imgs} ; do
        if [ -f $img.md5 ] ; then
          mv $img $ix2/images/
          mv $img.md5 $ix2/images/
      mount -o loop,ro $ix2/images/apps $ix2/apps
      cp -p $ix2/apps/usr/local/cfg/config.gz $ix2/images/
      umount $ix2/apps
      gunzip $ix2/images/config.gz
      md5=$(md5sum $img)
      md5=${md5% *}
      md5=${md5% }
      echo "$md5" > $img.md5
      cd $ix2/images/
      tar czvf ../ix2-boot.tgz *
      You will need to edit the first line of the script to match the filename of the firmware tarball you placed in the folder with the script.
    3. Log on to the ix2 using SSH. (the first part of this post will walk you through gaining SSH access to your ix2)
    4. Verify you have an available loopback device to use as a mountpoint: losetup -f
      • if losetup indicates that there are no free loop devices, you need to create the next one in sequence.
      • Determine the next-needed loop device with: losetup -a
        The output will probably look a bit like this:
        root@ix2:/# losetup -a
        /dev/loop0: [0900]:15970 (/sysroot/boot/images/apps)
        /dev/loop1: [0900]:47905 (sysroot/boot/images/config)
        /dev/loop2: [0900]:15972 (/sysroot/boot/images/oem)
      • As indicated above, the first 3 loops are in use (loop0 through loop2), so the next one needed is loop3.
      • Use the following command to make loop3 (the 4th loop device):
        mknod -m 0600 /dev/loop3 b 7 3
    5. Change to the directory of the share you created:
      cd /mnt/soho_storage/samba/shares/sharename
    6. Make the script executable:
      chmod 700 scriptname
    7. Execute the script:
      The script will take a while to run: the original tarball must be unpacked, several files get copied, a new tarball is built. It's time-consuming, especially on low-powered hardware like the ix2. But nothing you're doing is affecting the existing system; you're simply using a handy linux environment to get some work done.
    8. The final result will be a new file in the *-files folder called ix2-boot.tgz. Back on your main workstation, copy this new tarball to your USB drive in the following directory tree:
    9. Exit the SSH shell
  3. The drives that will be targeted for the conversion must be clean/clear drives; preferably, they will be new drives that have never been used, with no existing partition table. If you're going to re-use the existing ix2 drives, or re-use drives that you've had in another system, you first need to wipe them out: delete any existing partitions. The conversion won't work if you have data on the drives. This is where your mettle as a gadget geek gets tested: do you throw caution to the wind and wipe the ix2 drives, or do you get new drives and "play it safe?"
  4. Place the new/wiped drives into the ix2.
  5. Plug in the USB drive to any of the three ports.
  6. Press and continue to hold the reset button on the back of the unit while plugging in the power supply. The ix2 will automatically boot when power is applied, and if you don't keep the reset button depressed for the entire boot sequence, you will not activate the installation. Watch your USB activity light; although it will come on when the unit activates the drive, you need to keep the reset button depressed until regular activity (i.e., blinking) is observed. In my experience, this takes approximately 90 seconds from the moment you plug in the power supply.
  7. Wait. The update will occur, and you'll see plenty of activity for the drive indicator on both the USB drive and the ix2. After less than 10 minutes, however, the ix2 will automatically power off.
  8. Remove the USB drive. Reconnect the network. Power up the ix2.
If you successfully updated the image, the indicator lamps on the front of the unit will blink furiously for a few minutes. Once the main power indicator (bottom lamp) is solid, you should be able to reach the management console via browser (Hopefully, you have some way to tell what the IP address is; your router or other DHCP server should have a way to indicate the address—I use a DHCP reservation, so I always get the same address, regardless of configuration state for the unit).

If you don't get that far—the bottom lamp blinks, the drive lamp blinked a couple of times, but otherwise, no activity—don't despair: I've seen other postings that indicate success when performing the procedure twice; I never needed it (I've repeated the update three times so far). Worst-case: you pull the drives and replace your original ix2 drives. Nothing is stored locally on the device, so a clean drive swap is possible.

The second alert lamp will blink until data protection is complete (either because you switched to a "none" option or it finished setting up the mirror); you can expect the drive activity lamp to be lit solid during the data protection phase.

By default, the firmware wants the ix2 to be using mirrored protection. You can alter that through the setup, and with the cloud version, you have the option of using "linear" or "raid0" if you want to get optimal performance at the expense of data protection.

Once you have the device configured the way you prefer, recreate your shares and iscsi volumes and put your data back on the unit: you're now Cloud-Ready!

The science and tech fields have historically worked by advancing based on prior works; we don't reinvent the wheel, we take the various works of our predecessors and advance "the art" through mash-ups that fail more often than they succeed.

This is one of those mash-ups, and thanks go to the following...

Thursday, September 15, 2011

iomega px6-300d: first look

The iomega px6-300d is a storage appliance that can be provisioned with up to six SATA drives, both traditional rotating-disk and SSD. It is one of four NAS/SAN appliances in the "px" line from iomega, and is the largest-capacity "desktop" model.
Under the hood, it contains a dual-core Intel Atom processor (running at 1.8GHz) with 2GB RAM, sports a pair of Gigabit Ethernet ports, three USB ports (1 x USB 3.0, 2 x USB 2.0), and runs a customized version of Linux called "EMC LifeLine." The unit can be purchased with various pre-installed drive configurations (6, 12 & 18TB raw capacity); along with the px4 models, iomega has also (finally!) offered a "diskless" configuration.
Retail Box
I purchased a diskless unit ($750 from B&H) for use in my home lab, and independently acquired a matched set of six 2TB Hitachi 7200RPM SATA-III drives (HDS723020BLA642, $110 from NewEgg). This 12TB (raw) setup cost me less than $1500; in contrast, the list price for a pre-populated 12TB unit is $3,299.99.
The unit came in a nice retail box, and was well-padded and packed. In addition to the unit itself, the package contained the external power supply, power cord, CAT5e Ethernet cable, a "Quick Start Guide" and a "Solutions CD."
px6-300d front
px6-300d rear
The front of the unit provides drive access, power button, information display (and a pair of input buttons) and one USB port; the rear of the unit has a pair of 90mm fans, the power connection, reset button, Gigabit Ethernet ports and a pair of USB ports. The rear also sports (what appears to be) a surprising and undocumented x1 PCIe expansion slot.
The Quick Start Guide instructs the owner to install software from the Solutions CD, which will then assist in installing the array; having experience with the ix line, I simply created a DHCP reservation (the MAC address is documented with a sticker on the rear of the unit), connected it to Having purchased a diskless unit, I was curious to see how the system would behave when booted with no drives. True to form, this situation was handled gracefully.
front door open, showing drive trays
drive trays with 2.5", 3.5" drives

trays line up SATA connector
Up to six drives can be installed using supplied hot-swap trays, which support both 3.5" and 2.5" form-factor drives. In addition to the matched drives, I also had a 2.5" SSD available for testing, but the supplied screws didn't fit the drive properly; it was a bit of a struggle to find screws that would not only work with the drive, but also work with the sled: the smaller drives must be attached to the sled from the bottom, so care must be given to make sure the screws do not protrude too far from the bottom of the sled.
The unit I received was pre-installed with an out-of-date firmware (3.1.10; 3.1.14 most current as of this posting), which booted correctly even without any drives installed in the unit. This is a distinct departure from the "ix" line of NAS boxes, which require at least one functioning drive to store the firmware (which also explains why the older lines didn't support a diskless purchase option). The management interface will not allow you to do anything until at least one drive is installed, however, so that was the first order of business.

After installing the first drive, the management interface permitted me to add the unit to my Windows domain, and I also went through the exercise of downloading and updating the firmware. This is a very easy process, so novice users shouldn't be afraid of it.
The unit supports a wide variety of RAID levels, and the user can combine arbitrary combinations of drives into "storage pools" so that the unit can provide different levels of performance based on your needs, so I will be testing the various permutations with increasing spindle counts.
Storage Pool Protection Options
2None, RAID0, RAID1
3None, RAID0, RAID5
4None, RAID0, RAID1/0, RAID5
5None, RAID0, RAID5
6None, RAID0, RAID1/0, RAID5, RAID6
Once you create a storage pool, you can create one or more NAS volume (for CIFS, NFS or other share-types) or iSCSI volumes in it. When creating a file-sharing volume, the unit can recognize space utilization on a file-by-file basis, and will display free/used space as one would expect. A volume used for iSCSI will appear to be 100% allocated as soon as it is created, even if no initiator has ever connected and accessed the LUN. Additionally, multiple shares can be created for a single file-sharing volume, but there's a one-to-one mapping between an iSCSI volume and a target. Security for either file shares or iSCSI is optional; the unit defaults to read/write permissions for everyone.
There are a number of nice features that I'll investigate further, but the first order of business is performance testing. With luck, I'll learn that the device can scale with each added spindle; it will also be interesting to see how much of a performance hit the device will take when using RAID6 instead of RAID5 in the 6-spindle configuration.

Sunday, August 28, 2011

Time Zone support on iPhone

Okay, this is one place where the lovely "intuitiveness" of the iPhone falls right on its nose: time zone support.

I've had the pleasure of travelling from my normal haunts in the Central time zone to the Pacific time zone, but the displeasure of having all my appointments continue to show the appointment in Central time. For instance, an 8am meeting in Pacific time would still show up on the phone as being scheduled for 10am, which is the equivalent Central time (Pacific is UTC-8, Central is UTC-6).

The root cause of the problem is a setting I made in the "Mail, Contacts, Calendars" Settings page, not the "General" page--which is where the other time-related settings are done.

At the very bottom of the settings page (iOS 4), there's a section for adjusting calendar sync; it has a Time Zone Support submenu.

Setting Time Zone support on results in the phone "locking in" on the selected time zone--in my case, I set "Chicago" ages ago, when I first got my iPhone--and ignoring the "embedded" time in the calendars. There's an explanation on the submenu: Time Zone Support always shows event dates and times in the time zone selected for calendars. When off, events will display according to the time zone of your current location.


I'm not the only one with the misunderstanding. Google the phrase "iphone time zone problem," and you'll find a bunch of folks with similar problems: appointments set in one time zone will result in them "sticking" to the old time zone instead of moving to the new time zone with you.

Here's where the setting becomes counter-intuitive: If the device supports time zones, it implies that it "recognizes" that there are different time zones: an appointment set for 8am PDT should show up as 8am when the phone is in the Pacific Time Zone, while showing 10am when in the Central Time Zone. The opposite should also be true: disabling Time Zone support should result in the phone ignoring all time zone metadata and displaying all times as input, regardless of the physical location of the phone.

I don't have a problem with the iPhone operating this way; my objection is the way the settings are presented to the user. My recommendation is that the equivalent settings be duplicated on the General Settings page (Date/Time), and that the setting itself be called "Time Zone Override".

Friday, August 19, 2011

VMworld 2011 Las Vegas: a vExpert's take

VMware's annual conference—VMworld—is in its 8th year in the US and is being held in Las Vegas, Nevada. On-site registrations and hands-on labs begin on Sunday, August 28; breakout sessions and opening keynote begins the next day.

This will be my 4th year to attend the conference, and each year I've taken away some incredible experiences, both technical and social. Although I don't have the attendance record of some folks (a friend from Omaha has been to all of them since the first back in 2004), I've learned a thing or two about the conference that are worth sharing for anyone who's never been—or hasn't been back recently.

  • If you haven't booked your travel yet—or can change it—consider arriving on Saturday, Aug 27, and leaving on Friday, Sept 2. Seasoned travelers may not give a whit, but if you are normally a stay-home worker, setting aside the full day on either end of the trip means you'll be less rushed to get back-and-forth to the venue. Other benefits: 
    • You may end up with some savings in both airfare and lodging because of the Saturday night stay. 
    • You won't have to lug around your entire kit the last day of the conference because of check-out times.
    • More and more activities (unofficial and official) are available on Sunday. In previous years, nothing but registration was available on Sunday. As the conference has matured and grown, there are now parties, meetings and other unofficial events on Sunday. And for the first time in memory, the Self-Paced Labs are open at the same time as Registration.
    • No trips back to the office before the Labor Day Weekend!
  • Don't spend a lot of money on a great room. It's more important that where you stay is close to the Sands Expo Center than having luxury. It's pretty simple: unless you're agoraphobic and can't stand crowds, you probably won't be spending much time in your room (Of course, if you're in that group, maybe you should reconsider the trip: you're going to be joined by 20,000 of your closest friends in virtualization). You don't want to go too cheap, however; pick a spot on the Strip, and you should (literally) be safe.
  • Even if you have a room at the Palazzo or Venetian, you'll be doing lots of walking. If you didn't already buy comfy new shoes for the trip, it's probably too late: you'll never break them in before leaving, and the only thing worse than having bad shoes at VMworld is trying to break in a new pair of shoes.
  • While you should always budget for evening meals (breakfast & lunch are provided in your conference fees—assuming you've got the stomach for it—along with light snack foods in the conference venues), if you end up having to buy anything for yourself to eat from Sunday evening through Thursday afternoon, you're either on a very strict diet or you're doing it wrong.
    • First cozy up to your vendors; let them know you're going, and that you want to be informed of any events they have planned for the conference. Your regular salesperson isn't likely to be there—they'll be staying home trying to keep their commissions coming in!—but he/she should be able to get you on any lists for private events.
    • Don't ignore the spam you receive from vendors doing a pre-conference marketing blitz. You might not want to purchase their products at this point, but listening to their pitch may be a small price to pay for a nice meal & drinks.
    • Talk to your friends at other companies and folk you know through the local VMUG (you are a member, aren't you?) and trade invitations around.
  • Some schedule highlights:
    • Sunday: get registered—the lines will be huge, so plan to stand around a bit—and maybe hit the Self-Paced Labs. If you get in after registration, but before 7pm, consider going to the second annual "v0dgeball" tournament (register here). It's being held in the Sportscenter near the airport, so it might be easier to schlep directly over there on arrival rather than heading into the Strip and back after you check in. No word yet on whether there will be shuttle service, but if there is, you'll also save cab fare.
    • Monday night, you'll drink & eat at the opening of the Solutions Exchange (vendor booths). This is the big first-night reception, and the vendors do a pretty good job of "pulling out the stops." This is the night when the best swag is given out, as well as a good opportunity to troll for entrance bracelets/bands for all the various parties being held on Tuesday night. This year, you can also add the CXIparty to your list of events; it's the party that's most congruent with the all-night feeling of Las Vegas.
    • Tuesday night is Vendor Party Night. Nothing is explicitly scheduled for the conference, so most vendors do private parties that night. If you haven't received a half-dozen invitations to overlapping events, talk to your peers to get them. You can also use your time in the expo hall during the day to wheedle invitations. Don't worry about accepting multiple invitations; keep your options open and plan to migrate from party to party until you find one that serves the food you like or otherwise sits best with your temperament.
    • Wednesday night is the big blow-out party for the conference, and this year The Killers will be performing. Dinner will be served. Keep an ear out for word-of-mouth invitations to after-parties. The Self Paced Labs guys have thrown a rave for the last several years—they actually have labs available during the party—and many other vendors do small-group events that you may have an opportunity to get swept up in.
  • Drink lots of water. This can't be emphasized enough. Las Vegas is in the desert, and the venues you'll be in have the A/C set to remove almost every bit of humidity from the air. You'll lose lots of hydration just breathing, and if you consume any amount of alcohol, your body will need additional water to metabolize it. It's a joke that the biggest cause of a hangover is imbibing in the first place, but if you take that as a given, the second biggest cause is dehydration. I've learned to force myself to consume a bottle of water for every serving of alcohol I drink; yes, it makes you go to the head more often, but I've also never missed an 8am meeting the next morning because I was providing sacrifice to the local porcelain gods... Just don't drink a bunch of alcohol and try and offset it later with a bunch of water. Pace yourself and alternate from the very beginning, and your body will thank you later.
  • Do bring a laptop with you. Don't bring it to the expo or breakouts unless you're a blogger with up-to-the-minute deadlines. If you're anything like me, you can get the most work done while working at your desktop, stay productive with an Internet-connected laptop, and love the portability of an iPad or smart phone. Keeping your laptop in your room will make it possible to answer those emergencies that inevitably require your expertise, yet leave you free to skip lugging it around with you. You will get a laptop bag as part of your conference registration, and it's best used for stuffing with give-aways.
  • Other road-warrior tips:
    • Pick up a $100 Apple Airport Express. These little gems transform a single, wired Internet connection in your hotel room into a whole-room wireless network that you can share among your various devices (laptop, iPad, smartphone) using a secure protocol like WPA2. You'll also end up with much better coverage than relying on hotel or conference-hall wireless.
    • Instead of bringing multiple chargers for your devices, bring along a Tripp-Lite TRAVELER3USB. Not only can you share a single power outlet for charging several of your own devices, you may become a hero to your fellow travellers if you share a spare plug or two with them. Having the USB ports are a bonus, and unlike other mobile surge protectors, they have the 2A "oomph" to charge an iPad.
    • An external battery pack can be a lifesaver when you're going to be away from your room all day and your smartphone doesn't go all day on a single charge. While you could always charge this stuff while in your room, these external packs are far more portable than a laptop, can be easily stuffed in a cargo pocket, and allow you to "top off" the battery on your phone while on the move.
    • Pack a hoodie or other light jacket. Yes, it's the summer in the desert, but the conference venues are usually kept pretty cool to handle the heat-output of thousands of attendees; sitting in the breakout rooms can get uncomfortable if you don't have a layer you can add and remove as needed. Unless you're going to a high-profile meeting where you need to look your finest, there's nothing wrong with walking around the venues with a jacket tied around your waist (save the space in your backpack for the goodies!).

Wednesday, August 10, 2011

On the non-technical side...

Okay, the word is out. I can karaoke, and you don't have to get me drunk to do it.

I was at an industry conference for work, and the closing-night party was at a local micro-brewery. The bar had a nice selection of yummy beers, and they did a stand-up job in the kitchen. The half-dozen pool tables were busy with attendees, but the big draw for the private party, however, was the karaoke machine. While I was late to that part of the party—the weather outside was uncommonly nice for early August—I jumped right in to support the fun of the other attendees.

If you've never done karaoke, you should give it a try. If you have, you know that it's most fun when you get a whole group of folks—like a chorus—to perform songs that everyone will join in for.

But once you get warmed up, it's also energizing and exhilarating to find a song you know fairly well and can perform by yourself. I found She Caught the Katy by the Blues Brothers in the catalog, and did that one solo; I had a couple of dropped phrases—it's been a while since I listened to that song with any regularity—but on the whole, it was well-received.

And that kicked off a spate of other group songs where I was drafted to lead the chorus.

It was a lot of fun, and my previous theatre and voice training meant that I didn't lose my voice the next day even though most of the stuff we sang was a bit high for my natural range. I had my ego stroked a bit by all the compliments—which were very gratifying—but I also know that I'm gifted more with courage than musicality, so there won't be any "American Idol" auditions in the works.

Thursday, July 14, 2011

VMware vSphere 5 Licensing—it's not all about vRAM

With the announcement of features and new licensing model for VMware vSphere 5 during a live+webcast presentation, much of the ensuing kerfuffle was aimed at the new licensing model, rather than excitement over the new features. I have to admit, I'm one of those folks who aren't happy with the licensing changes; to be fair, I haven't been happy with the licensing moves VMware has been making since the introduction of "Enterprise Plus."

I understand and can accept the rationale from VMware, including the way the vRAM pooling is supposed to work. I'm also the first to admit that I've got a bias in favor of VMware: I am one of the leaders of the Kansas City VMware User Group in addition to being a long-time customer.

No, my concern is the way VMware keeps tying specific "entitlements" (their term) to the various Editions of vSphere. In this case, I'm not just thinking of the vRAM entitlement—the piece that's generating all the outrage—but the features that are available accross editions.

VMware's licensing whitepaper gives a nice overview of the new model, as well as some examples of how the new model could work in practice, paying particular attention to the vRAM portion. My opinion is that VMware pays short shrift to the other, non-vRAM details that distinguish the different Editions of vSphere.
On the vRAM side: If you have a small cluster of 2-socket hosts with modest amounts of physical RAM (e.g., 96GB/host), you will be unimpacted by the new license model if you're already on Enterprise Plus (2 sockets x 48GB vRAM= 96GB); in this scenario—assuming you have current service-and-support contracts—you'll upgrade straight to the vSphere 5 Enterprise Plus licenses and be "off to the races." Your physical RAM won't ever exceed your entitlement, and if you're over-subscribing your guest memory, you're begging more problems than vRAM entitlements, because it also means you've not left yourself any wiggle-room for N+1 availability. In fact, if you're in that situation, you may have over bought from a vRAM perspective: A 5-host cluster with 10 sockets of Enterprise Plus has a pool of 480GB vRAM. If all those hosts have 96GB physical RAM, your N+1 memory allocation shouldn't exceed 384GB, so you wouldn't need a vRAM pool bigger than 384GB. You can't quite achieve that with 10 sockets of Enterprise (which has a 32-GB entitlement), but you can do it with 12 sockets, which is a tiny bit less expensive (list price) than 10 sockets of Enterprise Plus ($34,500 vs $34,950). Of course, that assumes you're pushing the limits of your physical hardware and not obeying the 80% rule. In that case, you could get away with 10 seats of Enterprise and save a pretty big chunk of cash.

My suggestion to VMware: consider two changes to the licensing model with respect to vRAM entitlements. 1) allow vRAM pooling across editions, rather than keeping it edition-specific. 2) create vRAM "adder licenses" so that organizations can add blocks of vRAM to their pools (without paying for all the additional features of a full processor license at any given edition). Doing both eliminates the need for different SKUs for editions as well as vRAM increments.

Back to the 5-host cluster example...

The problem with going the route of choosing Enterprise over Enterprise Plus just to manage the vRAM pool—and I'm certain that VMware has all this in mind—is that you must give up some pretty cool vSphere features (e.g., host profiles, dvSwitch) if you aren't on Enterprise Plus, including some new features in vSphere 5 (e.g., storage DRS). These features of vSphere make a lot of sense for smaller enterprises that have to watch every single dollar spent on IT, especially when shared storage (which, in my opinion, is the one thing that really makes virtualization sing) is probably the single most expensive item in a virtualization project.
In this case, I'd like to see VMware push some of these more interesting new features down the Edition stack. In general, anything that helps the business better utilize their (typically) very expensive storage investment makes sense. If VMware keeps the storage features at the most expensive end of the spectrum, organizations may be more inclined to consider alternatives for their perceived value rather than paying the premium for Enterprise Plus, especially now that there's the added burden of vRAM entitlements to consider.

Tuesday, July 5, 2011

iPhone Battery Life: use it or lose it?

I think most iPhone users have discovered that their little "magical" friend has a propensity for chewing through battery life like high-schoolers chew through Doritos. Tips and tricks for extending life abound, but all the ones I've read amount to disabling functionality of the phone or applications.

I accept that keeping the GPS receiver on at all times will be a big drain; there isn't a lot of value of keeping it on when I spend the majority of my day inside and away from windows that could allow reception.

No, the things I don't want to disable are important functions to me. Things like push messaging; that's why I have a smartphone in the first place: to stay in touch and have messages at my fingertips.

So I husband my battery use, and most days am lucky to get through it with at least 30% still showing in the indicator.

But over the holiday weekend, I noticed something: I was regularly staying above 50% by the end of the day. Curious. I wasn't using it any less; if anything, I was on it more, taking pictures, checking social media, etc. What was the big difference?

I think I found it, and you may find it a surprise...

I use Bluetooth (BT) for in-car hands-free and for my Sony-Ericson HBH-IS800 headset. I normally leave the BT radio enabled so the phone automatically connects with the car; the way Apple buried the BT controls in the Settings app is pretty annoying for quick or frequent BT enable and disable, so I take the lazy way and leave it on. And I normally leave the HBH turned off to retain battery life, so the normal state for the iPhone is Bluetooth On/Unpaired.

And that seems to be a huge battery drain; an even bigger drain than having a paired device connected to it.

Here's what happened: this weekend, I kept my phone in my pocket, but didn't keep the HBH with me; I left it plugged into its charger until time came that I might want to use it. I noticed at one point during the day that my phone was paired up with it, however; I didn't realize it could/would do that while charging. I also noticed that the phone wasn't chewing through the battery like usual. So I did a qualitative experiment: One day, I would keep the phone paired with my HBH at all times; the next day I'd keep it un-paired, as usual; and the final day, I'd actively enable and disable the BT radio whenever I actually needed it.

Again; this was qualitative, not quantitative: there are too many other variables at play to put real numbers to it. However, by the end of the experiment, I was convinced: If you want to extend battery life without turning off the BT radio (by far, the best choice), make sure it's paired.

As I understand it, it goes a bit like this: the preferred state for the device—when BT is enabled—is to be paired. With anything. If it's unpaired, the radio throws extra power into trying to see if one of its partners are "out there" trying to reach it. When it's paired—and able to readily "stay in touch" with the partner—the radio draws less power than when it's "seeking."

Rule of thumb: when it comes to BT power consumption, radio off < radio paired < radio seeking

Friday, July 1, 2011

VMware vExpert 2011

I'm not usually in to the chest-thumping thing, but I'm proud to have been selected as a VMware vExpert for the third year in a row (which also happens to be every year the award has been given).

While I like to think I'm a bit of an expert when it comes to VMware, the award is really a recognition of the work I've done to promote VMware—and virtualization in general—through my volunteer work on the leadership team of the Kansas City VMware User Group (kcvmug).

It's been great to be a part of the team that helped grow our VMUG from a couple dozen regular attendees to a group that regularly sees 80+ members at "regular" meetings and 400+ attendees at our annual all-day "regional" meeting.

My thanks go out to VMware for supporting the program, John Troyer for his "above and beyond" efforts to launch and add value to the program over the years, and his anonymous team of judges for giving me the nod. But most of all, I want to thank my fellow kcvmug leaders, past (Kevin Davidson, Ron Armstrong, Ryan MeltonJoe Adams) and present (Ben ClaytonTony Lux); without you, the VMUG wouldn't be what it is today.

Sunday, June 19, 2011

SQL Server Failover Cluster -- Add-a-node gotcha

This isn't the sort of thing someone does all the time, so you're bound to be surprised by one thing or another...

We run instances of Microsoft SQL Server 2005 on hardware failover clusters (I'd love to run them all on VMware, but there are a number of reasons why we don't.). We had an occasion to replace the hardware for the nodes, and an associate & I were trying to get this accomplished.

Putting the 2 new nodes into the cluster was smooth as butter. That left me with a 5-node cluster (we were going to ultimately evict the older 3 nodes), and I set forth to add the new nodes to the first SQL instance on my list.

But I kept getting stopped by the "remote task will not start" error. I googled for a solution, and kept coming up with remote access issues: in essence, make sure you aren't using Remote Desktop to connect to the new passive node. I wasn't using RDP on any of the nodes, so I couldn't understand why it kept failing. Just about every combination of reboots and node selection was attempted. I made sure I was logged out from the new nodes. Still, no joy.

In final desperation, I logged out of all the nodes, not just the ones I was adding, then logged back in to the instance's active node and ran the setup from there. Voila! I was able to add a new node to the instance. For whatever strange reason, you cannot be logged into ANY of the nodes other than the one from which you're installing. Period. Not using RDP like they document, but also not using VNC or even the actual system console. NONE.

Sunday, May 29, 2011

Carrier shenanigans

I'm writing this post while visiting my in-laws in southern Missouri. They live in a rural area on a 100+ acre farm abutting the National Forest. It's a beautiful part of the state, but because they're over 10 miles from the nearest town with "real" Internet options, the only options they have is dial-up, spotty cellular and HughesNet residential satellite. It's true that they could "pony up" (big time) and pay for infrastructure to get cable or T.1—DSL is just plain not an option—so they're pretty much stuck with three bad choices for today's media-rich Internet.

After doing much research, they went with HughesNet. On the surface, it looks like a pretty good option: near-T.1 speeds for some capital (dish & receiver aren't included) and $40/mo.

If you check out their website yourself, you'll note, however, that they have an item called a "download allowance" that is even worse than the bandwidth caps being instituted by the "terrestrial" carriers. For my in-law's plan, it's 200MB per day. That's right: per day. On average, that's over 5GB per month, but on a daily basis, it's punitive.

It's supposed to work on a rolling 24-hour basis, but in my experience, it always expires at ~10am Central; if you exceed it by more than 10%, they start throttling your access. And by "throttle", I really mean "choke it off to the point you'll give up and go do something else." I ran a couple of tests from while in this state, and downloads averaged 60kbps. By comparison, non-throttled (on a clear day) runs about 450kbps. It's true that they don't come close to meeting their advertised maximums, but the throttling is unreal.

I run into this every time we visit. I'm the go-to computer guy in the family, and my mother-in-law always has lots of stuff for me to fix: patches, service packs, upgrades, etc. I blow through the 200MB in a couple hour's work in the morning, and pretty much can't get anything done the rest of the day.

Initially, I was unaware of the root cause; it just seemed like the satellite service sucked. But this visit I replaced their bad router with a new one and discovered that they had the previous router (that I didn't install) mis-configured: the satellite receiver is already a NAT router, so we originally had a double-NAT setup. This isn't always a problem, but it did "hide" the administrative "control center" for the satellite service. Once I put the new router into bridge mode, it exposed the other router and I discovered why the service seemed like such a wreck.

We knew that going with satellite service would result in high latency levels; even at the speed of light, you have to expect a packet that has to travel over 44,000 miles (ground-to-orbit and back) to take some time. In general, you can expect a round-trip data exchange to take on the order of 0.5 seconds; however, once you have a stream in progress (i.e., Netflix), you'd expect the advertised 1Mbps to be acceptable.

And it is.

Just barely.

As long as you don't blow past that 200MB...

Which can happen in an hour's time with standard definition, and as quick as 15 minutes with high def (which you really can't get because of the latency & lack of actual throughput).

To be fair, Hughes tries to be straightforward with its customers. Their marketing website gives the definition: "The Download Allowance is the amount of data which can be downloaded without restriction within a rolling 24-hour period". On the other hand, the control center has a little different spin on the issue; it's called the "fair access policy". Among other things, the Fair Access Policy "...may occur if there is a high amount of data download by computers connecting via your terminal, over a long period of time. Certain viruses can cause such activity... Right. Most viruses I know of send a lot of data (like spam or DDoS bots) if they do anything, not download bunches.

The HughesNet FAP FAQ says it more plainly; I'll paraphrase: the stuff you want to do using the Internet that involve rich media--like watching Netflix & Youtube--isn't something we want running on our network. Don't expect to spend the day listening to Pandora. Have your kids send DVDs or thumbdrives with pictures of the grandkids through the mail rather than posting them to flickr or shutterfly. If you mess with that stuff, you'll wish you were back on dialup.

Luckily, they do have a "relaxed period" where things like Windows Update can do it's thing; it's from 2am to 7am Eastern. But if you're the visiting IT guy, you're screwed. Don't ever try to install a new machine on this network; you won't get all the Windows Update patches installed before the pipes close down on you.

As with other carrier shenanigans, this penalizes customers in an arbitrary fashion. If the network isn't being utilized, let the bytes flow. The only time the policies should be implemented is when the network is actually saturated. Applying special handling policies for behavior that has no actual impact on the network performance is just rotten.

Monday, April 25, 2011

Non-destructive Drive Expansion in the StorCenter

If you can't tell from the series of posts I've already published, I'm having some fun playing with the iomega ix2-200 that I received from Chad Sakac of EMC. In reviewing those posts, I realized that I didn't publish anything on the trick to expanding the storage on the ix2 (which should also apply to all the models in the StorCenter series) without destroying your data.

This technique is fairly straightforward, and while it takes time and a bit of work at the command line via "Support mode", you will also be best served if all your data is still backed up. Note: to get at the support page on a "Cloud Edition" unit, the URL is /diagnostics.html


  1. Upgrade your drives with bigger models.
    1. It's not strictly required, but I suggest you upgrade starting at the highest labelled drive, working your way to the lowest labelled drive
    2. In order to make full use of each drive's capacity, they should all be identical.
    3. Shut down your unit each time you swap a drive (unless you're using a model that is explicitly hot-swappable).
    4. Allow the unit to fully redistribute the protected data before swapping the next spindle
  2. Enable SSH access to your unit.
    1. There's an unlinked page on the system called support.html; access it by putting the address directly into your browser's address bar.
    2. Check Allow remote access for support (SSH and SFTP) and click Apply
  3. Use an SSH client to logon to your unit
    • username: root
    • default password: soho
    • If you have security enabled, you will need to append the primary administrator's password to the default password. For example, if your primary user's password is ducksauce, the SSH password is sohoducksauce.

Magic Time!

The devices can be expanded because of the storage subsystem being used in the Linux kernel. The process is straightforward: expand the "outer" container before expanding the inner container(s).
  1. Dismount the user storage volume:
    root@ix2-200d:/# umount -l /mnt/soho_storage
  2. Expand the RAID pseudo-device:
    root@ix2-200d:/# mdadm --grow /dev/md1 --size=max
  3. Expand the LVM physical volume:
    root@ix2-200d:/# pvresize /dev/md1
  4. Determine the free space in the LVM volume group:
    root@ix2-200d:/# vgdisplay
    --- Volume group ---
      VG Name               md1_vg
      Free  PE / Size       476930 / 931.50 GB
  5. Expand the LVM logical volume by the amount of free blocks:
    root@ix2-200d:/# lvextend -l +476930 /dev/md1_vg/md1vol1
  6. Mount the expanded volume:
    root@ix2-200d:/# mount -t xfs -o noatime /dev/mapper/md1_vg-md1vol1 /mnt/soho_storage
  7. Expand the xfs file system:
    root@ix2-200d:/# xfs_growfs /dev/md1_vg/md1vol1
  8. Reboot the system (so the web management tools will recognize the expansion):
    root@ix2-200d:/# telinit 6
Your device will reboot and (if you have email notifications correctly configured) it will notify you that "data protection is being reconstructed", a side effect of expanding the outermost RAID container.

Wednesday, April 20, 2011

iomega "cloud edition" updates hardware, too

I've got several of iomega's NAS boxes from their ix product line. I was looking over the specs on the current "cloud edition" models to see what new "awesomesauce" was baked into the new firmware (hence making it a "personal cloud" device) and noticed that the hardware also underwent some quiet revisions. The ix2 is still shipping with 2 spindles and 256MB RAM, but the CPU has been updated from the old 200MHz ARM to a much-improved (but unnamed) 1GHz processor. The other models were similarly updated, with the ix4 getting a 1.2GHz CPU and the stellar ix12 getting an Intel E8400 (3GHz dual-core).

In general, it means the -200 and -300 designations no longer indicate the CPU speed as it did with their predecessors, the non-cloud editions.

UPDATE 14-June-2011

I've discovered that I've been in error in asserting that the older models (i.e., "non cloud") were 200MHz ARM units. Like the new units, the old ones are based on the Marvell Kirkwood ARM-compatible system-on-chip parts, which run at 1 or 1.2 GHz. I don't have enough data on the innards of the ix12 (old vs new), but I'm completely wrong on the ix2 & ix4...

Saturday, April 16, 2011

ix2-200 FrankenDisk Comparison

Okay, so you're a bit of a whack-job gadget geek, you read my previous post about upgrading an iomega ix2-200 by replacing the stock drives with SSDs, and now you want to know what sort of benefits might be gained for such a stunt.

Look no further... Here are my results.

The first chart is meant to make it clear that the improvements in replacing the 5900RPM spinning disk with SSD is marginal compared with the performance of putting the SSDs into a system that can really take advantage of them. Yes, the improvement is there, but the small, 200MHz ARM processor with 256MB RAM in the ix2-200 has nothing on a 2GHz quad-core with 8GB (my home desktop). But if you're interested in this experiment, you already knew that: you want to see how the ix2 fared...

The 4K read/write tests are meant to benchmark the outer-envelope for typical filesystem use: your standard Windows desktop is using NTFS with 4K clusters, so all your I/O is based on these units, whether or not you're actually reading/writing something of a different size. By doing 100% random, any cache in the system is pretty much overwhelmed, giving the real distinction between the two drive types.

The big-chunk sequential reads and writes represent workload that reflects video streaming (sequential read) or backup (sequential writes). Like the 4K random tests, this sort of workload also gets little benefit from cache and reflects the real throughput of  the drives.

Naturally, the reads fared better than the writes, and it's interesting to see the distinctions between the unit's iSCSI versus CIFS performance. In summary, you can better-than-double the performance of the device through upgrading to SSD.

Turbocharge your ix2-200 with SSD

The iomega ix2-200 is a nice little device for home or small-business NAS use, but it really suffers in the performance department: it only has a pair of 5900 RPM SATA drives under the hood, not to mention a slow (200MHz) CPU a single-core 1GHz CPU and only 256MB RAM. It's key benefit is the small package, which in turn equates to nice portability for a shared-storage device. However, if you're willing to break your warranty and get your hands dirty with a screwdriver and SSH client, you can turn it into something of a speed demon with some aftermarket upgrades.

Disclaimer: I don't advocate doing this; it's a bad idea for several reasons: you will sacrifice a ton of capacity (and cash) to gain performance improvements in a device that is really too underpowered to truly take advantage of the upgrade.

That said: I am also a gadget and hardware guy. I love playing with this stuff—especially storage and networking—and this was too much of an opportunity to miss. I did this more as an experiment to prove that a) it could be done and b) to see what kind of improvements were the result.

The ix2-200 itself will set you back at least $200 to start with, and 128GB SSDs (as used in the example) will also set you back $200 each plus the $20 each for the drive converters. For most users, you could find a much better way to spend $650-700 on shared storage. But if you need really, really fast, shared storage in a small, portable package—and capacity isn't as important as speed—this little Frankenstein's monster may be just the ticket.

Here's what you need, and how to get it done...


  1. Get your hands on an iomega ix2-200. If you can find a used/broken model, so much the better; and because we're gutting the factory drives, pick up the lowest-capacity model (1TB raw in 2 x 500GB) you can get your hands on. It must be able to boot into the management console from at least one drive; I don't have an image that you can lay down on a new replacement drive, and I don't advocate going onto the Internet in the hope you can find one that is truly "clean".
  2. Get a pair of the biggest SSD that you can afford. Don't worry about getting a "peak performance" model: even the slowest MLC-type units will still out-perform spinning disk, and still have more throughput than the ix2's SATA bus can handle. A matched pair is best because you can then use RAID0 for maximum performance.
  3. Assuming that your SSDs will be the 2.5" form-factor, get a pair of Icy Dock 2.5"-to-3.5" HDD Converters from your favorite reseller; I bought mine from Newegg.
    Note: Icy Dock has recently released a Dual 2.5"-to-3.5" Converter with built-in RAID-aggregation capability that would potentially allow you to put four SSDs into the unit. Depending on the way the ix2 recognized the drives, you could do JBOD (so the ix2 would "see" all 4 drives and permit you to run RAID5 in the GUI software), RAID1 (and use manually-built RAID0 in the ix2) or RAID0 (and use GUI-built Mirroring). I haven't tried it, so your mileage may vary...
  4. Assemble your new drives: put the SSDs in their Icy Dock converters
  5. If you're using an ix2 that you already own, back up all your data currently on it. What you're doing is decidedly destructive, and you will end up with less capacity. You don't have to worry about the system partitions, just the user data.

Replace the original drives with your SSDs...

  1. Change the protection method to "none". The only way the ix2 will allow you to do this is to remove all your shares and/or iSCSI volumes, and the only way to do that is to delete all your data (aren't you glad you backed everything up?).
  2. Shut down the ix2. The device infrastructure wasn't meant to support hot-swap.
  3. Remove one drive from the device (I always start with HDD2, but the system is designed so you could replace either one). Replace it with an SSD (in its converter, if needed).
  4. Restart the ix2. When it finishes booting and you can see the management dashboard, the unit will have replicated the system partition (/dev/md0) to the new drive, and should notify you that it recognizes the drive substitution. By removing the protection in Step 1, there's no wait while the system replicates the data partition, and the available data (as shown in the dashboard) should be roughly equivalent to the sum of the two drive sizes.
  5. Navigate to Disk Management in the control panel and verify that both disks are recognized by the system. If the new drive isn't recognized—the system displays no drive for the disk position you replaced, or the size is incorrect—you should stop and verify that the new drive is "good" in a regular PC. Even if the drive works, it is possible that the SATA implementation used by the SSD is not compatible with the ix2 hardware.
  6. Shut down the ix2. Replace the other drive.
  7. Restart the ix2. When it finishes booting, the appliance will have again replicated the system partition for reliability, and the remaining space on the drive will have been added to the pool of available storage.
  8. Navigate to Disk Management and verify that both disks are recognized and that you have the full storage capacity you were expecting.
Assuming everything went correctly for you, your ix2 will show the storage capacity as a single, unprotected pool of space. What it doesn't tell you is that the drives are assembled into a "linear" array: only after the first drive gets full will the other drive be used. You now have three choices:
  1. Live with it.
    You gain lots of speed by having the SSDs in there, but until you "fill up" the first disk's user data partition, that second one is doing nothing for you (although it is protecting the boot partition). If this is your choice, you're done. Enjoy using your device.
  2. Switch the protection mode to "Mirrored".
    You use both drives equally, and if one "dies", the other one carries on because it has an exact copy of the data on the failed drive. There is a write penalty for this when compared to "no protection"—but it's definitely worth it if your only backup is tedious and annoying to restore—and reads should be slightly faster than a single drive, as the appliance can perform simultaneous I/O on the drives. The major downside, of course, is that the second expensive disk you put in the ix2 isn't "doing" anything but keeping your data safe. Personally, I only worry about this sort of thing for the boot/system/app partitions on these sort of appliance systems, and that's covered automatically for the ix2. If you choose this option, you can make the change through the device GUI, and when it's finished applying protection, you're done and can start using your device.
  3. Convert the storage type to RAID0.
    On the negative side, RAID0 doesn't give any data protection, but then neither does linear: in either case, if one of the elements dies, so does all your data. On the positive side, however, all I/O will be distributed—in parallel—between both drives; there's no write penalty; you still get 100% of the available space for storage (no loss for protection). Too many people trust RAID "data protection" features to safeguard their data without having good backups. When you intentionally use RAID0, you know you have to keep good backups, so you force yourself to do so before trusting your data to the device. If you decide to choose this path, continue on...

Friday, April 15, 2011

Booting an NTFS-formatted thumbdrive

There are times when you'd like to boot from a thumbdrive that is formatted for NTFS. You dutifully follow instructions for copying bootable code to the drive, but it refuses to boot. What's wrong?
Unless you set the boot sector to make the drive bootable, no amount of "diddling" the data on the drive will make a difference.

So: how do you get the boot sector set?

Check your workstation to see if you have a copy of "bootsect.exe" on your system. If you do, you're already set; if you don't, you'll need to grab a copy from your installation media.

Once you have it, format your thumbdrive with NTFS, then issue the following command on the drive letter for the thumbdrive:
bootsect /nt60 x:
where "x:" is the letter for the thumbdrive:

Add your custom files, and the thumbdrive should behave as you'd expect.

Thursday, April 14, 2011

Drive Performance

UPDATE: more data from other arrays & configurations
In an earlier post I said I was building a table of performance data from my experimentation with my new iomega ix2-200 as well as other drive configurations for comparison. In addition to the table that follows, I'm also including a spreadsheet with the results:
The Corners 1 block seq read

4K random read

4K random write

512K seq write

512K seq read

local SSD RAID0 10400 2690 3391 63.9 350.6 2 x Kingston "SSD Now V-series" SNV425
ix2 SSD CIFS 3376 891 308 25.7 40.4 2 x Kingston "SSD Now V-series" SNV425
ix2 SSD iSCSI 4032 664 313 29.4 38.5 2 x Kingston "SSD Now V-series" SNV425
local 7200 RPM SATA RAID1 7242 167 357 94.3 98.1 2 x Western Digital WD1001FALS
ix4 7200RPM CIFS** 2283 133 138 32.5 39.4 4 x Hitachi H3D200-series;
**jumbo frames enabled
ix2 7200RPM CIFS 2362 125 98 9.81 9.2 2 x Hitachi H3D200-series
ix2 7200RPM iSCSI 2425 123 104 9.35 9.64 2 x Hitachi H3D200-series
ix4 7200RPM iSCSI** 4687 117 122 37.4 40.8 4 x Hitachi H3D200-series;
**jumbo frames enabled
ix4a stock CIFS 2705 112 113 24 27.8 4 x Seagate ST32000542AS
ix4 stock iSCSI 1768 109 96 34.5 41.7 4 x Seagate ST31000520AS
ix4a stock iSCSI* 408 107 89 24.2 27.2 4 x Seagate ST32000542AS;
*3 switch "hops" with no storage optimization introduce additional
ix2 stock CIFS 2300 107 85 9.85 9.35 2 x Seagate ST31000542AS
ix2 stock iSCSI 2265 102 84 9.32 9.66 2 x Seagate ST31000542AS
ix4 stock CIFS 4407 81 81 32.1 37 4 x Seagate ST31000520AS
DROBO PRO (iSCSI) 1557 71 68 33.1 40.5 6 x Seagate ST31500341AS + 2 x Western Digital
WD1001FALS; jumbo frames
DROBO USB 790 63 50 11.2 15.8 2 x Seagate ST31000333AS + 2 x Western Digital WD3200JD
DS2413+ 7200RPM RAID1/0 iSCSI 12173 182 194 63.53 17.36 2 x Hitachi HDS722020ALA330 + 6 x HDS723020BLA642
DS2413+ 7200RPM RAID1/0 NFS 2 x Hitachi HDS722020ALA330 + 6 x HDS723020BLA642
DS2413+ SSD RAID5 iSCSI 19238 1187 434 69.79 123.97 4 x Crucial M4

PX6-300 1 block seq read

4K random read

4K random write

512K seq write

512K seq read

Protocol RAID Disks
iSCSI none 1 16364 508 225 117.15 101.11
RAID1 2 17440 717 300 116.19 116.91
RAID1/0 4 17205 2210 629 115.27 107.75
6 17899 936 925 43.75 151.94
RAID5 3 17458 793 342 112.29 116.34
4 18133 776 498 45.49 149.27
5 17256 1501 400 115.15 116.12
6 18022 1941 106552.64149.1
RAID0 2 17498 1373 740 116.44 116.22
3 18191 1463 1382 50.01 151.83
4 18132 771 767 52.41 151.05
5 17692 897 837 56.01 114.35
6 18010 1078 1014 50.87 151.47
RAID66 17173 2563 870 114.06 116.37
Protocol RAID Disks 1 block seq read

4K random read

4K random write

512K seq write

512K seq read

NFS none 1 16146 403 151 62.39 115.03
RAID1 2 15998 625 138 63.82 96.83
RAID1/0 4 15924 874 157 65.52 115.45
6 16161 4371 754 65.87 229.52
RAID5 3 16062 646 137 63.2 115.15
4 16173 3103 612 65.19 114.76
5 15718 1013 162 59.26 116.1
6 16161 1081 201 63.85 114.63
RAID0 2 15920 614 183 66.19 114.85
3 15823 757 244 64.98 114.6
4 16258 3769 1043 66.17 114.64
5 16083 4228 1054 66.06 114.91
6 16226 4793 1105 65.54 115.27
RAID66 15915 1069 157 64.33 114.94

About the data

After looking around the Internet for tools that can be used to benchmark drive performance, I settled on the venerable IOmeter. Anyone who has used it, however, knows that there is an almost infinite set of possibilities for configuring it for data collection. In originally researching storage benchmarks, I came across several posts that suggest IOmeter along with various sets of test parameters to run against your storage. Because I'm a big fan of VMware, and Chad Sakac of EMC is one of the respected names in the VMware ecosystem, I found his blog post to be a nice start when looking for IOmeter test parameters. His set is a good one, but requires some manual setup to get things going. Also in my research, I came across a company called Enterprise Strategy Group which not only does validation and research for hire, they've published their custom IOmeter workloads in an IOmeter "icf"configuration file. The data published above was collected using their workload against a 5GB iobw.tst buffer. While the table above represents "the corners" for the storage systems tested, I also captured the entire result set from the IOmeter runs and have published the spreadsheet for additional data if anyone is interested.

px6-300 Data Collection

The data in the px6-300 tables represents a bit of a shift in the methodology: the original data sets were collected using the Windows version of iometer, while the px6-300 data was collected using the VMware Labs ioAnalyzer 1.5 "Fling". Because it uses the virtual appliance, a little disclosure is due: the test unit is connected by a pair of LACP-active/active 1Gb/s links to a Cisco SG-300 switch. In turn, an ESXi 5.1 host is connected to the switch via 4x1Gb/s links, each of which has a vmkernel port bound to it. The stock ioAnalyzer's test disk (SCSI0:1) has been increased in size to 2GB and is using an eager-zeroed thick VMDK (for iSCSI). The test unit has all unnecessary protocols disabled and is on a storage VLAN shared by other storage systems in my lab network. The unit is otherwise idle of any workloads (including the mdadm synchronization that takes place when configuring different RAID levels for disks, a very time-consuming process); there may be other workloads on the ESXi host, but DRS is enabled for the host's cluster, and if CPU availability were ever an issue in an I/O test (it isn't), other workloads would be migrated away from the host to provide additional resource.

The Takeaway

As expected, the SSD-based systems were by-far the best-performing on a single-spindle basis. However, as one might expect, an aggregate of spindles can provide synergy that meets or exceeds the capability of SSD, and locally-attached storage can also make up the difference in I/O performance. The trade-off, of course, is the cost (both up-front and long-term) versus footprint.

Wednesday, April 13, 2011

iomega StorCenter drive swaps

In my previous post about the iomega ix2, I wasn't sure that resizing the storage could be done without shell access by using GUI commands. I've now played with it enough to know that a) you can, but it is a destructive operation and b) the same technique can be used to shrink the available storage instead of grow it.

First, the technique:
  1. Swap in your new drive.
  2. Let it rebuild.
  3. Swap in the second drive.
  4. Before it gets done rebuilding, use the "delete disk" option:

    You'll have to jump through a bunch of hoops to make this happen, least of which is deleting all the data and shares (like I said, it's a destructive technique!), but once it's going, you might not get any indication that it's actually "doing" anything.
  5. Restart the unit
When it starts, if you have email alerts configured, you'll get a message that it's rebuilding the storage. Log back into the GUI and check your Dashboard; it should now show the new, increased capacity.

So why would anyone ever want to go smaller with one of these units? Well, suppose you had a pair of SSD drives that you picked up for cheap from Woot! or some other retailer. And suppose you picked up a pair of Icy Dock 2.5"-to-3.5" HDD Converters so that they'd fit correctly in the drive frame for the ix2? Well, you would then have the opportunity to create a smoking fast NAS with those SSDs.

I don't know enough about the StorCenter's Linux kernel (or Linux kernels in general) to tell if the unit can or does use TRIM to keep the write speeds optimal. But let's be fair: even without it an SSD blows away spinning disk at any speed. Given the cost/capacity ratio of SSDs, however, you'd have to be pretty starved for performance to try such a thing—and would certainly be better served by putting SSDs in a higher performance box than an ix2-200!