Tuesday, November 8, 2016

Virtual SAN Cache Device upgrade

Replacing/Upgrading the cache+buffer device in VSAN

Dilemma: I've got a VSAN cluster at home, and I decided to switch from single diskgroups-per-host to dual to give myself a bit more availability as well as additional buffer capacity (with all-flash, there's not much need for a read cache).

My scenario has some unique challenges for this transformation. First, although I already have the new buffer device to head the new disk group, I don't actually have all the new capacity disks that I'll need for the final configuration: I'll need to use some of the existing capacity disks if I want to get the second disk group going before I have the additional capacity devices. Second, I have insufficient capacity in the remainder of the VSAN datastore to perform a full evacuation while still maintaining policy compliance (which is sort of why I'm looking to add capacity in addition to splitting the one disk group up).


The nominal way to perform my transformation is:
  1. Put the host into maintenance mode, evacuating all registered VMs
  2. Delete the disk group, evacuating the data so all VMs remain storage policy-compliant.
  3. Add the new device
  4. Rebuild disk group(s)
I already took a maintenance outage during the last patch updates and added my new cache+buffer device to each host, so "Step 3" is already completed.
And then I hit on something: While removing the buffer device from a diskgroup will cause the decommissioning of the entire disk group, individual capacity devices can be removed without upsetting more than the objects being stored on that device alone. I have sufficient capacity in the remainder of the disk group—not to mention on the other hosts in the cluster—to operate on individual capacity elements.

So, here's my alternative process:

  1. Remove one capacity device from its disk group with full migration

  2. Add the capacity device to the new disk group.

It takes longer because I'm doing the evacuation and reconfiguration "in series" rather than "in parallel," but it leaves me with more active & nominal capacity+availability than doing it on an entire diskgroup at once.

My hosts will ultimately have two disk groups, but they'll break one "rule of thumb" by being internally asymmetric: My buffer devices are 400GB and 800GB NVMe cards, respectively, so when I'm fully populated with ten (10) 512GB capacity disks in each host, four (4) will be grouped with the smaller and six (6) will be grouped with the larger. When you keep in mind that Virtual SAN won't use more than 600GB of a cache+buffer device regardless of its size, it actually has some internal symmetry: each capacity disk will be (roughly) associated with 100GB of buffer, for a ~5:1 buffer:capacity ratio.

CLI alternative

Although this entire process can be performed using the Web Client, an alternative is to write a CLI script. The commands needed are all in the esxcli storage or vsan namespaces; combined with some shell/PowerShell scripting, it is conceivable that one could:
  • Identify storage devices.
    esxcli storage core device list
  • Identify any existing disk group, cache+buffer and capacity devices
    esxcli vsan storage list.
  •  Remove one of the capacity disks with migration
    esxcli vsan storage remove -d <device> -m evacuateAllData
  • Create a new disk group using an available flash device from the core device list as the new group's cache+buffer device, and the recently evacuated device as the capacity device
    esxcli vsan storage add -s <cache+buffer device> -d <device>
  • Loop through the remaining capacity devices, first removing then adding them to the new disk group. The esxcli vsan storage remove command is blocking when run from the ESXi console, so your script should wait for full evacuation and availability before the next step in the script is executed.

Thursday, October 13, 2016

Adding floppy for PVSCSI drivers when creating a VM in vCenter Web Client

Someone asked in a private slack channel if it was "just him" or can you really not add a floppy image when creating a VM using the Web Client. This is relevant any time you want to build a VM using the PVSCSI drivers so they'll always be available, even if VMware Tools is uninstalled.
The answer—at least with v6.0U2—is "no."
In this scenario, the vmimages folder won't expand; it offers the "arrowhead" showing there is content to be discovered within, but when you select it, you get no content...

Fortunately, there's a workaround: if you go ahead and save the new VM (without powering on) and then edit it, modifying the source for the floppy image, the vmimages folder will correctly expand and populate, allowing you to select one.

UPDATE: It turns out we were talking about two different Web Clients! My assumption was that we were referring to the vCenter Web Client, while the person asking was referring to the new(ish) Host Web Client.

The defect and workaround as I've documented it only apply to the vCenter Web Client. The Host Web Client will not behave correctly even in the workaround; this is a solid defect. There are other workarounds—use the C# client, copy the IMG file to an accessible datastore, etc.—but none are as good as the defect being eliminated in the first place.

Friday, February 26, 2016

NTFS, dedupe, and the "large files" conundrum.

Microsoft did the world a huge favor when they added the deduplication feature to NTFS with the release of Windows Server 2012. We can have a discussion outside of this context on whether inline or post-process dedupe would have been better (the NTFS implementation is post-process), but the end result is something that seems to have minimal practical impact on performance but provides huge benefits in storage consumption, especially on those massive file servers that collect files like a shelf collects dust.

On the underside, the dedupe engine collects the duplicate blocks and hides them under the hidden "System Volume Information" folder and leaves pointers in the main MFT. You can do a disk size scan and see very little on-disk capacity taken by a given folder, yet a ginormous amount of disk is being consumed in that hidden folder.


See that little slice of color on the far left? That's the stub of files that aren't sitting in the restricted dedupe store. The statistics tell a different story:


200GB of non-scannable data (in the restricted store) versus 510MB stored in the "regular" MFT space. Together they comprise some 140K files in 9K folders, and the net action of dedupe is saving over 50GB in capacity on that volume:


The implementation is fairly straightforward, and I've found few instances where it didn't save the client a bunch of pain.

Except when used as a backup target.

Personally, I though this was the perfect use case—and it is, but with the caveats discussed herein—because backup tools like Veeam can perform deduplication within a backup job, but job-to-job deduplication isn't in the cards. Moving the backup repository to a deduplicating volume would save a ton of space, giving me either space to store more data or more restore points for existing backups.

Unfortunately, I ran into issues with it after running backups for a couple of weeks. Everything would run swimmingly for a while, then suddenly backups would fail with filesystem errors. I'd wipe the backup chain and start again, only to have it happen again. Fed up, I started searching for answers...

Interestingly, the errors I was receiving (The requested operation could not be completed due to a file system limitation.) go all the way back to limitations on NTFS without deduplication, and the early assertions by Microsoft that "defragmentation software isn't needed with NTFS because it protects itself from fragmentation." Anyone else remember that gem?!? Well, the Diskeeper folks were able to prove that NTFS volumes do, in fact, become fragmented, and a cottage industry of competing companies popped up to create defrag software. Microsoft finally relented and not only agreed that the problem can exist on NTFS, but they licensed a "lite" version of Diskeeper and included it in every version of Windows since Windows 2000. They also went so far as to add additional API calls to the filesystem and device manager so that defragger software could better operate in a safe manner than "working around" the previous limitations.

I digress...

The errors and the underlying limitation have to do with the way NTFS handles file fragmentation. It has special hooks to readily locate multiple fragments across the disk (which is, in part, why Microsoft argued that a fragmented NTFS volume wouldn't suffer the same sort of performance penalty that an equivalently-fragmented FAT volume would experience), but the data structures to hold that information is a fixed resource. Once volume fragmentation reaches a certain level, the data structures are exhausted and I/O for the affected file is doomed. The fix? Run a defragger on the volume to free up those data structures (every fragment consumes essentially one entry in the table, so the fewer fragments that exist, the fewer table resources are consumed, irrespective of total file size) and things start working again.

Enter NTFS deduplication

Remember that previous description of how the dedupe engine will take duplicate blocks from the volume—whether they're within a single file or across multiple—and put it in the System Volume Information folder, then leave a pointer in the main MFT to let multiple files (or the same file) access to that block?

Well, we just deliberately engineered a metric crapton (yes, that's a technical description) of intentional fragmentation on the volume. So when individual deduplicated files grow beyond a certain size (personal evidence says it's ~200GB, but posts I've found here and there say it's as little as 100GB while MS says it's 500GB https://support.microsoft.com/en-us/kb/2891967) you can't do anything with the file. Worse, defrag tools can't fix it, because this fragmentation isn't something that the algorithms can "grab"; the only real fix—other than throwing away the files and starting over—is to disable dedupe. And if you're near the edge of capacity due to the benefit of dedupe, even that's no option: rehydrating the file will blow past your capacity. Lose-lose.

Luckily, Microsoft identified the issue and gave us a tool when building volumes intended for deduplication: "large files" flag in the format command. Unfortunately, as you might guess when referring to "format," it's destructive. The structures that are laid down on the physical media when formatting a volume are immutable in this case; only an evacuation and reformat fixes the problem.

Given that restriction, wouldn't it be helpful to know if your existing volumes support large files (ie extreme fragmentation) before you enable deduplication? Sure it would!

The filesystem command "fsutil" is your friend. From an administrative command prompt, run the following command + arguments (this is an informational argument that makes no changes to the volume, but requires administrative access to read the system information):

fsutil fsinfo ntfsinfo <drive letter>



Notice the Bytes Per FileRecord Segment value? On a volume that does not support high levels of fragmentation, you'll see the default value of 1024. You'll want to reformat that volume with the "/L" argument before enabling dedupe for big backup files on that bad boy. And no, the ability to do that format argument is not available in the GUI when creating a new volume; you've got to use the command line.

What does it look like after you've reformatted it? Here you go:


The Bytes Per FileRecord Segment value jumps up to the new value of 4096.

You'll still want to adhere to Microsoft's dedupe best practices (https://msdn.microsoft.com/en-us/library/windows/desktop/hh769303(v=vs.85).aspx), and if you're reformatting it anyway, by all means make sure you do it with the 64K cluster size so you don't run into any brick walls if you expect to expand the volume in the future. Note that the fsutil command also shows the volume's cluster size (Bytes per Cluster) if you're wanting to check that, too.

Special thanks to fellow vExpert Frank Buechsel, who introduced me to using fsutil for this enquiry.