tag:blogger.com,1999:blog-77649631364145416842024-03-18T05:27:15.462-05:00brain flossDribs and drabs of stuff that others might find useful, too.Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.comBlogger108125tag:blogger.com,1999:blog-7764963136414541684.post-3742865822388311602022-06-07T13:53:00.005-05:002022-07-21T08:15:10.518-05:00Synology DSM and Veeam 11<p>For a long time, Veeam has been telling its users to <b>not</b> use "low-end NAS boxes" (eg, Synology, QNAP, Thecus) as backup repositories for Backup & Replication (VBR), even though these Linux-based devices should be compatible if they have "x86" architecture (as opposed to ARM).</p><p>The reality is that none of these devices use "bog standard" Linux distributions, and due to their appliance-based nature, have some significant limitations on what can be done to their custom distributions.</p><p>However, there are many folks—both as home users or within small/budget-limited businesses—who are willing to "take their lumps" and give these things a shot as repositories.</p><p>I am one of them, particularly for my home "lab" environment. I've written about this use case (in particular, the headaches) a couple of times in this blog [<a href="https://blog.millard.org/2014/11/use-synology-as-veeam-b-linux-repository.html" target="_blank">1</a>, <a href="https://blog.millard.org/2014/11/repair-synology-dsm51-for-use-as-linux.html" target="_blank">2</a>], and this post joins them, addressing yet another fix/workaround that I've had to implement.</p><h4 style="text-align: left;">Background</h4><div>I use a couple of different Synology boxes for backup purposes, but the one I'm dealing with today is the DS1817+. It has a 10GbE interface for connectivity to my network, a quad-core processor (the Intel Atom C2538) and 8GB RAM (upgradable to 16GB, but I haven't seen the demand that would require it). It is populated with 8x1TB SATA SSDs for ~6TB of backup capacity.</div><div><br /></div><div>I upgraded DSM to 7.0 a while back, and had to make some adjustments to the NFS target service to continue to support ESXi datastores via NFS 4.1</div><div><br /></div><div>Yesterday, I updated it to 7.1-42661 Update 2, and was greeted to a number of failed backup jobs this morning.</div><div><br /></div><h4 style="text-align: left;">Symptoms</h4><div>All the failed jobs have uniform symptoms: <i>Timeout to start agent</i></div><div><i><br /></i></div><div>With further investigation, I saw that my DS1817+ managed server was "not available", and when attempting to get VBR to re-establish control, kept getting the same error with the installation of transport services:</div><div><br /></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div></div></blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><div style="text-align: left;"><span style="font-family: courier;">Installing Veeam Data Mover service Error: Failed to invoke command /opt/veeam/transport/veeamtransport --install 6162: /opt/veeam/transport/veeamtransport: error while loading shared libraries: libacl.so.1: cannot open shared object file: No such file or directory</span></div><div style="text-align: left;"><div><br /></div></div><span style="font-family: courier;">Failed to invoke command /opt/veeam/transport/veeamtransport --install 6162: opt/veeam/transport/veeamtransport: error while loading shared libraries: libacl.so.1: cannot open shared object file: No such file or directory</span></blockquote><h4 style="text-align: left;">Workaround</h4><h4 style="text-align: left;"><span style="font-weight: normal;">After failing to find a fix after some Linux-related searches, I discovered a thread on the Veeam Community Forum that addressed this exact issue [<a href="https://forums.veeam.com/veeam-backup-replication-f2/veeam-v11-and-synology-linux-repository-t72376.html" target="_blank">3</a>].</span> </h4><div>This is apparently a known issue with VBR11 and Synology boxes, and as Veeam is moving further and further away from the "on the fly" deployment of the transport agent to a permanently-installed "Data Mover" daemon (which is necessary to provide the Immutable Backup feature), it becomes a bigger issue. Veeam has no control over the distribution—and would just as soon have clients use other architectures—and Synology would probably be happy with customers considering their own backup tool over competing options...</div><div><br /></div><div>At any rate, some smart people posted workarounds to the issue after doing their own research, and I'm re-posting for my own reference because it worked for me.</div><div><br /></div><div><ol style="text-align: left;"><li>Download the latest ACL library from Debian source mirrors. The one I used—and the one in the Forum thread—is <a href="http://ftp.debian.org/debian/pool/main/a/acl/libacl1_2.2.53-10_amd64.deb" target="_blank">http://ftp.debian.org/debian/pool/main/a/acl/libacl1_2.2.53-10_amd64.deb</a></li><li>Unpack the .deb file using 7zip</li><li>Upload the data.tar file to your Synology box. Feel free to rename the file to retain your sanity; I did.</li><li>Extract the tarball to the root directory using the "-C /" argument:<br /><span style="font-family: courier;">tar xvf data.tar -C /</span></li><li>If you are using a non-root account to do this work, you'll need to use "sudo" to write to the root. You will also need to adjust owner/permissions on the extracted directories & files:<br /><span style="font-family: courier;">sudo tar xvf data.tar -C /<br />sudo chown -R root:root /usr/lib/x86_64-linux-gnu<br />sudo chmod -R 755 /usr/lib/x86_64-linux-gnu</span></li><li>Create soft links for these files in the boxes filesystem:<br /><span style="font-family: courier;">sudo ln -sf /usr/lib/x86_64-linux-gnu/libacl.so.1 /usr/lib/libacl.so.1<br />sudo ln -sf /usr/lib/x86_64-linux-gnu/libacl.so.1.1.2253 /usr/lib/libacl.so.1.1.2253</span></li><li>Last, get rid of any previous "debris" from failed transport installations<br />sudo rm -R /opt/veeam</li></ol><div>Once the Synology is prepped, you must go back into VBR and re-synchronize with the Linux repository:</div></div><div><ol style="text-align: left;"><li>Select the "Backup Infrastructure" node in the VBR console</li><li>Select the Linux node under Managed Servers</li><li>Right-click on the Synology box being updated and select "Properties..." from the popup menu.</li><li>Click [Next >] until the only option is [Finish]. On the way, you should see that the Synology is correctly identified as a compatible Linux box, and the new Data Mover transport service is successfully installed.</li></ol><h4 style="text-align: left;">Summary</h4></div><div>I can't guarantee that this will work after a future update of DSM, and there may come a day when other libraries are "broken" by updates to VBR or DSM. But this workaround was successful for me.</div><div><br /></div><div><div><h4>Update</h4></div><div>The workaround has persisted through a set of updates to DSM7. I have seen this come up with DSM6, but this workaround <u>does not</u> work on that; too many platform incompatibilities, I suspect. Need to do some more research & experimentation for DSM6...</div></div>Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com41tag:blogger.com,1999:blog-7764963136414541684.post-68932736964512311042020-02-28T00:54:00.000-06:002020-02-28T00:54:14.951-06:00Update: maintaining the pi-hole HA pairIn an <a href="http://blog.millard.org/2020/02/putting-pi-hole-to-work.html" target="_blank">earlier post</a>, I shared how I got pi-hole working in my environment, thanks to a number of posts on a <a href="https://www.reddit.com/r/pihole/comments/d5056q/tutorial_v2_how_to_run_2_pihole_servers_in_ha/" target="_blank">reddit thread</a>. Since then, I've been living with the setup and tweaking my configuration a bit.<br />
<br />
This post documents one of the tweaks that might be useful for others...<br />
<br />
If you're using the method documented by <a href="https://www.reddit.com/user/Panja0/" target="_blank">Panja0</a>, you know that there's a script in the pi-hole distribution (gravity.sh) that must be edited in order to synchronize files between the nodes of the HA pair. Well, he reminds you in the tutorial that it'll need to be <i>re-edited every time you update</i> pi-hole, or the synchronization won't occur.<br />
<br />
As you might guess, I didn't remember when I updated a week ago, and couldn't understand why my settings weren't getting synchronized. So I went back to the post, reviewed my settings, and face-palmed myself when I discovered my oversight. <i>I had failed to re-edit gravity.sh</i><br />
<br />
After I did the necessary edits, I realized that, even if I'd remembered about it, I'd <i>still</i> need to refer to the original post to get the right command line, etc., for the edits.<br />
<br />
I didn't want to spend the time to figure out how to trigger a script to make the update for me upon a pi-hole update, but I sure could figure out the script to do the correct updates!<br />
<br />
I mean... come on: what better use of automation than to use a script to a) check to see if the update has already been performed, and b) if not, perform the update?<br />
<br />
<pre><code style="background-color: #eeeeee; border: 1px solid #999; display: block; padding: 20px;">#!/bin/bash
# make sure the pihole-gemini script is being run by gravity.sh
GEMINI='su -c /usr/local/bin/pihole-gemini - <gemini user>'
GRAVITY=/opt/pihole/gravity.sh
TRIGGER=$(sed -e '$!{h;d;}' -e x $GRAVITY)
if [ "$TRIGGER" != "$GEMINI" ]
then
# insert the gemini commandline before the last line of the script
sed -i "$ i$GEMINI" $GRAVITY
fi
</code></pre>
<br />
If you decide to use the script, just make sure that you make any necessary modifications for the first two script variables to match your installation. You also need it on both nodes of your HA pair!<br />
<br />
In my setup, I'm saving this script in the /etc/scripts directory, which I'm using for other "keepalived" scripts. I'll remember to run it next time I update pi-hole, and that's all I'll need to recall!Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com184tag:blogger.com,1999:blog-7764963136414541684.post-24440593260637861052020-02-01T00:16:00.001-06:002020-02-01T00:16:45.157-06:00Putting Pi-hole to workI've been reading about my friends' use of <a href="https://pi-hole.net/" target="_blank">Pi-hole</a> on their home networks, and I've been curious about trying it to see how well it does. I've resisted doing so, primarily because of the single point of failure a pi-hole system represents: if it's unavailable, you get no DNS.<br />
<br />
And we all know, it's never DNS...except when it is.<br />
<br />
An alternative, naturally, it to run a pair of systems. Why not? Raspberry Pi devices are relatively cheap, and the software is all no-charge.<br />
<br />
For most home users, that might be fine, but I <a href="http://blog.millard.org/p/home-lab.html" target="_blank">run a lab</a> in my home that also provides services to the household, so I had more permutations to worry about: what happens if my Pi dies? what happens if my domain controllers are unavailable? Etc.<br />
<br />
The solution I've settled on is to run a primary Pi-hole server as a VM in my lab environment—which gives me more than enough performance and responsiveness, even under the most demanding of situations—and a secondary with a Raspberry Pi, so that even if the VM environment goes "pear shaped," I still get DNS resolution.<br /><br />
In order to accommodate several types of outages, yet avoiding the need to both double-up the configuration work (with the potential of missing an update and having weird results to troubleshoot) while providing pre-configured support for a couple of likely failure and maintenance scenarios, I've mated the two systems together in a failover cluster by configuring the "keepalive" daemon along with some scripting to keep the two systems in sync for the blocking function, while leaving some configuration elements (upstream DNS servers for one) independent of each other.<br />
<br />
I didn't do the "heavy lifting" on the sync and keepalive aspects; those were provided by reddit user <a href="https://www.reddit.com/user/Panja0/" target="_blank">Panja0</a> in this post: <a href="https://www.reddit.com/r/pihole/comments/d5056q/tutorial_v2_how_to_run_2_pihole_servers_in_ha/">https://www.reddit.com/r/pihole/comments/d5056q/tutorial_v2_how_to_run_2_pihole_servers_in_ha/</a><br />
<br />
I'm running ubuntu server 19.10 (Eoan Ermine... whatever) instead of Raspbian Stretch/Buster, so there have been a number of changes I've had to make to the systems to adapt:<br />
<br />
<ul>
<li>To get keepalived installed, I needed libipset11, not libipset3 (mentioned in the comments of the HA tutorial)</li>
<li>I had to modify the rsync command arguments in the synchronization script due to changes between Debian versions that I'm running versus the original post (mentioned in the comments of the HA tutorial)</li>
<li>I had to permit my <a href="https://phpraxis.wordpress.com/2016/09/27/enable-sudo-without-password-in-ubuntudebian/" target="_blank">rsync user to skip password re-aut</a>h by editing the sudoers file; I think this may also be a version-specific issue.</li>
<li>I <a href="https://www.tecmint.com/install-ntp-server-and-client-on-ubuntu/" target="_blank">added an NTP client</a> to utilize my GPS-based hardware time server; this is super important when using a Raspberry Pi without a real-time clock HAT add-on.</li>
<li>The primary system uses the lab's DNS (domain controllers) for its upstream DNS servers. In addition to avoiding the need to configure additional conditional forwarding rules for dnsmasq, this gives the Pi-hole server the identity of the clients via DNS</li>
<li>The secondary uses OpenDNS servers—I have a household account with several filtering options enabled already—with a dnsmasq configuration for conditional forwarding on the domain.</li>
</ul>
<div>
Given my homelab, it was pretty trivial to set this up as a VM, but what really sold it to me was getting the Raspberry Pi running in concert. I originally started with a Pi 3 Model B that I had lying around after an old project that I'd quit, but the performance difference between the two platforms was so noticeable that going with a true primary/secondary setup made the most sense. I considered upgrading to the Pi 4, but decided that my desire to avoid purchasing micro-HDMI adapters outweighed the value in the more-robust, newer model. I did decide to go ahead and upgrade from the 3 to the 3+, however, when I discovered that my local MicroCenter had them for $34US; I also paired the new unit with a passive heatsink case, which has allowed the Pi to run significantly cooler (30°F) than the original setup, which utilized aluminium heatsinks and a non-vented plastic case.</div>
<div>
<br /></div>
<div>
Aside from this "vanilla" setup, I also took note of the additional block lists that my friend <a href="https://tsmith.co/about/" target="_blank">Tim Smith</a> wrote about in <a href="https://tsmith.co/2018/setting-up-additional-pi-hole-blocklists-forwarding-for-homelab/" target="_blank">a blog post</a>. I need to let this "bake" for a while before considering it finished, but I'm liking what I'm seeing so far.</div>
Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com137tag:blogger.com,1999:blog-7764963136414541684.post-70221326293194085712019-09-19T21:46:00.000-05:002019-09-19T21:46:17.439-05:00New VM cleanupWhen creating a new VM in vSphere, you get a number of virtual devices & settings by default that you probably don't have any interest in keeping:<br />
<br />
<ul>
<li>Floppy drive (depending on version & type of client in use)</li>
<li>Floppy adapter</li>
<li>IDE ports</li>
<li>Serial ports</li>
<li>Parallel ports</li>
</ul>
<div>
Given that some of these are redundant (why keep the IDE adapter when you're using SATA for the optical device?) while others are polled I/O in Windows (OS must keep checking to see if there's activity on the port, even if there will never be any), it just makes things more streamlined if you cleanup these settings when creating a new VM...then using the cleaned-up VM as a template for creating new VMs later on.</div>
<div>
<br /></div>
<div>
Step 1: create a new VM</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="724" data-original-width="875" height="528" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZWbagUnhi6wkRlpuAvb6J4BH9FEnzDAqa5eEWS5QRj1v0exOl2fWN89UFqFiCxrhs35s6MFP5a5a1pYEtwU9an6GiOpfJxC50edtqH7qhIXX6uMgj2a3n4oGEOVlpOEJBqJFKGsMIC-pE/s640/Step+1+create+a+new+vm.png" width="640" /></div>
<div>
Step 2: Set VM name and select a location</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="723" data-original-width="867" height="532" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzoOqJFluXnawYnUkMjNk_MghNpQMjgnW9VukEPVUURCU6LmP8miCpLtlLMNqUzlYcdCv96VN0M1bpC40csHiahbkdAymtyXEGI7i0Zl2yj-lDaNn_GtzqOqJ6cLDvVNqz5mDS_GTwEQfp/s640/Step+2+set+name+and+folder.png" width="640" /></div>
<div>
Step 3: Select a compute resource</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="721" data-original-width="874" height="526" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOHvvccrK70dUlvfIdXmGdZLnxpVAbbPhWaZcF-WHAmQBNbebrZyBYpy-0tH6o5zXEf3kFUb0FrDfun3AxdhRRQ-RcjREn6BeqRGLYmFtmGjcGUkY5ZRJoMjFs2nUJDSKADW0VCmCWV8sJ/s640/Step+3+set+compute+resource.png" width="640" /></div>
<div>
Step 4: Select storage</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="723" data-original-width="873" height="528" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCF3DCfZWrx6RacUtgAHoAoPrUDR2Npnoyr8WBnSvStZOokBEkFC8tws861wqqCZbLeb0dTJQQQSgImNDj2gqveUtluc9HcQvogdje-rUrs-9N5SnyMRmNoSEkrnjKL-dTHhHmkPUMRLyd/s640/Step+4+set+storage.png" width="640" /></div>
<div>
Step 5: Set compatibility no higher than your oldest version of ESXi that the template could be deployed on.</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="725" data-original-width="872" height="532" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFGLcUpvoHnzmBVI2CxJEGZaZBM-KA-CPhSiOcWB6N4LK9DcwgbjAGPBl5Nu8d80n-aI7if12-jWfaSUuDjXxG7eZW7__IGl2_BdF41V1IZxxIg6YqCNC5vL3OJ6grJbB2T0NKkBiwz03g/s640/Step+5+set+compatibility.png" width="640" /></div>
<div>
Step 6: Select the guest OS you'll install</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="721" data-original-width="875" height="526" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirY265k_LsOtbTlAJ8VZXTcZy4JK4DFdU_c0hsLtqNsr7kYYop8NNrE40WVjxLVQ0ntn1UxcF9vXvtnhP27b97W-k8dSFjKqAGHdMKRilrwF0fgZswZ5NEo8YcBcBpTGjTBhUbXuCdGpyQ/s640/Step+6+set+guest+OS.png" width="640" /></div>
<div>
Step 7a: Customize hardware: CPU, Memory, Hard Drive</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="730" data-original-width="877" height="532" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzu6OjKvPbaSooj2ow7B9LZIpJKAj7CuyDuhWAhEhRLpX1oewIcGT-xQjR2nJyLQyvromTjhthhB3L_MDJCWyIvbaJEzFJBGqcTwbMGprBQ0lpTgAjFmqtRpD5WhzlRkMkQmOeM8_k77PM/s640/Step+7+edit+hardware.png" width="640" /></div>
<div>
Step 7b: Attach NIC to a general-purpose or remediation network port</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="471" data-original-width="583" height="516" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj4qK2HNer5lB0gvW_3obVNBaKI9NLf47u6zoskNDD1gAMAZF1A9dv7LYeeF5sxULwYgMQbuu1qj7P8HfIFz1U6Z5iJVfeQM2cgEXlZgU1s88mM5sSIuFBRHPlAcES1LD45AomMz1aVAVCv/s640/Step+7+edit+hardware-set+network.png" width="640" /></div>
<div>
Step 7c: Don't forget to change <b>the NIC type!</b> If you don't the only way to change it later is to remove & re-add the correct type, which will also change the MAC address and, depending on the order you do the modifications, could put the new virtual NIC into a different virtual PCIe slot on the VM hardware, upsetting other configurations in the guest (like static IP addresses).</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="721" data-original-width="877" height="524" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9rySSLmWYQnN9_L9xl1-gYobYweXF3BKtlc0awrFKYIC0IYXyM54Ocpy54vR6gWg_TfvOIZHeNILltMDamCNt0xbQvGj2jJaTVWxLJZzJFSYuAk59h2fp1VP0-k3tPMGl0UFD2CYsDJWV/s640/Step+7+edit+hardware-select+NIC+type.png" width="640" /></div>
<div>
Step 7d: Jump to the Options tab and set "Force BIOS setup"</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="726" data-original-width="875" height="530" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6-G4EWGeVxe35dXqhH4EC_xubIVsObgOCIYIelE01PFzICNOBE7G65sZW5imoARfidvw0qeB0bZVpLZz8wsdDOO8yx8yqagyOO2cEyjgEijEjRcZbPZXA1QQYBJKXOmQXvSzDIETZaC8R/s640/Step+7+edit+hardware-force+bios+setup.png" width="640" /></div>
<div>
Step 8: Finish creating the VM</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="725" data-original-width="876" height="528" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyYc7pEB9e5F8n_FoIFjksGkH5RTdnU_7xab6VYOWhcQit4-p7tf4yuB5fo8vhvNnV8jgzuH2uKyErDbE3yhFdyccrWzOZvj1pZFR62-3WqE_KSv7EfBt9dIbdqqmUHK6vMfCHzqdedNil/s640/Step+8+finish.png" width="640" /></div>
<div>
Step 9: Open remote console for VM</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="249" data-original-width="341" height="233" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjK1H9jNjTDSAvS2siO_9YTyqwUBKT9Gr41PRhxTI6daoAYDeNwOdYRfxDrpI1qnLHLFOcWPKLMbQu27ScYO-I-R5xW03Ut3cDJQ7PHsDAWjTMzLN_pTMW91zOnSq_JB8pmRp8ID8zcM9yn/s320/Open+remote+console.png" width="320" /></div>
<div>
Step 10: Power On the VM. IT should pause at the BIOS editor screen.</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="214" data-original-width="346" height="197" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3WfEhb3fahRUV32T8G50XOXVdHTACNbFIMqJIY6YXOqL5Px2vGSnrT4bhGeuynetICQN3qJeH6NZx_IEhPPY_0oLUubNHVj5Ggk7cN8s3SodMiqDDNXCSy_RixUA-9-dxsGJSgA0USU-U/s320/power+on+VM.png" width="320" /></div>
<div>
Step 11: On the Advanced page, set Local Bus IDE to "Disabled" if using SATA; set it to "Secondary" if using IDE CD-ROM (Even better: Change the CD-ROM device to IDE 0:0 and set it to "Primary").</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="249" data-original-width="365" height="272" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJhVkLWMmi0VAOn9Bz-UZ74X1Qb6qsk0pLhJIZQjPqs-QAnLhz1sMddTEK77ixByZ0_MPMDv75pi_VQJSsJZI_DUQ2rP3m-FzdZmiZCPR49Ul-qX0Q_TLjy1Hwcm90vG4cOdWe7DQlGHfW/s400/BIOS-advanced+IDE+%255Bafter%255D.png" width="400" /></div>
<div>
Step 12: Descend into the "I/O Device Configuration" sub-page; by default, it'll look like the screenshot below:</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="260" data-original-width="396" height="262" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBMxiXTLQIE3ftENCPuMStORAbdKepF2QQ2_60MoEMbqvDlDWpEf568h4MqjY4SGsLj2W1f2rrXZzkACsnVh0deOX87ECo5a6unHK0fuFQaaQvm6-Rxu6hjTAjvn0615DOKRo59wexYwty/s400/BIOS-advanced+IO+config+%255Bbefore%255D.png" width="400" /></div>
<div>
Step 13: Using the arrow keys & space bar, set each device to "Disabled", then [Esc] to return to the Advanced menu.</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="224" data-original-width="376" height="237" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicsrN2DWJTRs3xqPEV-Audozzo_ZLOkbaarR81j_1UFmkzh5ZPFrbk1pSyg3pd-dN-725-3TPO2Sszx_aMJ7bB_wstDcUSanB90OF-dt93JtANlk6CTJnsQaHGMhde5jeTGA2ZFkmDfRxZ/s400/BIOS-advanced+IO+config+%255Bafter%255D.png" width="400" /></div>
<div>
Step 14: Switch to the Boot page. By default, removable devices are first in the boot order.</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="177" data-original-width="363" height="195" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9b4xJ8ppNLqHUCm-uxJ4-5TJKdC8YBuFBOXSe1rdu6Sf98W8P_NHE8RLvxEGvt4NZHfGy9nz7tI9MPAPEpQMCFxRpSFLRz1u74dchbdzoUAscmVHAgquuepNw_dkAjwa9TpFm6p42lNSc/s400/BIOS-boot+%255Bbefore%255D.png" width="400" /></div>
<div>
Step 15: Use the minus [-] key to lower the priority of removable devices. This won't hurt the initial OS setup, even on setup ISOs that normally require a key-press to boot off optical/ISO media: the new VM's hard drive has no partition table or MBR, so it'll be skipped as a boot device even when it's first. Once the OS is installed, you'll never have to worry about a removable media causing a reboot to stall.</div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" data-original-height="162" data-original-width="364" height="177" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUyDJDrK_3e4mmpgC0c4RSFRw9c8PQ7iG1q0l8JNJFk8gjM-50QlDXEpmtTw6bVzZ5R_GPZADfQsepmtHEbgYRpfZjxNJQWuVghUrWN7haxqTBwZW_-74cdzO3wdDbRHfmLRygwzQWj1Oo/s400/BIOS-boot+%255Bafter%255D.png" width="400" /></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
Step 16: Press [F10] to save the BIOS config, then use the console to attach to an ISO (local or on a datastore) before exiting the BIOS setup page.</div>
<img border="0" data-original-height="163" data-original-width="852" height="122" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRvferKdYrFBNgENzcjQGHs4_tapYtE4P6WYgncoojkKAZ1aRVnL5ZZ7L1P6dDAVVxl5ozUy9Tkcs9YNWvx84B7LJ92xwWz1QdQrgHpNZ9XisOzQPLzSHImU0FvdQRHCRPcwBRPWpwD0Mp/s640/connect+to+ISO.png" width="640" /><br />
<br />
Step 17: Install the guest OS, then add VMware Tools. Perform any additional customization—e.g., patching, updates, and generalization—then convert the new VM to a template.<br />
<br />
You're set! No more useless devices in your guest that take cycles from the OS or hypervisor.<br />
<br />
<b>Additional Note on modifying existing VMs:</b><br />
Aside from the need to power down existing VMs that you might want to clean up with this same procedure, the only issue I've run into after doing the device + BIOS cleanup is making sure I get the right combination of IDE channels & IDE CD-ROM attachment. The number of times I've set "Primary" in BIOS but forgot to change the CD-ROM to IDE 0:0 is ... significant.<br />
<br />
<b>Additional Note on Floppy Drives:</b><br />
Floppy drive handling is a special case, and will very much depend on which version of vSphere—and therefore, the management client—you're using. If you have the "Flex" client (or are still using v6.0 and have the C# client), the new VM will have a floppy disk device added by default. Naturally, you want to remove it as part of your Hardware Customization step during new VM deployment.<br />
If you're happily using the HTML5 Web Client, you may find that the floppy is neither present, nor manageable (for adding/removing or attaching media)... This is the 0.1% of feature parity that I still find lacking in the H5 client. Hopefully, it'll get added, if for no better reason than to allow an admin to remove floppy devices that are still part of VMs that were created in older versions.</div>
Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com120tag:blogger.com,1999:blog-7764963136414541684.post-34099676044884060292019-01-08T22:44:00.001-06:002019-01-08T22:44:27.733-06:00Merry Christmas: Apple Macintosh SEChristmas, 2018.<br />
My brother has given to me a circa-1989/1990 Apple Macintosh SE HDFD. It's in a "carrying" case, includes an external 800K floppy drive, Apple Desktop Bus keyboard and mouse, power cord, manuals, and System 6 install disks.<br />
<br />
The system has 2.5MB RAM, a 20MB SCSI hard drive, and a 1.44MB internal floppy.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-o13GiWT4lCOfhBHsD1jDW45yHwXERi-_WY-byrcp5W_KNq-09QrwqgS7qVfZBpoUqWQBVzqoSYwg-_4ahEIGz6INE03CyD8dD53GNILZgl7-EyuBE549uTimSemzxhyECVS4v6iWKW3F/s1600/MacSE_2500KB_RAM.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-o13GiWT4lCOfhBHsD1jDW45yHwXERi-_WY-byrcp5W_KNq-09QrwqgS7qVfZBpoUqWQBVzqoSYwg-_4ahEIGz6INE03CyD8dD53GNILZgl7-EyuBE549uTimSemzxhyECVS4v6iWKW3F/s400/MacSE_2500KB_RAM.jpg" width="300" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">2.5MB RAM</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhI-KV-z26dtuZ3Yae7FujGOyjqu1ZJaLo2zVMsvTKkdSAUXrqE4ghXwZcu9EjjXNEfELm8RGXW_KiodLF_Lue_4V-juTROuE3faEJGTcYtlNAEDUC_ZxlsUGCvLfB5HXVhpEPRE9bznTTH/s1600/MacSE_20MB_HD.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhI-KV-z26dtuZ3Yae7FujGOyjqu1ZJaLo2zVMsvTKkdSAUXrqE4ghXwZcu9EjjXNEfELm8RGXW_KiodLF_Lue_4V-juTROuE3faEJGTcYtlNAEDUC_ZxlsUGCvLfB5HXVhpEPRE9bznTTH/s400/MacSE_20MB_HD.jpg" width="300" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">20MB Hard drive (with "stuff" on it)</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfU9na5YlMQvrloic2ncAL9KMxx7xuKGs-Wgr3K63M78A9xMVVANhqVGGxDx7ul3B0_0J97lWxbzA5ah4s9diUhlhzDWLdYwYiCsq8IUpjtKfnAS74KVWgf9RroNLaewjWPWB8Mkf-Tp4i/s1600/MacSE_Desktop.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="1200" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfU9na5YlMQvrloic2ncAL9KMxx7xuKGs-Wgr3K63M78A9xMVVANhqVGGxDx7ul3B0_0J97lWxbzA5ah4s9diUhlhzDWLdYwYiCsq8IUpjtKfnAS74KVWgf9RroNLaewjWPWB8Mkf-Tp4i/s400/MacSE_Desktop.jpg" width="300" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">System 6, at your service...</td></tr>
</tbody></table>
My wife wanted to know what I'd do with it... well, the answer is: play with it.<div>
<br /></div>
<div>
The first thing I did was look into "useful" upgrades: Network, Memory, Capacity.</div>
<div>
<br /></div>
<div>
I found an Asante MacCon adapter for the SE</div>
<div>
I found 4 x 1MB RAM SIMMs for the SE</div>
<div>
I found this gizmo: <a href="http://www.codesrc.com/mediawiki/index.php/SCSI2SD" target="_blank">SCSI2SD</a></div>
<div>
<br /></div>
<div>
DING-DING-DING!</div>
<div>
<br /></div>
<div>
I can work with this.</div>
<div>
<br /></div>
<div>
And then I ran across this: <a href="http://macrepository.org/">macrepository.org</a></div>
<div>
<br /></div>
<div>
<a name='more'></a>From the very beginning, the 800K floppy drive was malfunctioning. Among other things, there was an old floppy stuck inside, and it wouldn't eject even using the manual override: It was frozen.</div>
<div>
<br /></div>
<div>
But that's okay: I have the internal 1.4M drive; didn't really want an 800K drive anyway.</div>
<div>
<br /></div>
<div>
Shortly after booting it for the first time, I discovered that the 20MB hard drive has issues. I decided to go ahead and re-initialize it to give it a chance to remap bad sectors and build a bad sector table from whatever couldn't be remapped.</div>
<div>
<br /></div>
<div>
Then I ran into issues with the 1.4M floppy; unfortunately, it's never clear whether the drive is messed up or the media, so I plugged away with eject-insert-eject-insert until I got the drive reinitialized and the system reloaded. Yay! Booting to System 6 off the hard drive took all of 30s from power-on to Finder being ready to go.</div>
<div>
<br /></div>
<div>
One of the first things I did when I got home with the thing was crack the case and start cleaning the dust bunnies from the thing.</div>
<div>
<br /></div>
<div>
The floppy drive came out, and it was cleaned & re-lubed. I removed the 800K drive internals from the external unit, but kept the case: getting a 1.4M drive to replace it seemed like a splendid upgrade.<br /><br />
Although System 6 would run blissfully on this, I've found plenty of references that indicate I'll need System 7 to take advantage of the Asante MacCon card. So be it, but the size increase means I can't do much with it until I have the 4MB upgrade completed.<br />
<br />
The folks at macrepository.org have old copies of the System7 installers, so I'm set there. I found the Asante drivers using the Internet Way Back Machine (yes, it caches binaries as well as HTML files).<br />
<br />
The SCSI2SD gave me some trouble at first. Although there are copies of "patched" versions of the Apple SC HD utility that can recognize, initialize and work with non-Apple hard drives, when you can override the internal of the device to masquerade as an Apple-original device, you do so:<br />
<br />
<ul>
<li>Vendor: " SEAGATE" (yes, there is a single space in front of SEAGATE for a total of 8 characters)</li>
<li>Product ID: " ST225N" (10 spaces before the model name for a total of 16 charaters)</li>
<li>Revision: "1.0 " (1 space at the end, for a total of 4 spaces)</li>
</ul>
<div>
I had a 1GB SD card lying around; it was too small for just about anything else I might want it for, but it was going to be virtual overkill for the Mac. I prepped the SCSI2SD with it, and configured it for two devices: a 20MB system drive, and a 900MB data drive.</div>
</div>
Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com262tag:blogger.com,1999:blog-7764963136414541684.post-35308358111334001922017-05-17T10:31:00.000-05:002017-05-17T13:25:20.215-05:00VBR v10 new hotnessSitting in the general session is not typically the way I'd compose a new post, but I'm pretty stoked by some new, long-desired features announced for the next version of Veeam Backup and Replication (VBR), version 10.<br />
<br />
First is the (long awaited) inclusion of physical endpoint backup management via VBR console. We've had Endpoint Backup for a while, which is awesome, and we've been able to use VBR repositories to store backups, but all management was at the endpoint itself. In addition to centralized management, the newest version of the managed endpoint backup (alright, alright... Agent) will support Microsoft Failover Clusters at GA!<br />
<br />
Second is the new feature that significantly expands VBR's capability: the ability to backup NAS devices. Technically, it's via SMB or NFS shares, so you could target any share--including one on a supported virtual or physical platform--but the intention is to give great backup & recovery options for organizations that utilize previously-unsupported platforms for NAS, like NetApp, Celera, etc.<br />
<br />
Third--and most exciting to me, personally--is the addition of a replication mode utilizing VMware's new "VMware APIs for I/O Filtering" (VAIO). This replication mode uses a snapshot-free capture of VMDK changes on the source, with and the destination being updated on a (configurable, default of 15s) by-the-second interval. This new replication method is branded "Veeam CDP" (Continuous Data Protection). There are competing products on the market that offer similar capability, but Veeam is advertising that they are the first to leverage VAIO while other products are using either undocumented/unsupported APIs, or old APIs intended for physical replication devices.<br />
<br />
There are a number of other nice, new features coming--Object storage support, Universal APIs for storage integration, etc.--but these three will be the big, compelling reasons to not only upgrade to Version 10 when it arrives (for current customers) but to upgrade your vSphere environments if you haven't already embraced Version 6.x.Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com41tag:blogger.com,1999:blog-7764963136414541684.post-42331272486284729952017-04-15T22:16:00.000-05:002017-04-16T23:43:51.745-05:00Upgrading to vSphere 6.5 with NSX already installedThis has been a slow journey: I have so many different moving parts in my lab environment (all the better for testing myriad VMware products) that migrating to vSphere 6.5 was taking <i>forever</i>. First I had to wait for Veeam Backup & Replication to support it (<i>can't live without backups!</i>), then NSX, <i>then </i>I had to decide whether to discard vCloud Director (yes, I'm still using it; it's still a great multitenancy solution) or get my company to give me access to their Service Provider version...<br />
<br />
I finally (finally! after over a year of waiting and waiting) got access to the SP version of vCD, so it was time to plan my upgrade...<br />
<br />
My environment supports v6.5 from the hardware side; no ancient NICs or other hardware anymore. I was already running Horizon 7, so I had two major systems to upgrade prior to moving vSphere from 6.0U2 to 6.5a:<br />
<br />
<ul>
<li>vCloud Director: 5.5.5-->8.0.2-->8.20.0 (two-step upgrade required)</li>
<li>NSX: 6.2.2-->6.3.1</li>
</ul>
<div>
There was one hiccup with those upgrades, and I'm sure they may be familiar to people with small labs: the NSX VIBs didn't install without "manual assistance." In short, I had to manually place each host into maintenance mode, kick off the "reinstall" to push the VIBs into the boot block, then restart the host. This wouldn't happen in a larger production cluster, but because mine is a 3-node VSAN cluster, it doesn't automatically/cleanly go into Maintenance Mode.</div>
<div>
<br /></div>
<div>
Moving on...</div>
<div>
<br /></div>
<div>
Some time ago, I switched from an embedded PSC to an external, so I upgraded that first. No problems.</div>
<div>
<br /></div>
<div>
Upgrading the stand-alone vCenter required a couple of tweaks: I uninstalled Update Manager from its server (instead of running the migration assistant: I didn't have anything worth saving), and I reset the console password for the appliance (yes, I'd missed turning off the expiration, and I guess it had expired). Other than those items? Smooth sailing.</div>
<div>
<br /></div>
<div>
With a new vCenter in place, I could use the embedded Update Manager to upgrade the host. I had to tweak some of the 3rd-party drivers to make it compatible, but then I was "off to the races."</div>
<div>
<br /></div>
<div>
After the first host was upgraded, I'd planned on migrating some low-priority VMs to it in order to "burn in" the new host and see if some additional steps would be needed (ie removing VIBs for unneeded drivers that have caused PSODs in other environments I've upgraded). But I couldn't.</div>
<div>
<br /></div>
<div>
Trying to vMotion running machines to the new host, I encountered network errors. "VM requires Network X which is not available". Uh oh.</div>
<div>
<br /></div>
<div>
I also discovered that one of the two DVS (Distributed Virtual Switch) for the host was "out of sync" with vCenter. And no "resync" option that would normally have been there...</div>
<div>
<br /></div>
<div>
Honestly, I flailed around a bit, trying my <i>google fu</i> and experimenting with moving VMs around, both powered-on and off, as well as migrating to different vswitch portgroups. All failing.</div>
<div>
<br /></div>
<div>
Finally, something inspired me to look at my VXLAN status; it came to me after realizing I couldn't ping the vmknic for the VTEPs because they sit on a completely independent IP stack, making it impossible to use vmkping with a VTEP as a source interface.</div>
<div>
<br /></div>
<div>
Bingo!</div>
<div>
<br /></div>
<div>
The command esxcli network vswitch dvs vmware vxlan list resulted in no data for that host, but valid config information for the other hosts.</div>
<div>
<br /></div>
<div>
A quick look at NSX Host Preparation confirmed it, and a quick look at the VIBs on the host nailed it down: esx-vsip and esx-vxlan were still running 6.0.0 versions.</div>
<div>
<br /></div>
<div>
I went back through the process I'd used for upgrading NSX in the first place, and when the host came back up, DVS showed "in sync", NSX showed "green" install status and—most important of all—VMs could vMotion to the host and they'd stay connected!<br />
<br />
UPDATE: The trick, it seems, is to allow the NSX Manager an opportunity to install the new VIBs for ESXi v6.5 before taking the host out of maintenance mode. By manually entering Maintenance Mode prior to upgrading, VUM will <u>not</u> take the host out of Maintenance, giving the Manager an opportunity to replace the VIBs. Once the Manager shows all hosts upgraded and green-checked, you can safely remove the host from Maintenance and all networking will work.</div>
Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com10tag:blogger.com,1999:blog-7764963136414541684.post-55771139278364578572017-02-20T22:41:00.001-06:002017-02-21T00:07:08.756-06:00ADFS and SNISNI, or Server Name Indicator, is an extension to TLS (Transport Layer Security, the evolutionary child of SSL/Secure Socket Layer) that permits multiple certificates (and therefore encrypted sessions) to be bound to the same TCP port.<br />
<br />
Starting with ADFS v3.0 (aka ADFS for Windows Server 2012R2), Microsoft uses SNI by default. On the whole, this has little impact on most users of ADFS, but for one small, important subset: users that sit behind reverse proxy or hardware SSL-offload devices.<br />
<br />
Readers of this blog know that I use the Citrix NetScaler VPX Express as a <a href="http://blog.millard.org/search/label/proxy" target="_blank">reverse proxy</a> for my home lab; until I tried to stand up an ADFS server (running Server 2016) behind it—I'm going to start digging into Office 365 in a serious way and want the most seamless user experience—I'd never had a problem with it.<br />
<br />
I just could NOT figure out why the ADFS system was immediately rejecting connections via NetScaler, while it was perfectly happy with local connections.<br />
<br />
I knew things were problematic as soon as I did packet captures on the NetScaler: the [SYN]-[SYN, ACK]-[Client Hello] were immediately followed by [RST, ACK] and a dropped connection.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisuO8DHEqvZPBamcQ_bb9WjF7eNwS7Y2dQF9xLdkA-MjPXfK5PrZDt1Mw9X5HfK6cisgUQKzxbO7ddcN7vURnpA2Mquj4YFl7yCFJOBqx7DwXO6tXEGl5Kh1s3VY796E3-Yx6aQrbOXVxA/s1600/Screenshot+-+2_20_2017+%252C+10_17_55+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="155" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisuO8DHEqvZPBamcQ_bb9WjF7eNwS7Y2dQF9xLdkA-MjPXfK5PrZDt1Mw9X5HfK6cisgUQKzxbO7ddcN7vURnpA2Mquj4YFl7yCFJOBqx7DwXO6tXEGl5Kh1s3VY796E3-Yx6aQrbOXVxA/s640/Screenshot+-+2_20_2017+%252C+10_17_55+PM.png" width="640" /></a></div>
<br />
Once I "fired up" a copy of Wireshark and pulled some captures at the ADFS host, however, I was able to compare the difference between the NetScaler-proxied connections that were failing, and on-prem connections that were successful.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDeUC8olWgbtuHBPjDU__a2GUdK0WCMexNFi_QUd2naalxhkrb7D-plXE-uZa-k_iXAT6vkxh8PMxy3jUOzcqm55nCVlc2ZURCSadLoJM9vQr81LJxivU_QGuRoH1pjMqTC3LiWZd22uHz/s1600/Screenshot+-+2_20_2017+%252C+10_21_59+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="116" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjDeUC8olWgbtuHBPjDU__a2GUdK0WCMexNFi_QUd2naalxhkrb7D-plXE-uZa-k_iXAT6vkxh8PMxy3jUOzcqm55nCVlc2ZURCSadLoJM9vQr81LJxivU_QGuRoH1pjMqTC3LiWZd22uHz/s640/Screenshot+-+2_20_2017+%252C+10_21_59+PM.png" width="640" /></a></div>
<br />
<br />
At that point, I could explicitly compare the two different [Client Hello] packets and see if I could tell the difference between the two...<br />
<br />
Unfortunately, I started with comparing the protocols, ciphers and hash algorithms. It took a while to get the TLS1.2 setup <i>just right</i> to mimic the local connection, but no joy. But then I went after the extensions: only one extension was in the "misbehaving" [Client Hello]<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRkTW39XMeUiL3wOM0T3_GXS3PAnJPO_ft40zLqpty3B56hkUe4vzUIxrNxPiqH3waBcMfw6_lQ-047jDuvgP4tgX2s5Uk55SdlZ_cbgoyPsqo9RbJCbfdhUl66mEiGSN1kR1k8YhO0oh1/s1600/Screenshot+-+2_20_2017+%252C+10_31_41+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRkTW39XMeUiL3wOM0T3_GXS3PAnJPO_ft40zLqpty3B56hkUe4vzUIxrNxPiqH3waBcMfw6_lQ-047jDuvgP4tgX2s5Uk55SdlZ_cbgoyPsqo9RbJCbfdhUl66mEiGSN1kR1k8YhO0oh1/s1600/Screenshot+-+2_20_2017+%252C+10_31_41+PM.png" /></a></div>
There are a <i>bunch</i> of extensions in the "working" [Client Hello]:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpRCk0rbi9dJlyzxoG2A7HsN6VUT5-A7EQt2eGl3g93gY8Jyqqaid-LysrfzmPDozbOE3zhydDG3PRPwkv27Q8fSJwssA7MZ_A4m-eEBSPJt_yGiTVStyo5Tr35ePG4SNu8ITRIOTgM-li/s1600/Screenshot+-+2_20_2017+%252C+10_33_36+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpRCk0rbi9dJlyzxoG2A7HsN6VUT5-A7EQt2eGl3g93gY8Jyqqaid-LysrfzmPDozbOE3zhydDG3PRPwkv27Q8fSJwssA7MZ_A4m-eEBSPJt_yGiTVStyo5Tr35ePG4SNu8ITRIOTgM-li/s1600/Screenshot+-+2_20_2017+%252C+10_33_36+PM.png" /></a></div>
<i>holy crap</i><br />
<i><br /></i>
To make my task easier, I switched back to <i>google-fu</i> to see if I could narrow down the search; <i>voila!</i><br />
<i><br /></i>
I <a href="https://newsignature.com/articles/federation-adfs-3-0-sni-support/" target="_blank">found an article</a> that talked about handling ADFS clients that don't support the SNI extension, and the lightbulb went on: my browsers do SNI, but with the NetScaler acting as a proxy <i>SNI support is disabled by default</i>.<br />
<br />
Luckily there are two fixes:<br />
<ol>
<li>Update the ADFS server with a "blanket" or "fallback" binding for the ADFS service (see <a href="https://blogs.technet.microsoft.com/applicationproxyblog/2014/06/19/how-to-support-non-sni-capable-clients-with-web-application-proxy-and-ad-fs-2012-r2/">https://blogs.technet.microsoft.com/applicationproxyblog/2014/06/19/how-to-support-non-sni-capable-clients-with-web-application-proxy-and-ad-fs-2012-r2/</a>)</li>
<li>Update the NetScaler service entry (in the SSL Parameters section) to support SNI for the expected client hostname.</li>
</ol>
<div>
I went with the latter; that way I don't modify any more of the ADFS host than necessary, and because the NetScaler is essentially acting as a client while it's doing its proxy duties, that seemed to make the most sense.</div>
<div>
<br /></div>
<div>
Within a minute of adding the SNI extension, the ADFS system worked as expected.</div>
Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com24tag:blogger.com,1999:blog-7764963136414541684.post-1156952025253985332017-02-15T14:34:00.002-06:002017-02-21T00:05:47.497-06:00SSL Reverse Proxy using Citrix NetScaler VPX Express<h4>
Part 6 in a series</h4>
In <a href="http://blog.millard.org/search/label/proxy" target="_blank">previous posts</a> I covered the configuration of the NetScaler VPX Express for use as an intelligent reverse proxy, allowing the use of a single public IP address with multiple interior hosts.<br />
<br />
In recent days, I've been working on adding Horizon View to my home lab; in addition to requisite Connection Servers, I'm using the EUC Access Point virtual appliance as a security gateway instead of Security Servers paired with dedicated Connection Servers.<br />
<br />
The procedure I outline for the creation of a content-switching configuration works as you'd expect...to a point.<br />
<br />
I found that I kept getting "Tunnel reconnection is not permitted" errors when trying to login using the dedicated Horizon Client; this was extremely frustrating because HTML access (using nothing but an HTML5-compatible browser) was working flawlessly.<br />
<br />
Upon reviewing the client logs, I noticed that the response from the tunnel connection (HTTP/1.1 404 Not Found) was <i>from IIS</i>, not a Linux or other non-Windows webserver. In my configuration, my content-switching plan uses a Windows IIS server as the fall-through (default/no-match).<br />
<br />
Theory: for whatever reason, while the registration process for the Horizon Client was being properly switched to the Access Point, login via tunnel was <b>not</b>.<br />
<br />
By capturing a trace (including SSL decoding) at the NetScaler and reviewing it in Wireshark, I was able to see that the client is using <b>two different</b> host strings, one during the initial login followed by a second one during tunnel creation.<br />
<br />
What's the difference? The initial login doesn't include the port number in the host string; the tunnel request includes it...
<br />
<blockquote>
Login: vdi.corp.com<br />
Tunnel: vdi.corp.com:433</blockquote>
The fix is to add an additional match criteria for your content switching policy:<br />
<blockquote>
Before: <code>HTTP.REQ.HOSTNAME.EQ("vdi.corp.com")</code><br />
After: <code>HTTP.REQ.HOSTNAME.EQ("vdi.corp.com")||HTTP.REQ.HOSTNAME.EQ("vdi.corp.com:443")</code></blockquote>
You can also create an additional policy with the "fqdn:443" match, but editing the policy was faster to implement.<br />
<br />
UPDATE: I've done some more digging, and there are additional arguments/functions that would also work—and would've worked transparently had I used them in the first place—instead of the <span style="font-family: monospace;">EQ("")</span> expression:<br />
<blockquote>
<span style="font-family: monospace;">HTTP.REQ.HOSTNAME.CONTAINS("vdi.corp.com")<br />
HTTP.REQ.HOSTNAME.SERVER=="vdi.corp.com"<br />
HTTP.REQ.HOSTNAME.STARTSWITH("vdi.corp.com")<br />
HTTP.REQ.HOSTNAME.PREFIX('.',0).EQ("vdi")</span></blockquote>
Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com6tag:blogger.com,1999:blog-7764963136414541684.post-24832704300960112712016-12-23T11:29:00.000-06:002016-12-23T11:29:40.690-06:00Apple Watch First Impressions<h2 style="height: 0px;">
...from a former Pebble user</h2>
<div>
<br /></div>
<div>
When Pebble announced their acquisition by FitBit, I was cautious about the future of the product: I backed the original Pebble on Kickstarter, as well as the Pebble Steel, Time Steel and finally, Time 2 when the opportunities presented themselves. But then recent things like having a total reset screwing up all my settings (and needing to do a factory reset to get things back) and a limited lifetime (and no more warranty support) for the existing units, I decided to look elsewhere for a good smartwatch.<br />
<br /></div>
<div>
As a longtime iPhone/iPad user I'd looked at the specs for Apple Watch when it was first released, and between the significant cost difference from the Pebble (like 4x more expensive, depending on the edition and band choices) and significant hardware limitations (Single-day battery life? Really? Not water resistant?), the sale of Pebble was making my smartwatch options pretty bleak.<br />
<br /></div>
<div>
However, the recently released Series 2 from Apple addressed 2 of the 3 biggest faults I had with the platform (nothing is going to address the cost problem: this is Apple we're talking about, and all of its options are boutique-priced) by adding significant strides to battery life along with 50M water resistance.<br />
<br /></div>
<div>
So I pulled the trigger and yesterday was able to take delivery of a 42mm Stainless Steel with Milanese Loop band in Space Black.</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx586RmWpkIruZSRv_dGbHfe_yjWojOnuHhypo2g1Tj6HZd5mBnkV8ysiEuqnrUazMM86yzmHHTEFe-QlfsFVnG4pXgfhSeokKq7si0ICQZRZHjdr82jjeoPCgNi2NFn8pkygVMjnq2fkI/s1600/Screenshot+-+12_23_2016+%252C+9_48_38+AM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="161" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx586RmWpkIruZSRv_dGbHfe_yjWojOnuHhypo2g1Tj6HZd5mBnkV8ysiEuqnrUazMM86yzmHHTEFe-QlfsFVnG4pXgfhSeokKq7si0ICQZRZHjdr82jjeoPCgNi2NFn8pkygVMjnq2fkI/s320/Screenshot+-+12_23_2016+%252C+9_48_38+AM.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">42mm Apple Watch Series 2 in Space Black<br />with Milanese Loop band</td></tr>
</tbody></table>
<div>
If you're interested in an un-boxing, you can search elsewhere. Suffice it to say that, in typical Apple fashion, the watch was simultaneously beautifully and over-packaged; a fair expectation for an $800 timepiece, whether it comes from Apple or not, but the amount of material waste from the packaging hails back to when Apple thought they were competing in the luxury timepiece market rather than the fitness wearables market. They really, really could've gone with less.<br />
<br /></div>
<div>
I started by placing the watch on the charging disc for a few hours to make sure it was well charged, then I went through the pairing process. Unlike Pebble, the Watch doesn't use two different Bluetooth profiles (one standard and one low-energy), and pairing with my iPhone 6s running iOS 10.2 was smooth and less error-prone compared to my usual experience with Pebble pairing. If there's one thing to be said for getting the two devices from the same manufacturer, it's the effortless user experience with pairing.<br />
<br /></div>
<div>
Before purchasing, I visited a local Apple store to get a feel for my choices in cases and bands. I selected the 42mm over the 38mm because of the larger display and my old eyes. The stainless steel case is a heftier feel over aluminium (or ceramic), which I definitely prefer, and there was a noticeable difference between the 38mm and 42mm as well, solidifying my choice of that size. Lighter watches tend to slide around to the underside of my wrist, while heavier ones seem to stay in place on the top. And if I have to deal at all with the watch on the underside of my wrist, the sapphire crystal of the stainless steel & ceramic cases was a must. I also prefer the heavier link band, but between the $500 premium and its "butterfly clasp" (which I hate), there was no way I was going with the Apple link band. The Milanese felt "weighty" enough in comparison to the link band, and its "infinite adjustability" had some appeal as well.<br />
<br /></div>
<div>
Once I had the watch paired and on my wrist, I started digging into the features I'd come accustomed to on the Pebble. Probably the biggest surprise was the dearth of watch face choices: unlike the Pebble ecosystem, with thousands of watch faces to choose from—everything from <a href="https://apps.getpebble.com/en_US/application/5650ea0ea69d979e4400001b" target="_blank">utilitarian designs</a> to homages to <a href="https://apps.getpebble.com/en_US/application/555c1924ac15b6f992000076" target="_blank">Star Trek</a> to the silly "<a href="https://apps.getpebble.com/en_US/application/537d17e007dd651cce00000c" target="_blank">Drunk O'Clock</a>" face—the handful of faces available in the Watch ecosystem was a big surprise.<br />
<br /></div>
<div>
Worse, while all the Watch faces are customizable to some degree, all of them have the limitation of disallowing the customization of "time" itself. The face I'm most accustomed to on the Pebble—<a href="https://apps.getpebble.com/en_US/application/52cc44e045ffdd31dd000180" target="_blank">YWeather by David Rincon</a>—is nearly reproducible on the Watch using the "Modular" face, but the options—or "Complications" as Apple terms them—aren't very flexible and make "time" a less-prominent feature in the face. Which, in my opinion, sort of defeats the purpose in a <b>watch</b> face.</div>
<table align="center" style="text-align: center;"><tbody>
<tr><td><img border="0" height="125" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJ1eeTb5OLK2ZnOy5FQ2B9MVb7k-MoeXed6dAkkEXTV9POHnCSySkf1jFFbz1sPtsub63Ji5E3-z2xzTGoEuo_3dJahAebdw3Ouicf1M51zft_Fnir2bMHJveQWPGSATM0Qfz6nsXyyChJ/s200/modular.png" /></td>
<td><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicEkwqZdw8-4WLqFVHbXdmX7hPlkHHRsW3FmSWIj8ntMxXaK_T-wbG6qVDYRsDFRmUc2TFEXu6qx6XAL8h3p35hAoDpxbq8sUhNe0Bhyg11h8ETIOB88JrpOP62E3B97pUdBCpruUomi0Q/s1600/Screenshot+-+12_23_2016+%252C+10_47_04+AM.png" /></td></tr>
<tr><td>Apple Watch<br />
"Modular"</td><td>Pebble<br />
"YWeather"</td></tr>
</tbody></table>
<br />
<div>
If I could just move the Time to the center section and make it more prominent, while moving the date to the upper-right, it'd be good enough...<br />
<br /></div>
<div>
Notifications are also very different on the Apple Watch; the most significant seems to be the suppression of all notifications when the phone is actively being used, which I'm extremely unhappy with. Among other things, it means that I'm not getting notifications when I've got the phone plugged into power and showing a route in Waze. Even when the phone is locked & screen is off, I'm finding that notifications I usually received on the Pebble are missing/silent on the watch: I've yet to get a notification from Slack, which is one of the busiest apps on my phone after Mail itself.<br />
Yes, I've made sure that things like "cover to mute" is disabled and "mirror phone" is set for pretty much <b>all</b> of the integrations on the watch, but the only type of notification that I get seems to be Messages and Calendar.<br />
<br />
Application integration is nice for many apps I have on the phone; being able to quickly raise/lower the garage door using GarageIO on the watch instead of the phone is nice, as is checking the home alarm. However, it does seem that some watch app integrations require the phone-based app to be running (or at least "backgrounded") in order for the watch component to function. It's not consistent, so I'm still trying to figure out which ones need to be running in order to work.<br />
<br />
The blob of apps in the App Layout sucks, however. While I have the ability to move apps around to change their proximity to the "central" Clock app, the fact that there are so many that I'd just as soon never see—even after telling Watch to uninstall the integration—is mind-boggling when you consider the minimalist design elements used everywhere else in all Apple products.<br />
<br />
At any rate, I'm still getting used to this thing, but from my perspective, I like parts of it, but other parts are still inferior to Pebble</div>
Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com4tag:blogger.com,1999:blog-7764963136414541684.post-787655561006669912016-11-08T21:36:00.001-06:002016-11-08T21:36:03.734-06:00Virtual SAN Cache Device upgrade<h3>
Replacing/Upgrading the cache+buffer device in VSAN</h3>
Dilemma: I've got a VSAN cluster at home, and I decided to switch from single diskgroups-per-host to dual to give myself a bit more availability as well as additional buffer capacity (with all-flash, there's not much need for a read cache).<br />
<br />
My scenario has some unique challenges for this transformation.
First, although I already have the new buffer device to head the
new disk group, I don't actually have all the new capacity disks that
I'll need for the final configuration: I'll need to use some of the
existing capacity disks if I want to get the second disk group going before I have the additional capacity devices. Second, I have
insufficient capacity in the remainder of the VSAN datastore to perform a
full evacuation while still maintaining policy compliance (which is sort of why I'm looking to add capacity in addition to splitting the one disk group up).<br />
<br />
<br />
The nominal way to perform my transformation is:<br />
<ol>
<li>Put the host into maintenance mode, evacuating all registered VMs</li>
<li>Delete the disk group, evacuating the data so all VMs remain storage policy-compliant.</li>
<li>Add the new device</li>
<li>Rebuild disk group(s)</li>
</ol>
I already took a maintenance outage during the last patch updates and added my new cache+buffer device to each host, so "Step 3" is already completed. <br />
And then I hit on something: While removing the buffer device from a diskgroup will cause the decommissioning of the entire disk group, individual capacity devices can be removed without upsetting more than the objects being stored on that device alone. I have sufficient capacity in the remainder of the disk group—not to mention on the other hosts in the cluster—to operate on individual capacity elements.<br />
<br />
So, here's my alternative process:<br />
<br />
<ol>
<li>Remove one capacity device from its disk group with full migration<br /><div class="separator" style="clear: both; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEio_mulbwoi5AGjlwrvWPx03O_9ti_rvmRGtBHXAhHgcvSTk-7cHghQBrg974Q94qDeeFfZB9_bhv7j4XEDc2ipXU_jJ_tlCibLqxExQ9CDQsO0QVZcSERW-GRbj3Z89RsvmNomhJqMEjrd/s1600/Screenshot+-+11_8_2016+%252C+8_51_44+PM.png" /></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZb9JAqFEwlOP19IlTzT0Q_1eQB6CWOXLlseAvMaan9_AgwBTpedNRl5jd5QTolFCvY3vZxSyjRGydW6UFMY86QU28haxbNPPvZCnfM-v1xpXWXtdGRDZEXe9Sbv4K72ZU6eSKANO7kT2u/s1600/Screenshot+-+11_8_2016+%252C+8_52_00+PM.png" /></div>
</li>
<li>Add the capacity device to the new disk group.</li>
</ol>
<br />
It takes longer because I'm doing the evacuation and reconfiguration "in series" rather than "in parallel," but it leaves me with more active & nominal capacity+availability than doing it on an entire diskgroup at once.<br />
<br />
My hosts will ultimately have two disk groups, but they'll break one "rule of thumb" by being internally asymmetric: My buffer devices are 400GB and 800GB NVMe cards, respectively, so when I'm fully populated with ten (10) 512GB capacity disks in each host, four (4) will be grouped with the smaller and six (6) will be grouped with the larger. When you keep in mind that Virtual SAN won't use more than 600GB of a cache+buffer device regardless of its size, it actually has some internal symmetry: each capacity disk will be (roughly) associated with 100GB of buffer, for a ~5:1 buffer:capacity ratio.<br />
<br />
<h3>
CLI alternative</h3>
Although this entire process can be performed using the Web Client, an alternative is to write a CLI script. The commands needed are all in the <code>esxcli storage</code> or <code>vsan</code> namespaces; combined with some shell/PowerShell scripting, it is conceivable that one could:<br />
<ul>
<li>Identify storage devices.<br /><code>esxcli storage core device list</code></li>
<li>Identify any existing disk group, cache+buffer and capacity devices<br /><code>esxcli vsan storage list</code>.</li>
<li> Remove one of the capacity disks with migration<br /><code>esxcli vsan storage remove -d <device> -m evacuateAllData</code></li>
<li>Create a new disk group using an available flash device from the core device list as the new group's cache+buffer device, and the recently evacuated device as the capacity device<br /><code>esxcli vsan storage add -s <cache+buffer device> -d <device></code></li>
<li>Loop through the remaining capacity devices, first removing then adding them to the new disk group. The <code>esxcli vsan storage remove</code> command is blocking when run from the ESXi console, so your script should wait for full evacuation and availability before the next step in the script is executed.</li>
</ul>
Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com3tag:blogger.com,1999:blog-7764963136414541684.post-72110223690705671452016-10-13T05:58:00.000-05:002016-10-13T08:47:47.593-05:00Adding floppy for PVSCSI drivers when creating a VM in vCenter Web ClientSomeone asked in a private slack channel if it was "just him" or can you really not add a floppy image when creating a VM using the Web Client. This is relevant any time you want to build a VM using the PVSCSI drivers so they'll always be available, even if VMware Tools is uninstalled.<br />
The answer—at least with v6.0U2—is "no."<br />
In this scenario, the vmimages folder won't expand; it offers the "arrowhead" showing there is content to be discovered within, but when you select it, you get no content...<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhU-yaGMPWYB-qK19WqxeOWFJtTHfqqrQ4oFTFvfjrSZquB7wjalu3ruVU0ApZzc6Jq1TGhrhr684VFynZzcKy56yRrLZ-u5Xuw8B40_YZOOMHn0FRrVGboP5AvploxgEX3qXhHbcRQkFCC/s1600/Screenshot+-+10_13_2016+%252C+5_52_17+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhU-yaGMPWYB-qK19WqxeOWFJtTHfqqrQ4oFTFvfjrSZquB7wjalu3ruVU0ApZzc6Jq1TGhrhr684VFynZzcKy56yRrLZ-u5Xuw8B40_YZOOMHn0FRrVGboP5AvploxgEX3qXhHbcRQkFCC/s1600/Screenshot+-+10_13_2016+%252C+5_52_17+AM.png" /></a></div>
<br />
Fortunately, there's a workaround: if you go ahead and save the new VM (without powering on) and <u>then</u> edit it, modifying the source for the floppy image, the vmimages folder will correctly expand and populate, allowing you to select one.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6klouYtgA3aP8qiWCXL6RJHGp_IzTXxcTUAi1EyMP3TFGwiKpeZPN3RZlIJYX_wJJu-wUbQtrp9gVCJuobPpZRet_jYEvd_jBJxEMucLkQo9olRTdHXz3IkOiXAgO8ODqostat__xodlG/s1600/Screenshot+-+10_13_2016+%252C+5_57_30+AM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6klouYtgA3aP8qiWCXL6RJHGp_IzTXxcTUAi1EyMP3TFGwiKpeZPN3RZlIJYX_wJJu-wUbQtrp9gVCJuobPpZRet_jYEvd_jBJxEMucLkQo9olRTdHXz3IkOiXAgO8ODqostat__xodlG/s1600/Screenshot+-+10_13_2016+%252C+5_57_30+AM.png" /></a></div>
<br />
UPDATE: It turns out we were talking about two different Web Clients! My assumption was that we were referring to the vCenter Web Client, while the person asking was referring to the new(ish) Host Web Client.<br />
<br />
The defect and workaround as I've documented it only apply to the vCenter Web Client. The Host Web Client will <b>not</b> behave correctly even in the workaround; this is a solid defect. There are other workarounds—use the C# client, copy the IMG file to an accessible datastore, etc.—but none are as good as the defect being eliminated in the first place.Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com4tag:blogger.com,1999:blog-7764963136414541684.post-65629749621128256052016-02-26T16:15:00.000-06:002016-02-26T21:00:31.204-06:00NTFS, dedupe, and the "large files" conundrum.Microsoft did the world a huge favor when they added the deduplication feature to NTFS with the release of Windows Server 2012. We can have a discussion outside of this context on whether inline or post-process dedupe would have been better (the NTFS implementation is post-process), but the end result is something that seems to have minimal practical impact on performance but provides huge benefits in storage consumption, especially on those massive file servers that collect files like a shelf collects dust.<br />
<br />
On the underside, the dedupe engine collects the duplicate blocks and hides them under the hidden "System Volume Information" folder and leaves pointers in the main MFT. You can do a disk size scan and see very little on-disk capacity taken by a given folder, yet a ginormous amount of disk is being consumed in that hidden folder.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOPm3SGBt2FuVOJt-7f6PFPBra42bmexUWYrxYLeEsP9UrPYsJ9ItF0qk0NXrygflcBIaTnPd-7FxKZPxSr-7fuJrRv2N32UgguAoJLYz-SoxnTHTxOrc7_d6j0LmBB5Sc0PvojWyeJRb9/s1600/Dedupe_in_action_1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOPm3SGBt2FuVOJt-7f6PFPBra42bmexUWYrxYLeEsP9UrPYsJ9ItF0qk0NXrygflcBIaTnPd-7FxKZPxSr-7fuJrRv2N32UgguAoJLYz-SoxnTHTxOrc7_d6j0LmBB5Sc0PvojWyeJRb9/s1600/Dedupe_in_action_1.png" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
See that little slice of color on the far left? That's the stub of files that aren't sitting in the restricted dedupe store. The statistics tell a different story:</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuVH7DUyvCZdJghH-u3_6tmMQoaN-l4xe8P1I6E5j64yeEQwdtaPkr8vqKZZzHKYUNSG1EFuVUby04Bkay7Kkca6xl2uWkCWS59py4XgfkLhrJ26vHMFAbIA9glE_Dy6mRdmgCKSzo2ZMv/s1600/Dedupe_in_action_2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjuVH7DUyvCZdJghH-u3_6tmMQoaN-l4xe8P1I6E5j64yeEQwdtaPkr8vqKZZzHKYUNSG1EFuVUby04Bkay7Kkca6xl2uWkCWS59py4XgfkLhrJ26vHMFAbIA9glE_Dy6mRdmgCKSzo2ZMv/s1600/Dedupe_in_action_2.png" /></a></div>
<br />
200GB of non-scannable data (in the restricted store) versus 510MB stored in the "regular" MFT space. Together they comprise some 140K files in 9K folders, and the net action of dedupe is saving over 50GB in capacity on that volume:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSdK0Kax7iTYcUDaHGhp6Qq7JhvEQ-ogy-v7W587KLG5iSnsVP43o6VAmzhltWegCFSOVsuXT5pn6X2WbIlvwhrNSIDtW-sDWQHl2VYKlkNG2Jn7fyUP7ZJzLmPJnG5VXlvFgT9U2Z7uM8/s1600/Dedupe_Performance.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="105" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSdK0Kax7iTYcUDaHGhp6Qq7JhvEQ-ogy-v7W587KLG5iSnsVP43o6VAmzhltWegCFSOVsuXT5pn6X2WbIlvwhrNSIDtW-sDWQHl2VYKlkNG2Jn7fyUP7ZJzLmPJnG5VXlvFgT9U2Z7uM8/s640/Dedupe_Performance.png" width="640" /></a></div>
<br />
The implementation is fairly straightforward, and I've found few instances where it <u>didn't</u> save the client a bunch of pain.<br />
<br />
Except when used as a backup target.<br />
<br />
Personally, I though this was the perfect use case—and it is, but with the caveats discussed herein—because backup tools like Veeam can perform deduplication within a backup job, but job-to-job deduplication isn't in the cards. Moving the backup repository to a deduplicating volume would save a ton of space, giving me either space to store more data or more restore points for existing backups.<br />
<br />
Unfortunately, I ran into issues with it after running backups for a couple of weeks. Everything would run swimmingly for a while, then suddenly backups would fail with filesystem errors. I'd wipe the backup chain and start again, only to have it happen again. Fed up, I started searching for answers...<br />
<br />
Interestingly, the errors I was receiving (<i>The requested operation could not be completed due to a file system limitation.</i>) go all the way back to limitations on NTFS <u>without</u> deduplication, and the early assertions by Microsoft that "defragmentation software isn't needed with NTFS because it protects itself from fragmentation." Anyone else remember that gem?!? Well, the Diskeeper folks were able to prove that NTFS volumes do, in fact, become fragmented, and a cottage industry of competing companies popped up to create defrag software. Microsoft finally relented and not only agreed that the problem can exist on NTFS, but they licensed a "lite" version of Diskeeper and included it in every version of Windows since Windows 2000. They also went so far as to add additional API calls to the filesystem and device manager so that defragger software could better operate in a safe manner than "working around" the previous limitations.<br />
<br />
I digress...<br />
<br />
The errors and the underlying limitation have to do with the way NTFS handles file fragmentation. It has special hooks to readily locate multiple fragments across the disk (which is, in part, why Microsoft argued that a fragmented NTFS volume wouldn't suffer the same sort of performance penalty that an equivalently-fragmented FAT volume would experience), but the data structures to hold that information is a fixed resource. Once volume fragmentation reaches a certain level, the data structures are exhausted and I/O for the affected file is doomed. The fix? Run a defragger on the volume to free up those data structures (every fragment consumes essentially one entry in the table, so the fewer fragments that exist, the fewer table resources are consumed, irrespective of total file size) and things start working again.<br />
<br />
<b>Enter NTFS deduplication</b><br />
<br />
Remember that previous description of how the dedupe engine will take duplicate blocks from the volume—whether they're within a single file or across multiple—and put it in the System Volume Information folder, then leave a pointer in the main MFT to let multiple files (or the same file) access to that block?<br />
<br />
Well, we just deliberately engineered a metric crapton (yes, that's a technical description) of <b>intentional fragmentation</b> on the volume. So when individual deduplicated files grow beyond a certain size (personal evidence says it's ~200GB, but posts I've found here and there say it's as little as 100GB while MS says it's 500GB <a href="https://support.microsoft.com/en-us/kb/2891967">https://support.microsoft.com/en-us/kb/2891967</a>) you can't do anything with the file. Worse, defrag tools can't fix it, because this fragmentation isn't something that the algorithms can "grab"; the only real fix—other than throwing away the files and starting over—is to disable dedupe. And if you're near the edge of capacity due to the <u>benefit</u> of dedupe, even that's no option: rehydrating the file will blow past your capacity. Lose-lose.<br />
<br />
Luckily, Microsoft identified the issue and gave us a tool when building volumes intended for deduplication: "large files" flag in the format command. Unfortunately, as you might guess when referring to "format," it's destructive. The structures that are laid down on the physical media when formatting a volume are immutable in this case; only an evacuation and reformat fixes the problem.<br />
<br />
Given that restriction, wouldn't it be helpful to know if your existing volumes support large files (ie extreme fragmentation) before you enable deduplication? Sure it would!<br />
<br />
The filesystem command "fsutil" is your friend. From an administrative command prompt, run the following command + arguments (this is an informational argument that makes no changes to the volume, but requires administrative access to read the system information):<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">fsutil fsinfo ntfsinfo <drive letter></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeR10gne1tssefMBCit3Oca3a8Ao71R93oO6CGetJjBcsPExPik1KW7PMvWbaB1tz5_czJp3PwNHWuX8OqGRUq1c7-99axOpnnkR1qmq0mhZUEKm92luUCf2kOf9qv1-pQGYz9jcqo-eN7/s1600/without_largefiles.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeR10gne1tssefMBCit3Oca3a8Ao71R93oO6CGetJjBcsPExPik1KW7PMvWbaB1tz5_czJp3PwNHWuX8OqGRUq1c7-99axOpnnkR1qmq0mhZUEKm92luUCf2kOf9qv1-pQGYz9jcqo-eN7/s1600/without_largefiles.png" /></a></div>
<br />
Notice the <i>Bytes Per FileRecord Segment</i> value? On a volume that <b>does not</b> support high levels of fragmentation, you'll see the default value of <b>1024</b>. You'll want to reformat that volume with the "/L" argument before enabling dedupe for big backup files on that bad boy. And no, the ability to do that format argument is <b>not</b> available in the GUI when creating a new volume; you've got to use the command line.<br />
<br />
What does it look like after you've reformatted it? Here you go:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8L0iIZn-3ZDkSgrAULA58Y6CIfeOxZTXvioon36vAOQIhSbtSelraK-Jmmql-k5-C7TFQ9bZ_7IYJsinaSXy21UIRjlogTOtjjP63zlpxBKxWRsEmTeGmYaefaw9YXUcAmqHlTQGLHvoi/s1600/with_largefiles.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj8L0iIZn-3ZDkSgrAULA58Y6CIfeOxZTXvioon36vAOQIhSbtSelraK-Jmmql-k5-C7TFQ9bZ_7IYJsinaSXy21UIRjlogTOtjjP63zlpxBKxWRsEmTeGmYaefaw9YXUcAmqHlTQGLHvoi/s1600/with_largefiles.png" /></a></div>
<br />
The <i>Bytes Per FileRecord Segment</i> value jumps up to the new value of <b>4096</b>.<br />
<br />
You'll still want to adhere to Microsoft's dedupe best practices (<a href="https://msdn.microsoft.com/en-us/library/windows/desktop/hh769303(v=vs.85).aspx">https://msdn.microsoft.com/en-us/library/windows/desktop/hh769303(v=vs.85).aspx</a>), and if you're reformatting it anyway, by all means make sure you do it with the 64K cluster size so you don't <a href="http://blog.millard.org/2015/06/maximum-ntfs-volume-expansion.html">run into any brick walls</a> if you expect to expand the volume in the future. Note that the fsutil command also shows the volume's cluster size (Bytes per Cluster) if you're wanting to check that, too.<br />
<br />
<i>Special thanks to fellow vExpert <a href="https://twitter.com/fbuechsel" target="_blank">Frank Buechsel</a>, who introduced me to using fsutil for this enquiry.</i>Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com5tag:blogger.com,1999:blog-7764963136414541684.post-43682422331308649872015-12-19T13:55:00.002-06:002016-01-14T13:21:21.294-06:00Veeam 9 and StoreOnce CatalystHPE has offered their StoreOnce deduplication platform as a <a href="http://www8.hp.com/us/en/products/data-storage/free-sovsa.html">free, 1TB virtual appliance</a> for some time (the appliance is also available for licensed 5TB and 10TB variants). As a competitor for other dedupe backup targets, it offers similar protocols and features: virtual tape library, SMB (although they persist in calling it CIFS), NFS...and a proprietary protocol branded as <i>Catalyst</i>.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx52DQii_SYGLHcUviw8C2rMtre12JzZ_prbMXzFohMwiL8nzGlrAC8w0W3N6l6y9lKnp7vkoBk7BQRuB0KPuv8Nwut6z2wACr1ytm5y6r_jW7c2AQGBurnYi9dBcjz4XN6PWxmm4wUUxg/s1600/Screenshot+-+12_19_2015+%252C+12_43_48+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx52DQii_SYGLHcUviw8C2rMtre12JzZ_prbMXzFohMwiL8nzGlrAC8w0W3N6l6y9lKnp7vkoBk7BQRuB0KPuv8Nwut6z2wACr1ytm5y6r_jW7c2AQGBurnYi9dBcjz4XN6PWxmm4wUUxg/s400/Screenshot+-+12_19_2015+%252C+12_43_48+PM.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">StoreOnce protocols</td></tr>
</tbody></table>
<a href="http://bcove.me/7hq5eda4">Catalyst is part of a unified protocol from HPE</a> that ties together several different platforms, allowing "dedupe once, replicate anywhere" functionality. Like competing protocols, Catalyst also provides some performance improvements for both reads and writes as compared to "vanilla" file protocols.<br />
<br />
Veeam has supported the StoreOnce platform since v8, but only through SMB (err... CIFS?) protocol. With the immanent release of Veeam 9—with support for Catalyst—I decided to give the free product a try and see how it works with v8, v9, and what the upgrade/migration process looks like.<br />
<br />
HPE offers the StoreOnce VSA in several variants (ESXi stand-alone, vCenter-managed and Hyper-V) and is very easy to deploy, configure and use through its integrated browser-based admin tool. Adding a storage pool is as simple as attaching a 1TB virtual disk to the VM (ideally, on a secondary HBA) before initialization.<br />
<br />
Creating SMB shares is trivial, but if the appliance is configured to use Active Directory authentication, share access must be configured through the Windows Server Manager MMC snap-in; while functional, it's about as cumbersome as one might think. StoreOnce owners would be well-served if HPE added permission/access functionality into the administrative console. Using local authentication eliminates this annoyance, and is possibly the better answer for a dedicated backup appliance...but I digress.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGZtXpZlOr-_UbUmOPUOzYllhYIMbPBCvZEsIL34pFLpfUDnb4XB71hGjTER2RwT295oSl8q_VUHpJHE3vZ_P6czCw8DbXgBod0xlYriSIStUwJtEgavqC66wXQY3cDSBNSsrvvR_Op_Z6/s1600/Screenshot+-+12_19_2015+%252C+12_52_58+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGZtXpZlOr-_UbUmOPUOzYllhYIMbPBCvZEsIL34pFLpfUDnb4XB71hGjTER2RwT295oSl8q_VUHpJHE3vZ_P6czCw8DbXgBod0xlYriSIStUwJtEgavqC66wXQY3cDSBNSsrvvR_Op_Z6/s1600/Screenshot+-+12_19_2015+%252C+12_52_58+PM.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">StoreOnce fileshare configuration</td></tr>
</tbody></table>
Irrespective of the authentication method configured on the appliance, local authentication is the only option for Catalyst stores, which are also trivial to create & configure. In practice, the data stored in a Catalyst store is not visible or accessible via file or VTL protocols—and vice-versa; at least one competing platform of which I'm familiar doesn't have this restriction. This functional distinction does make it more difficult to migrate stored data from one protocol to another; among other possible scenarios, this is particularly germane when an existing StoreOnce+Veeam user wishes to upgrade from v8 to v9 (presuming StoreOnce is also running a firmware version that is supported for Veeam's Catalyst integration) and has a significant amount of data in the file share "side" of the StoreOnce. A secondary effect is that there is no way to utilize the Catalyst store without a Catalyst-compatible software product: in my case, ingest is only possible using Veeam, whether it's one of the backup job functions or the in-console file manager.<img height="16" id="qjkbqltx5wcz" src="data:image/gif;base64,R0lGODlhEAAQAOUdAOvr69HR0cHBwby8vOzs7PHx8ff397W1tbOzs+Xl5ebm5vDw8PPz88PDw7e3t+3t7dvb2+7u7vX19eTk5OPj4+rq6tbW1unp6bu7u+fn5+jo6N/f3+/v7/7+/ra2ttXV1f39/fz8/Li4uMXFxfb29vLy8vr6+sLCwtPT0/j4+PT09MDAwL+/v7m5ubS0tM7OzsrKytra2tTU1MfHx+Li4tDQ0M/Pz9nZ2b6+vgAAAAAAAAAAAAAAAAAAAAAAAAAAACH/C05FVFNDQVBFMi4wAwEAAAAh+QQFMAA5ACwAAAAAEAAQAAAGg8CcMAcICAY5QsEwHBYPCMQhl6guGM5GNOqgVhMPbA6y5Xq/kZwkN3Fsu98EJcdYKCo5i7kKwCorVRd4GAg5GVgAfBpxaRtsZwkaiwpfD0NxkYl8QngARF8AdhmeDwl4pngUCQsVHDl2m2iveDkXcZ6YTgS3kAS0RKWxVQ+/TqydrE1BACH5BAkwADkALAAAAAAQABAAAAZ+wJwwJ1kQIgNBgDMcdh6KRILgQSAOn46TIJVSrdZGSMjpeqtgREAoYWi6BFF6xCAJS6ZyYhEIUwxNQgYkFxwBByh2gU0kKRVHi4sgOQuRTRJtJgwSBJElihwMQioqGmw5gEMLKk2AEkSBq4ElQmNNoYG2OVpDuE6Lrzmfp0NBACH5BAUwADkALAAAAAAQABAAAAaFwJwwJ1kQCDlCwTAcMh6KhDQnVSwYTkJ1un1gc5wtdxsh5iqaLbVKyVEWigq4ugZgTyiA9CK/JHIZWCsICCxpVWV/EzkHhAgth1UPQ4OOLXpScmebFA6ELHAZclBycXIULi8VZXCZawplFG05flWlakIVWravCgSaZ1CuksBDFQsAcsfFQQAh+QQJMAA5ACwAAAAAEAAQAAAGQcCccEgsGo/IpHLJzDGaOcKCCUgkAEuFNaFRbq1dJCxX2WKRCFdMmJiiEQjRp1BJwu8y5R3RWNsRBx9+SSsxgzlBACH5BAkwADkALAAAAAAQABAAAAaJwJwwJ1kQCDlCwTAcMh6KhDQnVSwYTkJ1un1gc5wtdxsh5iqaLbVKyTEWigq4ugZglRXpRX5J5DJYAFIAaVVlfhNrURqFVQ9DYhqCgzkzCGdnVQBwGRU0LQiXCRUAORQJCwAcOTChoYplBXIKLq6vUXRCCQ22olUEcroJB66KD8FNCjUrlxWpTUEAIfkEBTAAOQAsAAAAABAAEAAABobAnDAnWRAIOULBMBwyHoqENCdVLBhOQnW6fWBznC13G8nZchXNllql5Bg2xA1cZQOwShwCMdDkLgk5GVgAUgAie3syVDkTbFIaiIkIJ0NiGnp7HiNonRVVAHEuFjlQFVQVAI0JCzYjrKCPZQWnf1unYkMVWrFbBLVoUIaPD8C6CwCnAMhNQQA7" width="16" /><br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYEhqluGg1Sqxi16dhl35yLmEl5F29FTRF4QtK9kToT7-axanMcrOcG9-9rB17w-HvgEncqLjDxC4t15UQRmMQz_g6Wkn7BBW02WXNSFwbAa1lYI4rNr6Xzn4Cv-T6BGqlpZRmjqmCNT9m/s1600/Screenshot+-+12_19_2015+%252C+1_28_22+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYEhqluGg1Sqxi16dhl35yLmEl5F29FTRF4QtK9kToT7-axanMcrOcG9-9rB17w-HvgEncqLjDxC4t15UQRmMQz_g6Wkn7BBW02WXNSFwbAa1lYI4rNr6Xzn4Cv-T6BGqlpZRmjqmCNT9m/s1600/Screenshot+-+12_19_2015+%252C+1_28_22+PM.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Veeam 9 file manager</td></tr>
</tbody></table>
As of this writing, I have no process for performing the data migration from File to Catalyst without first transferring the data to an external storage platform that can be natively managed by Veeam's "Files" console. Anyone upgrading from Veeam 8 to Veeam 9 will see the existing "native" StoreOnce repositories converted to SMB repositories; as a side effect, file-level management of the StoreOnce share is lost. Any new Catalyst stores <u>can</u> be managed through the Veeam console, but the loss of file-management for the "share side" means there is no direct transfer possible. Data must be moved twice in order migrate from File to Catalyst; competing platforms that provide simultaneous access via file & "proprietary" protocols allow migration through simple repository rescans.<br />
<br />
Administrative negatives aside, the StoreOnce platform does a nice job of optimizing storage use with good dedupe ratios. Prior to implementing StoreOnce (with Veeam 8, so only SMB access), I was using Veeam-native compression & deduplication on a Linux-based NAS device. With no other changes to the backup files, migrating them from the non-dedupe NAS to StoreOnce resulted in an immediate 2x deduplication ratio; modifying the Veeam jobs to dedupe appliance-aware settings (eg, no compression at storage) saw additional gains in dedupe efficiency. After upgrading to Veeam 9 (as a member of a partner organization, I have early to the RTM build)—and going through the time-consuming process of migrating the folders from File to Catalyst—my current status is approaching 5x, giving me the feeling that dedupe performance may be superior on the Catalyst stores as compared to File shares. As far as I'm concerned, this is already pretty impressive dedupe performance (given that the majority of the job files are still using sub-optimal settings) and I'm looking forward to increasing performance as the job files cycle from the old settings to dedupe appliance-optimized as retention points are aged out.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmTXNBBOnPNGc_sgVvEukcwMMvJlKnl_Xfq6KFIFvTGfuYLQmifG6yU9Xz8Pe8Qm0brxR9pcGOwNue8DIpz-3dWP73jNZzVj_H3GclGBgi3wC5_qM4NpRJLcBTuN673Ej6fTYm4NtcORvh/s1600/Screenshot+-+12_19_2015+%252C+1_36_08+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmTXNBBOnPNGc_sgVvEukcwMMvJlKnl_Xfq6KFIFvTGfuYLQmifG6yU9Xz8Pe8Qm0brxR9pcGOwNue8DIpz-3dWP73jNZzVj_H3GclGBgi3wC5_qM4NpRJLcBTuN673Ej6fTYm4NtcORvh/s1600/Screenshot+-+12_19_2015+%252C+1_36_08+PM.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Appliance performance during simultaneous read, write operations</td></tr>
</tbody></table>
StoreOnce appliance performance will be variable, based not only on the configuration of the VM (vCPU, memory) but also on the performance of the underlying storage platform; users of existing StoreOnce physical appliances will have a fixed level of performance based on the platform/model. Users of the virtual StoreOnce appliance can inject additional performance into the system by upgrading the underlying storage (not to mention more CPU or memory, as dictated by the capacity of the appliance) to a higher performance tier.<br />
<br />
<i>Note: Veeam's deduplication appliance support—which is required for Catalyst—is only available with Enterprise (or Enterprise Plus) licensing. The 60-day trial license includes all Enterprise Plus features and can be used in conjunction with the free 1TB StoreOnce appliance license to evaluate this functionality in your environment, whether you are a current Veeam licensee or not.</i><br />
<br />
<h2>
Update</h2>
<div>
With the official release of Veeam B&R v9, Catalyst and StoreOnce are now available to those of you holding the Enterprise B&R licenses. I will caution you, however, to use a different method of converting from shares to Catalyst than I used. Moving the files <u>does</u> work, but it's not a good solution: you don't get to take advantage of the per-VM backup files that is a feature of v9 (if a backup starts with a monolithic file, it will continue to use it; only creating a new backup—or completely deleting the existing files—will allow per-VM files to be created. This is the preferred format for Catalyst, and the dedupe engine will work more efficiently with per-VM files than it will with monolithic files; I'm sure there's a technical reason for it, but I can vouch for it in practice. Prior to switching to per-VM files, my entire backup footprint, even after cycling through the monolithic files to eliminate dedupe-unfriendly elements like job-file compression, consumed over 1TB of raw storage with a dedupe ratio that never actually reached 5:1. After discarding all those jobs and starting fresh with cloned jobs and per-VM files, I now have all of my backups & restore points on a single 1TB appliance with room to spare and a dedupe ratio currently above 5:1.</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFQY2e_h5nt2HKGq_LwKgxta3UpoN7aqTE5XrnmEXKErZuty4oxKWsJUgcoYZA7Jqko6I51TPVitlzc6EAwy_XI18sKeGlCgXJ2ILLPR-YEn2595SEcIqotbuvSgst2OKzMEXfqgW-4T6d/s1600/Screenshot+-+1_14_2016+%252C+1_15_32+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFQY2e_h5nt2HKGq_LwKgxta3UpoN7aqTE5XrnmEXKErZuty4oxKWsJUgcoYZA7Jqko6I51TPVitlzc6EAwy_XI18sKeGlCgXJ2ILLPR-YEn2595SEcIqotbuvSgst2OKzMEXfqgW-4T6d/s1600/Screenshot+-+1_14_2016+%252C+1_15_32+PM.png" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgXWfjaWVwomi3CWqcEdQ8_DWtBv8x5cw_MxrAvob6uxhqSR-uIugNxgBUNeZ5tkfnsB0JzBs-8tK_T8n0cW_PjeOck7vHjIZ1w9hOfzeVZieuJxltp5KIBMhvRNQtStSmWak4EIcZXLRWa/s1600/Screenshot+-+1_14_2016+%252C+1_18_33+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgXWfjaWVwomi3CWqcEdQ8_DWtBv8x5cw_MxrAvob6uxhqSR-uIugNxgBUNeZ5tkfnsB0JzBs-8tK_T8n0cW_PjeOck7vHjIZ1w9hOfzeVZieuJxltp5KIBMhvRNQtStSmWak4EIcZXLRWa/s1600/Screenshot+-+1_14_2016+%252C+1_18_33+PM.png" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div>
I'm still fine-tuning, but I'm very pleased with the solution.</div>
Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com10tag:blogger.com,1999:blog-7764963136414541684.post-48754034956061795152015-11-23T16:49:00.002-06:002015-11-23T16:49:52.080-06:00Long-term self-signed certsWhile I'm a big proponent of using an enterprise-class certificate authority—either based on internal offline root/online issuing or public CAs—there are some instances when using a self-signed cert fits the bill. Unfortunately, most of the tools for creating a self-signed cert have defaults that result in less-than-stellar results: the digest algorithm is sha1, the cert is likely to have a 1024-bit key, and the extensions that define the cert for server and/or client authentication are missing.<br />
<br />
With a ton of references discoverable on The Interwebz, I spent a couple of hours trying to figure out how to generate a self-signed with the following characteristics:<br />
<br />
<ul>
<li>2048-bit key</li>
<li>sha256 digest</li>
<li>10-year certificate life (because, duh, I don't want to do this every year)</li>
<li>Accepted Use: server auth, client auth</li>
</ul>
<div>
It took pulling pieces from several different resources, documented herein:</div>
<div>
<br /></div>
<h4>
Required Software</h4>
<div>
OpenSSL (command-line software)</div>
<div>
Text editor (to create the config file for the cert)</div>
<h4>
Steps</h4>
<div>
<ol>
<li>Create a text file that specifies the "innards" of the cert:<br />
<blockquote>
<code>[req]<br />default_bits = 2048<br />encrypt_key = no<br />distinguished_name = req_dn<br />prompt = no<br /><br />[ req_dn ]<br />CN={replace with server fqdn}<br />OU={replace with department}<br />O={replace with company name}<br />L={replace with city name}<br />ST={replace with state name}<br />C={replace with 2-letter country code}<br /><br />[ exts ]<br />extendedKeyUsage = serverAuth,clientAuth</code></blockquote>
</li>
<li>Run the following openssl command (all one line) to create the new private key & certificate:<br />
<code>openssl req -x509 -config {replace with name of config file created above} -extensions "exts" -sha256 -nodes -days 3652 -newkey rsa:2048 -keyout host.rsa -out host.cer</code></li>
<li>Run the following openssl command to bundle the key & cert together in a bundle that can be imported into Windows:<br />
<code>openssl pkcs12 -export -out host.pfx -inkey host.rsa -in host.cer</code></li>
</ol>
</div>
<h4>
What's happening</h4>
<div>
The text file sets up a number of configuration items that you'd either be unable to specify at all (the extensions) or would have to manually input during creation (the distinguished name details).</div>
<div>
<br /></div>
<div>
The request in the second step creates a 2048-bit private key (host.rsa) and a self-signed certificate (host.cer) with a 10-year lifetime (3652 days) with the necessary usage flags and SHA256 digest.</div>
Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com1tag:blogger.com,1999:blog-7764963136414541684.post-91653845704189007362015-06-05T22:43:00.000-05:002015-06-05T23:33:42.166-05:00Resurrecting a TomTom XLI'm a longtime fan of TomTom GPS devices, and thanks to my friends over at <a href="http://www.woot.com/" target="_blank">w00t</a>, I've bought quite a few over the last score years, gifting some and reselling others.<br />
<br />
While my most reliable mapping/routing service (recently) has been Waze on my iPhone, I've had an older TomTom XL·S 310/340 that I've kept in the company car, because sometimes Waze isn't always available or accurate—more because of Verizon CDMA limitations than anything else, but that's a different story—and having a dedicated device is super convenient.<br />
<br />
I've been doing a bunch of travel in that company car, and the out-of-date map on the TomTom has become a bit of an annoyance, so unlike the XL I have for the personal car with lifetime map updates, I had a conundrum: do I purchase a new map ($45), subscribe to a year of updates ($49), punt and live with just the iPhone, or purchase a new device for home and move the one with lifetime maps to the company car and let the XL·S go to the electronics graveyard?<br />
<br />
Because the device had been working flawlessly otherwise—with the exception of essentially zero battery life—I went ahead and selected the Map Update service.<br />
<br />
After attaching the device to my PC and downloading several updates to the TomTom Home management application, the purchased map update was immediately available as an installable option. This old unit only had 2GB of local storage, so the old map had to be deleted before installing the new update; I bravely went ahead with the update process.<br />
<br />
And after a goodly while, received errors that Home was unable to copy a file to the device, so it aborted the process. The management app itself suggested disconnecting, reconnecting and retrying the update, so I did that.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6YrxBI7GrSls8VypKDrZY-ZKrlUE6YodPI1oPWFdrrffmvsyyl_rsZcyCK7PZjzd6bza1QDf7PYT9Uo7kU8D3wC9Tz925p8jBwzQU8edxyldWIyDdKXibL876G3VmsYYVhiN-vByCD1jm/s1600/TomTom+Home+Failures.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6YrxBI7GrSls8VypKDrZY-ZKrlUE6YodPI1oPWFdrrffmvsyyl_rsZcyCK7PZjzd6bza1QDf7PYT9Uo7kU8D3wC9Tz925p8jBwzQU8edxyldWIyDdKXibL876G3VmsYYVhiN-vByCD1jm/s1600/TomTom+Home+Failures.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">A common sight: errors writing to internal storage</td></tr>
</tbody></table>
Unfortunately, repeating the process didn't help: it might error out at a different file, but over and over, it would still fail.<br />
<br />
As it happens, however, when the TomTom is attached to the PC, it shows up as a removable USB drive. When interacting with the Home application, it can create backup copies of the filesystem on the PC, and by comparing the data on the properly-updating home XL, I was able to make some assumptions about the XL·S filesystem. Instead of relying on the Home application to properly transfer the map to the device, I let Windows do it, copying the map data from the downloaded ZIP file to the removable device that was the TomTom's internal storage.<br />
<br />
One problem: I was missing a file from the map download.<br />
<br />
TomTom uses DRM to keep non-subscribers from using their maps. I was fine with that: as a subscriber, I should have rights to use those maps. However, some searching on the interwebz didn't net me any solutions. Luckily, I also thought to look on my PC where Home was running; there was a second download that had an "activation.zip" file. Inspecting it, I found a .dct file; a quick google search informed me that this was my DRM key.<br />
<br />
By putting the map and the DRM key on the TomTom manually, I now had a map that was usable by the device.<br />
<br />
Or did I?<br />
<br />
While I knew I could operate the device and use the map via the Home management app, the device refused to boot independently. Again, I used my google-fu and discovered that I should be able to wipe the local storage and get Home to reinstall the boot image and application software. And after wiping, but prior to doing the install, I performed Windows filesystem checks to make sure the TomTom local storage was functional and free of errors.<br />
<br />
The Home tool worked as documented, but once again, after trying to add the map update, copy/install errors became my bane. I tried again to use Windows to copy the map update and DRM file, and lo... success! Not only would the device operate with the Home app, but it worked when independently powered.<br />
<br />
So that's the trick:<br />
<br />
<ol>
<li>Wipe the TomTom local storage. Completely.</li>
<li>Let Home reinstall the boot image and mapping application. This could require several restarts of the device, including hard resets (press and hold the power button until the TomTom logo appears and the drum sound is played).</li>
<li>Extract the PC-based map to the TomTom local storage.</li>
<li>Extract the .dct file to the map folder on the TomTom local storage.</li>
<li>Restart the TomTom.</li>
</ol>
<div>
Update:</div>
<div>
The device was working perfectly, so I continued with adding the MapShare corrections, and as the image above shows, I ran into another file transfer error. Following this error, the device refused to restart properly, getting stuck at the indemnity acknowledgement screen and spontaneously restarting. I reconnected the device and removed the most recent files from the map folder—the ones that didn't match the files received in the map update or the DRM file—and restarted the device, and it recovered nicely.<br />
<br />
Update 2:<br />
Before anyone asks: the .dct file that's the DRM key is specifically created by TomTom for my use on <i>this device alone</i> and is unusable on any other device, with any other map. The device serial number and map thumbprint are both part of the decryption key for DRM, so even if I didn't care about TomTom's IP rights and the possibility of litigation for it (which I actually do on both accounts), sharing the DRM file with the world wouldn't help anyone. So no, I <b>will not</b> share any of the files I received from TomTom in this update process.</div>
Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com5tag:blogger.com,1999:blog-7764963136414541684.post-60739124234936968232015-06-03T17:04:00.000-05:002015-06-05T22:44:11.817-05:00Maximum NTFS Volume ExpansionA peer recently had an issue when working on a client system: After adding a second shelf of SAS-attached drives to a physical Windows Storage Server and doubling the available capacity of the environment from ~20TB to ~40TB, he was unable to extend the existing NTFS volume after extending the SAS array group.<br />
<div>
<br /></div>
<div>
The error was "The volume cannot be extended because the number of clusters will exceed the maximum number of clusters supported by the filesystem."</div>
<div class="separator" style="clear: both; text-align: center;">
<img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPEuXnbtFydRsl46XlpU1ExT0jRzs4VHqc3FNiTAvjEkMimT5BwEdPnQTDOz6EP2MhvVF1_Hmfp_MRI_M_S4gyclY8-Oc5tmvXYatBBv90AO8XOKzbDgLv0Ixv-ufT76xZdVdscv1EUee_/s1600/Screenshot+-+6_3_2015+%252C+4_27_40+PM.png" /></div>
<div>
The original volume was reportedly formatted "using the defaults," which under most circumstances would mean it was using 4K clusters. Why wouldn't it allow extending the volume?</div>
<div>
<br /></div>
<div>
<b>Because NTFS </b>(as currently implemented)<b> has a cluster limit of 2<sup>32</sup>-1 clusters per volume.</b><br />
<br />
When you "do the math," that cluster limit does impose some hard limits on the maximum size of the NTFS volume, irrespective of the actual drive space that is available for the volume. And trying to use tricks like dynamic disks and software RAID won't help: those tricks modify the underlying disk structure, <b>not</b> the NTFS filesystem that "rides" on top of it.<br />
<br />
<table align="center" border="1" cellpadding="5" cellspacing="0">
<tbody>
<tr>
<td align="center" colspan="6">Max NTFS Volume by Cluster Size</td>
</tr>
<tr>
<td align="center">cluster<br />
size (B)</td>
<td align="center">Bytes</td>
<td align="center">KB</td>
<td align="center">MB</td>
<td align="center">GB</td>
<td align="center">TB</td>
</tr>
<tr>
<td align="right">512</td>
<td align="right">2,199,023,255,040</td>
<td align="right">2,147,483,648</td>
<td align="right">2,097,152</td>
<td align="right">2,048</td>
<td align="right">2</td>
</tr>
<tr>
<td align="right">1024</td>
<td align="right">4,398,046,510,080</td>
<td align="right">4,294,967,295</td>
<td align="right">4,194,304</td>
<td align="right">4,096</td>
<td align="right">4</td>
</tr>
<tr>
<td align="right">2048</td>
<td align="right">8,796,093,020,160</td>
<td align="right">8,589,934,590</td>
<td align="right">8,388,608</td>
<td align="right">8,192</td>
<td align="right">8</td>
</tr>
<tr>
<td align="right">4096</td>
<td align="right">17,592,186,040,320</td>
<td align="right">17,179,869,180</td>
<td align="right">16,777,216</td>
<td align="right">16,384</td>
<td align="right">16</td>
</tr>
<tr>
<td align="right">8192</td>
<td align="right">35,184,372,080,640</td>
<td align="right">34,359,738,360</td>
<td align="right">33,554,432</td>
<td align="right">32,768</td>
<td align="right">32</td>
</tr>
<tr>
<td align="right">16384</td>
<td align="right">70,368,744,161,280</td>
<td align="right">68,719,476,720</td>
<td align="right">67,108,864</td>
<td align="right">65,536</td>
<td align="right">64</td>
</tr>
<tr>
<td align="right">32768</td>
<td align="right">140,737,488,322,560</td>
<td align="right">137,438,953,440</td>
<td align="right">134,217,728</td>
<td align="right">131,072</td>
<td align="right">128</td>
</tr>
<tr>
<td align="right">65536</td>
<td align="right">281,474,976,645,120</td>
<td align="right">274,877,906,880</td>
<td align="right">268,435,456</td>
<td align="right">262,144</td>
<td align="right">256</td>
</tr>
</tbody></table>
</div>
<br />
We knew that we had a functioning 20TB volume, so we verified my theory that the volume was actually formatted with 8K clusters (the smallest size that would support 20TB) using DISKPART's FILESYSTEM command. Sure enough: 8192 was the cluster size.<br />
<br />
We gave the client several options for addressing the issue, including the purchase of software that could "live adjust" the cluster sizing. In the end, the client chose the "migrate->reformat->migrate" option, and while it took a long time to perform (20TB is a <b>lot </b>of data!), it was successful.Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com5tag:blogger.com,1999:blog-7764963136414541684.post-47629333344807278172015-04-07T12:31:00.001-05:002015-04-07T13:31:32.559-05:00Don't say "Customer"<a href="http://www.merriam-webster.com/dictionary/customer" target="_blank">customer</a><br />
<dl>
<dd>one that purchases a commodity or service</dd></dl>
According to the above definition from Webster's, it's quite likely that you deal with people that can be labelled with that term. Even those of you who are not in specific sales roles may have used the term to refer to those you serve (eg, "the internal customer").<br />
<br />
Good or bad, that term has a certain "transactional" connotation to it; the retail model is pretty clear: a person goes to a store, buys a product or receives a service, then leaves. You don't have a relationship with the clerk running the point-of-sale system; you get your stuff and go. Transactional.<br />
<br />
For gasoline purchases and groceries, this may be the right model and the right term. But is that the right model for you in IT? Consider it instead from the reverse point of view:<br />
<br />
What is your relationship with your barber/hair stylist (Some of you reading this might not use one: your <a href="http://www.merriam-webster.com/dictionary/tonsorial" target="_blank">tonsorial</a> needs may be non-existent or you may be able to handle things for yourself. But I bet you can probably remember a time when it was a regular requirement...)? Do you have a certain place to which you always return? When you go there, is there a particular person to whom you look (or schedule or wait) because you get a consistently good cut, or because the conversation is more pleasant, or some other (in)tangible benefit? Do you like the personal service that comes with being known by your first name? Would you have a hard time switching to a different barber/stylist because of the trust you've given to your current one? <b>That</b> is the sort of relationship you should seek to have with your...<u>clients</u>.<br />
<dl>
<dt><a href="http://www.merriam-webster.com/dictionary/client" target="_blank">client</a></dt>
<dd><ol>
<li>one that is under the protection of another : dependent</li>
<li>a: a person who engages the professional advice or services of another<br />b: customer</li>
</ol>
</dd></dl>
While a client <u>can</u> be a form of customer, the superior definitions imply a more intimate relationship between the two parties. Dependency (used in the primary definition) can have both negative and positive connotations, but in this context, we're essentially talking about repeat business. This is what we're after, both as provider and consumer: as a provider, it keeps me in business and employed; as a consumer, I can either add the process of determining "from whom" as the same time I'm trying to figure out the "what" for my need. As a client, you put a certain level of trust into the relationship: you trust that your provider will have your interests at heart; that you won't be taken advantage of; that you can rely on the quality of the work.<br />
<br />
That, then, pushes a certain level of responsibility on the provider. Be(come) the trusted adviser by not abusing the trust. Provide good advice. Provide repeatably good service and/or products. Own your mistakes and gently guide your client away from making them on their own. This is how one treats his/her clients—especially if the goal is to keep them!<br />
<br />
If you're not already in this frame of mind, I challenge you to make this shift in a simple yet subtle way: Even if you're in the "business of selling widgets," <i>even if you're running a convenience store selling gasoline and snack food</i>, train yourself to stop using the word "customer" and start using the word "client" instead. Words have power; they convey ideas and have implications. Changing the use of that one word should change the way you look at the people you serve; when your outlook changes, the way you act and react in the relationship should follow. All of your clientele may not perceive the difference, either overtly or subconsciously; some still want to be "merely" customers, ignoring the relationship and simply needing a widget or two. Making this adjustment won't "fix" that relationship, but neither should it affect your ability to be there to serve them when they choose you. But the shoppers, the fence-sitters? With this one subtle change, you could influence them in a way that sends them into your care with more frequency.<br />
<br />
<i>Disclaimer: I currently work for a value-added reseller—a "VAR" in industry parlance—but have also spent a long time as a purchaser of products and services. I believe this concept is valid in either case.</i>Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com5tag:blogger.com,1999:blog-7764963136414541684.post-86795349935977107852015-03-08T18:51:00.001-05:002015-03-08T18:58:20.608-05:00Fix vShield Manager after modifying vSphere VDS uplinksIf you've been following my posts about <a href="http://blog.millard.org/2015/02/homelab-2015-hello-10gbps.html" target="_blank">upgrading my home lab</a>, you know that I removed the add-in 1Gbps NICs and consolidated the motherboard-based 1Gbps NICs on one DVS (distributed virtual switch) in order to add 10Gbps support to my hosts. In that process, I not only rearranged the physical NICs for the uplinks, I also updated the uplink names in order to keep my environment self-documenting.<br />
<br />
Things pretty much went as planned, but I didn't expect vShield Manager (vSM) to choke on the changes: when updating the uplink names for the DVS that provided the VXLAN port group, I expected vSM to recognize the changes and handle creation of new VXLAN networks without issue. I was wrong.<br />
<br />
The first symptom that I had an issue was the inability of vCloud Director (vCD) to create a new Organizational Network on a deployed Edge device:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgj42zaDCADkYsEOpQM3q1zKsC9jo8aSlclJNp-z4Bi8-OPnARaRB7Szq7bH08nYQHFHd5h6JKV7YAveVc2AiEmbzogJ177ZB-BpantYGFYI7RXKhdf9AfLCZtpxOpvuwnQgDTuSqw0dw1D/s1600/Screenshot+-+3_8_2015+,+4_22_56+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgj42zaDCADkYsEOpQM3q1zKsC9jo8aSlclJNp-z4Bi8-OPnARaRB7Szq7bH08nYQHFHd5h6JKV7YAveVc2AiEmbzogJ177ZB-BpantYGFYI7RXKhdf9AfLCZtpxOpvuwnQgDTuSqw0dw1D/s1600/Screenshot+-+3_8_2015+,+4_22_56+PM.png" height="536" width="640" /></a>
</div>
So: something is off with the teaming policy. Time to look at vSM to determine whether vCD is sending a bad request to vSM, or if vSM itself is the source of the issue. The easiest way to check is to manually create a new network in vSM; if it succeeds, vCD is sending a bad request, otherwise I need to troubleshoot vSM--and possibly vCenter, too.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrMojwmKg9bp5H-xkOu3D-AhkMm1B8N-qyIaSbM2R0_Xg5alEK5_Vq0NT13ZuK3A6Lq_2_rPvcRU8kt5RTPWhpcaMKOEWy-zgokBGyRF4QqTfpQgOSIINc4-uqEf-ppSJ12bRZPPKaGhl6/s1600/Screenshot+-+3_8_2015+,+4_17_12+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrMojwmKg9bp5H-xkOu3D-AhkMm1B8N-qyIaSbM2R0_Xg5alEK5_Vq0NT13ZuK3A6Lq_2_rPvcRU8kt5RTPWhpcaMKOEWy-zgokBGyRF4QqTfpQgOSIINc4-uqEf-ppSJ12bRZPPKaGhl6/s1600/Screenshot+-+3_8_2015+,+4_17_12+PM.png" /></a>
</div>
Boom: the problem is reproduced even for a test directly in vSM. Time to verify the teaming in the base portgroup in vCenter.
<br />
Oops. I hadn't updated the portgroup for VXLAN after moving the uplinks around, although I had done so for the other portgroups on the DVS.
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTm5D5HLplAKiMvHejgF2L1bilckj0MVNXPGa27Ze3Y1nUX8IBoRHaUmdmv6PuZAzLunkK_dkQTvKV7AV_F5FdguUomzQdMgYM0puG5BEJyJQPSurjQ733w2wKlI6wBh7N1kHFyVoYubS9/s1600/Screenshot+-+3_8_2015+,+4_19_34+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTm5D5HLplAKiMvHejgF2L1bilckj0MVNXPGa27Ze3Y1nUX8IBoRHaUmdmv6PuZAzLunkK_dkQTvKV7AV_F5FdguUomzQdMgYM0puG5BEJyJQPSurjQ733w2wKlI6wBh7N1kHFyVoYubS9/s1600/Screenshot+-+3_8_2015+,+4_19_34+PM.png" height="148" width="320" /></a>
</div>
Unfortunately, updating the portgroup to use all the available uplinks didn't help. However, in the process, I discovered an unexpected error in vCenter itself:
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEglAxXj7KOhbNkFtIfEmmbll2eEFWEkr4yuRHk2ElJLZBg_aIS7fXMtihPu72vEj1jgxkXJipap7OkpRSNlT0GUqemZgy-gDQjvhh3LLwzfyvClS2bF98IUn8Zkj6IPl43ESA0TggWe0U75/s1600/Screenshot+-+3_8_2015+,+4_21_40+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEglAxXj7KOhbNkFtIfEmmbll2eEFWEkr4yuRHk2ElJLZBg_aIS7fXMtihPu72vEj1jgxkXJipap7OkpRSNlT0GUqemZgy-gDQjvhh3LLwzfyvClS2bF98IUn8Zkj6IPl43ESA0TggWe0U75/s1600/Screenshot+-+3_8_2015+,+4_21_40+PM.png" height="224" width="320" /></a>
</div>
vSM was making an API call to vCenter that included one of the old uplink names, one which no longer existed on the DVS. To test the theory, I added a couple of additional uplink ports to the DVS and renamed one to match the missing port. It worked, but not as expected:
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihQjvj1oipJYwz5Fi9twau5yEcY38xJ-tOC6a8yz42Z3d6ojH595ImOpCHNZC6H6-XeEUfl6Sjs3bA9M4GQTRE-6btwd-q0ijNsrKywWTDaEzosu8kX68H5UcEAmNPxJHlnd7vxLfhNRTr/s1600/Screenshot+-+3_8_2015+,+4_28_50+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihQjvj1oipJYwz5Fi9twau5yEcY38xJ-tOC6a8yz42Z3d6ojH595ImOpCHNZC6H6-XeEUfl6Sjs3bA9M4GQTRE-6btwd-q0ijNsrKywWTDaEzosu8kX68H5UcEAmNPxJHlnd7vxLfhNRTr/s1600/Screenshot+-+3_8_2015+,+4_28_50+PM.png" /></a>
</div>
vSM was able to send a proper API call to vCenter, but the portgroup had sub-optimal uplink settings: of the two active uplinks, only one had an actual, physical uplink associated with it. This was not a redundant connection, even though it looked like it.<br />
<br />
Time to restart vSM to get it to re-read the vCenter DVS config, right? Wrong. Even with a restart & re-entering the vCenter credentials, the state persisted.<br />
<br />
At this point, my Google-fu failed me: no useful hits on a variety of search terms. Time to hit the <a href="http://communities.vmware.com/" target="_blank">VMware Community Forums</a> with a question. Luckily, I received a <a href="https://communities.vmware.com/message/2483122#2483122" target="_blank">promising answer</a> in just a day or two.<br />
<br />
I learned that one can use the REST API for vSM to reconfigure it, which can get it back in line with reality. But how do you work with arbitrary REST calls? It turns out, there's a <a href="https://addons.mozilla.org/en-us/firefox/addon/restclient/" target="_blank">REST client plug-in</a> for Firefox, written to troubleshoot and debug REST APIs. It works a treat:
<br />
<ol>
<li>Set up the client for authenticated headers</li>
<li>Retrieve the DVS configuration as an XML blob in the body of a GET call</li>
<li>Modify the XML blob so that it has the correct properties</li>
<li>PUT the revised XML blob back to vSM.</li>
</ol>
Voila! Everything works.<br />
<br />
Specifics:
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPig2Nt8yzyQ_91U3ta6mZNbRRi8BlWVB6KoZBM09BlN2zMGykicKWwPK8iXvDd_gCXGZNtP7gWKu1tkQSttavajo_BFqqmiA2qwqSmvlLZupoTPCaGshsFpYLlO197TUY9ePYgTtUjF5E/s1600/Screenshot+-+3_8_2015+,+5_22_51+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPig2Nt8yzyQ_91U3ta6mZNbRRi8BlWVB6KoZBM09BlN2zMGykicKWwPK8iXvDd_gCXGZNtP7gWKu1tkQSttavajo_BFqqmiA2qwqSmvlLZupoTPCaGshsFpYLlO197TUY9ePYgTtUjF5E/s1600/Screenshot+-+3_8_2015+,+5_22_51+PM.png" height="417" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">1) Use an Authenticated GET on the switches API</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRsg_3uDONMFj-HlwGJuZFkrVfj3g10UJe6WbRZbjAmE4_pMOmfKMOdco1TIqZpAz-yKYXYKGZ3f0RlW86NwXZZsVU6gHO13EJhp6XiKuOrJprre_KzujuL3jE5uCwlKSNSYp-gECftA6S/s1600/Screenshot+-+3_8_2015+,+5_28_27+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRsg_3uDONMFj-HlwGJuZFkrVfj3g10UJe6WbRZbjAmE4_pMOmfKMOdco1TIqZpAz-yKYXYKGZ3f0RlW86NwXZZsVU6gHO13EJhp6XiKuOrJprre_KzujuL3jE5uCwlKSNSYp-gECftA6S/s1600/Screenshot+-+3_8_2015+,+5_28_27+PM.png" height="370" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">2) Using the objectId of the desired DVS, get the specific switch data</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOM22oHx3pxC59JbUJ4oYDg-ihz8OLFQnpPtmnTtsGE8RR7b828q5_lmR0ZRPqpPm4z1_vgIxqvjwjTqI7Usgvezi0AYlEOLgIJ6EC0Uwvgpaot_TWPzdxzxUcKKEE5V3NGP8Ud1H6XJch/s1600/Screenshot+-+3_8_2015+,+5_31_19+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOM22oHx3pxC59JbUJ4oYDg-ihz8OLFQnpPtmnTtsGE8RR7b828q5_lmR0ZRPqpPm4z1_vgIxqvjwjTqI7Usgvezi0AYlEOLgIJ6EC0Uwvgpaot_TWPzdxzxUcKKEE5V3NGP8Ud1H6XJch/s1600/Screenshot+-+3_8_2015+,+5_31_19+PM.png" height="196" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">3) Update the XML blob with the correct uplink names</td></tr>
</tbody></table>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirKOx8tk64LFXIcMYzFcax-9HKC4Nrhs2INvX2AQaMxQmfdhc-RH2VCf4gCYxkQ80QQngIw4vO9aXJX8JoKTro_e3r8fNM85E__oheeK1Fob9fFjAkzynhF1goE19Eff6Z5H7MKeqOt1m3/s1600/Screenshot+-+3_8_2015+,+5_26_50+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirKOx8tk64LFXIcMYzFcax-9HKC4Nrhs2INvX2AQaMxQmfdhc-RH2VCf4gCYxkQ80QQngIw4vO9aXJX8JoKTro_e3r8fNM85E__oheeK1Fob9fFjAkzynhF1goE19Eff6Z5H7MKeqOt1m3/s1600/Screenshot+-+3_8_2015+,+5_26_50+PM.png" height="210" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">4) PUT the revised XML blob</td></tr>
</tbody></table>
As soon as this blob was accepted with a <span style="font-family: Courier New, Courier, monospace;">200 OK</span> response, I re-ran my test in vSM: success! vCD was also able to successfully create the desired portgroup, too.<br />
<br />
Key takeaways:<br />
<ol>
<li>REST Client for Firefox is awesome for arbitrary interaction with a REST API</li>
<li>Sometimes, the only way to accomplish a goal is through the API; a GUI or CLI command may not exist to fix your problem.</li>
<li>This particular fix allows you to arbitrarily rename your uplinks without having to reset the vShield Manager database and completely reinstall it to get VXLAN working again.</li>
</ol>
Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com4tag:blogger.com,1999:blog-7764963136414541684.post-78650308097377693492015-02-23T17:22:00.000-06:002015-02-23T17:22:18.993-06:00Planning for vSphere 6: VMCA considerationsWith the immanent release of vSphere 6, I've been doing prep work for upgrades and new installs. There's a lot of information out there (just check out <a href="http://vsphere-land.com/news/vsphere-6-0-link-o-rama.html" target="_blank">the vSphere 6 Link-O-Rama</a> at <a href="https://twitter.com/ericsiebert" target="_blank">Eric Siebert</a>'s <a href="http://vsphere-land.com/" target="_blank">vSphere Land</a> for an idea of the breadth & depth of what's already written), but not as much as I'd like to make good decisions in order to future-proof the setup.<br />
<br />
I'm sure I join lots of VMware admins in looking forward to the new features in vSphere 6—long-distance vMotion, cross-datacenter & cross-vCenter vMotion, multi-vCPU fault tolerance (FT), etc.—but along with these features come some foundational changes in the way vSphere management & security are architected.<br />
<br />
Hopefully, you've already heard about the new PSC (Platform Services Controller), the functional descendant of the SSO service introduced in vSphere 5.1. SSO still exists as a component of the PSC, and the PSC can be co-installed ("embedded") on the same system as vCenter, or it can be independent. Like SSO on vSphere 5.5, the PSC has its own internal, dedicated database which it replicates with peer nodes, similar to what we've come to know and expect from Microsoft Active Directory.<br />
<br />
This replication feature not only helps for geographically-distributed enterprises—allowing a single security authority for multiple datacenters—but high availability in a single datacenter through the use of 2 (or more) PSCs <i>behind a load balancer</i>. Note the emphasis on the load balancer: you will end up with the abstraction of the PSC with a DNS name pointing at an IP address on your load balancing solution, rather than the name/IP of a PSC itself.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFAIS-AhbWWlQggh1OGA7iibtgiGjCHtHfVgx8pk7KfUkWgtbIy-hBIbBsYE9sguIHrsdBV4Wg-dN-mCL6n4sdCbnJp-nyzFWZGNM6Uyr0vptk_l774o19SNv3AwpsIAuszPydKLrg0i3z/s1600/PSC_HA.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFAIS-AhbWWlQggh1OGA7iibtgiGjCHtHfVgx8pk7KfUkWgtbIy-hBIbBsYE9sguIHrsdBV4Wg-dN-mCL6n4sdCbnJp-nyzFWZGNM6Uyr0vptk_l774o19SNv3AwpsIAuszPydKLrg0i3z/s1600/PSC_HA.png" height="320" width="164" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">PSC in load-balanced HA configuration</td></tr>
</tbody></table>
<br />
This delegation means you must plan ahead of time for using load balancing; it's really not the sort of thing that you can "shim" into the environment after implementing a single PSC.<br />
<br />
Joining SSO in the PSC "black box" are several old and some brand new services: identity management, licensing...and a new Certificate Authority, aka VMCA (not to be confused with vCMA, the vCenter Mobile Access fling) .<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjV9j2c1ClMcZmQsO61C2guUnq_XmqjIgbT1HH1Qg8TN8G8JC-IBOJRSDeaqO3LZ6G54vop-sb5f3tZKW59Nt5XJbf3C0-aElR7KMTmBgmmB3RVNG2tOEXOP37u1uGkaMHFRKO_WtaSxAdW/s1600/PSC_components.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjV9j2c1ClMcZmQsO61C2guUnq_XmqjIgbT1HH1Qg8TN8G8JC-IBOJRSDeaqO3LZ6G54vop-sb5f3tZKW59Nt5XJbf3C0-aElR7KMTmBgmmB3RVNG2tOEXOP37u1uGkaMHFRKO_WtaSxAdW/s1600/PSC_components.png" height="307" width="640" /></a></div>
It's that last item—the Certificate Authority—that should make you very nervous in your planning for vSphere 6. The documentation so far indicates that you have upwards of four different modes for your CA implementation that are independent of your PSC implementation choices:<br />
<ul>
<li>Root (the default). Source of all certs for dependent services, this CA's public key/cert must be distributed and placed in your trusted CA store.</li>
<li>Intermediate. Source of all certs for dependent services, but the CA itself gets its trust from a parent CA, which in turn must have its public key/cert distributed. In the case of corporate/Enterprise CA/PKI infrastructure, this will already be in place <i>and will be my go-to configuration</i>.</li>
<li>None/External-only. All services receive their certs from a different CA all together. This model is equivalent to removing all the self-signed certificates in pre-6 and replacing them with signed certificates. With the proliferation of services, each using its own certificate, this model is becoming untenable.</li>
<li>Hybrid. In the hybrid model, the VMCA provides certificates to services that provide internal communication (either service-to-service or client-to-service) while public CA-signed certs are used in the specific places where 3rd-party clients will interact. In this model, the VMCA may act as either root or intermediate CA.</li>
</ul>
<div>
Confused? Just wait: it gets more complicated...</div>
<div>
<br /></div>
<div>
Migrating from one model to another will have risks & difficulties associated with it. The default installer will set you up with a root CA; you will have the option to make it an intermediate at time of install. As near as I can tell from the available documentation, you will need to reinstall the PSC if you start with it as a root CA and decide you want it instead to be an intermediate (or vice-versa). This is consistent with other CA types (eg, Microsoft Windows), so there's no surprise there; however, it's not clear what other replicated services will be impacted when trying to swap CA modes, as it will require previously-issued certificates to be revoked and new certificates to be issued.</div>
<div>
<br /></div>
<div>
You can switch some or all of the certificates it manages with 3rd-party (or Enterprise) signed certs, but once you do, you will have to deal with re-issue & expiration on your own. I can't find anything documenting whether this is handled gracefully & automatically with VMCA-signed certs & services, similar to the centralized/automated certificate management that Windows enjoys in an Enterprise CA environment.</div>
<div>
<br /></div>
<div>
There isn't any documentation on switching from 3rd-party (or Enterprise) <i>back</i> to a VMCA-signed certificate. Presumably, it'll be some CLI-based witchcraft...if it's allowed at all.</div>
<div>
<br /></div>
<div>
Finally, keep in mind that DNS names factor heavily into certificate trust. Successfully changing name and/or IP address of an SSO server—depending on which was used for service registration—can be challenging enough. Doing the same with a system that is also a certificate authority will be doubly so.</div>
<div>
<br /></div>
<div>
So: what's the architect going to do?</div>
<div>
<br /></div>
<div>
For small, non-complex environments, I'm going to recommend what VMware and other bloggers recommend: stick with the single, combined PSC and vCenter server. Use the VCSA (vCenter Server Appliance) to save on the Windows license if you must, but I personally still prefer the Windows version: I'm still far more comfortable with managing the Windows OS environment & database than I am with Linux. Additionally, you're going to want Update Manager—still a Windows service—so I find it easier to just keep them all together.</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiaA7OGmDo_HLbdo-N4Go36iuqG5RpW0V4r0rrdEX3oeSMLJVqjqStFGafoR-9UM9Y-SIUmuHjQrzOpmIWqEzRLHfUq_osU5BJdNm0lRpcM5NYc-iCdg-PKrVw6tigsEIo58BLiVhyphenhyphenoeZf/s1600/embedded.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiaA7OGmDo_HLbdo-N4Go36iuqG5RpW0V4r0rrdEX3oeSMLJVqjqStFGafoR-9UM9Y-SIUmuHjQrzOpmIWqEzRLHfUq_osU5BJdNm0lRpcM5NYc-iCdg-PKrVw6tigsEIo58BLiVhyphenhyphenoeZf/s1600/embedded.png" /></a></div>
<div>
<br /></div>
<div>
This also suggests using the VMCA as a root CA, and I'll stick with that recommendation <i>unless you have an Enterprise CA already</i>. If you have the Enterprise CA, why not make it an intermediate? At a minimum, you'll would eliminate the need for yet another root certificate to distribute. More importantly, however, is that it's vastly easier to replace an intermediate CA—even through the pain of re-issuing certificates—than a root CA.</div>
<div>
<br /></div>
<div>
What constitutes small, non-complex? For starters, any environment that exists with one—and only one—vCenter server. You can look up the maximums yourself, but we're talking about a single datacenter with one or two clusters of hosts, so less than 65 hosts for vSphere 5.5; in practice, we're really talking about environments with 20 or fewer hosts, but I have seen larger ones that would still meet this category because—other than basic guest management (eg, HA & DRS)—they aren't really using vCenter for anything. If it were to die a horrible death and be redeployed, the business might not even notice!</div>
<div>
<br /></div>
<div>
Even if you have a small environment by those standards, however, "complex" enters the equation as soon as you implement a feature that is significantly dependent on vCenter services: Distributed Virtual Switch, Horizon View non-persistent desktops, vRealize Automation, etc. At this point, you now need vCenter to be alive & well pretty much all the time.</div>
<div>
<br /></div>
<div>
In these environments, I was already counseling the use of a full SQL database instance, not SQL Express with all of its limitations. Even when you're inside the "performance bubble" for that RDBMS, there are a host of other administrative features you must do without that can compromise uptime. With vSphere 6, I'm continuing the recommendation, but taking it a step further: use AlwaysOn Availability Groups for that database as soon as it's certified. It's far easier to resurrect a cratered vCenter server with a valid copy of the database than rebuilding everything from scratch; I know VMware wants us all to treat the VCSA as this tidy little "black box," but I've already been on troubleshooting calls where major rework was required because no maintenance of the internal PosgreSQL database was ever done, and the whole-VM backup was found wanting...</div>
<div>
<br /></div>
<div>
Once you've got your database with high availability, split out the PSC from vCenter and set up <b>at least</b> two of them, the same way you'd set up at least two Active Directory domain controllers. This is going to be the hub of your environment, as both vCenter and other services will rely on it. Using a pair will also require a load balancing solution; although there aren't any throughput data available, I'd guess that the traffic generated for the PSC will be lower than the 10Mbps limit of the <b>free</b> and excellent Citrix NetScaler VPX Express. I've <a href="http://blog.millard.org/search/label/NetScaler" target="_blank">written about it before</a>, and will be using it in my own environment.<br />
<br />
Add additional single and/or paired PSCs in geographically distant locations, but don't go crazy: I've seen blogs indicating that the replication domain for the PSC database is limited to 8 nodes. If you're a global enterprise with many geographically-diverse datacenters, consider a pair in your primary, most critical datacenter and single nodes in up to 6 additional datacenters. Have more than 7 datacenters? Consider the resiliency of your intranet connectivity and place the nodes where they will provide needed coverage based on latency and reliability. If you're stumped, give your local Active Directory maven a call; he/she has probably dealt with this exact problem already—albeit on a different platform—and may have insight or quantitative data that may help you make your decision.<br />
<br />
Finally, I'm waiting with anticipation on an official announcement for FT support of vCenter Server , which will eliminate the need for more-complex clustering solutions in environments that can support it (from both a storage & network standpoint: FT in vSphere 6 is completely different from FT in previous versions!). Until then, the vCenter Server gets uptime & redundancy more through keeping its database reliable than anything else: HA for host failure; good, tested backups for VM corruption.</div>
Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com5tag:blogger.com,1999:blog-7764963136414541684.post-81571288318822845192015-02-10T18:32:00.001-06:002015-02-10T18:32:13.871-06:00HP StoreVirtual VSA: The Gold StandardHP has owned the Left Hand storage system since late 2008, and has made steady improvements since then. The product had already been officially supported on a VM; not only did the acquisition not destroy that option, but HP has embraced the product as a cornerstone of their "software-defined storage" marketing message.<br />
<br />
Although other products existed back in 2008, a virtualized Left Hand node was one of the first virtual storage appliances (VSA) available with support for production workloads.<br />
<br />
Fast-forward to August, 2012: HP elects to rebrand the Left Hand product as StoreVirtual, renaming the <i>SAN/iQ</i> to <i>LeftHand OS</i> in order to preserve its heritage. The 10.0 version update was tied to the rebranding, and the VSA arm of the portfolio—HP never stopped producing "bare-metal" arrays based on their 2U DL380 server chassis—promised to bring additional enhancements like increased capacity (10TB instead of 2TB) and better performance (2 vCPUs instead of 1) along with price drops.<br />
<br />
The 11.0 version was released with even more features (11.5 is the production/shipping version for both bare-metal and VSA), chief of which—in my opinion—is <i>Adaptive Optimization</i> (AO), the ability for node-attached storage to be characterized in one of two tiers.<br />
<br />
Note that this isn't a Flash/SSD-specific feature! Yes, it works with solid state as one of the tiers—and is the preferred architecture—but any two performance-vs-capacity tiers can be configured for a node: a pair of 15K RPM SAS drives as Tier 0 performance with 4-8 NL SAS drives as Tier 1 capacity is just as legitimate. HP cautions the architect, however, not to mix nodes with varying AO characteristics in the same way it cautions against mixing single-tier nodes in one cluster.<br />
<hr />
Personally, I've played with the StoreVirtual VSA off and on over the years. The original hold-back for getting deeply into it was the trial duration: 30 to 60 days is insufficient to "live with" a product and really get to know it. In early 2013, however, HP offered NFR licensing to qualified members of the VMware vExpert community, and those licenses had year-long duration associated with them.<br />
<br />
Unfortunately, however, the hosts I was running at home were pretty unsuited to supporting the VSA: limited RAM and 2-4 grossly inferior desktop-class SATA hard drives in each of 2 hosts. I'd still load up the VSA for test purposes; not for performance, but to understand the LeftHand OS better and how failures are handled, configurations are managed, and how the product interacts with other software like Veeam Backup & Recovery. But then I'd also tear down the cluster when I finished with it in order to regain consumed resources.<br />
<br />
When PernixData FVP was still in pre-GA beta, I was able to make some system upgrades to add SSD to newer hosts—still with essentially zero local capacity, however—and was able to prove to myself that a) solid state works very effectively at increasing the performance of storage and b) there is a place for storage in the local host.<br />
<br />
With the release of the first VMware Virtual SAN beta, I decided it was time to make some additional investments into my lab, and I was able to not only add a third host (the minimum for supported VSAN deployment) but also provision them all with a second SSD and enterprise SATA disks for the experiment. In that configuration, I was able to use one SSD for iSCSI-based performance acceleration (using the now-GA FVP product) and a second SSD for VSAN's solid state tier. My hosts remained limited in the number of "spinning disk" drives that could be installed (four), but in aggregate across three hosts, the total seemed not only reasonable, but seemed to work in practice.<br />
<br />
Unfortunately, I was plagued by hardware issues in this configuration: rarely a week went by without either FVP or VSAN complaining about a drive going offline or being in "permanent failure," and it seemed like the weeks when that didn't occur, the Profile Driven Storage service of vCenter—which is critical to making use of VSAN in other products like vCloud Director or Horizon View—would need to be restarted. Getting FVP or VSAN working correctly would usually require rebooting the host reporting an issue; in some cases, VMs would need to be evacuated from VSAN to provide the necessary free space to retain "availability."<br />
<br />
In short, the lab environment with my 1Gbps networking and consumer-grade disk & HBA made VSAN and FVP a little too much work.<br />
<br />
But I still had that VSA license... If I could get a better HBA—one that would perform true hardware-based RAID and have deeper queue, not to mention other enterprise SATA/SAS capabilities—I'd be able to leverage the existing disk investment with the VSA and have a better experience.<br />
<br />
I was able to source a set of Dell PERC H700 adapters, cables and cache batteries from eBay; these were pulled from R610 systems, so dropping them into mine was trivial and the set cost considerably less than a single kit from Dell. Although I could have rebuilt the VSAN and FVP environments on the new HBA—each disk in the system would need to be set up as a single-spindle RAID0 'virtual volume'—I went with a RAID1 set for the pair of SSD and a RAID5 for the spindles. I would be able to continue leveraging PernixData for acceleration using the RAM-backed function, but I was done messing with VSAN for now.<br />
<hr />
Setting up the v11.5 VSA initially gave me pause: I was booting from SD card, so I could use 100% of the SSD/HDD for it, but how to do it? If the LeftHand OS had drivers for the PERC array—possible: the core silicon of the H700 is a LSI/Symbios product which might be supported in spite of being a Dell OEM—I could do a DirectPath I/O if there was another datastore available on which to run the VSA. A second, similar alternative would be to manually create Physical RDM mappings for the RAID volumes, but that still left the problem of a datastore for the VSA. Yes, I could run the VSA on another array, but if the host ever had issues with that array, then I'd also end up with issues on my LeftHand cluster—not a good idea!<br />
<br />
My final solution is a hybrid: The HDD-based RAID group is formatted as a VMFS5 datastore, and the VSA is the only VM using it. A large, 1.25TB 'traditional' VMDK is presented using the same datastore (leaving ~100GB free for the VSA boot drive and files); the SSD-based RAID group is presented as Physical RDM. This configuration permitted me to enable AO on each node, and get an SSD performance boost along with some deep storage from the collection of drives across all three nodes.<br />
<br />
In practice, this array has been more trouble-free than my VSAN implementation on (essentially) identical hardware. A key difference, however, has been the performance with respect to inter-node communication: With VSAN, up to four interfaces can be simultaneously configured for inter-node communication, increasing bandwidth and lowering latency. Even with the lower performance characteristics of the disks and HBA in each host, saturating two of the four gigabit interconnects I had configured was possible with VSAN (when performing sequential reads & writes, eg, backups & storage vMotion), so the single gigabit connection available to VSA was very noticeable.<br />
<br />
I have since migrated my network environment to use 10Gbps Ethernet for my back-haul network connectivity (iSCSI, NAS, vMotion) and have objective evidence of improved performance of the LeftHand array. I'll be updating this post with subjective test results when the opportunity presents itself.Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com8tag:blogger.com,1999:blog-7764963136414541684.post-50649627324133515192015-02-10T16:37:00.001-06:002015-02-10T16:37:27.300-06:00Citrix NetScaler UI changes<i>Which is worse?</i><br />
<br />
<ul>
<li><i>Searching for the solution to a problem and not being able to find it</i></li>
</ul>
<i><div style="text-align: center;">
<i>— or —</i></div>
</i><ul>
<li><i>Finding the <b>exact</b> solution to a problem in a blog, but discovering that it's an older post using out-dated products and documenting an API or UI that no longer exists?</i></li>
</ul>
<br />
This question comes from some feedback I received on a <a href="http://blog.millard.org/search/label/NetScaler" target="_blank">series of posts</a> I put together that documents my use of the Citrix NetScaler VPX Express virtual appliance as a reverse proxy.<br />
<br />
Citrix is doing the right thing: they're rebuilding the GUI in the NetScaler to eliminate Java (as much as possible). It has been a slow-going process, starting with the 10.0 version (as of this writing, 10.5 is current, and there are still one or two places that use a Java module), and one of the drawbacks is that the new HTML-only UI elements can't duplicate the Java UI—so things are...different.<br />
<table><tbody>
<tr><td><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgC43YG_YwZb_2pbZR0VTMnCRMJxd7WNmzxWvLaP6-TsH91-FMSHj5vXJ8sQifDQZVs-HViJf4KOmeXBhQky7McRUAhD7A8x7MtztigP88HNc6YvN7HGaCv7aUXYrnXrG5MwEQWy4g_y40P/s1600/ns_ha_setup.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgC43YG_YwZb_2pbZR0VTMnCRMJxd7WNmzxWvLaP6-TsH91-FMSHj5vXJ8sQifDQZVs-HViJf4KOmeXBhQky7McRUAhD7A8x7MtztigP88HNc6YvN7HGaCv7aUXYrnXrG5MwEQWy4g_y40P/s1600/ns_ha_setup.png" height="239" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">HA Setup, v10.0 & earlier</td></tr>
</tbody></table>
</td><td><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFjONFxwZ-Q-TF_vk6soeDczMseVSIFnrejN0n46QcclgiCFRY0L-Rw5KTaaqDgzKcGArOVmwBza6PRw3cSEqOmS5woNU63BPGtH6679uy6oyDInzc_I6O6f3o2e0Afm6WS5Kx3Ee01-T9/s1600/Screenshot+-+2_10_2015+,+4_08_30+PM.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFjONFxwZ-Q-TF_vk6soeDczMseVSIFnrejN0n46QcclgiCFRY0L-Rw5KTaaqDgzKcGArOVmwBza6PRw3cSEqOmS5woNU63BPGtH6679uy6oyDInzc_I6O6f3o2e0Afm6WS5Kx3Ee01-T9/s1600/Screenshot+-+2_10_2015+,+4_08_30+PM.png" height="320" width="279" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">HA Setup, v10.5</td></tr>
</tbody></table>
</td></tr>
</tbody></table>
In the screencaps above, you see the older Java-based dialog box and the newer HTML page. They have some of the same data, but they are neither identical, nor are they found in the same exact place from the principal UI.<br />
<br />
How does a blogger serve his/her audience? Does one ignore the past and soldier on, or does one revisit the old posts and update them for a new generation of software? If I had positioned myself as a NetScaler expert, that answer is obvious: UI changes in and of themselves would be post-worthy, and revisiting old functions to make them clear under the new UI would make perfect sense.<br />
<br />
In this case, however, I have only had a couple of requests for revised instructions using the equivalent UI; I'm not a NetScaler guru, and to be perfectly frank, I haven't the time needed to redo the series. If I get a <b>lot</b> more feedback that this series needs to be updated, I'll think about a second edition using the new UI, but as of now it's going to stay the way it is.Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com4tag:blogger.com,1999:blog-7764963136414541684.post-89402155467510682962015-02-10T14:17:00.002-06:002015-02-10T14:17:54.758-06:00Homelab 2015: Hello 10Gbps!<div class="separator" style="clear: both; text-align: center;">
</div>
New year, new post documenting the home lab. I've accomplished a number of upgrades/updates since my last full roundup of the lab, so rather than posting this as another delta, I'm doing this as a full re-documentation of the environment.<br />
<h3>
Compute: VMware vSphere 5.5</h3>
<ul>
<li>3 x Dell R610, each spec'd as follows:</li>
<ul>
<li>(2) Intel Xeon E5540 @ 2.53GHz</li>
<li>Hyperthreading enabled for 16 logical CPUs per host.</li>
<li>96GiB RAM</li>
<li>Boot from 4 or 8GB SD card</li>
<li>Dell PERC H700 6Gbps SAS/SATA HBA</li>
<ul>
<li>(4) 500GB Seagate Constellation.2 (ST9500620NS) SATA</li>
<ul>
<li>RAID5</li>
<li>Formatted as local vmfs5 datastore</li>
</ul>
<li>(2) 240GB Intel 530 SATA SSD</li>
<ul>
<li>RAID1</li>
<li>RDM for StoreVirtual VSA (see below)</li>
</ul>
</ul>
<li>Quad port Broadcom BCM5709 Gigabit Copper (embedded)</li>
<li>Dual port Mellanox MT26448 10GigE (8-lane PCIe), 850nm SFP+ optics</li>
<li>iDRAC 6 Enterprise</li>
<li>Redundant power</li>
</ul>
</ul>
<h3>
Storage: IP-based</h3>
<div>
<ul>
<li>iomega StorCenter ix2-200 "Cloud Edition"</li>
<ul>
<li>(2) 1TB Seagate Barracuda (ST1000DM003) 7200RPM SATA</li>
<li>RAID1</li>
<li>(1) 1000Base-T</li>
<li>LifeLine OS v3.2.10.30101</li>
<li>NFS export for VMs</li>
</ul>
<li>2 x Lenovo (iomega/EMC) px6-300d</li>
<ul>
<li>(6) 2TB Hitachi Deskstar (HDS723020BLA642) 7200RPM SATA</li>
<li>RAID5</li>
<li>(2) 1000Base-T, bonded, multiple VLANs</li>
<li>LifeLine OS v4.1.104.31360</li>
<li>2TB iSCSI Target for VMs</li>
</ul>
<li>Synology DS2413+</li>
<ul>
<li>(12) 2TB Seagate Barracuda (ST2000DM001) 7200RPM SATA</li>
<li>RAID1/0</li>
<li>(2) 1000Base-T, bonded, multiple VLANs (CLI-added)</li>
<li>DSM 5.1-5022 Update 2</li>
<li>NFS exports:</li>
<ul>
<li>ISOs (readonly, managed by SMB)</li>
<li>Logs</li>
<li>VMs</li>
</ul>
</ul>
<li>Synology DS1813+</li>
<ul>
<li>(8) 256GB Plextor PX-256M6S SSD</li>
<li>RAID5</li>
<li>(4) 1000Base-T</li>
<ul>
<li>(1) Management network</li>
<li>(2) iSCSI network (multi-homed, not bonded)</li>
</ul>
<li>~1.6TB iSCSI Target (block mode)</li>
</ul>
<li>HP StorVirtual "Lefthand OS"</li>
<ul>
<li>(3) ESXi Virtual Appliances, 1 on each host</li>
<ul>
<li>1280GB VMDK on local storage; tier 1</li>
<li>223GB RDM on SSD volume; tier 0</li>
</ul>
<li>4486.49GB Raw, 2243GB RAID1</li>
<li>(2) 1TB volumes for VMs</li>
<ul>
<li>Thin provisioning</li>
<li>Adaptive Optimization</li>
</ul>
<li>iSCSI network: 10GbE</li>
<li>Management: 1GbE</li>
</ul>
</ul>
<h3>
Networking:</h3>
</div>
<div>
<ul>
<li>(2) Cisco SG500X-24</li>
<ul>
<li>(4) 850nm SFP+ optics for 10GbE</li>
<li>(24) 1000Base-T MDI/MDI-X</li>
<li>Primary ISL: 10GbE</li>
<li>Backup ISL: (1) 2x1GbE LACP LAG</li>
<li>STP Priority: 16384</li>
</ul>
<li>Cisco SG300-28</li>
<ul>
<li>(28) 1000Base-T MDI/MDI-X</li>
<li>(2) 2x1GbE LACP LAG for link to SG500X-24</li>
<li>STP Priority: 32768</li>
</ul>
<li>Google Fiber (mk.1)</li>
<ul>
<li>"network box"</li>
<li>"storage box"</li>
<li>"fiber box"</li>
</ul>
<li>Various "dumb" (non-managed) 1GbE switches</li>
<li>Apple Airport Extreme (mk.4)/Express (mk.2)</li>
</ul>
<div>
<h3>
Miscellaneous:</h3>
</div>
</div>
<div>
<ul>
<li>(4) APC BackUPS XS1500</li>
<li>Internet HTTP/SSL redirection via Citrix NetScaler VPX (HA pair)</li>
<li>Remote access via:</li>
<ul>
<li>TeamViewer 10</li>
<li>Microsoft RDS Gateway</li>
<li>VMware Horizon View</li>
<li>Citrix XenApp</li>
</ul>
</ul>
</div>
<h3>
Connectivity Diagrams:
</h3>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLSP2oLZOW_VACvUdZMYBLE-rD6z7L183vHs2FbUVMp6H5Bn04DV6f_rPgJ80ssmsmbxRxkgx0mSHY19AG5TNNDSWh45CUeb5RpcNVsNcRwgTtJ1hGc_1G2kWl251btXz5dhxzps5Vuw6O/s1600/vSphere+Host+Networking.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLSP2oLZOW_VACvUdZMYBLE-rD6z7L183vHs2FbUVMp6H5Bn04DV6f_rPgJ80ssmsmbxRxkgx0mSHY19AG5TNNDSWh45CUeb5RpcNVsNcRwgTtJ1hGc_1G2kWl251btXz5dhxzps5Vuw6O/s1600/vSphere+Host+Networking.png" height="640" width="596" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Host Configuration</td></tr>
</tbody></table>
<br />
<div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXA6_YKl5XjhZ9JZ5T_MHo3egFbdVh0RibT5_kG3kwTZaa8_EEeVSylqcvXWAqsTUpRS7H6pjTDK2uMc4t2L2Rp0mQJRqPSZZe7Yx-6wuR-18abmUXsYned4Gn4Vt6CvE8MKeMF24L8Syp/s1600/vSphere+Host+Networking.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXA6_YKl5XjhZ9JZ5T_MHo3egFbdVh0RibT5_kG3kwTZaa8_EEeVSylqcvXWAqsTUpRS7H6pjTDK2uMc4t2L2Rp0mQJRqPSZZe7Yx-6wuR-18abmUXsYned4Gn4Vt6CvE8MKeMF24L8Syp/s1600/vSphere+Host+Networking.png" height="620" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Environment</td></tr>
</tbody></table>
</div>
Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com3tag:blogger.com,1999:blog-7764963136414541684.post-18594456878049617592015-02-01T00:01:00.001-06:002015-02-10T08:47:37.319-06:00Homelab Upgrades: stay tunedI'm forever "messing" with my home lab. My latest set of updates will be based on a plan to get myself upgraded from all-gigabit to using 10Gbps for inter-host communication.<br />
<br />
If I only had two hosts, it'd be fairly straightforward—especially if my hosts had on-board 10Gbase-T: build some Cat6 crossover cables and link the machines directly together. But I have three hosts (from my early experimentation with VSAN) and none of them have 10Gbps ports.<br />
<br />
Why the move to 10Gb Ethernet?<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEic1fBfMOsbsOTTzaz0_4Ca5P1HSWF_BlLEPSozdcybwEIS_tHlmu5aYPHcLE3wJmrUdXNIWEh-JwPPPXTY0SiIUujCQnKe4uO1LEHq_W66vOqN9P3enkOVH_4zsC27FEB2yw2Bh45lgGcA/s1600/Screenshot+-+1_31_2015+,+9_36_01+PM.png" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEic1fBfMOsbsOTTzaz0_4Ca5P1HSWF_BlLEPSozdcybwEIS_tHlmu5aYPHcLE3wJmrUdXNIWEh-JwPPPXTY0SiIUujCQnKe4uO1LEHq_W66vOqN9P3enkOVH_4zsC27FEB2yw2Bh45lgGcA/s1600/Screenshot+-+1_31_2015+,+9_36_01+PM.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Dual RJ-45 10Gbps NIC</td></tr>
</tbody></table>
<br />
My experimentation with VMware VSAN and PernixData FVP have led me to the conclusion that, while they certainly function in a 1Gbps environment, they are seriously limited by it (FVP especially so, as it cannot perform any multi-NIC bindings in its current incarnation).<br />
<br />
With the growing prevalence of SSD in the datacenter—and the dropping price-per-gigabyte making it approachable for the homelab builder—the bandwidth and latency limitations in gigabit networks make 10Gbps networks almost a necessity as soon as you drop in that first SSD. Anything less, and you don't get full value for that dollar.<br />
<blockquote class="tr_bq">
<i>The same applies to your older 2/4Gbps Fibre Channel storage networks, but FC is pretty much unattainable by most homelab builders. That said: If you're spending top-dollar on an SSD FC array in your enterprise, don't hobble it with a slow network. For that matter, 8Gbps might even be too slow... Plus, with Ethernet upgrades, you get more bang for the buck: in addition to providing a boost to your storage network performance—especially NFS if your filer has 10Gbps—you can run vMotion and FT over it; an upgraded FC network only gives you a boost in block storage performance.</i></blockquote>
In my professional life, I currently work as a "delivery boy" for a soup-to-nuts value-added reseller. I spend a lot of time in client datacenters, performing all sorts of installations and upgrades. The ones I've participated in that involve upgrades from 1Gbps backbones to 10Gbps (or better, with 40Gbps being the common number even in non-Infiniband implementations by taking advantage of QSFP+ ports) were just amazing in the performance jump. In many ways, it brings back memories of putting in my first Fast Ethernet (100Mbps) switches for backbone services while the fan-out to the client workstations was all 10Mbps 10Base-2 "thin-net" coaxial. But I digress...<br />
<br />
So my requirement is straightforward: I am working towards a 10Gbps solution for three hosts, and I'm satisfied if it only provides host-to-host intercommunication. That means that guest networking (and management) can still go over existing 1Gbps (simply because the demand is being fulfilled by 1Gbps, making 10Gbps overkill).<br />
<br />
That meant, at a minimum, three single-port PCIe host adapters, a 10Gbps switch, and the appropriate interconnect cabling. But I wasn't going to be satisfied with "the minimum," even for my lab: I really, really want to have redundant 10Gbps connections for each host. You see, I don't rebuild everything every 60-90 days like some folks. This environment is fairly stable, supports a number of higher-level lab & training projects (like VDI and vCloud Director), and pretty much runs like production. It just does it in my basement instead of in a datacenter. With some specific VMware-related exceptions, I do most of my truly experimental work inside an Organizational Virtual Datacenter (OvDC) provisioned by vCloud Director; in that "bubble," I can even spin up virtual ESXi systems. So my requirements are a little more involved: <u>dual</u> 10Gbps in the hosts with multiple 1Gbps as fallback; this means a single 10Gbps switch would be acceptable, but not single ports in the hosts.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyn3Ive3NrzifOi1tKgSWpmW__EVihbOexKCJas3zWGyn8weXZ_K0LTRl1FR1jUl2C4eBr16ewuX8uENJRhY6D8VpDzBDHIXyoQKFaMzFzxjBy9kY8lRHed7RLJvSIP0IU4hAOw5bFtWN6/s1600/netgear_10GbE.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyn3Ive3NrzifOi1tKgSWpmW__EVihbOexKCJas3zWGyn8weXZ_K0LTRl1FR1jUl2C4eBr16ewuX8uENJRhY6D8VpDzBDHIXyoQKFaMzFzxjBy9kY8lRHed7RLJvSIP0IU4hAOw5bFtWN6/s1600/netgear_10GbE.png" height="41" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Netgear ProSafe XS708E</td></tr>
</tbody></table>
Depending on how you approach the problem, you can end up with several different solutions. If you start with the switch—and the popular 10Gbps switch for homelab use these days seems to be the <a href="http://www.netgear.com/business/products/switches/unmanaged-plus/10g-plus-switch.aspx" target="_blank">Netgear ProSafe XS708E</a>, with eight unmanaged copper (10Gbase-T) ports—then you need adapters & cabling that will work with it. Conversely, if you start with the adapters, you'll need a switch (and cabling) that will work with them. Many of the folks in the VMware community have been building their hosts with 10Gbps copper ports so for them, the copper switch made sense. I originally started down that path, and came up with the following bill-of-materials (BOM):<br />
<br />
<ul>
<li>10Gbps Switch (Netgear XS708E): $850-$900</li>
<li>3 x 2-port 10Gbps PCIe adapters (Q-Logic QLE3442-CU): $1000-$1200</li>
<li>6 x 6ft Cat6 (Monoprice): $10</li>
<li>Total: $1860-2100 US (tax, shipping not included)</li>
</ul>
<br />
That seemed like a pretty expensive option, so I hunted around and really didn't find many other deals. Variations exist when you check with eBay, but you're still in the $2000 price range. <i>Lesson 1</i>: if you don't already have existing 10Gbps ports in your hosts, you're looking at $350/port as a reasonable estimate for the <b>low end</b> for the price of entry.<br />
<br />
I also chose to approach it from the other direction: get a line on the least-expensive compatible adapters, and build out from there.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0jXsTdtQKIUsdotenb3prp3XpJTLYhx8R7KZvLYnGWFrVyfpApurnbXeoVyHCkQC_uG7Ltvs_50OOuy-BKQKrSiJF2YtvCgMCnKhYKAjJkJ6Xdl8D6f79PgTnIgzuzHEHOHqRCz-AO-yd/s1600/BCM57712A.JPG" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0jXsTdtQKIUsdotenb3prp3XpJTLYhx8R7KZvLYnGWFrVyfpApurnbXeoVyHCkQC_uG7Ltvs_50OOuy-BKQKrSiJF2YtvCgMCnKhYKAjJkJ6Xdl8D6f79PgTnIgzuzHEHOHqRCz-AO-yd/s1600/BCM57712A.JPG" height="150" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Broadcom BCM57712A dual-SFP+</td></tr>
</tbody></table>
After finding some Broadcom-based dual-port SFP+ adapters for $70 apiece, I started to research SFP+ transceivers. Although they exist as 1Gbps SFP modules, you cannot get RJ45/Copper modules for SFP+/10Gbps; the specification for SFP+ simply doesn't provide sufficient power for UTP/STP cabling. That meant I'd have to get SFP+ transceivers (for both adapter and switch) as well as fiber cabling—or twinax direct-connect cables—to make this work. Plus a switch with SFP+ ports.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuUFUTTywfijbxvPHpZ2hEhEwb2hJdvoz3sFscsMxZZ9rB9Y0ejtZdRF8qW5dljIBYzaB5SLhEyhIUfY8LvA25itATor-_MXOreppb59b8NJTxAUZsTUS0IEV7wSwn0aSBh5-neBsHZiGH/s1600/sfp10g01.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuUFUTTywfijbxvPHpZ2hEhEwb2hJdvoz3sFscsMxZZ9rB9Y0ejtZdRF8qW5dljIBYzaB5SLhEyhIUfY8LvA25itATor-_MXOreppb59b8NJTxAUZsTUS0IEV7wSwn0aSBh5-neBsHZiGH/s1600/sfp10g01.jpg" height="46" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">FiberStore SFP+ transceivers</td></tr>
</tbody></table>
As it turns out, the former problem (transceivers & cables) is readily solved: a China-based manufacturer/reseller (<a href="http://www.fiberstore.com/" target="_blank">FiberStore</a>) has compatible products at what I'd call "Monoprice Prices" for both their transceivers ($18/ea) and their twinax direct-connect cables ($22 for a 2M).<br />
<blockquote class="tr_bq">
<i>As much as I'd prefer the direct-connect option for this project, I was a little worried about Cisco/Broadcom compatibility; yes, FiberStore has satisfaction guarantees, but shipping back-and-forth to mainland China from the US would be a big pain if they didn't work the first time. Or the second. Or the third. I also didn't know until after purchasing them that the BCM57712A cards were OEM'd for Cisco.</i></blockquote>
<blockquote class="tr_bq">
<i>Unless you're working with single-source for both the switch and endpoint—no matter what they say about standards—you can still have a direct-connect cable that won't work while transceivers would have worked fine. So I went the "spendy" route of selecting guaranteed-compatible 850nm multimode LC transceivers & fiber optic patch cables.</i></blockquote>
The big stickler, then, is finding a switch. Unfortunately, high-count SFP+ ports aren't typically found in consumer-grade gear, making the search more challenging: under typical sales models, SFP+ transceivers will set you back ~$100 apiece, which makes it a daunting financial proposition for a lab setup, even if the switch itself was cheap. After researching several lines of products, I was about to give up on the idea when I came across references to Cisco's "SG" line of small business switches. I've had the SG300-28 in my lab for a long time, so I know it well and like the product line. The <a href="http://www.cisco.com/c/en/us/support/switches/sg500xg-8f8t-16-port-10-gigabit-stackable-managed-switch/model.html" target="_blank">SG500XG-8T8F</a> with eight SFP+ and eight copper 100M/1G/10Gbps ports looked like a winner, but the prices (~$2500 street) were ridiculous compared to the copper-only Netgear. I found some alternative candidates from a variety of manufacturers on eBay, but some models were 10Gbps <b>only</b> (so I'd still need something to downlink them to 1Gbps if I wanted to access the 1Gbps-based storage in my environment) and others had "Buy Now" pricing in excess of $2500. Still too spendy.<br />
<br />
But then I came across Cisco's $900 <a href="http://www.cisco.com/c/en/us/support/switches/sg500x-24-24-port-gigabit-4-port-10-gigabit-stackable-managed-switch/model.html" target="_blank">SG500X-24</a>. After doing a bunch of reading—reviews, product guides, user manuals—I decided that this could be what I was after. In addition to having 24 copper 10M/100M/1Gbps ports, it also boasted 4x 10Gbps ports for uplinking (one pair of which would work as 5Gbps downlink ports when stacking—true, multi-chassis stacking—with other 1Gbps models in the Cisco SG500 line). Two variants existed, one with SFP+ ports, the other with 10Gbase-T. Alone, this switch wouldn't fit my requirement for dual host ports, but a <b>pair</b> of them—with one SFP+ port used to link the switches together—would fit the bill. Would it make budget?<br />
<br />
2 x SG500X-24: $1800<br />
3 x BCM57712A: $210<br />
6 x SFP+ Tranceivers: $252<br />
6 x 6ft LC patch cables: $36<br />
1 x 1ft LC patch cable: $3<br />
Total: $2300<br />
<br />
Holy. Smokes. For a ~25% increase, I could build a <b>dual switch</b> configuration; one that would afford me some additional redundancy that wouldn't exist in the single switch setup. I checked with my CTO, and have approval: as long as it stays under $2500, I had the green light.<br />
<br />
Go. Go! Go Baby Go!!!<br />
<br />
UPDATE:<br />
It turns out that those BCM57712A adapters I found were supplied with low-profile brackets and were originally designed to go inside the Cisco rackmount-series UCS servers. Not only were there no full-height brackets available—I checked with Broadcom (no answer), a local manufacturer rep listed on the Broadcom site ("check with the distributor"), a local distributor suggested by the manufacturer rep ("check with Broadcom") and the vendor from whom I purchased them ("check with Broadcom") and struck out—but the only way I was going to get the cards' firmware updated—not critical because ESXi 5.5 recognized the NIC and worked fine with the out-of-date firmware, but still very desirable, as research also suggested that there were good fixes in newer versions—was to put them into a compatible Cisco server and run the Host Update Utility (huu) on the system. It's my lab: if I'd been able to update the firmware <b>or</b> been able to source the full-height brackets, I'd have moved forward with them. Instead: Time for Plan B.<br />
<br />
The next-cheapest dual-SFP+ adapters I'd found when doing my original were Mellanox-based HP adapters for ~$130 apiece. This was at the edge of my budget, and if I couldn't recoup the $210+ I'd already spent on the Broadcom adapters, I'd be even deeper in the hole (not to mention barreling right through the $2500 budget), but I'm going forward with this little experiment. I'll try and unload the other adapters on eBay, although this could be "throwing good money after bad" as easily as "in for a penny, in for a pound." We'll see.<br />
<br />
UPDATE:<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZ5cbdqn6Rv87c41GQAx-HpQcFblyQfuj4eTd3spNJSB_NT2I2-1Phl5JiJrfXLqc7XZS0zj9GIQvZ3Fml36kJ5RyDGVw2uVR4jfAdsG9IyDhzuQcxuG6pBTyybSYfC5u9g89QgQFH8LyN/s1600/mellanox-g2.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZ5cbdqn6Rv87c41GQAx-HpQcFblyQfuj4eTd3spNJSB_NT2I2-1Phl5JiJrfXLqc7XZS0zj9GIQvZ3Fml36kJ5RyDGVw2uVR4jfAdsG9IyDhzuQcxuG6pBTyybSYfC5u9g89QgQFH8LyN/s1600/mellanox-g2.jpg" height="132" width="200" /></a></div>
The Mellanox adapters (HP G2 [516937-B21]) arrived with full-height brackets. The firmware was out-of-date, but not to the point that the hardware and ESXi didn't recognize the card.Thanks to "Windows To Go" functionality in Windows 8.1, I was able to boot one of my hosts into Windows—with the help of a big USB drive—without screwing up my ESXi installation and update the card firmware of all 3 cards. ESXi had no problem recognizing the NIC (although this model was originally sold new in the vSphere 4.0 days) with both the as-received firmware as well as the latest-greatest. I'd also gone ahead and acquired the transceivers & cables after verifying that the Broadcom NICs were recognized by ESXi. With those in hand, I was able to do some basic point-to-point hardware checks for the adapters: the transceivers worked just as well in the HP cards as the Cisco cards (Enterprise guys: keep that in mind when you're looking at the transceiver prices from the OEM. There are enormous margins attached to these little gizmos, and purchasing from an original manufacturer could save your organization a <b>ton</b> of money). At this point, I'm in for less than $1000, but have a go/no-go on the switches. If I could get a single compatible switch for $1000, I'd be right where I was with the copper-based solution. Unfortunately, even eBay was a dry well: nothing for less than $1800.<br />
<br />
Go. No-Go. Go. No-Go.<br />
<br />
I am almost ready to pull the plug: I haven't ordered the switches yet (they're the single largest line-items on the BOM) because while they would be new and pretty sure to work as expected, the Frankenstein's monster of the adapters, SFP+ transceivers and fiber cables had no such guarantee. My project plan required me to start with the small and grow to the large as I proved each bit of infrastructure would work as envisioned. At this run rate, however, I'm going to exceed my budget; I should quit here and chalk it up to a learning experience. But I also wouldn't be able to do much with the existing infrastructure without a switch. Sure, I could try and create some weird full-mesh setup with multiple vMotion networks, but that would require a <b>ton</b> of manual overrides whenever I wanted to do anything. That, in turn, would make DRS essentially useless.<br />
<br />
So: do I take the loss (or sit on the hardware and watch for dropping prices or a surprise on eBay) or push through and go over budget?<br />
<br />
UPDATE:<br />
Screw it. I want this so bad I can taste it. All I need is the the switches and I can make this work. Time to get crackin'.<br />
<br />
UPDATE:<br />
Nuts. I can only get one of the two switches I need for $900 from the supplier I found back in my original research. The next-cheapest price is an additional $60. I am again faced with the dilemma: do I stop here (this will get me the sub-optimal but fully-functional single 10Gbps port on each host), wait until another becomes available at the same price, or pay the additional cost to get the second switch?<br />
<br />
UPDATE:<br />
Screw it, part 2. I'm going ahead with the higher-price switch. This is going to work, and it's going to be awesome. I can feel it. I am going to save myself a few bucks, however, and select the "free" option on the shipping. Almost seems silly at this point...<br />
<br />
UPDATE:<br />
The first switch arrived before expected (Friday instead of the following Monday), so I get to start putting things together this weekend.<br />
<br />
UPDATE:<br />
Okay, I'm disappointed. All the documentation seems to read like you can use one to four of the 10Gbps ports for stacking—and you can—but what they don't say is that you <b>must configure</b> two or four ports for stacking, My original plan to use 3 ports for the hosts and 1 port for the multi-chassis stack has been foiled. Fortunately, I was only about 50% certain that it would work, so I've set the switch in "standalone" mode and moved forward with getting things cabled & configured. This is purely temporary until I get the second switch in and do some more validation; at some point, there will be "the great migration" so that I can evacuate the 1Gbps switches I'm currently using.<br />
<br />
Once I finish, I'll be ejecting a 24-port dumb, unmanaged 10/100/1000 switch (in favor of my old SG300) and a pair of 28-port managed 10/100/1000 switches from the environment. This will have the effect of reducing the port count on each host from 9 (which includes the iDRAC) down to 7, but with two of those being the much-thinner 10Gbps ports. The cabling back to the switches will also be a bit more sveldt: two fiber cables take much less room in a wire loom than two (much less four) Cat6 cables take.<br />
<br />
The hosts, each, have four gigabit ports on the motherboard, so I'm going to keep them in service as management and guest data networks.<br />
<br />
The 10Gbps network will serve iSCSI (both host & guest use), NFS (I've hacked my Synology arrays to do trunked VLANs on bonded adapters) and vMotion. At the moment, I've reconfigured my in-host storage for use by HP StoreVirutal VSAs; this means I also can't have VSAN running anymore (not as robust as I'd like with my consumer-grade SSDs, and the Storage Profiles Service keeps 'going sideways' and making vCD unusable). I was able to source three of the H710 HBA adapters—along with SAS cables and backup battery for cache memory—intended for use in my hosts for a song. This should give me not only RAID5 for the spinning disks (something unavailable in the simple SAS HBA I'd been running before), but a more powerful HBA for managing both the SSD and the spinning disks.<br />
<br />
For the same reason, I don't have an FVP cluster using SSD; luckily, however, I'm running v2, so I'm playing with the RAM-based acceleration and looking forward to seeing improvements once I get its network running on the 10G. My long-term hope is that I can find a cost-effective PCIe-based flash board that can serve as FVP's acceleration media; that, along with the 10Gbps network, should make a big difference while simultaneously giving me back all the RAM that I'm currently utilizing for FVP.<br />
<br />
UPDATE:<br />
The second switch has been received & I'm getting it configured with the first switch; I'm also having to make some changes to the old switches to add connectivity to the new switches. It'll be a fun migration, but it should be physically possible to dismount the old switches and lay them down next to the rack; that will free the rack space for the new switches, and I can perform a rolling conversion of the cabling for my hosts, followed by moving the arrays one-by-one (they all have bonded uplinks, so disconnecting them, one link at a time, should be safe).<br />
<br />
UPDATE:<br />
AARRRGGGHHH! I'm re-using one of the old switches as an aggregation switch—any device that has only one connection/link will be plugged into that switch, while anything with multiple connections will be connected to the new switches. All three switches are cross-connected, and RSTP is used to keep broadcast loops from forming. Fine so far. But I needed to update the STP priorities for my switches to make sure the new switches would prefer their 10Gbps link, and in the process I discovered that one of my cross-connect LAGs wasn't built properly. It wasn't until STP did a recalc and decided to use that LAG as the designated port—and traffic stopped flowing—that I discovered my error. And the symptoms were manifold & annoying...thank goodness the LeftHand VSA is so resilient.<br />
<br />
UPDATE:<br />
Yay! Everything is done, including the rebuild of the LeftHand nodes to accommodate some architectural changes needed as a result of the LAG/STP outage. Looking forward to getting this new setup documented & published...which makes this the last update, and a new post in the drafts folder!!!Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com2tag:blogger.com,1999:blog-7764963136414541684.post-61339150578758069682014-12-25T23:15:00.000-06:002015-06-05T23:16:40.298-05:00Synology DS2413+Based on the recommendations of many members of the vExpert community, I purchased a Synology DS2413+. This is a 12-bay, Linux-based array that can be expanded to 24 spindles with the addition of the DX1211 expansion chassis. My plan was to eliminate a pair of arrays in my home setup (an aging Drobo Pro and my older iomega px6-300d), keeping a second array for redundancy.<br />
<br />
The array is a roughly cube-shaped box which sits nicely on a desk, with easy access to the 12 drive trays and "blinky lights" on the front panel. It also sports two gigabit (2x1000Mb/s) network ports that can be bonded (LACP is an option if the upstream switch supports it) for additional throughput.<br />
<br />
Synology has a <a href="http://www.synology.com/products/product.php?product_name=DS2413%2B&lang=us" target="_blank">page full</a> of marketing information if you want more details about the product. The intent of this post is to provide the benchmark information for comparison to other arrays, as well as information about the device's comparative performance in different configurations.<br />
<br />
The Synology array line is based on their "DSM" (DiskStation Manager) operating system, and as of this iteration (4.1-2661), there are several different ways to configure a given system. The result is a variety of different potential performance characteristics for a VMware environment, depending on the number of spindles working together along with the configuration of those spindles in the chassis.<br />
<br />
The two major classes of connectivity for VMware are represented in DSM: You can choose a mix of NFS and/or iSCSI. In order to present either type of storage to a host, disks in the unit must be assembled into volumes and/or LUNs, which are in turn published via shares (NFS) or targets (iSCSI).<br />
<br />
DSM supports a panoply of array types—Single-disk, JBOD, RAID0, RAID1, RAID5, RAID6, RAID1+0—as the basis for creating storage pools. They also have a special "SHR" (Synology Hybrid RAID) which automatically provides for dynamic expansion of the storage capacity when an even number of drive sizes are present; both single-drive- and dual-drive-failure protection modes are available with SHR on the DS2413+.<br />
<br />
When provisioning storage, you have essentially two starting options: do you completely dedicate a set of disks to a volume/LUN ("Single volume on RAID"), or do you want to provision different portions of a set of disks to different volumes and/or LUNs ("Multiple volumes on RAID")?<br />
<br />
iSCSI presents a different sort of twist to the scenario. DSM permits the admin to create both "Regular files" and "Block-level" LUNs for iSCSI. The former reside as sparse file on an existing volume, while the latter is done with a new partition on either dedicated disks (Single LUNs on RAID) or a pre-existing disk group (Multiple LUNs on RAID). The "Regular files" LUN is the only option that allows for "thin provisioning" and VMware VAAI support; the Single LUN option is documented as highest-performing.<br />
<br />
For purposes of comparison, the only mode of operation for the iomega <a href="http://blog.millard.org/search/label/px6" target="_blank">px6-300d</a> (which I've written about several times on this blog) is like using "Multiple Volumes/LUNs on RAID" in the Synology, while the older iomega <a href="http://blog.millard.org/search/label/ix2" target="_blank">ix2-200d</a> and <a href="http://blog.millard.org/search/label/ix4" target="_blank">ix4-200d</a> models operate in the "Regular files" mode. So the DSM software is far more versatile than iomega's StorCenter implementations.<br />
<br />
So that leaves a lot of dimensions for creating a test matrix:<br />
<ul>
<li>RAID level (which is also spindle-count sensitive)</li>
<li>Volume/LUN type</li>
<li>Protocol</li>
</ul>
<table border="1" cellpadding="0" cellspacing="0">
<tbody>
<tr>
<th colspan="3">DS2413+</th>
<th rowspan="2">1 block seq read<br />
<br />
(IOPS)</th>
<th rowspan="2">4K random read<br />
<br />
(IOPS)</th>
<th rowspan="2">4K random write<br />
<br />
(IOPS)</th>
<th rowspan="2">512K seq write<br />
<br />
MB/s</th>
<th rowspan="2">512K seq read<br />
<br />
MB/s</th>
</tr>
<tr> <th>Protocol</th> <th>RAID</th>
<th>Disks</th> </tr>
<tr> <th rowspan="14">iSCSI</th> <th>none</th> <th>1</th> <td>16364</td> <td>508</td> <td>225</td> <td>117.15</td> <td>101.11</td> </tr>
<tr> <th>RAID1</th> <th>2</th> <td>17440</td> <td>717</td> <td>300</td> <td>116.19</td> <td>116.91</td> </tr>
<tr> <th rowspan="2">RAID1/0</th> <th>4</th> <td>17205</td> <td>2210</td> <td>629</td> <td>115.27</td> <td>107.75</td> </tr>
<tr> <th>6</th> <td>17899</td> <td>936</td> <td>925</td> <td>43.75</td> <td>151.94</td> </tr>
<tr> <th rowspan="4">RAID5</th> <th>3</th> <td>17458</td> <td>793</td> <td>342</td> <td>112.29</td> <td>116.34</td> </tr>
<tr> <th>4</th> <td>18133</td> <td>776</td> <td>498</td> <td>45.49</td> <td>149.27</td> </tr>
<tr> <th>5</th> <td>17256</td> <td>1501</td> <td>400</td> <td>115.15</td> <td>116.12</td> </tr>
<tr> <th>6</th> <td>15768</td> <td>951</td> <td>159</td><td>60.41</td><td>114.08</td> </tr>
<tr> <th rowspan="5">RAID0</th> <th>2</th> <td>17498</td> <td>1373</td> <td>740</td> <td>116.44</td> <td>116.22</td> </tr>
<tr> <th>3</th> <td>18191</td> <td>1463</td> <td>1382</td> <td>50.01</td> <td>151.83</td> </tr>
<tr> <th>4</th> <td>18132</td> <td>771</td> <td>767</td> <td>52.41</td> <td>151.05</td> </tr>
<tr> <th>5</th> <td>17692</td> <td>897</td> <td>837</td> <td>56.01</td> <td>114.35</td> </tr>
<tr> <th>6</th> <td>18010</td> <td>1078</td> <td>1014</td> <td>50.87</td> <td>151.47</td> </tr>
<tr> <th>RAID6</th><th>6</th> <td>17173</td> <td>2563</td> <td>870</td> <td>114.06</td> <td>116.37</td> </tr>
<tr>
<th>Protocol</th> <th>RAID</th>
<th>Disks</th>
<th>1 block seq read<br />
<br />
(IOPS)</th>
<th>4K random read<br />
<br />
(IOPS)</th>
<th>4K random write<br />
<br />
(IOPS)</th>
<th>512K seq write<br />
<br />
MB/s</th>
<th>512K seq read<br />
<br />
MB/s</th>
</tr>
<tr> <th rowspan="14">NFS</th> <th>none</th> <th>1</th> <td>16146</td> <td>403</td> <td>151</td> <td>62.39</td> <td>115.03</td> </tr>
<tr> <th>RAID1</th> <th>2</th> <td>15998</td> <td>625</td> <td>138</td> <td>63.82</td> <td>96.83</td> </tr>
<tr> <th rowspan="2">RAID1/0</th> <th>4</th> <td>15924</td> <td>874</td> <td>157</td> <td>65.52</td> <td>115.45</td> </tr>
<tr> <th>6</th> <td>16161</td> <td>4371</td> <td>754</td> <td>65.87</td> <td>229.52</td> </tr>
<tr> <th rowspan="4">RAID5</th> <th>3</th> <td>16062</td> <td>646</td> <td>137</td> <td>63.2</td> <td>115.15</td> </tr>
<tr> <th>4</th> <td>16173</td> <td>3103</td> <td>612</td> <td>65.19</td> <td>114.76</td> </tr>
<tr> <th>5</th> <td>15718</td> <td>1013</td> <td>162</td> <td>59.26</td> <td>116.1</td> </tr>
<tr> <th>6</th> <td></td> <td></td> <td></td> <td></td> <td></td> </tr>
<tr> <th rowspan="5">RAID0</th> <th>2</th> <td>15920</td> <td>614</td> <td>183</td> <td>66.19</td> <td>114.85</td> </tr>
<tr> <th>3</th> <td>15823</td> <td>757</td> <td>244</td> <td>64.98</td> <td>114.6</td> </tr>
<tr> <th>4</th> <td>16258</td> <td>3769</td> <td>1043</td> <td>66.17</td> <td>114.64</td> </tr>
<tr> <th>5</th> <td>16083</td> <td>4228</td> <td>1054</td> <td>66.06</td> <td>114.91</td> </tr>
<tr> <th>6</th> <td>16226</td> <td>4793</td> <td>1105</td> <td>65.54</td> <td>115.27</td> </tr>
<tr> <th>RAID6</th><th>6</th> <td>15915</td> <td>1069</td> <td>157</td> <td>64.33</td> <td>114.94</td> </tr>
</tbody></table>
<br />
While this matrix isn't a complete set of the available permutations for this device, when I stick with the 6-disk variations that match the iomega I already have in the lab, I've been stunned by the high latency and otherwise shoddy performance of the iSCSI implementation. Further testing with additional spindles did not—counter to expectations—improve the situation.<br />
<br />
I've discovered the Achilles' Heel of the Synology device line: regardless of their protestations to the contrary about iSCSI improvements, their implementation is still a non-starter for VMware environments.<br />
<br />
I contacted support on the subject, and their recommendation was to create the dedicated iSCSI target volumes. Unfortunately, this also eliminates the ability to use VAAI-compatible iSCSI volumes, as well as sharing disk capacity for NFS/SMB volumes. For most use cases of these devices in VMware environments, that's not just putting lipstick on a pig: the px6 <b>still</b> beat the performance of a <b>12-disk RAID1/0</b> set using all of Synology's tuning recommendations.<br />
<br />
NFS performance is comparable to the PX6, but as I've discovered in testing the iomega series, NFS is not as performant as iSCSI, so that's not saying much... What to do, what to do: this isn't a review unit that was free to acquire and free to return...<br />
<br />
Update:<br />
I've decided to build out the DS2413+ with 12x2TB drives, all 7200RPM Seagate ST2000DM001 drives in a RAID1/0, and use it as an NFS/SMB repository. With over 10TB of <u>formatted</u> capacity, I will use it for non-VMware storage (backups, ISOs/media, etc) and low-performance-requirement VMware workloads (logging, coredumps) and keep the px6-300d I was planning to retire.<br />
<br />
I'll wait and see what improvements Synology can make with their iSCSI implementation, but in general, don't see using these boxes for anything but NFS-only implementations.<br />
<br />
Update 2:<br />
Although I was unsatisfied with the DS2413+, I had a use case for a new array to experiment with Synology's SSD caching, so I tried a DS1813+. Performance with SSD was improved over the non-hybrid variation, but iSCSI latency for most VMware workloads was still totally unacceptable. I also ran into data loss issues when using the NFS/VAAI in this configuration (although peers on Twitter responded with contrary results).<br />
<br />
On a whim, I went to the extreme of removing <b>all</b> the spinning disk in the DS1813+ and replacing them with SSD.<br />
<br />
Wow.<br />
<br />
The iSCSI performance is still "underwhelming" when compared to what a "real" array could do with a set of 8 SATA SSDs, but for once, not only did it exceed the iSCSI performance of the px6-300d, but it was better than anything else in the lab. I could only afford to populate it with 256GB SSDs, so the capacity is considerably lower than an array full of 2TB drives, but the performance of a "Consumer AFA" makes me think positively about Synology once again.<br />
<br />
Now I just need to wait for SSD prices to plummet...Jim Millardhttp://www.blogger.com/profile/18351359075519822053noreply@blogger.com2