HTTP Reverse Proxy using Citrix NetScaler VPX Express

Part 4 in a series

So far: the first three parts of this series dealt with the introduction of a problem (multiple servers behind a NAT firewall that use the same port) and solution (Citrix NetScaler VPX Express); laying the groundwork for configuring the solution; an overview of what we'll be configuring.

Because it is possible to set up content switching with a single host (the degenerate case), this is the method we'll begin with. While it doesn't really do much for us, simply repeating the steps for a second (and subsequent) will result in a working solution. Other guides lay down the steps with two hosts already in mind, and teasing apart the pieces to apply it to your situation might be more difficult.

Groundwork

Some planning must be done prior to doing this setup. The first is a set of IP addresses that you'll need to have handy. This post will use the following addresses; substitute them with your own:

Host	IP
CS Virtual Server	192.168.106.37
Target Server A	192.168.106.38
Target Server B	192.168.106.39

Enable Features

The bare-bones install of the NetScaler has a number of features enabled, but the ones we need for content switching are disabled. Open the System configuration tree and select Settings

Select "Configure basic features" and make sure the following features are enabled (checked):

• Load Balancing
• Content Switching

If you selected "Traffic Management" in the left menu before and after enabling the feature, this is what you'd see:

Default, features disabled

LB and CS enabled

Begin the setup by expanding "Load Balancing" under "Traffic Management" and select "Servers":

In the center section, click [Add...] and create the server. The "Server Name" is an identifier used in the NetScaler; it does NOT have to be the FQDN or short name for the server.

Then switch to the Services option

and create a protocol-specific entry for the server, including a monitor

(I like to use http because it doesn't require any customization; a custom http-ecv monitor can be created to check for the explicit function of the target server, but that's beyond the scope of this series).

I also recommend using a naming convention that includes the type of object you're creating ('svc' for the service) and the protocol it's tied to ('http'); that will make it more obvious where a given object comes from when you see them bound in other places.

Switch to the Virtual Servers menu

and click [Add...] to build the virtual server.

Make sure you uncheck the "Directly Addressable" option; this eliminates the need to give the virtual server its own address (we want to give an address to the Content Switching virtual server) and select the service we just created.

Switch to the Content Switching menu and select "Policies"

Click [Add...] to create a policy to trigger sending the traffic based on the hostname used in the HTTP header.

Select the Virtual Servers option under Content Switching

and click [Add..] to create a new virtual server.

This server gets the IP address to which we'll be forwarding traffic.

Click "Insert Policy" to insert a new policy

Select the new policy from the drop-down, then pull down the list of targets, selecting the new load balancing server. You will get a warning about the "Goto Expression"

Select [Yes], then [Create] to make the server.

At this point, your setup should function for the first server you configured!

Now: go back to the step for creating the outside server and repeat except for creating a new Content Switching server.

Now: Open the existing server

and add another policy, using the new server's policy and LB virtual server entry:

You can test this internally by either updating your DNS server entries or adding a line to your machine's HOSTS file:

192.168.106.37 serverA serverB

Point your browser at http://serverA after you make the change, and voila!, you get to the target. Switch to http://serverB, and you get that target instead.

Once you've verified the functionality from the inside, update the forwarding on your NAT firewall and test using an outside address (eg, use a cell phone that's not on your home WiFi).

Parts in this series:

1. Intro to using a reverse proxy
2. Getting NetScaler VPX going in your lab
3. Intro to Content Switching in the NetScaler
4. Configure a basic content-switching application (HTTP) (this post)
5. Configure an advanced content-switching application (SSL-SSL proxy)

Citrix NetScaler Content Switching Overview

Part 3 in a series

In the first part of the series, I discussed the problem facing a user with a single outward-facing public IP address, when he/she wants to host multiple services behind a NAT router that use the same port. My proposed solution is the Citrix NetScaler VPX Express, for which I documented the installation and initial configuration in the second part.

Content Switching, NetScaler-style

Before diving into the configuration steps, I thought it would be helpful to diagram the interconnections and data flow when a content switching setup is completed; I sure could have used it when I started down the path of figuring this out for myself...

Terms

Load Balancing Server

A host external to the NetScaler, specified by name and IP address

Load Balancing Service

A specification of the sort of traffic (protocol & port) that a Server is expecting, and that the NetScaler is switching. One or more "monitor" rules should be attached to the service so that the NetScaler can determine whether the target server is actually available.

Load Balancing Virtual Server

A specification of an "internal" endpoint used by the NetScaler to route traffic. In the case of SSL traffic, a copy of the target server's private key and certificate chain will enable SSL offload.

Content Switching Policy

A definition or rule that is used to identify or select traffic in order to correctly forward requests.

Content Switching Virtual Server

An IP+Port+Protocol specification that the NetScaler uses to listen for incoming traffic. This address is where your NAT router will forward the traffic to be switched. The CSVS has its own SSL certificate pair, and can thus decrypt and review all incoming requests against policies and take action.

Traffic flow

1. An external request is received by the NetScaler on the IP and Port configured as a Content Switching virtual server.
2. The NetScaler inspects the traffic and if it matches a policy rule, forwards the traffic to the target configured for the rule.
3. The target Load Balancing server accepts the traffic, passing it along to the server+service specified.
4. The internal server receives a request as if it were being originated from the NetScaler's Subnet IP.
5. If the first policy rule didn't match, subsequent policy rules are evaluated, sending traffic to targets at the first match. If no policy rules match, a default target may be configured; otherwise, traffic is discarded.

Takeaway

Content Switching elements are configured in the reverse order of traffic flow:

1. Servers
2. Services
3. Load Balancing Virtual Servers
4. Policies
5. Content Switching Virtual Server(s)

Other items:

1. When using multiple Content Switching Virtual Servers, each Policy—by name—can only be configured for one CSVS at a time. As long as they have different names, however, they can have the same matching criteria (eg: HTTP.REQ.HOSTNAME.EQ("fqdn")).
2. One-to-many relationships can exist between a target server and CS Policies; you don't have to go crazy building complex boolean logic into the expressions, just add additional Policies pointing at the same target.
3. A variant of SSL Offload (SSL_BRIDGE) is not possible when using Content Switching. SSL_BRIDGE cannot inspect the headers used by the client in order to know where to send the traffic.

Parts in this series:

1. Intro to using a reverse proxy
2. Getting NetScaler VPX going in your lab 
3. Intro to Content Switching in the NetScaler (this post)
4. Configure a basic content-switching application (HTTP)
5. Configure an advanced content-switching application (SSL-SSL proxy)

Citrix NetScaler VPX Express installation

Part 2 in a series

In the previous post in this series, I posed a problem that could be solved by using the Content Switching capability of the Citrix NetScaler VPX Express. This post focuses on the steps required to get started with the NetScaler virtual appliance; a certain amount of preparation is required before you ever start configuring for reverse proxy operations.

Get the Virtual Appliance

The virtual appliance is currently offered with a 56-week free license; at this point in time, the license can be renewed at no cost, too. You will need a Citrix Login, however: they want you in their marketing database! Luckily, there's a short link that will take you to a starting page; after you log in, you'll be presented with a long list of available versions for XenServer, Hyper-V and VMware. Don't let the "ESX" label fool you: it works just as well on ESXi as ESX.

I've always downloaded the latest/greatest version that is available for my platform:

After downloading, you can unzip the file and import the OVF. I recommend doing it once, converting it to a template, then deploying two VMs from the original.

Don't worry about the network adapter warning when you configure the import parameters: you'll only need one in the "proxy on a stick" configuration.

 Before you convert the VM to a template, edit it and a) remove the second NIC and b) manually set the MAC Address:

The manual MAC is really important: The NetScaler license is tied to the address, and setting it ahead of time will make it easier to re-use a license if you have to re-install one of the two nodes; putting the manual MAC in the template will result in a manual MAC when you deploy a new VM from it.

This also allows you to go straight to the license portion without waiting for the environment to be loaded and partially-configured.

Licensing is available on the same page as the appliance

Unfortunately, Citrix's licensing process is a bit convoluted; it's also outside of the scope of this series. Once you have the license files in hand, however, you're good for a year...

After you deploy the VM, edit the MAC address (eg, 00:50:56:aa:bb:01) and power up your appliance. Open the console; the initial configuration happens using that interface.

Supply the NetScaler's individual address information, save and allow it to restart:

Feel free to "spin up" a second instance, modify its MAC and get it configured; we'll bind the two together into an HA pair in a minute...

Once the first NetScaler is up and initialized, point your favorite Java-enabled (v7b45 or earlier: the Java applets are not signed) browser. Log in using "nsroot" as the username and password; on v10 appliances, leave the deployment type as "NetScaler ADC."

In addition to the IP address you alread entered for management purposes, you must add an IP that it uses to interact with hosts on the subnet; we'll also upload that license file that we already retrieved because we knew the MAC address ahead of time. And although you'd be tempted to change the admin password from "nsroot" at this time, don't: I've run into issues getting the HA pair to work if the password is changed prior to pairing.

Continue and upload the license file:

If you matched the MAC addresses properly, you get the happy green bar! Continue/close until you get the prompt to restart the appliance, then reboot the NetScaler:

Repeat these same steps on your second NetScaler; when both appliances are licensed and you've logged back in, we can proceed with getting the devices paired.

Once you can log into both, pick one to be the Primary of the HA pair and select System->High Availability from the configuration menu:

Click [Add...] to start the wizard to add the other node (at this point, the first Java applet gets loaded on your browser, and you'll know fairly quickly if you'll be opening a different browser window to troubleshoot getting the damn thing to work correctly)

Enter the other node's IP address in the box, leave the checkboxes alone, and enter the default credentials even though they're the same as this node.

Assuming everything else is "happy" in the environment, you should get a confirmation that the HA pair is set up, and if you set the same Subnet IP on both nodes, it might even failover as part of the configuration being established.

You can feel free to browse around the NetScaler capabilities; set up NTP time, etc., as optional exercises. We'll pick up the configuration of a basic content switching configuration in the next part.

Parts in this series:

1. Intro to using a reverse proxy
2. Getting NetScaler VPX going in your lab (this post)
3. Intro to Content Switching in the NetScaler
4. Configure a basic content-switching application (HTTP)
5. Configure an advanced content-switching application (SSL-SSL proxy)

Multiple servers behind One IP

Part 1 in a series

As a systems consultant, I have to deliver a number of software and hardware solutions to customers; in order to do so, I have to know what I'm doing (I know: shocking!), and that takes study and practice. As followers of this blog know, I've got a nice little setup in my basement, and the process of teaching myself these technologies, I've gotten to the point where I want to actually use some of those things.

So, I do what lots of people do: I set up these services and enable port forwarding in my NAT firewall, sending the traffic on a port-by-port basis to the servers that are listening for connections.

Port Forwarding behind a single IP address

In some cases, I don't even have to set up a forward: the device on my network uses a protocol known as "Universal Plug 'N Play" (UPnP) to request port forwarding directly from the router. This all works well as long as I don't have more than one "inside" device listening on a given port. Unfortunately, this leaves me in a conundrum: most of these new & useful applications are using web services—or maybe just the TCP/80 and TCP/443 ports—which makes it hard to manage behind a home router.

The problem: only one destination per port

One alternative is to use non-standard ports for the connections; unfortunately, many applications don't permit the use of non-standard ports (erk!), or when they do, their use is incredibly cumbersome on the client.

Another option—less desirable, and not always functional—is co-installing everything on a single machine. This sets you up for all sorts of compatibility issues and potentially negative interactions. Plus: when we have a virtual environment available, isn't it desirable to have single-purpose machines to avoid all those annoyances?

Luckily, there's a solution: use a proxy (or, more precisely, a reverse proxy). A standard proxy will accept many different outgoing requests and act as a single point of contact for the returning requests. A reverse proxy takes a single incoming request, and after inspecting the request, decides which one among many destinations will receive the traffic.

Reverse Proxy to direct traffic

There are a lot of solutions out there that can provide this solution; this series of blog posts will focus on the Citrix NetScaler VPX Express. This solution has several things going for it, not only for use in the home lab:

1. It's free.
2. A pair is easy to configure for high availability (and is still covered under the no-cost license!)
3. It can be configured for multiple protocols, not just HTTP and/or HTTPS
4. A simple license update and it can support far more than the included 5Mb/s capacity.
5. "Playing" with it can directly translate into using the product in business scenarios.

Parts in this series:

1. Intro to using a reverse proxy (this post)
2. Getting NetScaler VPX going in your lab
3. Intro to Content Switching in the NetScaler
4. Configure a basic content-switching application (HTTP)
5. Configure an advanced content-switching application (SSL-SSL proxy)

Homelab, part troix

With VSAN in beta and having a requirement for a three-host minimum, I was getting antsy: how could I test it in real-life scenarios? Sure, I could've spun up a virtual ESXi instance on one of my two hosts, but that would defeat the purpose of duplicating customer scenarios.

So, I once again reached out to my friends at Stallard Technologies, Inc. for a third host to match the two I'd already purchased. Thankfully, they still had my preferred configuration available, and they even had it on sale for the same price I'd paid before. With my original setup designed around those older 2U hosts, I was still using the 2U brackets to hold each of the 1U servers. With the addition of the third server, I simply move it from side-to-side if I need access to the host that's closer to the wall.

At this point, I'd also reached the limit of the ports available on my Cisco SG300-28, so I needed some changes there, too: I wouldn't just buy 1 switch, but I'd instead pick up a pair so I'd have redundancy for the hosts, and I'd uplink the switches to the Cisco as my "core" switch, as well as cross-connect the new switches to prevent either uplink set as a single-point-of-failure. Given the costs and the number of ports I was planning, SG300 was out of the running due to cost considerations. Based on recommendations, I chose a pair of HP V1910-24G; together, they essentially cost the same as the SG300-28 did.

I again turned to the StarTech equipment brackets, this time for the 3U model to accommodate the three switches.

Network Central

The addition of all those switches were nice, but I was starting to get worried by all those devices with single power supplies: sure, I could plug them all into a UPS, but what happens when the UPS quits? I searched for and found a working automatic transfer switch (ATS) on Ebay for cheap, and added that to the mix, again using a StarTech 1U bracket to tuck it behind the desk.

With common VSAN configurations being designed around 10Gb/s, I knew that kicking the tires with 1Gb/s connections was going to require some careful design decisions: two connections per host, shared with the other IP storage in my environment was going to be a bit thin. A side-effect of adding more network ports, however, was the ability to also increase the number of pNIC ports in the hosts, so I added another dual-port Intel adapter to bring the per-host count up to 8 Gigabit ports.

In my current setup, I have three VDS switches:

1. Management & vMotion (2 x pNIC)
2. Guest networking (2 x pNIC)
3. IP Storage (4 x pNIC)

The first and last switches have vmkernel port groups; the IP Storage switch also has a VM portgroup for in-guest access to iSCSI. It looks a little like this:

In this configuration, I get 4x1Gb/s "pinned" connections for VSAN, two "pinned" connections for iSCSI and two vmkernel NICs for independently-addressed NFS shares. Between NIOC and physical load balancing that's available in the VDS switch, I don't think I'll overwhelm the iSCSI with VSAN (or vice-versa).

This configuration also makes it so no single-chip failure can take out an entire block of services: adjacent pNICs are on different assignments, and paired adapters are either on-board or expansion card.

With 9 connections from each host (8 data NICs for the host; 1 management NIC for the hardware), it was time to get a little more serious about cable management. I ended up buying custom colored patch cables from Monoprice, along with a length of "wire loom"; the loom allowed my to bundle the data cables together into a single, tidier bulk cable. As the diagram above shows, the colors were assigned to NICs and each pair was additionally marked with some black electrical tape. All motherboard-based ports were patched into one switch, while the remaining ports were patched into the second. It's now all very tidy and consistent.

When I purchased those original hosts, I didn't get drives with them; I pretty much presumed that I'd boot from SD card and use shared storage for 100% of the workloads.

Enter PernixData FVP: I'd already had a couple of Intel i520-series SSDs from testing the performance of cache in my iomega arrays (conclusion: it doesn't help), so the first disk installed in my hosts were 240GB SSD.

Enter HP StoreVirtual VSA: I've been a long-time fan of the former Lefthand VSA, and after receiving the latest version as a long-term NFR copy as a vExpert, I decided I needed some capacity to do some testing. I searched around and decided that the Seagate Constellation.2 enterprise SATA were the right choice: sure, they were limited by their interface and rotational speed, but they were also backed by a 5y manufacturer's warranty and had a decent price point for 500GB, all things considered. Two more spindles added to each host.

Enter the third host and VSAN: although the hosts already had what I'd need to do testing (SSD & HDD), I didn't want to tear down the other storage, so it was back to the well for more SSD and HDD.

As they stand today, all three hosts have full inventory of disk: 2x 240GB SSD, 4x 500GB HDD.

Revisiting the Home Lab

Although my home lab has performed flawlessly (aside from the occasional lost hard drive in the NAS boxes), the cornerstone equipment—two Dell PowerEdge 2950 hosts—were a little dated when acquired and have certainly shown their age as post vSphere 5.0 releases (and their accessory services & features) come out.

In short, they weren't cutting it anymore: not enough RAM, physical NICs that don't support NIOC, limited cores, limited CPU/RAM features for the hypervisor, etc.

But, having "saved my pennies" while working with the 2950s, I have treated myself to new kit.

I could have gone down the path of "Baby Dragon"—and had I more free time to do system builds and support my frankenservers—but instead chose recertified Dell PowerEdge R610s from Stallard in my home town of Kansas City, getting another set of enterprise-class servers with a 1-year warranty from STI to boot.

Of several criteria, I knew I wanted a recent generation of hardware with 96 to 128GB of RAM per host. Rackmount would be a good replacement for the existing hosts, as long as the "ears" on the new hosts would work with my crazy vertical setup. PCIe was a given, but with the enormous capacity available among 3 NAS boxes, local storage wasn't a consideration. Finally, I wanted a system that had an inboard SD card reader so I could boot via Flash.

There are many, many options to fit the bill: I watched eBay and Craig's List, checked with my company's NFR purchase options, and several recommended resellers of reconditioned gear. With STI in the area and the option to take direct delivery (no shipping!!), there just wasn't anything else that fit the bill.

I took delivery of the new servers on Friday, 13-Sept, and set about swapping new for old. After putting the first host into maintenance mode and removing it from storage & DVS, I dismounted it and swapped the 2-port Intel server NIC into the new server.

I then mounted the new server after dropping an 8GB "Class 10" SD card into the internal reader, and booted to the vSphere install CD.

For those of you who have done this, you know how quickly ESXi installs; a little additional configuration for storage and vmkernel nics and I was ready to perform my first vMotion to the new host (what I would normally consider a definitive test of a cluster setup). A final step to run Update Manager against the host to get it fully patched, and I was "In Production" on the first host.

I repeated the steps with my second old/new pair, and even with interruptions around the house, was done with the swap in a handful of hours. (Try THAT with a non-virtualized system!)

New lab specs:

• Dell PowerEdge R610
• 96GB RAM (12x8GB)
• 2x Intel E5540 (Quad-core, 2.53Ghz, Hyperthreading enabled)
• Dell SAS 6/iR; No internal storage
• Boot from 8GB SD Card
• 6x 1Gbps (4x Broadcom 5709, 2x Intel 82571EB)
• Redundant power
• iDRAC Enterprise for "headless" operation

Closing Notes:

By choosing to go with Enterprise class equipment, both now an previously, I also have a charitable organization that is quite interested in taking my old 2950s; while I may have out-grown them, these servers would be the first "true" servers they'd ever had. Additionally, the servers remain on the HCL for VMware, so it's quite likely I'll be able to get them to put in VMware rather than trying to do physical server installs on them.

Shrinking the Server 2012 VM by managing the WinSxS repository

With the release of Server 2012, administrators have more control over the data being held in the local %systemroot%\WinSxS (Windows Side-By-Side) folder through the use of PowerShell cmdlets.

The previously-supported Uninstall-WindowsFeature cmdlet has been enhanced with a new argument: -Remove.

When used, the cmdlet will not only uninstall the feature (if installed), it will remove the installer code from the WinSxS. Additionally, a feature that's not installed—but still available in the SxS folder—can be removed as well.

This is particularly valuable when a server VM is fully-deployed and you don't need any additional features; simply run the following cmdlet to remove all that extra cruft:

Get-WindowsFeature | where {$_.InstallState -Eq "Available"} | Uninstall-WindowsFeature -Remove

But what if you need one of those removed features back? There are several mechanisms available; the most transparent one is to use the Add-WindowsFeature cmdlet while connected to the Internet (or with network access to the local Windows Server Update Services host defined in the domain policy). In this use case, the system will retrieve a network copy of the feature and install it.

It might be more efficient, however, to use a readily-available ISO; in that case, you mount the ISO file to the VM and use the -source argument to specify the image for installing the feature:

Add-WindowsFeature $feature -Source:WIM:D:\sources\install.wim:1

There is a bit of a trick in there, too: what's that index number at the end of the source specification? The WIM (Windows IMage) file can contain multiple images; you specify the appropriate image index for the OS edition you're managing. How do you know which index to choose? Use the dism command:

dism /Get-WimInfo /WimFile:D:\Sources\install.wim

Personally, I'm going with a thin-and-trim template for my Server 2012 VMs:

1. Install the Server Core version
2. Add the "Minimal GUI" management interface
   Install-WindowsFeature Server-Gui-Mgmt-Infra
3. Remove all the available features (above)
4. Create a custom unattended sysprep configuration file

With this as a base template, I can easily add needed features (including the full server GUI) from a datastore-based ISO, always accessible to the VM.

certificates 101 (or: how I stopped worrying and learned to love the CSR)

Here's the short of it: I presented a seminar on the #vBrownBag series about the basics of SSL (TLS) certificates. You can find the recording below, as well as at Professional VMware and in the iTunes Store.

How I learned to love the CSR with Jim Millard (@millardjk) ProfessionalVMware #vBrownBag US from ProfessionalVMware on Vimeo.

The "home lab"

Back in 1998, I took an old tower system and put a copy of NetWare 5 on it, then connected a modem and shoved it under a desk. I connected it to two other machines using discarded 10Base-2 network adapters,  RG-59 cable and a couple of terminating resistors. That version was able to do on-demand dialup to an ISP, could share that connection with connected clients, and when you stuck with IPX for inter-machine communication, was immune from most worms or viruses that spread by NetBIOS shares.

No, that wasn't a work or customer environment, it was my home network.

It wasn't too much longer that I joined the beta team for Time Warner Cable, testing out the very first DOCSIS modems in the Kansas City area. Back then, it was faster for me to drive home, download patches (to an Iomega ZIP disk—remember those?) and drive back to work than to use the office network to do it.

My, how things have changed.

I'm still the geek with a home system that rivals those found in some small businesses, but that system serves a very important purpose. This home environment affords me the luxury to experiment, learn and play with systems that are similar enough to business environments to be useful, but not so critical that the occasional upset doesn't result in lost revenue.

Having seen others post information on how they built their labs, I thought I'd post mine, too. Not so much for bragging rights—although there's always a bit of that in IT—but to show yet another variant in the theme.

Compute: 2 x Dell PE2950 III with

• 2 x Intel Xeon E5345 @ 2.33 GHz
• 32 GB DDR2 (8 x 4GB DIMMs)
• Dell PERC 6/i
• Intel PRO/1000 PT Dual Port Server Adapter
• DRAC5
• 2 x 1TB 7200RPM (RAID1), 1 x 240GB SSD; 4 x 500GB 7200RPM (RAID1/0), 1 x 240GB SSD
• Dual power supplies
• 2 x APC BackUPS 1500

Network:

• Cisco SG300-28
• Netgear JGS524
• Apple Airport Extreme
• 2 x Apple Airport Express (1st Generation)
• Apple Airport Express (2nd Generation)
• Astaro ASG110

Storage:

• iomega StorCenter ix2 "Cloud Edition", 2 x 1TB "stock" drives
• 2 x iomega StorCenter px6-300d, 6 x 2TB HDS723020BLA642 (RAID5)
• Synology DS2413+, 12 x ST2000DM001 (RAID5)

The current status is the result of years of slow change. I picked up the first 2950 at Surplus Exchange for $200, with no drives, 4GB RAM and stock network (dual-port Broadcom). I replaced the RAM and added the Intel NIC and PERC 6/i through my friends at Aventis Systems. The second 2950 was from eBay, originally priced as part of a bulk lot with a "buy it now" exceeding $1000; it was only one of several identical systems with 8GB RAM (8 x 1GB), PERC 5/i and no DRAC. I was able to negotiate with the seller to send it without the RAM or PERC for $100, then added the same (plus DRAC) to match the first. In the latter case, it was more important to match MB & CPU than anything else, and I made out like a bandit because of it.

At the time, I put the big drives in the hosts because I didn't have shared storage with sufficient performance to host the VMs I needed to run regularly; running them locally was the only option. Later, I was able to add shared storage (first, an iomega ix4-200d, which actually had enough "steam" to be reasonable with these systems), which I've been slowly updating to the current status.

The PE2x50 is a line of 2U rackmount servers. Rather than leaving them on a table (a pain if you ever need to get into them) or putting them in a rack (lots of floor space consumed), I hung them on the wall. Seriously. Startech.com sells a 2U vertical-mount rack that's intended for network gear or other light gear; I bolted them into the poured-concrete walls of my basement, and hung the servers from them. The front "tabs" of the server are sufficiently sturdy to support the weight, and it gives me full access to the "innards" of the machines without pulling them from the wall.

PE2950 on startech.com vertical rack brackets.

The Cisco switch doesn't run IOS (it doesn't run iOS, either, but that's a different joke) but it is a layer 2* managed switch that does VLAN, QoS, LAG/LACP and other fine functions you expect to find in the enterprise. And yes, I would prefer to have 2, but seem to do fine without as long as I don't need to upgrade the firmware.

This environment is a bit more than just a lab, however; labs have the idea of impermanence to them, while I have a number of systems that I never dump or destroy. This setup is permanent residence of a pair of domain controllers, a pair of file servers, an Exchange 2010 MBX/HT host, a remote desktop server, a multi-purpose web server (it does regular web, Exchange CAS & RDS Gateway), SQL Server and of course, vCenter. The remaining capacity gets used by "play" with other projects in the way one would normally use a lab: VMware View, Citrix XenApp/XenDesktop/XenMobile, vCOps, vCIN (infrastructure navigator).

And as of Monday (29-April-2013) it was upgraded to the latest/greatest version of vSphere: 5.1 U1

Moving the vSphere 5.1 SSO database

Plenty of resources for moving MS SQL Server-hosted vCenter and Update Manager databases. But what about the database for the new Single Sign-On service?

Easy, as long as you get the SQL users moved and change the hostname string in two places.

The easy part is getting the users moved. There's a handy Microsoft KB article for transferring logins from one server to another. I've never had a problem with that.

The harder part is getting the SSO "bits" to accept a new hostname. Thankfully, Gabrie van Zanten was able to document this, along with some other pieces related to SSO database management.

So here's your steps:

1. Execute the sp_help_revlogin stored procedure on the existing SQL server to get the RSA_USER and RSA_DBA logons.
2. Merge the create user lines with the script from the vCenter SSO Install source. This makes certain you have all the necessary attributes for these users.
3. Shut down the SSO service.
4. Backup the current RSA database.
5. Restore the backup on the new server.
6. Execute the user creation lines from Step 2.
7. In a command shell, go to the SSO Server's utils folder (in a default install, the path is C:\Program Files\VMware\Infrastructure\SSOServer\utils) and use the rsautil script to modify the database location:
   rsautil configure-riat -a configure-db --database-host hostname
8. Verify your changes by inspecting .\SSOServer\webapps\ims\WEB-INF\classes\jndi.properties
9. Update the db.host field in the .\SSOServer\webapps\lookupservice\WEB-INF\classes\config.properties file.
10. Restart the SSO service.

Windows Sysprep and VM creation

I've seen a ton of blog posts, reference documents and white papers all instructing you—the virtualization admin—to build "template" VMs in the following fashion:

1. Create a VM and install the OS
2. Install standard applications
3. Set all your configuration settings
4. Patch & Update the OS and applications
5. Sysprep
6. Convert to Template

I'm here to tell you now: stop doing Step 5. Don't sysprep those golden images. At least, don't do it in your template process.

At the very least, using this model means you won't be able to update that template more than 3 times: doing a standard sysprep—without a custom unattended settings file—will "rearm" the software activation only so many times. If you run out of "rearms" you get the joy of rebuilding your golden image.

There is a way around the sysprep limit—see the SkipRearm post for my method—but that still leaves you with a VM template that's going to roll through the remainder of the Sysprep process the first time you turn it on—which you'll be doing every time you want to patch or update the image.

Instead, make Sysprep part of your new VM creation process. With VMware, you can easily convert a VM  back-and-forth from a template to a VM; in fact, for the longest time, I never even converted VMs to templates because there didn't seem to be much value in them: everything you could do to a template, you could do to a VM, while there are things you can do with a VM that you can't do to a template.

Instead, leave your golden image at Step 4; you will be revisiting it every month anyway, right?

Every time you need to spin up a VM from that point forward, you will have a (relatively) recently-patched starting point. In fact, if you're really efficient, you'll run the template VM before creating a new machine from it and patch that machine. Either way, you'll be patching a VM; but if you need to spin up more than one VM, the patching is already complete!

So here's my process:

A) Create your golden image

B) Update your golden image

C) Clone a new VM from the golden image

D) Run Sysprep (with or without SkipRearm and other unattended settings)

E) Repeat steps C-D as needed

F) Repeat step B as needed

Note: I realize there are certain setups that require you to leave a template/golden image at the post-Sysprep shutdown state. In those cases, just make sure you've got a snapshot prior to Sysprep so you can revert to a state before it was ever run.

It's not a watch, it's a Pebble

After being prompted by a tweet from Chris Grossmeier (@cgrossmeier) to check out a Kickstarter project he decided to back, I joined him in the ranks of backers for the single most successful project in Kickstarter history. Originally requesting $100,000 to build a modest little "smart watch," Pebble Technology founder Eric Migicovsky found his project with over $10 million in backing before "selling out."

With that sort of support, Migicovsky revised the scope and breadth of the project, including additional features for the device and plans to retail the watch to non-backers. After many delays—not surprising with Kickstarter projects, but wholly appropriate for the new scope and scale of this one—a Pebble was delivered to my eager hands.

The friendly box design

Inside the spartan box: Pebble watch & its USB power cord

Kickstarter Edition

When first "firing up" the watch, it simply prompts you to pair it with a supported smartphone; in my case, I'd already downloaded the Pebble app from the Apple App Store and was ready to get going.

iOS App

First impressions are everything. It took very little effort to accomplish the Bluetooth pairing, and a software update for the watch was already available for transfer: it shipped with v.1.5.3 and was updated to v1.7.1. With the hints from the iOS app, I was also able to get some of the interactive functions going between watch and phone; it's also the conduit for loading additional watch faces.

Status and tips

App & Watchface Loading

At this time, the SDK isn't publicly available, but a watch face design tool and app creator SDK are in the works. The watch comes with three "hard coded" watch faces, and five more are available in the iOS app. The built-in watch faces can't be deleted, and there's no function for hiding or reordering the menu: new faces always appear below the lowest permanent menu item (Settings).

Built-in Watch Face Options

Additional Menu Options

Default Watch Face

Strangely enough, while the Pebble has a configuration option for setting whether it's a 12- or 24-hour clock by default, one of the original, optional watch faces ("Big Time") was purpose-built to ignore the setting. Since my original inquiries about the behavior, the Pebble team has replaced the original design with a pair of watch faces—Big Time 12 and Big Time 24—to accommodate user desires rather than updating the single face to honor the system setting. This makes me wonder a bit about how sophisticated the API for custom watchfaces is going to be...

Watchfaces

Two faces instead of one

The Pebble is a work in progress: there are some gyrations that one must complete to get notifications for Mail and non-cell applications going (SMS and Call notifications work as soon as pairing is complete) for iPhone, and there are plenty of bugs being discussed on the Pebble forums. Luckily, the guys behind the project "get it," and have been serious about keeping backers updated.

Text Alert on phone

With "project update #32," they went through a laundry list of known issues. Although I'm personally experiencing some problems with my Pebble, it was heartening to see all those issues identified as "known problems" for my Pebble/Phone combination.

From a cosmetic standpoint, I've found that wearing the Pebble on the inside of my wrist is most comfortable; I've found other watches to work better that way, too, but there's the real potential for badly-scratching the watch face.

Watch "rolls away" on back of wrist.

Inside wrist, face stays in a good place.

The backlight is understated enough that it won't cause comments from others at the movie theater, but plenty bright to make the watch readable in a dark(ened) room. It comes on when pressing buttons as one would expect; it will also come on with the flick of the wrist, a cool feature now that the watch contains an accelerometer (not in the original scope).

Overall, I'm satisfied with the Pebble, and am looking forward to the improvements in the functionality as time goes on.

Re-engineering vCenter: a proposal

After fighting my own instances of SSO and vCenter in the vSphere 5.1 management suite, seeing posts from others that have run into the same issues or other new and interesting ones, and generally counseling people to hold off on upgrading to 5.1 because of vCenter issues rather than hypervisor issues, it struck me that I've not seen very many suggestions on how or what to fix.

I'm just as guilty: It's far easier to complain and expect someone else to fix the problem than to wade in provide solutions.

So I did a bit of thinking, and have a set of ideas for re-engineering vCenter to overcome perceived faults.

At any rate, here we go...

Solution 1: SSO as a "blackbox" appliance.

Single sign-on has probably received the worst press of all the new vCenter bits in vSphere 5.1. By divesting this particular piece of all its Windows- and SQL-compatible nature and being distributed as an appliance, the vCenter team could also focus on adding features that allow the single appliance to be scaled (or at least made highly-available as an intrinsic feature).
Problems solved:

• Native code. By standardizing on a single appliance OS, the development team could shelve the low-performing Java code—who's only redeeming value is the ready portability to run on both Windows and Linux platforms—and write using native code and eschew the interpreted languages. This should have the added bonus of being far more "tunable" for memory utilization, resulting in a svelte little appliance instead of a multi-gigabyte monster.
• Integral clustering & load balancing. By adding integrated clustering and shared virtual server technology, the addition of a second appliance immediately eliminates SSO as a single point of failure in the vCenter suite. While the current implementation has a degree of support for adding high availability to this most-crucial of services, the lack of official support for many clustering or high-availability technologies for dependencies (eg, database access, client load balancing) is embarrassing.
• Distributed database. By discarding the use of ODBC-connected databases and falling back on an open-source distributed database (with high levels of data integrity), the appliance can rely on internal database replication & redundancy rather than depending on some other system(s). Single appliances for small implementations are no longer dependent on external systems; multi-node clusters become interwoven, allowing scale-out without any other dependencies, yet behave transparently to systems that rely upon it.

Solution 2: If you're going "blackbox" with SSO, why not vCenter Server, too?

Yes, the vCenter Server Appliance (or VCSA) exists, but in its current iteration, it's limited compared to the Windows Application. Worse, because of a presumed desire to share as much code between the application and the appliance, a large portion—would it be fair to say essentially all of it?—of the server is written in Java. I don't know about you, but while that might serve the goal of making portable code, it certainly isn't what I'd want to use for a performance piece. So the same thing goes here as with SSO:

• Native code.
• Integral clustering (say goodbye to vCenter Heartbeat as an independent product)
• Distributed database (Say goodbye to those MS or Oracle licenses!)

Solution 3: Integrated appliance

If you're going to have SSO and vCenter with the same sort of "black box" packaging, why not combine everything (SSO, Inventory, vCenter, Client, etc.) into a single appliance? We have a degree of that with the VCSA, but without the additional "packaging" as suggested above as well as needing feature-parity with the Windows app. Update Manager should be included, and View Composer could be just another 'click to enable' service that's activated with a license key: when configuring the VCSA, the admin should have the ability to enable arbitrary services, and if the same service is configured on multiple instances of the VCSA, the admin should have the option of enabling that service to run as a member of a cluster instead of having an independent configuration.

Stop with the individual appliances for every little management function: include all of them as a service in every build of VCSA!

No Silver Bullet

These suggestions are neither the "silver bullet" for the current perceived failings in vCenter, and I'm sure my peers can come up with dozens of reasons why these ideas won't work—not to mention the difficulty in actually producing them in code.

If nothing else, however, I hope is sparks thought in others. Maybe some discussion into how things can be improved, rather than simple complaints of "would'a, could'a, should'a" can come from it.

Overriding Citrix VDI-in-a-Box "write reserve" for pooled desktops

It is possible to adjust the "write reserve" that is built into Citrix VDI-in-a-Box (ViaB) managers, in both 5.0 and 5.1 implementations.

Add the instruction:

config.vm.diskspace.reserved.existing=0.0

to the file

/home/kvm/kvm/install/servlet_container/webapps/dt/WEB-INF/etc/store/config.properties (5.0)

or

/home/kvm/kvm/install/servlet_container/webapps/dt/WEB-INF/etc/store/store.properties (5.1)

to remove all "write reserve" for a given manager. This value is read only at startup, so you must restart the manager for the change to take effect.

The default algorithm for ViaB is 10-15% of the image. The manager will assume that each desktop will use at least that much space and refuse to provision additional desktops if sufficient free space on the desktop drive is no longer available.

There are certain use cases where the default reserve is overly conservative; this gives the ViaB administrator additional control over the environment in those situations.

iOS device non-glare screens

I've never been happy with the high-gloss screens on iOS devices: in my opinion, they're fingerprint magnets, regardless of their "oleophobic" coatings.

Since getting my first iPad (the gateway device that suck[er]ed me into the clutches of iOS), I've put non-glare screen films on my devices. Not only does a good film improve the experience for me, I've saved an iPhone from serious damage thanks to it.

I haven't tried them all—they're awful pricey for much experimentation—but I have had both success and failure with various brand's implementation of the film. For the purpose of this post, I'll stick to making only positive recommendations, leaving out any negative reviews; you can find those all over the Internet...

The iLuv "Glare-free protective film" was the first I tried, based on an online recommendation. As it turns out, it was a great choice, and it gave me my first taste of the difficulties involved in getting screen protectors properly "installed" on a device—especially one with the square inches of coverage that an iPad represents. The only negative part of my experience was the lack of local purchase options for the various devices (iPad, iPod Touch, iPhone). Online was the only option for some of them, and after adding both tax & shipping, they were a bit of an investment.

That's when I was introduced to the Power Support HD Anti-Glare screen protectors, the only screen protector I've ever found sold in the Apple retail stores. While all screen protectors have some amount of impact on the original glossy screen—the stated reason why Apple didn't make matte screens a standard option—I've preferred the non-glare aspect of the Power Support film far more than any loss in color fidelity in the Retina display. Similarly, the Apple retail stores in my area only carried the iPhone version; getting them for iPod or iPad are also online experiences.

And with the introduction of a new form-factor—the iPad Mini—it becomes a waiting game to see who will support the new screen first (as of this writing, it was iLuv).

Links:

iLuv
Power Support

vSphere 5.1 Beacon Probes

As in previous versions of vSphere, an administrator for 5.1 can choose to use Link status only or Beacon probing to help determine the uplink status of multi-NIC physical port aggregation.

Failover Detection Options

The default is "Link status only," which merely watches the Layer 2 status for the network adapter. While very simple, it does not have the ability to determine whether "upstream" network problems are present on that link.

That's where Beacon probing comes in handy: By sending a specially crafted frame (it's not a packet; it's an Ethernet frame without any IP-related elements) from one pNIC to another, ESX(i) is able to determine whether a complete path exists between the two. When three or more pNICs are together in an uplink set (in either standard or distributed switches), it's possible to determine with high reliability when a link is "bad," even if the local connectivity is good.

VMware has several blog posts on the topic (What is beacon probing?, Beaconing Demystified: Using Beaconing to Detect Link Failures), and the interwebs are full of information on what it is and how it works for even the most casual searcher.

While working on a completely different system, I was doing some port monitoring and discovered that my ESXi 5.1 hosts were using beaconing. I don't have it turned on in my lab because I have just the one switch: If a link is down, you can immediately detect it without any need to "look downstream." It was kind of annoying to see those showing up in my packet capture, and while it would've been easy enough to filter them, I was more interested in trying to figure out why they were there in the first place: I was pretty sure I hadn't turned Beaconing on for any of my port groups.

Beacon probing frames captured in Wireshark

I went through the settings of all my port groups and verified: all were set to Link status only. What? So I turned to Twitter Tech Support with an inquiry and got a quick reply:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1024435

Unfortunately, setting the Net.MaxBeaconsAtOnce to 0 as suggested in the KB article didn't help: still seeing the beacons. But that suggestion helped me fine-tune some of my search criteria, and a memory was triggered: there's some new network health check capabilities in vSphere 5.1...

Virtual switch health check options in vSphere 5.1

By default, both of these health checks are disabled, but I remember seeing them and enabling them when I first set up 5.1. I wasn't sure which item was the source of the beacon frames, but it's simple and fast to check both options when the frames show up in the packet capture every 30 seconds!

Enabling VLAN and MTU will enable beaconing

Turns out that it's the VLAN and MTU that was putting those beacons out there. I was only watching the traffic for a specific VLAN (which is tagged for my pNICs), so the Teaming and failover option may also put beacons on the untagged network. But the mystery of beacon frames was solved!

Upgrading vSphere 5.1.0 to 5.1.0a

VMware released the 'a' update to their vSphere 5.1 binaries (both vCenter & Hypervisor) on 25-Oct-2012. I downloaded the ISOs for both ESXi (VMware-VMvisor-Installer-201210001-838463.x86_64.iso), vCenter (VMware-VIMSetup-all-5.1.0-880471.iso)  as well as the offline update (ESXi510-201210001.zip) because VMware vSphere Update Manager (VUM) doesn't perceive these as patches.

Update: Since first posting this, I've been informed that VUM is able to patch the ESXi hosts, whether you're running 5.1.0 or 5.1.0a versions of vCenter. I infer one of two things from this: I went after the updates too soon (before VMware had published the update for VUM to use), or my VUM install isn't getting the update info correctly. This change only affects the way you (can) go about updating the host; the vCenter server upgrade doesn't change.

Note: The offline update package for 5.1.0a is not for use with VUM; you'll have to either install from the ISO or use command-line methods to perform an upgrade of ESXi. The latter will be covered in this post.

Reminder: If you run vCenter in a VM, not on a physical host, use out-of-band console access that bypasses the vCenter Server! As soon as that vCenter service stops—which it must—your connectivity to the VM goes away. You can use the VIC if you connect directly to the ESXi host that's running the vCenter VM; that's the way I do mine. Windows Remote Console should only be used with the "/admin" switch, and even then, your mileage may vary. Any other remote access technique that mimics the physical console access is fine. Just don't use the VIC or Web Client remote access via the vCenter services that you're trying to upgrade. "Duh" might be your first response to this, but the first time you forget and connect out of habit, you'll hopefully remember this post and smile.

As with all upgrades, vCenter is first on the list, and in the new model introduced with 5.1, that starts with the SSO Service. That was recognized as an upgrade, and proceeded and succeeded without any additional user entry beyond the obligatory EULA acceptance.

Just to be sure, after SSO finished updating, I tried logging in using the "old style" client (VIC) from the shortcut on the vCenter desktop: no problem. Then I tried it with the Web Client: failure. On a hunch, I restarted the Web Client Service, but with no luck: "Provided credentials are not valid."

Oopsie.

One more test: I'd used the "Use Windows session authentication" option in the Web Client. This time, I tried using the same domain credentials, but manually enter them instead of using pass-through: Pass.

That may be a bug; it may be a problem with unmatched versions of SSO and Web Client. Moving on with the rest of the upgrade...

The next step is to upgrade the Inventory Service service; like SSO, it can upgrade without specific user input. However, when the service is replaced with the newer version, vCenter Server (and other services dependent on vCenter Service) is stopped and not restarted. Manually restarting the services will allow you back at your system again, just in case you get interrupted while working and need to get back on before updating the vCenter Server service to the new version...

Like the previous services, vCenter Server recognizes that it's an upgrade and click, click, click to complete. Naturally, the services are stopped in order to replace them, but the installer does restart them when it's done. Upgrading the VIC is another click,click,click operation, as is the Web Client.

It did not, however, fix the pass-through authentication issue in the Web Client.

I spent a while in conversation with Chris Wahl and Doug Baer on Twitter, trying to get it straightened out. Both are VCDX and super sharp, and they gave me lots of solid advice for improving bits of my vCenter setup, but this one wasn't budging. At this point, I've given up on it: there's a solid workaround, so it's not a dealbreaker. Watch this space, however: if/when I figure it out, I'll pass along my findings.

VUM isn't updated in this release, so that bit doesn't need to be upgraded or reinstalled. However, the offline package isn't going to work with it (as mentioned above), so the upgrade is done using one of the alternate methods. My preferred choice is to use the busybox shell via SSH.

To use this method, I first used the VIC to upload the offline update to a datastore visible to all my hosts. Next, I put the first host into Maintenance Mode. Because I took advantage of the sweet "ssh AutoConnect" plugin, the console of the host is a right-click away. Once at the console, the following command is executed:

esxcli software vib update -d /vmfs/volumes/[DATASTORE]/[PATCH_FILE].zip

After a short wait, the prompt was back informing me that the update was complete, and a reboot was required to implement. You can use your favorite method of restarting the host, and once it returns to connectivity with vCenter, you have to manually exit Maintenance Mode. Repeat on each host until you're fully updated.

This update didn't replace Tools with a new version; the tools installed as part of the GA version are recognized as current, so I didn't get to see if the promise of no-reboot Tools updates would come to fruition.