Tuesday, April 7, 2015

Don't say "Customer"

customer
one that purchases a commodity or service
According to the above definition from Webster's, it's quite likely that you deal with people that can be labelled with that term. Even those of you who are not in specific sales roles may have used the term to refer to those you serve (eg, "the internal customer").

Good or bad, that term has a certain "transactional" connotation to it; the retail model is pretty clear: a person goes to a store, buys a product or receives a service, then leaves. You don't have a relationship with the clerk running the point-of-sale system; you get your stuff and go. Transactional.

For gasoline purchases and groceries, this may be the right model and the right term. But is that the right model for you in IT? Consider it instead from the reverse point of view:

What is your relationship with your barber/hair stylist (Some of you reading this might not use one: your tonsorial needs may be non-existent or you may be able to handle things for yourself. But I bet you can probably remember a time when it was a regular requirement...)? Do you have a certain place to which you always return? When you go there, is there a particular person to whom you look (or schedule or wait) because you get a consistently good cut, or because the conversation is more pleasant, or some other (in)tangible benefit? Do you like the personal service that comes with being known by your first name? Would you have a hard time switching to a different barber/stylist because of the trust you've given to your current one? That is the sort of relationship you should seek to have with your...clients.
client
  1. one that is under the protection of another : dependent
  2. a: a person who engages the professional advice or services of another
    b: customer
While a client can be a form of customer, the superior definitions imply a more intimate relationship between the two parties. Dependency (used in the primary definition) can have both negative and positive connotations, but in this context, we're essentially talking about repeat business. This is what we're after, both as provider and consumer: as a provider, it keeps me in business and employed; as a consumer, I can either add the process of determining "from whom" as the same time I'm trying to figure out the "what" for my need. As a client, you put a certain level of trust into the relationship: you trust that your provider will have your interests at heart; that you won't be taken advantage of; that you can rely on the quality of the work.

That, then, pushes a certain level of responsibility on the provider. Be(come) the trusted adviser by not abusing the trust. Provide good advice. Provide repeatably good service and/or products. Own your mistakes and gently guide your client away from making them on their own. This is how one treats his/her clients—especially if the goal is to keep them!

If you're not already in this frame of mind, I challenge you to make this shift in a simple yet subtle way: Even if you're in the "business of selling widgets," even if you're running a convenience store selling gasoline and snack food, train yourself to stop using the word "customer" and start using the word "client" instead. Words have power; they convey ideas and have implications. Changing the use of that one word should change the way you look at the people you serve; when your outlook changes, the way you act and react in the relationship should follow. All of your clientele may not perceive the difference, either overtly or subconsciously; some still want to be "merely" customers, ignoring the relationship and simply needing a widget or two. Making this adjustment won't "fix" that relationship, but neither should it affect your ability to be there to serve them when they choose you. But the shoppers, the fence-sitters? With this one subtle change, you could influence them in a way that sends them into your care with more frequency.

Disclaimer: I currently work for a value-added reseller—a "VAR" in industry parlance—but have also spent a long time as a purchaser of products and services. I believe this concept is valid in either case.

Sunday, March 8, 2015

Fix vShield Manager after modifying vSphere VDS uplinks

If you've been following my posts about upgrading my home lab, you know that I removed the add-in 1Gbps NICs and consolidated the motherboard-based 1Gbps NICs on one DVS (distributed virtual switch) in order to add 10Gbps support to my hosts. In that process, I not only rearranged the physical NICs for the uplinks, I also updated the uplink names in order to keep my environment self-documenting.

Things pretty much went as planned, but I didn't expect vShield Manager (vSM) to choke on the changes: when updating the uplink names for the DVS that provided the VXLAN port group, I expected vSM to recognize the changes and handle creation of new VXLAN networks without issue. I was wrong.

The first symptom that I had an issue was the inability of vCloud Director (vCD) to create a new Organizational Network on a deployed Edge device:
So: something is off with the teaming policy. Time to look at vSM to determine whether vCD is sending a bad request to vSM, or if vSM itself is the source of the issue. The easiest way to check is to manually create a new network in vSM; if it succeeds, vCD is sending a bad request, otherwise I need to troubleshoot vSM--and possibly vCenter, too.
Boom: the problem is reproduced even for a test directly in vSM. Time to verify the teaming in the base portgroup in vCenter.
Oops. I hadn't updated the portgroup for VXLAN after moving the uplinks around, although I had done so for the other portgroups on the DVS.
Unfortunately, updating the portgroup to use all the available uplinks didn't help. However, in the process, I discovered an unexpected error in vCenter itself:
vSM was making an API call to vCenter that included one of the old uplink names, one which no longer existed on the DVS. To test the theory, I added a couple of additional uplink ports to the DVS and renamed one to match the missing port. It worked, but not as expected:
vSM was able to send a proper API call to vCenter, but the portgroup had sub-optimal uplink settings: of the two active uplinks, only one had an actual, physical uplink associated with it. This was not a redundant connection, even though it looked like it.

Time to restart vSM to get it to re-read the vCenter DVS config, right? Wrong. Even with a restart & re-entering the vCenter credentials, the state persisted.

At this point, my Google-fu failed me: no useful hits on a variety of search terms. Time to hit the VMware Community Forums with a question. Luckily, I received a promising answer in just a day or two.

I learned that one can use the REST API for vSM to reconfigure it, which can get it back in line with reality. But how do you work with arbitrary REST calls? It turns out, there's a REST client plug-in for Firefox, written to troubleshoot and debug REST APIs. It works a treat:
  1. Set up the client for authenticated headers
  2. Retrieve the DVS configuration as an XML blob in the body of a GET call
  3. Modify the XML blob so that it has the correct properties
  4. PUT the revised XML blob back to vSM.
Voila! Everything works.

Specifics:
1) Use an Authenticated GET on the switches API
2) Using the objectId of the desired DVS, get the specific switch data
3) Update the XML blob with the correct uplink names
4) PUT the revised XML blob
As soon as this blob was accepted with a 200 OK response, I re-ran my test in vSM: success! vCD was also able to successfully create the desired portgroup, too.

Key takeaways:
  1. REST Client for Firefox is awesome for arbitrary interaction with a REST API
  2. Sometimes, the only way to accomplish a goal is through the API; a GUI or CLI command may not exist to fix your problem.
  3. This particular fix allows you to arbitrarily rename your uplinks without having to reset the vShield Manager database and completely reinstall it to get VXLAN working again.

Monday, February 23, 2015

Planning for vSphere 6: VMCA considerations

With the immanent release of vSphere 6, I've been doing prep work for upgrades and new installs. There's a lot of information out there (just check out the vSphere 6 Link-O-Rama at Eric Siebert's vSphere Land for an idea of the breadth & depth of what's already written), but not as much as I'd like to make good decisions in order to future-proof the setup.

I'm sure I join lots of VMware admins in looking forward to the new features in vSphere 6—long-distance vMotion, cross-datacenter & cross-vCenter vMotion, multi-vCPU fault tolerance (FT), etc.—but along with these features come some foundational changes in the way vSphere management & security are architected.

Hopefully, you've already heard about the new PSC (Platform Services Controller), the functional descendant of the SSO service introduced in vSphere 5.1. SSO still exists as a component of the PSC, and the PSC can be co-installed ("embedded") on the same system as vCenter, or it can be independent. Like SSO on vSphere 5.5, the PSC has its own internal, dedicated database which it replicates with peer nodes, similar to what we've come to know and expect from Microsoft Active Directory.

This replication feature not only helps for geographically-distributed enterprises—allowing a single security authority for multiple datacenters—but high availability in a single datacenter through the use of 2 (or more) PSCs behind a load balancer. Note the emphasis on the load balancer: you will end up with the abstraction of the PSC with a DNS name pointing at an IP address on your load balancing solution, rather than the name/IP of a PSC itself.
PSC in load-balanced HA configuration

This delegation means you must plan ahead of time for using load balancing; it's really not the sort of thing that you can "shim" into the environment after implementing a single PSC.

Joining SSO in the PSC "black box" are several old and some brand new services: identity management, licensing...and a new Certificate Authority, aka VMCA (not to be confused with vCMA, the vCenter Mobile Access fling) .

It's that last item—the Certificate Authority—that should make you very nervous in your planning for vSphere 6. The documentation so far indicates that you have upwards of four different modes for your CA implementation that are independent of your PSC implementation choices:
  • Root (the default). Source of all certs for dependent services, this CA's public key/cert must be distributed and placed in your trusted CA store.
  • Intermediate. Source of all certs for dependent services, but the CA itself gets its trust from a parent CA, which in turn must have its public key/cert distributed. In the case of corporate/Enterprise CA/PKI infrastructure, this will already be in place and will be my go-to configuration.
  • None/External-only. All services receive their certs from a different CA all together. This model is equivalent to removing all the self-signed certificates in pre-6 and replacing them with signed certificates. With the proliferation of services, each using its own certificate, this model is becoming untenable.
  • Hybrid. In the hybrid model, the VMCA provides certificates to services that provide internal communication (either service-to-service or client-to-service) while public CA-signed certs are used in the specific places where 3rd-party clients will interact. In this model, the VMCA may act as either root or intermediate CA.
Confused? Just wait: it gets more complicated...

Migrating from one model to another will have risks & difficulties associated with it. The default installer will set you up with a root CA; you will have the option to make it an intermediate at time of install. As near as I can tell from the available documentation, you will need to reinstall the PSC if you start with it as a root CA and decide you want it instead to be an intermediate (or vice-versa). This is consistent with other CA types (eg, Microsoft Windows), so there's no surprise there; however, it's not clear what other replicated services will be impacted when trying to swap CA modes, as it will require previously-issued certificates to be revoked and new certificates to be issued.

You can switch some or all of the certificates it manages with 3rd-party (or Enterprise) signed certs, but once you do, you will have to deal with re-issue & expiration on your own. I can't find anything documenting whether this is handled gracefully & automatically with VMCA-signed certs & services, similar to the centralized/automated certificate management that Windows enjoys in an Enterprise CA environment.

There isn't any documentation on switching from 3rd-party (or Enterprise) back to a VMCA-signed certificate. Presumably, it'll be some CLI-based witchcraft...if it's allowed at all.

Finally, keep in mind that DNS names factor heavily into certificate trust. Successfully changing name and/or IP address of an SSO server—depending on which was used for service registration—can be challenging enough. Doing the same with a system that is also a certificate authority will be doubly so.

So: what's the architect going to do?

For small, non-complex environments, I'm going to recommend what VMware and other bloggers recommend: stick with the single, combined PSC and vCenter server. Use the VCSA (vCenter Server Appliance) to save on the Windows license if you must, but I personally still prefer the Windows version: I'm still far more comfortable with managing the Windows OS environment & database than I am with Linux. Additionally, you're going to want Update Manager—still a Windows service—so I find it easier to just keep them all together.


This also suggests using the VMCA as a root CA, and I'll stick with that recommendation unless you have an Enterprise CA already. If you have the Enterprise CA, why not make it an intermediate? At a minimum, you'll would eliminate the need for yet another root certificate to distribute. More importantly, however, is that it's vastly easier to replace an intermediate CA—even through the pain of re-issuing certificates—than a root CA.

What constitutes small, non-complex? For starters, any environment that exists with one—and only one—vCenter server. You can look up the maximums yourself, but we're talking about a single datacenter with one or two clusters of hosts, so less than 65 hosts for vSphere 5.5; in practice, we're really talking about environments with 20 or fewer hosts, but I have seen larger ones that would still meet this category because—other than basic guest management (eg, HA & DRS)—they aren't really using vCenter for anything. If it were to die a horrible death and be redeployed, the business might not even notice!

Even if you have a small environment by those standards, however, "complex" enters the equation as soon as you implement a feature that is significantly dependent on vCenter services: Distributed Virtual Switch, Horizon View non-persistent desktops, vRealize Automation, etc. At this point, you now need vCenter to be alive & well pretty much all the time.

In these environments, I was already counseling the use of a full SQL database instance, not SQL Express with all of its limitations. Even when you're inside the "performance bubble" for that RDBMS, there are a host of other administrative features you must do without that can compromise uptime. With vSphere 6, I'm continuing the recommendation, but taking it a step further: use AlwaysOn Availability Groups for that database as soon as it's certified. It's far easier to resurrect a cratered vCenter server with a valid copy of the database than rebuilding everything from scratch; I know VMware wants us all to treat the VCSA as this tidy little "black box," but I've already been on troubleshooting calls where major rework was required because no maintenance of the internal PosgreSQL database was ever done, and the whole-VM backup was found wanting...

Once you've got your database with high availability, split out the PSC from vCenter and set up at least two of them, the same way you'd set up at least two Active Directory domain controllers. This is going to be the hub of your environment, as both vCenter and other services will rely on it. Using a pair will also require a load balancing solution; although there aren't any throughput data available, I'd guess that the traffic generated for the PSC will be lower than the 10Mbps limit of the free and excellent Citrix NetScaler VPX Express. I've written about it before, and will be using it in my own environment.

Add additional single and/or paired PSCs in geographically distant locations, but don't go crazy: I've seen blogs indicating that the replication domain for the PSC database is limited to 8 nodes. If you're a global enterprise with many geographically-diverse datacenters, consider a pair in your primary, most critical datacenter and single nodes in up to 6 additional datacenters. Have more than 7 datacenters? Consider the resiliency of your intranet connectivity and place the nodes where they will provide needed coverage based on latency and reliability. If you're stumped, give your local Active Directory maven a call; he/she has probably dealt with this exact problem already—albeit on a different platform—and may have insight or quantitative data that may help you make your decision.

Finally, I'm waiting with anticipation on an official announcement for FT support of vCenter Server , which will eliminate the need for more-complex clustering solutions in environments that can support it (from both a storage & network standpoint: FT in vSphere 6 is completely different from FT in previous versions!). Until then, the vCenter Server gets uptime & redundancy more through keeping its database reliable than anything else: HA for host failure; good, tested backups for VM corruption.