Importing a CA-signed certificate in VMware vSphere Log Insight

I came across a retweet tonight; someone looking for help getting VMware vSphere Log Insight (vCLog) to accept his CA-signed certificate for trusted SSL connections:

As luck would have it, I was able to get it going in my home environment a while back. In the course of studying for Microsoft certifications, I had the opportunity to get some real practice with their certificate authority role, and I have a "proper" offline root and online enterprise intermediate as an issuer running in my home lab environment.

With that available to me, I'd already gone through and replaced just about every self-signed certificate I could get my hands on with my own enterprise certs, and vCLog was just another target.

I will admit that in my early work with certificates and non-Windows systems, I had a number of false starts; I've probably broken SSL in my environment as many times as I've fixed it.

One thing I've learned about the VMware certs: they tend to work similarly. I learned early on that the private key cannot be PKCS#1 encoded; it must be PKCS#8 key. How can you tell which encoding you have?

If the Base64 header for your private key looks like this:

-----BEGIN PRIVATE KEY-----

you have a PKCS#1 key. If, instead, it looks like this:

-----BEGIN RSA PRIVATE KEY-----

then you have the PKCS#8 key. Unfortunately, many CAs that provide the tools needed to create the private key and a signed cert only provide you with the PKCS#1 key. What to do? Use your handy-dandy OpenSSL tool to convert it (note: there are live/online utilities that can do this for you, but think twice and once more just for good measure: do you really want to give a copy of your private key to some 3rd party?):

openssl rsa -in private.pkcs1 -out private.pkcs8

Once you have the properly formatted private key, you must assemble a single file with all the certs in the chain—in the correct order—starting with the private key. This can be done with a text editor, but make sure you use one that honors *NIX-style end-of-line characters (newline as opposed to carriage-return+linefeed like DOS/Windows likes to use).

Most public Certificate Authorities (I personally recommend DigiCert) are going to be using a root+intermediate format, so you'll end up with four "blobs" of Base64 in your text file:

-----BEGIN RSA PRIVATE KEY-----
[Base64-encoded private key blob]
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
[Base64-encoded intermediate-signed server certificate]
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
[Base64-encoded root-signed intermediate CA cert]
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
[Base64-encoded root CA cert]
-----END CERTIFICATE-----

Note that there's nothing in between the END and BEGIN statements, nor preceding or following the sections. Even OpenSSL's tools for converting from PKCS#12 to PEM-encoded certificates may put "bag attributes" and other "human readable" information about the certificates in the files; you have to strip that garbage out of there for the file to be acceptable.

If you follow these rules for assembly, your file will be accepted by vCLog's certificate import function, and your connection will be verified as a trusted connection.

An odd thing happened on the way to the VSAN...

Object placement

Cormac Hogan has a nice post on the nature of VSAN from an "Objects & Components" perspective, Rawlinson Rivera describes witness creation & placement, and Duncan Epping teaches the user how to see the placement of objects in VSAN.

Based on these (and many other articles written by them and other authors—check out Duncan's compendium of VSAN links) I thought I had a pretty good idea of how a VM would be laid out on a VSAN datastore.

Turns out, I was wrong...

Default use case

Take a VM with a single VMDK and put it on a VSAN datastore with no storage policy, and you get the default configuration of Number of Failures to Tolerate (nFT) = 1 and Number of Disk Stripes per Object (nSO) = 1. You'd expect to see the disk mirrored between two hosts, with a third host acting as witness:

Image shamelessly copied from Duncan

If you drill down into the object information on the Web Client, it bears out what you'd expect:

Multi-stripe use case

The purpose of the "Number of Disk Stripes per Object" policy is to leverage additional disks in a host to provide more performance. The help text from the nSO policy is as follows:

The number of HDDs across which each replica of a storage object is striped. A value higher than 1 may result in better performance (for e.g. when flash read cache misses need to get services from HDD), but also results in higher use of system resources. Default value 1, Maximum value: 12.

In practice, adding additional stripes results in VSAN adding a new "RAID 0" layer in the leaf objects hierarchy under the "RAID 1" layer. That first level is the per-host distribution of objects needed to meet the nFT policy rule; this second layer represents the per-host distribution of objects necessary to meet the nSO policy rule.

As you can see from the diagram, the stripes for a single replica aren't necessarily written to the same host. Somehow, I'd gotten the impression that a replica had a 1:1 relationship with a host, which isn't the way it's run in practice.

I'd also been misreading the details in the web client for the distribution of the components; when all your disks have a display name that only varies in the least-significant places of the "naa" identifier, it's easy to get confused. To get around that, I renamed all my devices to reflect the host.slot and type so I could see where everything landed at a glance:

As this screencap shows, the VM's disk is a 4-part stripe, split among all three of my hosts. One host (esx2) has all four components, so the other hosts need enough "secondary" witnesses to balance it out (three for esx3 because it hold one data component, one for host esx1 because it holds three data components). There's also a "tiebreaker" witness (on esx1) because the sum of the data components and secondary witnesses is an even number.

The other disks show similar distribution, but the details of disk utilization is not the same. The only thing I've found to be fairly consistent in my testing is that one host will always get an entire replica, while the other two hosts share components for the other replica; this occurs for all policies with nSO>1. If you have more than 3 hosts, your results will likely be different.