Friday, October 10, 2014

VeeamON 2014: A post-event challenge

Branded as the industry's first and only "Data Center Availability" conference, Veeam's freshman effort was a success by almost any measure.

Disclaimer: I work for a Veeam Partner and my conference attendance was comp'd in exchange for some marketing/promotional activities prior to the conference. I have also been a long-time user of Veeam Backup & Replication, since before my transition to the partner side of business due to my vExpert status in the VMware community.

Because I work for a partner, I arrived in Las Vegas on Sunday, October 5 to attend the partner-oriented social & networking events and to be ready for the 8:30am start on Monday morning for the partner keynote.

In a twist from other industry conferences I've attended, the keynote was MC'd by comedian Richard Laible, with a format intended to mimic those of late-night talk shows. It was successful, and the give-and-take between Richard and his "guest" was well-orchestrated and amusing.

In the first "interview," Veeam CEO Ratmir Timashev was able to tell the story of the founding of Veeam, underscore the company's love of their reseller-partners and reaffirmed the company's longstanding policy of staying 100% "channel-based" (no customer may purchase directly from Veeam); most important, he talked about the shift of Veeam from being "merely the best" backup product for virtualization, but to strive towards producing the best availability product for the enterprise.

Other Veeam employees took to the stage, and customer success stories were played out. In other words, much like any other keynote.

The remainder of the day was filled with breakout sessions covering a wide range of topics--both technical and business-oriented--for the partner crowd. The obligatory sponsor exposition opened for a happy hour/dinner reception, which also capped-off the scheduled activities for the day.

The second full day of events (Tuesday) was opened with a second keynote which echoed much of the messaging in the Partner keynote, but with an obvious new audience: the customer & prospects attending the event. In addition to even more entertainment (a pair from X-Pogo performed), some additional features of the forthcoming Version 8 for the "Availability Suite" (a rebranding of the former Backup & Management Suite) were shared, as well as even more customer testimonials which underscored Veeam's commitment not just to protecting data, but to making good on their aim to create the "always available datacenter."

The remainder of the day was again filled with breakout sessions, again ranging from business to technical topics. The day was scheduled late, however, with the optional party at the "LIGHT" nightclub in the Mandalay Bay hotel.

The third and final day opened with breakout sessions, these principally seemed to be presented by sponsor partners rather than Veeam employees with Veeam-specific topics. None of the sessions I attended, however, seemed too far off-base at a Veeam-oriented conference: the connection and/or synergy between the sponsor's product & Veeam's products was clear by the end of the session.

A final keynote by's co-founder, Alexis Ohanian, was both humorous and insightful, and essentially closed out the conference.

There are many other posts out there with even more details and insight into the conference; check out my fellow #vDBer Mike Preston's series from the conference at for more insight and reporting.

My retelling of this is all to aim towards one thought: Veeam did a great job on their first conference. The content was relevant, the sponsors were invested and made sense, and it was both informative and entertaining.

Here's the challenge: What about 2015?

Unless the breakout catalog is significantly expanded, I'm not sure how many folks will want/need to attend a second year. Don't get me wrong: I'm not saying that no one will attend. On the contrary: if they repeated next year with a cookie-cutter duplicate of this year, anyone who a) didn't attend and b) wants to learn more about Veeam's products and how they can boost the availability of the datacenter would find their time well-spent.

I'm saying that everyone that went was a first-timer, and they got that spot-on. They can still fine-tune it, but next year's first-time attendee will get great value whether they change it or not.

No, the problem is getting repeat attendees. The conference can increase their first-time attendee counts simply based on positive word-of-mouth recommendations, but the top end for that will be reached far sooner than getting both those new attendees and the repeat (alumni?) attendees.

As it was, the number that was rumored prior to the conference—around 1200 people comprised of attendees & Veeam staff—seemed to have some validity. The conference space at the Cosmopolitan was sized well for the attendees, and it was never crowded or crazy like VMworld can feel (with almost 20x the attendance). But I can't imagine that Veeam is going to be content with putting on a two-and-a-half-day conference for "only" 1000 people. Yes, you want a multi-day conference to help justify the travel costs, but let's be honest: the VMUG organization has chapters that manage to put together single-day conferences for that number of attendees.

This isn't meant as criticism: I'm identifying the challenge they now face, and send the call-to-action to Veeam to plan next year's event—as far as I know, TBA for place & time, yet expected from Doug Hazelman's parting "See you next year at VeeamON 2015"—with the goals of both increasing the number of new attendees compared to the inaugural "class" from this year, as well as compelling most (if not all) of this year's attendees to return.

Wednesday, April 9, 2014

Importing a CA-signed certificate in VMware vSphere Log Insight

I came across a retweet tonight; someone looking for help getting VMware vSphere Log Insight (vCLog) to accept his CA-signed certificate for trusted SSL connections:

As luck would have it, I was able to get it going in my home environment a while back. In the course of studying for Microsoft certifications, I had the opportunity to get some real practice with their certificate authority role, and I have a "proper" offline root and online enterprise intermediate as an issuer running in my home lab environment.

With that available to me, I'd already gone through and replaced just about every self-signed certificate I could get my hands on with my own enterprise certs, and vCLog was just another target.

I will admit that in my early work with certificates and non-Windows systems, I had a number of false starts; I've probably broken SSL in my environment as many times as I've fixed it.

One thing I've learned about the VMware certs: they tend to work similarly. I learned early on that the private key cannot be PKCS#1 encoded; it must be PKCS#8 key. How can you tell which encoding you have?

If the Base64 header for your private key looks like this:
you have a PKCS#1 key. If, instead, it looks like this:
then you have the PKCS#8 key. Unfortunately, many CAs that provide the tools needed to create the private key and a signed cert only provide you with the PKCS#1 key. What to do? Use your handy-dandy OpenSSL tool to convert it (note: there are live/online utilities that can do this for you, but think twice and once more just for good measure: do you really want to give a copy of your private key to some 3rd party?):
openssl rsa -in private.pkcs1 -out private.pkcs8
Once you have the properly formatted private key, you must assemble a single file with all the certs in the chain—in the correct order—starting with the private key. This can be done with a text editor, but make sure you use one that honors *NIX-style end-of-line characters (newline as opposed to carriage-return+linefeed like DOS/Windows likes to use).

Most public Certificate Authorities (I personally recommend DigiCert) are going to be using a root+intermediate format, so you'll end up with four "blobs" of Base64 in your text file:
[Base64-encoded private key blob]
[Base64-encoded intermediate-signed server certificate]
[Base64-encoded root-signed intermediate CA cert]
[Base64-encoded root CA cert]
Note that there's nothing in between the END and BEGIN statements, nor preceding or following the sections. Even OpenSSL's tools for converting from PKCS#12 to PEM-encoded certificates may put "bag attributes" and other "human readable" information about the certificates in the files; you have to strip that garbage out of there for the file to be acceptable.

If you follow these rules for assembly, your file will be accepted by vCLog's certificate import function, and your connection will be verified as a trusted connection.

Wednesday, April 2, 2014

An odd thing happened on the way to the VSAN...

Object placement

Cormac Hogan has a nice post on the nature of VSAN from an "Objects & Components" perspective, Rawlinson Rivera describes witness creation & placement, and Duncan Epping teaches the user how to see the placement of objects in VSAN.

Based on these (and many other articles written by them and other authors—check out Duncan's compendium of VSAN links) I thought I had a pretty good idea of how a VM would be laid out on a VSAN datastore.

Turns out, I was wrong...

Default use case

Take a VM with a single VMDK and put it on a VSAN datastore with no storage policy, and you get the default configuration of Number of Failures to Tolerate (nFT) = 1 and Number of Disk Stripes per Object (nSO) = 1. You'd expect to see the disk mirrored between two hosts, with a third host acting as witness:
Image shamelessly copied from Duncan
If you drill down into the object information on the Web Client, it bears out what you'd expect:

Multi-stripe use case

The purpose of the "Number of Disk Stripes per Object" policy is to leverage additional disks in a host to provide more performance. The help text from the nSO policy is as follows:
The number of HDDs across which each replica of a storage object is striped. A value higher than 1 may result in better performance (for e.g. when flash read cache misses need to get services from HDD), but also results in higher use of system resources. Default value 1, Maximum value: 12.
In practice, adding additional stripes results in VSAN adding a new "RAID 0" layer in the leaf objects hierarchy under the "RAID 1" layer. That first level is the per-host distribution of objects needed to meet the nFT policy rule; this second layer represents the per-host distribution of objects necessary to meet the nSO policy rule.

As you can see from the diagram, the stripes for a single replica aren't necessarily written to the same host. Somehow, I'd gotten the impression that a replica had a 1:1 relationship with a host, which isn't the way it's run in practice.

I'd also been misreading the details in the web client for the distribution of the components; when all your disks have a display name that only varies in the least-significant places of the "naa" identifier, it's easy to get confused. To get around that, I renamed all my devices to reflect the host.slot and type so I could see where everything landed at a glance:

As this screencap shows, the VM's disk is a 4-part stripe, split among all three of my hosts. One host (esx2) has all four components, so the other hosts need enough "secondary" witnesses to balance it out (three for esx3 because it hold one data component, one for host esx1 because it holds three data components). There's also a "tiebreaker" witness (on esx1) because the sum of the data components and secondary witnesses is an even number.

The other disks show similar distribution, but the details of disk utilization is not the same. The only thing I've found to be fairly consistent in my testing is that one host will always get an entire replica, while the other two hosts share components for the other replica; this occurs for all policies with nSO>1. If you have more than 3 hosts, your results will likely be different.