Tuesday, December 16, 2014

Remote Switchport Identification for ESXi

I was working by remote, trying to complete some work in a client's VMware environment when I discovered that one of the hosts didn't have the proper trunking to its network adapters. I had access to the managed switch, but for one reason or another, the ports weren't identified in the switch. Had the switch been from Cisco, the host itself could've told me what I needed: ESXi supports CDP on the standard virtual switch & uplinks.
But this was an HP switch.
Luckily, I had three things going for me:

  1. The HP switch supported LLDP
  2. I had access to temporary Enterprise Plus licensing
  3. The host had redundant links for the virtual switch.
How did that help? 

While the standard switch will only support CDP, the VMware Distributed Switch (VDS) supports either CDP or LLDP.

Here's how I managed to get my port assignments:
  1. Create a VDS instance
  2. Modify the distributed virtual switch (DVS) to use LLDP instead of CDP (the default)
  3. Update host licensing to temporary Enterprise Plus
  4. Add one (1) adapter to the DVS uplink group
  5. After 30 seconds, click on the "information" link for the adapter to retrieve switchport details
  6. Return adapter to the original standard switch
  7. Repeat steps 3-5 for additional adapters
  8. Remove host from DVS
  9. Return host licensing back to original license
  10. Repeat steps 3-9 for remaining hosts
  11. Remove DVS from environment

Saturday, November 8, 2014

Use Synology as a Veeam B&R "Linux Repository"

I posted a fix earlier today for adding back the key exchange & cipher sets that Veeam needs when connecting to a Synology NAS running DSM 5.1 as a Linux host for use as a backup repository. As it turns out, some folks with Synology devices didn't know that using them as a "native Linux repository" was possible. This post will document the process I used to get it going originally on DSM 5.0; it wasn't a lot of trial-and-error, thanks to the work done by others and posted to the Veeam forums.

Caveat: I have no clue if this will work on DSM 4.x, as it wasn't until I was already running 5.0 when I started to work on it.

  1. Create a shared folder on your device. Mine is /volume1/veeam
  2. Install Perl in the Synology package center.
  3. If running DSM 5.1 or later, update the /etc/ssh/sshd_conf file as documented in my other post
  4. Enable SSH (control panel --> system -->terminal & snmp)
  5. Enable User Home Service ( control panel --> user --> advanced)
Once this much is done, Veeam B&R will successfully create a Linux-style repository using that path. However, it will not be able to correctly recognize free space without an additional tweak, and for that tweak, you need to understand how B&R works with a Linux repository...

When integrating a Linux repository, B&R does not install software on the Linux host. Here's how it works: 
  1. connects to the host over SSH
  2. transmits a "tarball" (veeam_soap.tar)
  3. extracts the tarball into temporary memory
  4. runs some Perl scripts found in the tarball
It does this Every. Time. It. Connects.

One of the files in this bundle (lib/Esx/System/Filesystem/Mount.pm) uses arguments with the Linux 'df' command that the Synology's busybox shell doesn't understand/support. To get Veeam to correctly recognize the space available in the Synology volume, you'll need to edit the 'mount.pm' file to remove the invalid "-x vmfs" argument (line 72 in my version) in the file. However, that file must be replaced within the tarball so it can be re-sent to the Synology every time it connects. Which also means every Linux repository will get the change as well (in general, this shouldn't be an issue, because the typical Linux host won't have a native VMFS volume to ignore).

Requests in the Veeam forum have been made to build in some more real intelligence for the Perl module so that it will properly recognize when the '-x' argument is valid and when it isn't.

So how does one complete this last step? First task: finding the tarball. On my backup server running Windows Server 2012R2 and Veeam B&R 7, it's in c:\program files\veeam\backup and replication\backup. If you used a non-default install directory or have a different version of B&R, you might have to look elsewhere.

Second, I used a combination of  7-Zip and Notepad++ to manage the file edit on my Windows systems. Use whatever tool suits, but do not use an editor that doesn't respect *nix-style text file conventions (like the end-of-line character).

Once you edit the file and re-save the tarball, a rescan of the Linux repository that uses your Synology should result in valid space available results.

One final note: why do it this way? The Veeam forums have several posts suggesting that using an iSCSI target on the Synology--especially in conjunction with Windows 2012R2's NTFS dedupe capability--is a superior solution to using it as a Linux Repository. And I ran it that way for a long time: guest initiator in the backup host, direct attached to an iSCSI target. But I also ran into space issues on the target, and there aren't good ways to shrink things back down once you've consumed that space--even when thin provisioning for the target is enabled. No, it's been my experience that, while it's not as space-efficient, there are other benefits to using the Synology as a Linux repo. Your mileage may vary.

Repair Synology DSM5.1 for use as a Linux backup repository.

After updating my Synology to DSM 5.1-5004, the following morning I was greeted by a rash of error messages from my Veeam B&R 7 backup jobs: "Error: Server does not support diffie-hellman-group1-sha1 for keyexchange"

I logged into the backup host and re-ran the repository resync process, to be greeted by the same error.
Synology DSM 5.1 error
The version of SSH on the Synology was OpenSSH 6.6p2:

As it turns out, this version of SSH doesn't enable the required key exchange protocol by default; luckily, that's an easy edit of the /etc/ssh/sshd_config file. And to play it safe, I added not only the needed Kex parameter, I also added the published defaults.
KexAlgorithms diffie-hellman-group1-sha1,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
After restarting SSH in the DSM control panel, then re-scanning the repository, all was not quite fixed:

Back to the manfile for sshd_conf...

The list of supported ciphers is impressive, but rather than add all of them into the list, I thought it would be useful to get a log entry from the daemon itself as it negotiated the connection with the client. Unfortunately, it wasn't clear where it was logging, so it took some trial-and-error with the config settings before I found a useful set of parameters:
SyslogFacility USER
LogLevel DEBUG
At that point, performing a rescan resulted in an entry in /var/log/messages:
Armed with that entry, I could add the Ciphers entry in sshd_conf, using the options from the Veeam ssh client to the defaults available in this version of sshd:
Ciphers aes128-cbc,blowfish-cbc,3des-cbc,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com
One more rescan, and all was well, making it possible to retry the failed jobs.

Follow Up

There have been responses of both successes and failures from people using this post to get their repository back on line. I'm not sure what's going on, but I'll throw in these additional tips for editing sshd_config:
  1. Each of these entries (KexAlgorithms and Ciphers) are single line entries. You must have the keyword—case sensitive— followed by a single space, followed by the entries without whitespace or breaks.
  2. There's a spot in the default sshd_config that "looks" like the right place to put these entries; that's where I put them. It's a heading labelled "# Ciphers and keying." Just drop them into the space before the Logging section. In the screenshot below, you can see how there's no wrap, no whitespace, etc. This works for me.
  3. Restart the SSH service. You can use the command line (I recommend using telnet during this operation, or you'll loose your SSH connection as the daemon cycles) or the GUI control panel. If using the latter, uncheck SSH, save, check SSH.

Friday, October 10, 2014

VeeamON 2014: A post-event challenge

Branded as the industry's first and only "Data Center Availability" conference, Veeam's freshman effort was a success by almost any measure.

Disclaimer: I work for a Veeam Partner and my conference attendance was comp'd in exchange for some marketing/promotional activities prior to the conference. I have also been a long-time user of Veeam Backup & Replication, since before my transition to the partner side of business due to my vExpert status in the VMware community.

Because I work for a partner, I arrived in Las Vegas on Sunday, October 5 to attend the partner-oriented social & networking events and to be ready for the 8:30am start on Monday morning for the partner keynote.

In a twist from other industry conferences I've attended, the keynote was MC'd by comedian Richard Laible, with a format intended to mimic those of late-night talk shows. It was successful, and the give-and-take between Richard and his "guest" was well-orchestrated and amusing.

In the first "interview," Veeam CEO Ratmir Timashev was able to tell the story of the founding of Veeam, underscore the company's love of their reseller-partners and reaffirmed the company's longstanding policy of staying 100% "channel-based" (no customer may purchase directly from Veeam); most important, he talked about the shift of Veeam from being "merely the best" backup product for virtualization, but to strive towards producing the best availability product for the enterprise.

Other Veeam employees took to the stage, and customer success stories were played out. In other words, much like any other keynote.

The remainder of the day was filled with breakout sessions covering a wide range of topics--both technical and business-oriented--for the partner crowd. The obligatory sponsor exposition opened for a happy hour/dinner reception, which also capped-off the scheduled activities for the day.

The second full day of events (Tuesday) was opened with a second keynote which echoed much of the messaging in the Partner keynote, but with an obvious new audience: the customer & prospects attending the event. In addition to even more entertainment (a pair from X-Pogo performed), some additional features of the forthcoming Version 8 for the "Availability Suite" (a rebranding of the former Backup & Management Suite) were shared, as well as even more customer testimonials which underscored Veeam's commitment not just to protecting data, but to making good on their aim to create the "always available datacenter."

The remainder of the day was again filled with breakout sessions, again ranging from business to technical topics. The day was scheduled late, however, with the optional party at the "LIGHT" nightclub in the Mandalay Bay hotel.

The third and final day opened with breakout sessions, these principally seemed to be presented by sponsor partners rather than Veeam employees with Veeam-specific topics. None of the sessions I attended, however, seemed too far off-base at a Veeam-oriented conference: the connection and/or synergy between the sponsor's product & Veeam's products was clear by the end of the session.

A final keynote by reddit.com's co-founder, Alexis Ohanian, was both humorous and insightful, and essentially closed out the conference.

There are many other posts out there with even more details and insight into the conference; check out my fellow #vDBer Mike Preston's series from the conference at http://blog.mwpreston.net for more insight and reporting.



My retelling of this is all to aim towards one thought: Veeam did a great job on their first conference. The content was relevant, the sponsors were invested and made sense, and it was both informative and entertaining.

Here's the challenge: What about 2015?

Unless the breakout catalog is significantly expanded, I'm not sure how many folks will want/need to attend a second year. Don't get me wrong: I'm not saying that no one will attend. On the contrary: if they repeated next year with a cookie-cutter duplicate of this year, anyone who a) didn't attend and b) wants to learn more about Veeam's products and how they can boost the availability of the datacenter would find their time well-spent.

I'm saying that everyone that went was a first-timer, and they got that spot-on. They can still fine-tune it, but next year's first-time attendee will get great value whether they change it or not.

No, the problem is getting repeat attendees. The conference can increase their first-time attendee counts simply based on positive word-of-mouth recommendations, but the top end for that will be reached far sooner than getting both those new attendees and the repeat (alumni?) attendees.

As it was, the number that was rumored prior to the conference—around 1200 people comprised of attendees & Veeam staff—seemed to have some validity. The conference space at the Cosmopolitan was sized well for the attendees, and it was never crowded or crazy like VMworld can feel (with almost 20x the attendance). But I can't imagine that Veeam is going to be content with putting on a two-and-a-half-day conference for "only" 1000 people. Yes, you want a multi-day conference to help justify the travel costs, but let's be honest: the VMUG organization has chapters that manage to put together single-day conferences for that number of attendees.

This isn't meant as criticism: I'm identifying the challenge they now face, and send the call-to-action to Veeam to plan next year's event—as far as I know, TBA for place & time, yet expected from Doug Hazelman's parting "See you next year at VeeamON 2015"—with the goals of both increasing the number of new attendees compared to the inaugural "class" from this year, as well as compelling most (if not all) of this year's attendees to return.