vSphere 5.1 Beacon Probes

As in previous versions of vSphere, an administrator for 5.1 can choose to use Link status only or Beacon probing to help determine the uplink status of multi-NIC physical port aggregation.

Failover Detection Options

The default is "Link status only," which merely watches the Layer 2 status for the network adapter. While very simple, it does not have the ability to determine whether "upstream" network problems are present on that link.

That's where Beacon probing comes in handy: By sending a specially crafted frame (it's not a packet; it's an Ethernet frame without any IP-related elements) from one pNIC to another, ESX(i) is able to determine whether a complete path exists between the two. When three or more pNICs are together in an uplink set (in either standard or distributed switches), it's possible to determine with high reliability when a link is "bad," even if the local connectivity is good.

VMware has several blog posts on the topic (What is beacon probing?, Beaconing Demystified: Using Beaconing to Detect Link Failures), and the interwebs are full of information on what it is and how it works for even the most casual searcher.

While working on a completely different system, I was doing some port monitoring and discovered that my ESXi 5.1 hosts were using beaconing. I don't have it turned on in my lab because I have just the one switch: If a link is down, you can immediately detect it without any need to "look downstream." It was kind of annoying to see those showing up in my packet capture, and while it would've been easy enough to filter them, I was more interested in trying to figure out why they were there in the first place: I was pretty sure I hadn't turned Beaconing on for any of my port groups.

Beacon probing frames captured in Wireshark

I went through the settings of all my port groups and verified: all were set to Link status only. What? So I turned to Twitter Tech Support with an inquiry and got a quick reply:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1024435

Unfortunately, setting the Net.MaxBeaconsAtOnce to 0 as suggested in the KB article didn't help: still seeing the beacons. But that suggestion helped me fine-tune some of my search criteria, and a memory was triggered: there's some new network health check capabilities in vSphere 5.1...

Virtual switch health check options in vSphere 5.1

By default, both of these health checks are disabled, but I remember seeing them and enabling them when I first set up 5.1. I wasn't sure which item was the source of the beacon frames, but it's simple and fast to check both options when the frames show up in the packet capture every 30 seconds!

Enabling VLAN and MTU will enable beaconing

Turns out that it's the VLAN and MTU that was putting those beacons out there. I was only watching the traffic for a specific VLAN (which is tagged for my pNICs), so the Teaming and failover option may also put beacons on the untagged network. But the mystery of beacon frames was solved!