Etherchannel, LAG, portchannel and friends

By | June 30, 2015
EtherChannel physical diagram

An EtherChannel is a way to use multiple physical interfaces as a single logical one. That logical one appears as a single interface to MAC table, STP and management plane. That solves two main concerns: it adds bandwidth and it prevents STP from considering several parallel links a loop, thus preventing it from being blocked.

What on Earth are EtherChannels?

EtherrChannel basics

Side note: I know some people who call EtherChannels “Trunks” (which is not to be confused with what Cisco calls “Trunks”). That one, as well as the term “NIC Teaming”, comes from the world of servers. Functionally, it usually means the same, but technicalities might differ.

More side note: there are actually so many names for the same functionality: Ethernet trunk, NIC Teaming, Port Channel, Port Teaming, LAG (link aggregation), Link Bundling, Multi-Link Trunking (MLT), DMLT, SMLT, DSMLT, R-SMLT, NIC bonding, Network Fault Tolerance (NFT), Fast EtherChannel.

EtherChannel might be considered a form of virtualization, as one virtual entity (interface portchannel) represents several physical ones.

EtherChannel physical diagram

Isometric shapes are deep

Link aggregation does not change Ethernet frame format (no new headers, TLVs or sequence numbers), does not perform fragmentation or reassembly.

A port channel interface can be formed statically or dynamically, it can be an L2 access or trunk port, or it can be an L3 port of the switch.

EtherChannel logical diagram

Although it can be drawn as such on a logical diagram, a LAG is not a fat pipe.

The data transmitted across the portchannel interface is shared among the member links based on frame properties (per-frame sharing, which looks a lot like per-flow, but there is no state kept and decision is made for each frame independently, though deterministic so to prevent any reordering in a given flow). Also, despite some popular belief, this means that bandwidth is not increased for a single flow – it will always be transmitted over a single link and thus conform to its limitations. Thus, efficiency of EtherChannel can only be observed when the network has several pairs of active talkers.

More info: https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/cether/configuration/15-s/ce-15-s-book/ce-ieee-link-bndl-load-bal.html

 

“LAG is good, but it’s not as good as a fatter pipe”

 

Static Layer 2 EtherChannels

Static L2 EtherChannles are formed by administrator directly in configuration by configuring the member interfaces with

 And configuration of the respective portchannel interface:

Any further configuration is performed under the portchannel interface.

Note: on many (if nat all) platforms, configuration of member interfaces MUST be the same before joining the portchannel. At least switchport mode, vlans, speed and MTU should be the same. If the settings are not compatible, the physical interface gets suspended.

 

Dynamic L2 Etherchannels

Dynamic etherchannel is formed with special negotiation protocols. The administrator must enable one of them on the interface beforehand. Of course, both sides of connection must use the same protocol.

For trunk links, both protocols send the PDUs over the VLAN with the lowest ID (of those allowed).

Administrator can force usage of one protocol over another via

Both protocols can coexist on the same switch, but, naturally, a single portchannel cannot be formed with two protocols at the same time.

PAgP

Cisco proprietary protocol. Has two configurable modes: desirable and auto. At least one side for the link MUST be in desirable mode for the channel to form.

 Note: the non-silent option is used to facilitate connection to servers.

There aren’t a lot of configurable parameters for that:

  • pagp learn-method {aggregation-port|physical-port} – influences load distribution by forcing learning either on aggregate (default) or individual ports – in that case return traffic is sent over the same link it first arrived.
  • pagp port-priority {0-255} – default is 128, allows to influence which port is used fro PAgP communication

Command reference for Cat3750x (IOS 15) tells that neither of the two commands has any effect and they were left for interoperability with older platforms. See usage guidelines note on https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750x_3560x/software/release/15-0_1_se/command/reference/3750xcr/cli1.html#wp1863746

LACP

Open standard, IEEE 802.3ad. Two configurable options: active and passive. At least one side of the connection MUST be active for the channel to form.

LACP can handle up to sixteen physical ports in a single channel, but only up to eight will be active at any given time. The others will fall into a Standby mode (State == hot-sby in show commands), and become active in case of main interfaces’ failure. The interfaces that get to be active first are those with lower interface IDs. 

LACP has several parameters, which in most cases should be left in their default state: system priority (\in [1;65535], lower-better, 32768 is the default), port priority(\in [1;65535], lower-better, 32768 is the default), and administrative key (16 bit). LACP system-ID is a combination of System priority and the bridge MAC-address, it determines which switch in an LACP link controls port priorities. Port priority together with the port number form interface-ID (thus, active versus stand-by link selection can be influenced). Relevant commands include:

Other LACP features include configuration mismatch detection, and failure detection (crossed LAG, looped LAG, unidirectional forwarding).

More info:

https://www.cisco.com/c/en/us/td/docs/ios/12_2sb/feature/guide/gigeth.html

https://www.ieee802.org/3/hssg/public/apr07/frazier_01_0407.pdf

You can also influence LACP timers in a limited manner on Nexus switches: https://www.cisco.com/web/techdoc/dc/reference/cli/nxos/commands/l2/lacp_rate_fast.html

Interesting side note: it might be wise to configure any inter-switch link as a LACP LAG – even if it is just one physical interface (mind platform limitations though). You’ll reap some of the error-detection benefits, and new links can be added easily (without conversion to channel, i.e. downtime)

Layer 3 EtherChannel

The L3 channel setup is not much different from that of L2. Either Static, LACP or PAgP may be used for configuration. The only major difference: administrator must issue the

command on the physical interface prior to adding it to portchannel group.

Otherwise, Etherchannel configuration is the same.

Note: As far as I understand, some routers do not support dynamic channel negotiation and might have other limitations.

The portchannel interface inherits its MAC-address from the first member port of the group.

In case of a physical member port failure, that is hidden from upper-layer protocols (e.g. routing protocols receive no notification, so no reconvergence occurs at network layer)

EtherChannel Load Balancing

“Load balancing” is not really a good term. The switch does not monitor interface load and does not try to “balance” load between physical interfaces of the channel. Rather it distributes load using a deterministic algorithm, which is based on one or several parameters of each frame:

  • Src-mac, dst-mac or both
  • Src-ip, dst-ip or both
  • Src-port, dst-port or both (i.e. TCP or UDP ports)

Administrator can choose (within platform capability limits of course) which parameters the switch should use and configure it system-wide:

To see exact parameter values it is better to check context help for reference on system capability. To restore the defaults – negate the command:

The switch then takes the [parameter] value of an outgoing frame, hashes it into a number corresponding to the interface and places the frame to the physical interface’s egress buffer. The notable thing is that the hashing algorithm is only efficient for the number of interfaces of the power of two: two, four, eight. Any other quantity of physical interfaces will result in “unfair” distribution, when the first {two, four, eight} interfaces are loaded more (and evenly), and the other interfaces get “traffic leftovers”.

Also, the algorithm is prone to polarization, so it is wise to consider configuring it differently on different switches. For that, topology and traffic flow patterns must be considered.

More info:

https://www.cisco.com/c/en/us/support/docs/lan-switching/etherchannel/12023-4.html

https://www.cisco.com/c/en/us/support/docs/lan-switching/etherchannel/116385-technote-etherchannel-00.html

 

EtherChannel Protocol Limiting

What the heck is that? I mean, I’ve spent an hour digging google search (also, some Cisco Press books), and only came up with INE’s expanded blueprint (where I’ve got the bullet point in the first place), some blog telling strange unrelated things, and a youtube video talking completely different strange things. I mean, no Cisco document came up. Shame on me.

It can refer to:

  • lacp max-bundle max-bundles – allows to limit the number of active interfaces (1-8, 8 is default) in a LAG
  • pagp learn-method (see above)
  • any lacp/pagp configuration commands available besides just turning them on

 

EtherChannel Misconfig Guard

This is actually a spanning-tree feature (or hack, as some people call it). It prevents configuration mistakes from having effect on spanning tree by err-disabling misconfigured interfaces. It is supported with RPVST+, and MSTP.

As a LAG is supposed to be a point-to-point interface, this feature checks LAG consistency, because STP BPDUs received via the port MUST contain only one bridge-ID. So, if several different Bridge-IDs are detected, the port is blocked.

The only configuration command is

> spanning-tree etherchannel guard misconfig

Obviously, multi-chassis-LAG (MLAG) is not a point-to point (physically), so to work around this feature, the secondary chassis will assume bridge-id of the primary. This is the case with Stack-wise cross-stack LAG, Cisco Virtual Switching System, as well as Cisco Virtual Port Channel (vPC) configurations (different names on different platforms, very different from each other on a tech side, but top-level functionality is basically same).

Both sent and received BPDUs are examined by the detection mechanism. An EtherChannel is considered inconsistent if the channel detects greater than 75 BPDUs from different MAC addresses in more than 30 seconds. However, if 5 BPDUs are seen consecutively from the same MAC address, the detection counters are reset. These timers/counters can change in future software releases.

Source: https://www.cisco.com/c/en/us/support/docs/lan-switching/etherchannel/20625-127.html

 

 

 

Denis Borchev
Follow me

Denis Borchev

Engineer at Netcube LLC
I am a networking engineer, a geek and a generally nice person=)
Computer Networking Engineer with some experience; MSc Applied CS, CCIE #53271
Denis Borchev
Follow me

Latest posts by Denis Borchev (see all)