Menu
Field CTO, Hybrid Data Center
Very recently, I was deploying a new Cohesity C5016 appliance with 25Gb NICs, connecting up to a pair of Nexus 93180YC-FX3H switches. When using the 9K’s in a VPC pair, my personal preference is to configure the Cohesity nodes with LACP to get the most bandwidth possible (regardless if it’s 10Gb or 25Gb connectivity). Nothing super creative there, and I’ve done this dozens of times in the past with no issue, on both the Cohesity appliances and Nexus 9k’s. But this time, it was different…
This deployment included a pair of Nexus 93180YC-FX3H switches (I love this model, and if you didn’t know what the H stands for, it stands for Half—go figure). The FX3 switch is a 48 1/10/25Gbe and 8 100Gbe port switch with a fantastic set of features– But in some cases, we just don’t need the full 48 ports on each switch, so to help with the overall cost, we use the FX3H model, which is the same switch but only 24 of the ports are licensed. In my specific scenario, we were performing a refresh for a past customer from a 3-tier environment to a pair of new 9k’s, Nutanix nodes for hyperconverged storage and compute, and Cohesity for the backup solution. A perfect trifecta if I say so myself…
When bringing up the C5016 nodes with 25Gb NICs connected redundantly to the Nexus 93180YC-FX3H switches running 10.3.4a code, even though I had visible light on the fiber, I wasn’t getting any connectivity. So I moved forward with normal troubleshooting. Checked the SFP modules (and swapped them out), checked the fiber (and swapped that out as well). Rebooted the nodes, rebooted the switches, nothing. But go figure, the Nutanix nodes using the same 25Gb modules and the same switches (even the same ports) worked just fine.
Forward Error Correction (FEC) is a method used in data communication and storage systems to detect and correct a limited number of errors in data without the need for retransmission. It works by adding redundancy to the original data so that errors can be detected and corrected on the receiver’s end. This is particularly useful in high-speed networking environments where retransmissions can be costly, in terms of time and bandwidth.
Now, I’ve had this issue in the past when connecting a pair of Cisco Catalyst 9200/9300’s switches using 25Gb uplinks to the Nexus 93180, and had to tune the FEC parameters, but it just didn’t click that this might be the same issue.
Doing some searching on the Cohesity support site, I stumbled upon a KB article addressing a similar issue with Arista switches. So why not, let’s check it out.
A quick synopsis of the KB article:
The links on the nodes do not come up after upgrading the switch.
The issue might stem from a mismatch in the Forward Error Correction (FEC) mode between the NICs/SFPs and the switch configuration. The specific problem noted in the Cohesity article was related to Arista switches allowing FEC Mode RS to be set, which needed to match the NIC/SFP configuration.
The FEC mode supported is determined by the SFP and switch configuration. Cohesity devices default to negotiating FEC automatically using RS encoding. You might need to force RS encoding on the Nexus switch ports connected to the C5016 series NICs.
Here’s an example of checking and setting the FEC mode:
Ensure that the interface matches the one connected to your C5016 NICs. You might need to adjust these commands based on the specific syntax for your Nexus switch and IOS version.
After reading the Cohesity article, and remembering the experience with the C9200 connectivity with the Nexus, I dug into the port FEC options, as I didn’t just want to disable FEC, but find the right combination. After trying each of the below settings, I found that rs-ieee worked and I was able to establish connectivity between the Cohesity nodes and Nexus switches.
Quite often, I use blogging to help remind me of situations I have previously encountered—call it my personal knowledgebase. As higher bandwidth connections continue to be the norm, running into this on 25Gb and higher connections will remind me to check the FEC!
Thanks for reading, I hope you found this valuable if you ran into a similar situation!
If you have any questions or you’re looking for assistance with Cohesity, please reach out to info@eGroup-us.com or complete the form below.
Contact our team today to schedule a call with one of our experts.
