02-23-2016 11:03 AM
I work for a company that has a pretty close relationship with another independent telco (same General Manager, even), and I've seen this problem appear on BOTH of our networks, so I figure it's time to reach out to the community.
Basically, what happens is that when there's a topology change in the network, we experience crippling storms that appear to be from a loop or reflection.
In my case, this appeared after migrating our TV feed from a single link to dual links into each VPT (6450s) managed by Spanning Tree from a Juniper MX router. Video settings all set up properly, everything configured so that VPT1 normally sends TV out onto PG1. Whenever a card was recovering from a reboot - whether a software crash, reseat, or a circuit breaker reset - the EPS network went into a storm. The video router's DoS protection mechanisms activated reporting > 100Kpps of spanning tree traffic received on the ports from the VPTs. That's one hundred thousand packets per second on a port that should see ONE packet per second... We tried many fixes, and audited the network completely for preferred PG, double-fault PG, video settings, IGMP versions, and setting the right VLANs on ring profiles (one time a 6151 went berserk, found that one fast!). Nothing stuck. The only solution for us to be able to keep redundancy was to migrate from the Juniper MX router onto a Juniper EX switch and use a Redundant Trunk Groups, which is the same as a Cisco FlexLink.
In my colleague's case, this is appearing as he attempts to upgrade portions of his EPS ring to 10 Gbps. His CO stuff is already 10 Gbps (if I remember correctly). When he takes down PG2 to a remote, leaving everything running on 1 Gbps PG1, he completely loses that remote and he sees 7.5 Gbps traffic on some ports in his CO. My colleague's network has STP entering from some Juniper EXes as well as a Cisco 6500 that feeds TV in. I think he's been working on getting STP out of the picture entirely, although there is some complexity because the E7 that hangs off of one section is allegedly STP-aware - unlike B6. I believe Josh Levi is somewhat familiar with this case.
We are running various OS in the 7.3 train on our B6 stuff. We both have some E7 hanging off of B6.
Has anybody else had REALLY strange issues with spanning tree? Or possibly E7 hanging of of their B6?
02-24-2016 05:05 PM - edited 02-24-2016 05:07 PM
Found this in some release notes:
Fixed in 7.3.30
• Resolved Issue: MAC addresses can become "stuck" after network events such as link
failures, EPS failovers, or other disturbances. [BSIX-16114, BSIX-15683] Calix has
observed a rare occurrence where the B6 may retain old MAC addresses, or may time out MAC
addresses while statically configured. Calix is investigating this issue.
Can anybody at Calix comment on this? On further investigation we have a couple of cards running releases prior to 7.3.30, seems a few are even on 7.2 and 7.1 trains :O.