The Network Monkey

Series (#003) - The Myth of Auto-Negotiation
Written by: Jim Huddle CNE CNE5 CBS MCSE ES-RC, send e-mail

Ever watch Dennis Miller's show on late night HBO? Mr. Miller usually has a topic he begins joking about. After a few jokes he generally says something like " I don't mean to go off on a rant..." and then does just that for several minutes. The rant is often humorous, as is his intention, but is also rather biting about the serious point of the topic. At the end he says, "but that's just my opinion, I could be wrong."

I don't have any jokes, but I do have a rant, and it is just my opinion. Like as not I'm wrong, but I can only go on experience. My current particular rant is about a part of Ethernet switching called Auto-negotiation.

Ethernet switching was introduced to help alleviate the problem of collisions. Traditional shared Ethernet networks operate by allowing one device to pass one data packet on the wire at a time. The devices, whether they are servers, PCs, print servers, etc., all have to negotiate for time to send their packets of data across the wire. When the Network Interface Card (NIC) in a device senses that no packets are being transmitted, it will begin sending its own packet. If another device also begins sending at the same time there will be a collision on the wire and both devices will back off for a random amount of time before trying again. On a small network, this usually passes unnoticed. On a large or very busy network, the resulting time loss can be a significant problem.

Since the communications between devices attached to a switching hub are privately negotiated there are no collisions. This was a nice improvement. A couple of other advancements came along: duplexing and 100 megabit (Mb) Ethernet. Devices with compatible NICs could send and receive simultaneously. In theory doubled the speed of network communications. 100Mb Ethernet boosted the speed of traditional Ethernet tenfold.

All this is great, except for one problem. Networks ended up being built with a mixture of speed and bandwidth capabilities. Vendors of networking equipment had to come up with a switch that would accommodate the differing speeds and duplexing capabilities. What they came up with was Auto-negotiation. The idea was slick. Devices supporting auto-negotiation would negotiate the highest speed and determine the duplexing mode. So you could have 10Mb and 100Mb devices attached to the same switch and not have to worry about setting the individual ports to match the capabilities of the device being attached. Just plug and play. Ah, I'll bet that last phrase caught your attention.

Now the rant.

I've used switching equipment from many manufacturers, and more kinds of NICs than I'd rather think about. The one thing in common they all seem to have is that auto-negotiation, uh, doesn't.

You've probably been there. You get in a shiny new switch or switch-router. You may be using it to speed up the engineering department's CAD downloads or maybe it's an upgrade to your core. You are optimistic. After all, you didn't buy Joe's NoName Switch-Router, you bought a major manufacturer's best offering. So you attach a ton of hardware, upload the configuration file that you've meticulously created, bring the unit up and smile confidently as the box goes through it's self check and boot process. All the pretty lights turn green, the switch-router is up, and just as you're congratulating yourself on how bright you are one or more, probably more, of the ports start blinking red or just shuts down altogether.

Frown slightly. Check the box through the laptop you've judiciously attached to the switch-router's console port. Everything looks OK. Config is right, ports don't show any errors, and the box appears fat, dumb and happy. You do the stupid stuff like checking the cables and fiber links. You check the servers and routers just to be sure they're still running. Everything appears to be operating, but the ports are still dead or constantly initializing. So you recheck everything again, even applying the NOS and firmware patches you've been meaning to get around to. Finally you reboot the new box. It comes up again but this time all the ports stay up. You shrug, concluding it was just a warning from the Cosmic Muffin that you've been lax in sacrificing users.

So you pack up the laptop and begin thinking about the next project. A short time later you start getting calls from users complaining that they can't access the servers or the Internet. Going back to the box, you find several ports have decided to become blinking Christmas displays. You do the trouble shooting again and still all looks fine. Re-attaching the laptop, you scan the down ports. They still show fine, just down. Next your boss comes in wanting to know what you are doing about this problem. He or she has just got a call from the Marketing department and they are in a dither because they can't access that new Internet site that promises to teach them how to lie more sincerely. You're getting desperate. Others are panicking.

You select the port that connects to the Internet router, open its configuration screen and look at it's setup. You recall that about six months ago you had a problem with one user's PC that wouldn't connect until you turned off auto-negotiation on their NIC. You figure it's a long shot but worth a try. So you change the ports configuration and turn off auto-negotiation and manually set the connection speed and duplexing. Suddenly the holiday season is over and the port starts working showing only the reassuring green lights. You go to each dead port and manually set them and they all come up.

You watch the ports for a time, only half listening to the nattering of your bosses about how "This kind of thing just can't be allowed to happen" and "We need to have a meeting (or ten) to decide how to prevent the lack of meetings." Meanwhile you're studying the packet statistics and idling thinking that the overdue sacrifice is a toss up between the Marketing department and your own management. Either would offer significant enhancements to your sense of well being, but you worry the Muffin might get so serious a case of indigestion that it would curse you by placing a major whiner next to your cube.

While you have an inner debate on whether or not it would be worth it, one of the ports attached to a server goes down again. Your bosses begin wondering if maybe you shouldn't put the old equipment back online because there was never any trouble with it. Never mind that it was so old that the vendor no longer supported it and the support folks actually sniggered at you whenever you called about getting replacement parts. You go to the server and pull up it's network configuration. Sure enough, the NIC is set to auto-negotiate the link. You turn it off and set the connection's speed and duplexing manually. The link comes back up. Cursing your lot, you decide to check every device coming into the new switch-router and manually set the NIC as a precaution. You miss a switch that feeds one area, but you don't notice it as it's continuing to pass packets.

For a couple of months after that the network goes about its business of moving packets. The folks in Marketing become more proficient prevaricators and for now they are happy. The bosses mercifully go on extended retreats after cautioning you several times not to fix any problems while they are gone, and you've moved on the other projects that need to be finished.

One day you get a call from a user that uses the switch you neglected to set. She can't login. You check the core switch and sure enough that port has gone south. You check the user's switch and discover that it is still set to auto-negotiate the link to the core. After turning that off and setting the link manually, the link is restored. You wonder why it suddenly stopped working.

I suppose the bottom line is this: You can't trust auto-negotiation to work. Sometimes it works flawlessly for months and years then suddenly fails. It's become the first thing I check when I have any connectivity problems. Any new switches added to the core will now have auto-negotiation on the core link turned off by default. Auto-negotiation is a great idea and maybe one day it will work consistently, but not today. My advice is to turn it off and forget it's even a feature to consider using. But, like Mr. Miller says, that's just my opinion, I could be wrong.

Letters to the Editor are welcome and occasionally abused in public. Send e-mail to: whine@kickstartnews.com