Network

Our network took a huge dump this morning. We got most of the company back online and had to fight to isolate the one piece of cable that we needed to replace. Once one bit of network gizmos go down, they typically bring down healthy network gizmos. We spent another hour tracking down and resetting things to make sure we didn't get two failures simultaneously. We call these days "work days".

We were running around plugging and unplugging equipment. We were watching for lights on network jacks to blink specific colors and at specific rates. We noted dead connections. We rebooted printers. We cursed. We yelled and yelled at equipment. We swapped cables and breathed in dust. There are network bits still in daily use that are 15 years old. Those are the ones we trust. There are bits we replaced last week. These are suspect.

When we pull that cable out of the wall I will inspect it for bite marks because I'm starting to suspect gremlins.

People call from all over the company. "Hey, did you know the network is down?"

"Yup, we are on it." We answer.

"I can't get to my file." Someone else states.

"Yup, we are on it." We answer.

"Facebo... er, the CRM web page is down." complains the fourteenth person.

"Yup, we are on it." We answer.

There is a hidden advantage to VOIP. When the network dies, so do the phones.

I wish I could have videoed the whole thing. It is interesting to watch people work to diagnose a network issue. I found out today that we have a set of walky-talkys at the company specifically for these situations.