Crumbacher Business Systems Reconnects Santa Fe Highschool to Internet Day After Night-time Lightning Strikes D-Mark Room.
Santa Fe, NM is no stranger to lightning. Lightning strikes are common and during the rainy season, you can see lightning storms coming hours before they even hit. This particular evening, ISP nodes throughout Santa Fe were hit by lighting, shocking businesses and residences alike over coaxial.
Today’s Schools rely completely on their internet connections. From Teachers inputting grades into their cloud software, to Students working diligently on their Chromebooks, completing assignments written in Google Docs and submitting them via Google classroom or like software. In addition to Cloud based document storage and word processing, programming and digital media classes have cloud based Adobe subscriptions and require constant access to distant servers. The banks that they upload their data to and their cloud based accounting and fundraising software, and VoiP phones will fail the second the lines are cut.
The safety of each failure point in any networking infrastructure is paramount. With failover ISP connections and gateway devices, Double Conversion UPSs protecting critical devices like On-site Network Attached Storage, firewalls, modems, and switches, it is important to remember that disaster can still strike in unexpected ways.
So, when the previous evening’s impressive lightning storm rolled through town and fried a Comcast node and the lightning’s path of least resistance was the coaxial connection to The highschool’s Comcast modem (and everything downstream via Low Voltage Ethernet), all their expensive protection was rendered meaningless. The UPS surge protection was not even tripped.
The entire business was down and at a complete standstill. Payroll couldn’t be processed, Students couldn’t submit work that they could not access, teachers could not input grades, Staff could not call out or receive calls.
As we arrived in the DMARC room, we could smell the smoldering of wood and drop ceiling. A local electrician arrived and determined that the electrical wiring was intact. The facilities department determined that the fire was out, and Crumbacher determined that the charred Ethernet end sticking out of the Comcast modem was what fried the DMARC switch, Synology NAS, Century Link failover, Sonicwall TZ500, 2 computers, one HP laserjet, and one access point. Fortunately, the rest of the school was not in the Modular unit and was downlinked via fiber from the now unresponsive switch.
This lightning strike could have kept this school down for over a week, sending students home and stopping all operations.
A sense of urgency, A thorough understanding of networking basics, and a determination to keep small to medium sized businesses running, leads to minimized downtime.
“It was a Friday morning and before I even arrived at the office I got the texts from each member of the administration. I immediately checked my cloud switch and access point console which appeared to be completely offline. Before setting up my calendar for the day, I let my manager know that I had an emergency and needed to be on site immediately to see what was going on. I checked the modular building that houses both the High School Foundation, and the school’s Demark room.
I noticed the charred smell immediately and carefully checked the IT closet. All of our equipment, (except one Tripp Lite Double Conversion UPS) was offline and toasty. When I unplugged the low voltage from the Comcast modem, I was greeted with the sight of a blacked fried RJ45 end. Several phone calls to an electrician, facilities, Comcast and the school’s president later, I had Comcast on the way, several quotes for new equipment, requests for loaner equipment in the works and a fiber media converter ready to connect the school’s fiber pipeline to a new Comcast modem. When Comcast arrived, we discussed setting the static IPs and hooked the school up to an internet connection.
With the client up and running, I spent the next installing an overnighted switch, backup software, NAS, a new Foundation PC, printer, and a firewall.”
In the aftermath of this incident, Crumbacher Business Systems makes a point to manage our customer’s ISP relations and we now make sure our clients have inline surge suppressors on Comcast lines and grounding. You can keep full back ups of client information in the cloud and protect expensive hardware with insurance and backed up configuration files, but you can’t always protect against downtime from natural or man-made disasters. For some businesses, downtime is THE disaster.
You can have all of the protections and insurance in place, complete with a deep knowledgebase of technical solutions, but if the client has to suffer weeks of downtime because IT lacks a sense of urgency or does not communicate well, IT cannot be considered effective.