Rethinking Disaster Recovery After Surviving Hurricane Helene: An IT Leader’s Story

Lessons learned in the aftermath of a killer storm.

Before Hurricane Helene devastated the southeastern U.S. in September leaving cataclysmic destruction and numerous fatalities in its wake, midmarket IT leader Jon DeMersseman said that when it came to disaster recovery, he had never really thought about what happens if your entire business is affected by a disaster.

DeMersseman said the catastrophic storm led him to rethink his organization’s disaster recovery strategy and says organizations need serious discussions about DR and business continuity.

As the IT operations manager for ReWa, a wastewater utility in Greenville, South Carolina, DeMersseman gave harrowing details about surviving the storm and how it crippled business operations, as well as the critical lessons he learned about disaster preparedness.

Weathering The Historic Storm

“When Hurricane Helene came through ... we had 200-year level flooding,” DeMersseman said. Prior to the hurricane’s landfall, most of South Carolina and parts of Teneessee had already experienced heavy rainfall.

“Just a constant, steady rain for about a week that saturated the soil,” he recalled.

Then came Helene, and the winds.

“Now you’ve got 80 mph wind gusts, and these trees got blown over ... the roots just tore out of the ground,” he said.

The utility infrastructure, most of which is above ground there, was walloped. The local power company lost 6,000 poles, DeMersseman said.

“We have nine different water treatment plans. They were all on generator power at the same time,” he said.

However, the flooding rose to such a level that it entered ReWa’s fleet parking lot.

“All of our vehicles [were] sitting in electrified water because of the solar field, and we lost 20 vehicles because we couldn’t get them out of water because of danger to staff,” he said.

Then, “most of our sites lost their internet connection. One of our sites didn’t get their internet connection back for about over a month,” DeMersseman said.

He said his company ended up putting cellular internet connections at several sites so they could at least get management back on site, something they had to scramble to do at the last minute.

“We [had] purchased new, redundant firewalls that were going to have cellular backup, but unfortunately those were not implemented at the time of the disaster,” he said.

Thankfully, there were no fatalities or injuries among staff. Business operations were severely impacted, however.

“We had a lot of staff who were without power for an extended period of time at home. I didn’t have power for about six days, I didn’t have internet for about three weeks,” DeMersseman said.

Rethinking Disaster Recovery

Hurricane Helene’s impact led to DeMersseman reassessing his company’s disaster recovery and remediation strategy.

"The thing about this particular event is it was more regional in nature,” he said. “We could have been more distributed and still been impacted.”

Questions that arose in DeMesserman’s mind in the aftermath were how do you prepare for regional disaster? How do you prepare for multi-site disaster? What if you don’t have enough staff coverage?

DeMersseman said that the event made him realize the importance of testing.

“There were a number of issues that we had that if we had been doing testing more frequently, I think would have been helpful. “

He said while his company’s sites were testing their generators on a monthly bases, they were only testing to see if the generators would fire up.

“They weren’t actually putting any load on the generators to verify [if they] could handle running the entire wastewater site. What happened at multiple sites was when the generators fired up under load, various components within the site failed so the site couldn’t run immediately,” he explained.

Testing their overall communication system and having a plan on how they were going to communicate with staff and one another during a disaster would also have helped with remediation, DeMersseman said.

The disaster also made him realize the need for implementing an emergency operation center plan.

Such a plan would provide a way to just knowing where people were going to meet up, he said.

“Because of the breadth of the impact, we had staff that was essentially trapped in their homes or in their neighborhoods,” he said.

Since Helene, DeMersseman said his company hired a consultant who’s been helping them with disaster “lessons learned” and “figure out what we want to do as an organization going forward to adjust our disaster preparedness plans.”

Above photo: IT Manager Jon DeMersseman took this photo of the historic flooding at his company’s location.