When IT Hits the Fan: Managing Tech Crises Like a Pro

Ask your local cybersecurity junkie about the likelihood of an attack against your organization and 9 times out of 10 they’ll parrot: “It’s not if, it’s when.” Ask them what you should do in response to a critical disruption, however, and their answer will likely be a much less certain, “Well, that depends.”

What to do when IT hits the proverbial fan was the focus of a recent panel discussion at the Los Angeles Digital Government Summit on Aug. 13, where public- and private-sector representatives shared their unique experiences and the pitfalls to avoid.

Part of the challenge that comes with any technical disruption is figuring out exactly what’s gone wrong, explained AT&T Principal Architect Devon Winthrop, adding that it typically comes down to 1 of 4 main areas: cyber attacks, natural and man-made disasters, and emerging technologies.

“COVID caused a huge disruption within your guys’ network to where it changed the way you guys work, changed your day-to-day. That carries through to today,” he offered as an example.

Lillian Russell, chief privacy officer for the Los Angeles County Chief Executive Office, added that human staff can also add a level of unexpected disruption, pointing out that the adoption of new tools like e-signature and Microsoft Teams initially caused ripples for some workers.

“We’ve seen in the public sector where services get interrupted, meaning social services get interrupted because the business processes have to change with the technology. And then we had emergencies where staff didn’t know how to perform the job any other way …,” she said.

This sort of disruption, though less likely to grab headlines, must still be addressed with education, instruction and operationalization, Russell noted.

In addition to aligning staff with new technology, Dean Gialamas, former general manager for the Los Angeles County Internal Services Department, said there is a challenge in aligning the spectrum of other stakeholders who need to be considered when responding to a crisis.

Russell agreed, noting that this laundry list of stakeholders stretches from elected officials and the media to internal technical teams — all of which have different informational needs and clearances.

“I think sometimes for folks that have a technology-focused role, there’s a lot of attention with the technology emergency and not so much the focus on these other players because you’re dealing with a problem in front of you, with the machine in front of you, with the system in front of you,” she said. “You’re not thinking about, ‘OK, downriver, who else is going to be impacted?’ or ‘What do I need to say to them?’”

For AT&T’s part, Winthrop noted advance coordination and planning is critical to ensuring not only the continuity of broader service, but also that the right systems are protected in order of priority. Operations like emergency services and public safety answering points (PSAPs) are prioritized for obvious reasons.

In addition, this coordination helps to ensure the company is not pulling in technicians who might be personally impacted by natural disasters like wildfires away from their homes and families, Winthrop said.

All three participants agreed that coordination with vendor partners before an emergency is an ideal approach for any organization, though looping them in early into response efforts is the next best thing.

“The reality is, anybody that has a network should be planning on their network going down at some time or another. And who are those parties? Who are the quick numbers I pick up and call right away? You know, where’s my red phone?” Winthrop said.

Russell urged attendees to lean on their organization’s cyber insurance provider, who will often have a list of pre-approved vendors and resources, and the legal department.

“In the area of technology, making sure that not only the continuity of operations has a plan, but we’re actually testing it and validating it, and that we’re actually testing different components of it from year to year,” Gialamas said, adding that ensuring systems are redundant, protected and accessible is a key component to any successful response effort.