To “future-proof the city’s IT” and prevent loss of data, productivity and control, the department wanted to take on disaster planning. They found that it would take extra time to do it properly.
While making a case for disaster planning to city management about two years ago, Matthews said he’d realized his department would benefit from bringing in a consultant.
“Here’s where the professional approach comes in. Because I’d looked at (disaster recovery) plans online — download a plan, fill in your city’s information, customize it — we felt like there was a much bigger process that needs to happen.”
The city chose Info-Tech Research Group to assist with a detailed planning process involving multiple city departments.
This included a four-day intensive session to gather the full picture of what IT staffers do each day and to ask internal customers such as finance and human resources what they considered important in case of a disaster in which the city system goes offline.
“The physical issues that can happen at our location, and that’s what precipitated this … the big ice storm we had, right? Discussions around that and a dedication to doing disaster recovery, but along with that, the economy of operations aspect of this planning resulted in what we hope is better uptime no matter what the disaster is, whether it’s human made or mother nature,” Matthews recently told a conference audience.
A disaster could be a fire, water damage or other natural disaster, but ransomware and cybercrime are also top concerns. In Texas, weather is always of concern due to tornadoes and hurricanes. Now winter weather has moved up on the list thanks to 2021’s Winter Storm Uri.
According to Info-Tech, it’s the mundane service interruptions that are most likely to take systems offline, and these include software failures, hardware failures, network failures, power outages and building closures.
The four-day disaster recovery workshop covered:
- Identifying maturity and key systems
- Creating a business impact analysis
- Plotting current recovery and identifying gaps in processes
- Creating a road map
Going in, public safety already had a backup plan in place, and the IT team wanted to know who should be prioritized.
Finance and human resources turned out to be the biggest stakeholders due to various impacts. These include direct costs, city reputation, department reputations, compliance and safety. The “goodwill impacts” include employee morale and customer or citizen satisfaction in the case of a city outage.
After long days and many conversations, the IT team had more clarity on current workload and how building an on-premise backup would impact that workload. The consultant raised the question of collocating data or implementing a cloud backup.
In the end, the choice was cloud, partly for the ease of getting it online. The cost was not prohibitive and may be offset by time and work saved.
Matthews said an existing vendor offered a cloud backup service called Xi Leap Cloud. They now have a three-year contract for that product; the hope is that the backup remains unnecessary.
If there is downtime and the backup is engaged, the cost will reflect that. As it stands, data backups are made regularly and depend on the data needs of various departments.
Matthews, who has been sharing the experience with other IT leaders, said the planning session cost between $20,000 and $25,000. Allen has about 150,000 residents and more than 850 employees, set to increase by more than 75 this year.
“For us, it was more of a journey … to get a process in place, then to choose the right technology and execute,” Matthews recently said at a conference. “I don’t want to talk about it as a midsize city. I think some of the principles, the way we approached it, really would apply to all sizes.
“If we want to say that we are a top IT organization, if we want to be prepared for the future, we need to be doing the things that probably nobody has time for. … Which is plan and work through a disaster recovery.”
