Dallas CIO Reveals Details on Ransomware Attack and Response

Dallas CIO Bill Zielinski confirmed Wednesday that the May 3 Royal ransomware attack started with stolen credentials, allowing threat actors into the city network.

Zielinski said he was unable to elaborate publicly on how those credentials were stolen due to ongoing investigations. His department, Information and Technology Services (ITS), continues to work with the city attorney and the Texas Office of the Attorney General to understand what sensitive data or documents may have been stolen or copied by the Royal ransomware group. They also worked with federal agencies during the forensic investigation.

He and city CISO Brian Gardner gave an after-action report to the Dallas City Council during a Wednesday briefing. Neither a ransom payout nor cyber insurance were mentioned during the briefing, but $8.5 million was “allotted for mitigation, recovery and restoration” related to the attack.

Gardner said that “between April 7 and May 3, 2023, the bad actors performed reconnaissance and surveillance operations and during that time began to exfiltrate small amounts equating to a total estimate of 1.169 terabytes.”

“To bring context, the city currently manages a growing footprint of roughly 3.8 petabytes, roughly 3,800 terabytes of data. Although final reviews of all impacted files are still ongoing, we do notice a small portion of data consisted of sensitive information but also know there's a large portion of data that is non-sensitive.”

They explained to the council that the documents and information in question amounted to less than 1 percent of “total daily data holdings” or about 815,000 files. ITS continues to work on making sure anyone impacted by the data theft is notified.

Royal uses multiple ways to access systems, according to a March advisory from the national Cybersecurity and Infrastructure Security Agency (CISA). During Wednesday’s meeting, Zielinski alluded to the group’s dark web marketplace being recently shut down, but didn’t cite the source.

Zielinski explained that the stolen credentials cloaked the attack, and threat actors know how to mimic routine work, lowering the risk of detection. Royal’s reconnaissance started on April 7, and it appeared “as a normal process in the IT environment; with more than 14,000 city users, this can go undetected.”

On May 3 the city of Dallas shut down systems, taking the city website offline and giving updates to the media pages.

“The city employs a formal incident response plan, which defines the detailed steps and processes that must be followed to determine, classify, respond to and remediate an incident,” Zielinski said. “In response to the attack the Information and Technology Services Department activated the incident response plan, and the after-action report is the final deliverable.”

Gardner said that the ransomware delivery began at 2:04 a.m. May 3, and extortion notes began arriving at 996 endpoints. By 3 a.m., ITS teams were on alert and taking systems offline. In the end, he said that 6.4 percent of systems were affected.

The Dallas attack interrupted “all city operations,” according to the presentation, when ITS activated the incident response plan, which they have been refining for about four years. Some departments suffered outages longer than others, due to prioritization of critical services including public safety.

“Service restoration could not begin … until the malware was effectively removed from the city’s network,” Gardner said. “This event affected all city departments. This effort was in two parts: first, the malware infection itself. Secondly, ITS did take servers offline in an abundance of caution.”

By June 9, 90 percent of the network was restored.

Referencing IBM and Forrester reports on security breaches, Gardner highlighted that Dallas’ recovery was much quicker than average. In the presentation, he outlined these metrics:

• The “mean time” to identify a data breach is 204 days; however, the city identified the Royal attack in 27 days.
• The “mean time” to contain a breach is 73 days, while this one was contained in one day.

Gardner outlined the depth and breadth of the city’s operations, including IT, saying that “the city of Dallas is a major metropolitan hub that consists of a diverse 40-plus departments, each having a unique set of missions and objectives. As such, the city faces unique challenges in the terms of information security.”

The city, with a 13,415 employee headcount, uses more than 860 applications within 40-plus departments, and the IT department employs more than 200 full-time employees and outside vendors.

“The city is a logical choice for bad actors wishing to initiate and deliver a cybersecurity attack,” the CISO said. “The city's large organization deals with large volumes of data in support of various civic activities.”

Collaboration was key to the recovery and included fire rescue, the police department, emergency management, state and federal agencies and outside vendors including GTS Technology Solutions, Zielinski said.

“I would say this first and foremost: first thing we're looking to do is contain and eradicate the malware from our environment altogether. The second thing that we're looking to do is try to establish exactly where was this threat actor and what they may have touched,” he said. “As the forensics investigation progressed … we said at this point we do not have an indication that they have data.”

Here are some of the metrics and dates, compiled from the after-action presentation, comments and the city website.

TIMELINE

• April 7: Royal Group performed reconnaissance and staging
• May 3: City of Dallas Security Operations Center (SOC) notified
• May 3: City networks shut down as a preventive measure against further compromise
• May 4: Last known infection and reporting to authorities, agencies
• June 9: Ninety percent of systems restored
• Aug. 9: City Council approves $8.6 million for recovery
• Sept. 1: IT operations and services are restored and normalized at more than 99 percent
• Sept. 20: ITS reports that more than 100 outdated servers (technical debt) were retired
• Sept. 20: Total analysis still in the works

BUDGET AND WORKFORCE NUMBERS

• The city employs more than 13,000 people
• As of July 31, the city had spent 39,590 work hours on the Royal attack
• ITS cyber and risk programs employ 35 full-time “resources” as opposed to 18 in 2020
• The IT security budget was 2.5 percent of the overall IT budget in 2019
• The IT security budget is almost 10 percent for the 2023-24 fiscal year
• The 2022-23 ITS budget was $140 million with about $60 million originated in various department budgets