In California, where epic Sierra Nevada snowpack and “the Big Melt” have substantially increased the stakes for reservoir managers, officials are taking steps to protect the state’s water systems from hackers, terrorist attacks and natural disasters, such as the flooding that temporarily severed the Los Angeles Aqueduct — the city’s water lifeline to the Owens Valley.
But experts say the challenges are numerous. Many of the systems in California and nationwide are still operating with outdated software, poor passwords, aging infrastructure and other weaknesses that could leave them at risk.
“We’ve seen a steady rise in both the prevalence and the impact of cyber intrusions, as well as an extraordinary increase in ransomware attacks, which have become more destructive and more expensive,” said Joe Oregon, chief of cybersecurity for Region 9 of the federal Cybersecurity and Infrastructure Security Agency (CISA).
Andrew Reddie, an assistant professor of practice in cybersecurity at UC Berkeley’s School of Information, said much of the problem is “driven by the fact that the infrastructure is really, really old, and ultimately predates the era that we find ourselves in now, where we actually bake cybersecurity into these ... systems by design.”
“You can point to any number of critical infrastructure [projects], including things like dams and water treatment plants, that are not terribly well-protected in terms of passwords,” he said.
A lot of older infrastructure is not “air-gapped” from the Internet, he said, referring to a separation between operational technology and Internet technology. That could enable a bad actor to do things like change chemical levels or open sluices to manipulate flows in water channels or dams.
Compounding the problem is a lack of central regulation or uniform protocols. Multiple agencies — including the Environmental Protection Agency, the National Institute of Standards and Technology, the American Water Works Association and the Department of Homeland Security and CISA — provide some degree of risk management oversight, or offer frameworks and recommendations. But many of the day-to-day decisions are left up to individual operators.
“A lot of the responsibility does certainly fall on the stakeholders’ shoulders to manage their own information systems effectively to prevent any type of cyber compromise or cyber incidents,” said Oregon, of CISA.
The agency estimates that about 63 percent of the nation's 91,000 dams are privately owned. Federal, state and local governments and utilities own 35 percent, and the remaining 2 percent have “undetermined ownership.”
Despite the risks, experts said it’s important for water systems to be networked in order to expedite maintenance and monitoring. In California, reservoirs are often intentionally spread far apart to maximize rainwater capture and other benefits, so sending physical crews to respond to every potential problem would be time-consuming and expensive, said Ethan Schmertzler, chief executive of Dispel, a cyber defense firm.
“It all depends upon how water systems are connected, and most water systems in the United States are not — it’s not one national water system,” he said. “The good news is each community is divided into their own command and control systems. The downside is, they’re all divided into their own command and control systems.”
Though most standards are not mandatory, cybersecurity recommendations — and spending — have vastly improved in recent years, he said. Recent legislation through the National Defense Authorization Act will soon compel utilities to report cybersecurity threats to CISA, which will help the federal agency better spot trends, share information and render a response.
John Rizzardo, security coordinator with the State Water Project at the California Department of Water Resources, said the agency operates with an ethos of “layers upon layers of security,” for both physical and cyber threats. Because the agency is also an energy provider in the state, “we probably employ more security features than a lot of just the water industry,” he said.
That doesn’t mean it is immune, however. CISA pointed to the Oroville dam crisis of 2017 as an example of the nation’s need for “comprehensive oversight and guidance over dam resilience.” During that incident, hillside erosion on the dam’s emergency spillway threatened a major flood event and prompted the evacuation of about 200,000 people, though disaster was ultimately averted.
Rizzardo said the agency has since shored up the spillway and made significant security upgrades and is working to implement the same standards across all State Water Project facilities. The Department of Homeland Security runs national security drills for the dam sector every two years, he said, which the agency also participates in.
But even with the best protocols in place, “there’s still going to be a risk of a cyber or physical attack,” Rizzardo said. “It could happen — we’re doing our best to prevent it — but if it does happen, we do practice our emergency action plans regularly so that we're prepared if there is some kind of attack that we can try to mitigate, to reduce the consequences."
In January 2021, an unnamed water treatment plant in the San Francisco Bay Area also suffered a cyber attack, NBC News first reported. Hackers accessed the plant’s system through a remote access TeamViewer account and deleted programs used to treat drinking water. The programs were reinstalled the next day, and no failures were reported. (The Northern California Regional Intelligence Center, which compiled a report on the incident, said it could not provide more details as an investigation is ongoing.)
One of the largest water providers in the country is the Metropolitan Water District of Southern California (MWD), a massive regional wholesaler that supplies 26 agencies serving 19 million people, including the Los Angeles Department of Water and Power.
General Manager Adel Hagekhalil said in an email that America’s Water Infrastructure Act of 2018 served as a “catalyst for utilities to evaluate their resilience to risk and create emergency plans for responding to all hazards.”
Hagekhalil noted that community water systems serving more than 3,300 people are required to actively update their risk and resilience assessment and emergency response plans every five years.
Additionally, the MWD employs cybersecurity experts and constantly monitors network and computer activity to “detect unusual events quickly so they can be addressed,” he said. Computer and network access is tightly controlled, and employees are also required to take annual cybersecurity training. The agency also conducts periodic emergency management exercises at different facilities to simulate responses to physical threats such as earthquakes, floods, fires and terrorist attacks, which include first responders and law enforcement agencies, he said.
Reddie, of Berkeley, said more auditing would provide a better understanding of which systems are networked, as well as which systems follow best practices. He also recommended educating workforces about proper cyber hygiene.
Even with such steps in place, however, vulnerabilities remain.
“These individual firms need to be thinking about what’s their model for the type of threat actor that they’re likely to see,” he said. “Like, is this going to be a state actor? Is it going to be a disgruntled employee? Is it going to be, you know, a script kiddie in a basement?”
©2023 Los Angeles Times. Distributed by Tribune Content Agency, LLC.
