In 2015 Jerry Dawkins, CISO of TRUE Digital Security, invited me to work on a DHS funded project focused on a municipal water system. The goals were simple enough: Try out a new software product from one of the DOE National Labs and provide the city an assessment of the network security of a water treatment plant. Jerry had done a great job of setting up a win-win-win situation, and the Lab received feedback from use of their tool in a real world environment. The city received an cyber security assessment of a water plant, and TRUE had a new project to deliver. TRUE has a long history of working with Industrial Control Systems (ICS), and our clients include many energy companies who are highly dependent on properly functioning, safe and secure ICS. TRUE also works for a number of cities that operate water systems. Many of them realize the importance of security and engage with us in security training, assessments, security engineering, remediation and incident response of those systems.
Part of our project was to provide awareness of the issues with ICS security and to insure that the city personnel were aware of the issues. We gathered some basic information that described the issues including:
- Project Basecamp - This project was executed by Digital Bond to assess the security of a wide range of common ICS devices. The project resulted in a watershed moment for ICS security, the clear understanding of how fragile these devices are. As one of the project participants said "It was a bloodbath."
- ICS Alert (ICS-ALERT-12-046-01A) Increasing Threat to Industrial Control Systems (Update A) - This was a response to the publication of Project Basecamp findings. This included the clear recommendation that ICS devices be protected from the open internet.
- Project SHINE - This research project was designed to show potential threats and risks associated with ICS devices directly connected to the internet. The project showed "a rampant problem of public infrastructure exposure." The project found millions of ICS devices on the open internet and sadly they saw 2,000 to 8,000 new devices every day.
The project included the collection and analysis of netflow data from the router at the edge of the treatment plant. The reasons are simple; the network is a source of truth for how things are. It does what it is configured to do, not what we hope it does! We interacted with two teams at the city. One team from IT was responsible for the network, and one team from the water department was responsible for the plant devices, a common arrangement. The issue is that each team had expectations about the other team that didn't translate into device level configurations or operational realities. As a result, the city had a substantial number of devices exposed to the open internet, just as outlined in the above referenced projects. The netlow data provided the details showing exactly what was happening. The good news is that the IT and Water teams worked hard to resolve the issues. TRUE published an report for DHS on this project, a copy of the report can be downloaded here.
The interesting thing to me as security practitioner is how did we get here? The bottom line is that there had been no regulatory requirements to assess these systems. The energy sector has dealt with NERC CIP requirements since 2006. But there has been no regulatory pressure in the water industry until a change in law in October 2018 when the US Congress passed a law known as America's Water Infrastructure Act (AWIA). Under this new law drinking water systems are required to conduct risk and resilience assessments that include a cyber security component and revise emergency response plans and the Environmental Protection Agency (EPA) is directed to oversee these efforts. All Community Water Systems serving populations over 3,300 are required to adhere to these new requirements.
Keep an eye out for my next blog post will provide more detail on AWIA and the work we are doing to support our clients compliance with the new law.