Three lessons from a recent event.
- DC Water’s highest priority is the health and safety of our customers.
- DC Water’s infrastructure is failing too often.
- DC Water’s personnel pulls together to solve problems and serve the public.
Which event, you ask, could yield such a divergent set of lessons? It all started on a late Wednesday afternoon.
At about 5 pm on Wednesday, March 5, 2014 our internal power circuitry at our pump station at Fort Reno in Washington, DC failed. Power is essential to being able to provide clean drinking water to our customers, so we had a back-up system in place. The back-up – an on-site generator – automatically switched on. A switch-gear must then shift the power source from what has failed to the generator. The switch-gear then failed. Despite having power from Pepco, and back-up power from a generator, our own systems caused our pumps to fail.
Throughout the city, water is supplied by gravity from above ground tanks or below ground reservoirs. A reservoir at Fort Reno provides water to customers at our highest elevations, and can provide water for about 1 hour until it must be replenished by the pumps. With the pumps down, the water in the reservoir drained down, water in the water mains decreased in pressure and in some cases stopped flowing entirely. Lack of water pressure in the pipe can cause a negative pressure that draws moisture into the pipe from the ground outside. This infiltration could allow contaminants to get into the drinking water.
Realizing this sequence of events, DC Water issued a precautionary boil water advisory for the 1,019 properties in the elevations where a drop in pressure could cause contamination. We went into overdrive to inform our customers and coordinate our response with the city, sending alerts through every available means and responding to the special needs of customers like hospitals and schools. We also coordinated with Fire and EMS on plans for fire suppression when the pressure in hydrants was dropping.
Fortunately, our expert maintenance personnel were able to rewire the switch-gear to restore power and replenish the Fort Reno reservoir, which then repressurized the water system. We left the boil water advisory in place until two rounds of water monitoring – taken at difference places and difference times – confirmed that no contamination had actually occurred.
And no surprise, there was lots of media attention:
The first is to highlight again that our response demonstrates that our stated focus on water quality and customer safety is genuine. No water authority wants to issue a boil water alert – which calls into question the quality of our drinking water, our primary product. This reality is particularly true for DC Water, because we in the midst of such an aggressive campaign to convince our customers to use tap water rather than bottled water. And in this case, we never had any actual evidence that indicated there was contamination – just a theory on how contamination could occur under the observed circumstances. We did not hesitate, however, to issue the advisory as soon as we knew the risk was real. In truth, the health and safety of our customers is most important to us.
The second is that our infrastructure is failing. And critically important to us is infrastructure that expands beyond the classic pipes and valves to include electrical switch-gear and associated control technology. In this case, not only our internal electrical circuits failed, but the switch-gear necessary to shift to our back-up power (which we did have on hand) also failed. As it turns out, we have a capital project in the planning stages to update all the electrical and other control systems in all of our pump stations. Yet this is another stark example of how deferred maintenance in the past is haunting us today, even if we are upgrading and rebuilding as fast as we can.
The third is that our team is fantastic at banding together to solve problems when they arise. We had our electrical and maintenance crews mobilized immediately who were able to assemble a wiring resolution on the spot at Fort Reno. We had flushing crews at the ready and water quality experts quickly mobilized to collect and evaluate samples to detect any contamination. Our external affairs office worked with the water quality division to create press releases, informational materials, robo-call messages and a nearly constant Facebook and Twitter presence to get out the word – including walking door to door to deliver notifications to affected customers.
It was a fast moving event as you can see in this summary of our social media efforts assembled by External Affairs: The Boil Water Alert – From the Inside Looking Out.
I also have to give credit to the District government, which quickly convened a sequence of emergency calls to alert relevant agencies and cover a wide variety of important issues: how to provide fire suppression in areas with low pressure; how to provide water to hospitals, dialysis centers and other vulnerable populations; how to support the five schools in the area of potential contamination; and how to minimize the consequences to affected businesses.
I list below the employees who stepped up and helped respond to this crisis. The list is long and diverse, and highlights a final additional lesson. Solving crises requires coordination among many people, many skills and perspectives – and an organized structure and format is necessary to do so in order to succeed. I’ll write more about our emergency-planning program in a future blog.
|Charles Kiely||Incident Management Team|
|Jonathan Reeves||Incident Management Team|
|Chuck Sweeney||Incident Management Team|
|Lauren Preston||Incident Management Team|
|Jason Hughes||Incident Management Team|
|Carmen Gibson||Command Center, Fielding Customer Calls|
|Geneva Green||Command Center, Fielding Customer Calls|
|Annette Laughter||Command Center, Fielding Customer Calls|
|Erica Smith||Command Center, Fielding Customer Calls|
|Kim Harrison||Command Center, Fielding Customer Calls|
|Lashawn Jones||Command Center, Fielding Customer Calls|
|Pam Easter||Command Center, Fielding Customer Calls|
|Bunmi Akinyosoye||Command Center, Fielding Customer Calls|
|Sharon Roach||Command Center, Fielding Customer Calls|
|Timothy Cannon||Command Center, Fielding Customer Calls|
|Tyvon Leonard||Command Center, Fielding Customer Calls|
|Tammy Banks||Command Center, Fielding Customer Calls|
|Kim Crawford||Command Center, Fielding Customer Calls|
|Gloria Blair||Command Center, Fielding Customer Calls|
|Greg Collins||Command Center, Fielding Customer Calls|
|Gregory Vinson||Command Center, Fielding Customer Calls|
|Kim Arrington||Command Center, Fielding Customer Calls|
|Mary Larris||Command Center, Fielding Customer Calls|
|June Adams||Command Center, Fielding Customer Calls|
|Edith Lanum||Command Center, Fielding Customer Calls|
|Derrick Myers||Customer Notification Deployment|
|Keith Watts||Customer Notification Deployment|
|Victor Browm||Customer Notification Deployment|
|Eric Stevens||Customer Notification Deployment|
|Gregory Stevens||Customer Notification Deployment|
|Eli Philson||Customer Notification Deployment|
|Jeffery Crosson||Customer Notification Deployment|
|Charles Redd||Customer Notification Deployment|
|Reginald James||Customer Notification Deployment|
|Roy Marshal||Customer Notification Deployment|
|John Herring||Customer Notification Deployment|
|Michael Cooper||Customer Notification Deployment|
|Isidro Carranza||Customer Notification Deployment|
|Renard Blanchard||Distribution System Monitoring|
|Curtis Brown||Distribution System Monitoring|
|Francis Peters||Distribution System Monitoring|
|Chris Coit||Distribution System Monitoring|
|Art Smith||Distribution System Monitoring|
|Ervin Sawyers||Distribution System Monitoring|
|Donald Yearwood||Distribution System Monitoring|
|Sam Bridges||Distribution System Monitoring|
|Mike Jackson||Distribution System Monitoring|
|Tyrone Johnson||Distribution System Monitoring|
|Edward Boney||Distribution System Monitoring|
|Marquette Austin||Distribution System Monitoring|
|Jessica Edwards-Brandt||EPA Coordination|
|David Wall||Distribution System Monitoring|
|Calvin Williams||Mobile Command Van|
|Scott Brown||Mobile Command Van|
|Nicole Condon||Public Outreach|
|Pam Mooring||Public Outreach|
|Tamara Stevenson||Public Outreach|
|Will Pickering||Public Outreach|
|Nija Ali||Public Outreach|
|Jill McClanahan||Public Outreach|
|Danny Ballerini||Public Outreach|
|Louis Desjardins||Public Outreach|
|Omar Javed||Public Outreach|
|Renee Lawrence||Pump Station Restoration|
|Ravi Kammila||Pump Station Restoration|
|Carlos Almeida||Pump Station Restoration|
|Chip Yanager||Pump Station Restoration|
|Larry Hendrickson||Pump Station Restoration|
|Warren Small||Pump Station Restoration|
|Steve Lunsford||Pump Station Restoration|
|Leslie Howard||Pump Station Restoration|
|Tony Winbush||Pump Station Restoration|
|James Jones||Pump Station Restoration|
|Wayne Reed||Pump Station Restoration|
|Ray Thompson||Pump Station Restoration|
|Duane Jones||Pump Station Restoration|
|Eugene White||Pump Station Restoration|
|Hiram Tanner||Pump Station Restoration|
|Tom Dyson||Pump Station Restoration|
|John Kennedy||Contractor Liason|
|Maureen Schmelling||Water Quality Monitoring|
|Pierre Constant||Water Quality Monitoring|
|Silas Obasi WQ||Water Quality Monitoring|