Archive, Infrastructure

A Little Heat Brings Key Things to the Surface


Three lessons from a recent event.

  1. DC Water’s highest priority is the health and safety of our customers.
  2. DC Water’s infrastructure is failing too often.
  3. DC Water’s personnel pulls together to solve problems and serve the public.

Which event, you ask, could yield such a divergent set of lessons?  It all started on a late Wednesday afternoon.

At about 5 pm on Wednesday, March 5, 2014 our internal power circuitry at our pump station at Fort Reno in Washington, DC failed.  Power is essential to being able to provide clean drinking water to our customers, so we had a back-up system in place.  The back-up – an on-site generator – automatically switched on.  A switch-gear must then shift the power source from what has failed to the generator.  The switch-gear then failed.  Despite having power from Pepco, and back-up power from a generator, our own systems caused our pumps to fail.

Throughout the city, water is supplied by gravity from above ground tanks or below ground reservoirs.  A reservoir at Fort Reno provides water to customers at our highest elevations, and can provide water for about 1 hour until it must be replenished by the pumps.  With the pumps down, the water in the reservoir drained down, water in the water mains decreased in pressure and in some cases stopped flowing entirely.  Lack of water pressure in the pipe can cause a negative pressure that draws moisture into the pipe from the ground outside.  This infiltration could allow contaminants to get into the drinking water.

Realizing this sequence of events, DC Water issued a precautionary boil water advisory for the 1,019 properties in the elevations where a drop in pressure could cause contamination.  We went into overdrive to inform our customers and coordinate our response with the city, sending alerts through every available means and responding to the special needs of customers like hospitals and schools. We also coordinated with Fire and EMS on plans for fire suppression when the pressure in hydrants was dropping.

Fortunately, our expert maintenance personnel were able to rewire the switch-gear to restore power and replenish the Fort Reno reservoir, which then repressurized the water system.  We left the boil water advisory in place until two rounds of water monitoring – taken at difference places and difference times – confirmed that no contamination had actually occurred.

And no surprise, there was lots of media attention:

FOX 5: Some DC residents advised to boil drinking water

The first is to highlight again that our response demonstrates that our stated focus on water quality and customer safety is genuine. No water authority wants to issue a boil water alert – which calls into question the quality of our drinking water, our primary product. This reality is particularly true for DC Water, because we in the midst of such an aggressive campaign to convince our customers to use tap water rather than bottled water. And in this case, we never had any actual evidence that indicated there was contamination – just a theory on how contamination could occur under the observed circumstances. We did not hesitate, however, to issue the advisory as soon as we knew the risk was real. In truth, the health and safety of our customers is most important to us.

The second is that our infrastructure is failing. And critically important to us is infrastructure that expands beyond the classic pipes and valves to include electrical switch-gear and associated control technology. In this case, not only our internal electrical circuits failed, but the switch-gear necessary to shift to our back-up power (which we did have on hand) also failed. As it turns out, we have a capital project in the planning stages to update all the electrical and other control systems in all of our pump stations. Yet this is another stark example of how deferred maintenance in the past is haunting us today, even if we are upgrading and rebuilding as fast as we can.

The third is that our team is fantastic at banding together to solve problems when they arise. We had our electrical and maintenance crews mobilized immediately who were able to assemble a wiring resolution on the spot at Fort Reno. We had flushing crews at the ready and water quality experts quickly mobilized to collect and evaluate samples to detect any contamination. Our external affairs office worked with the water quality division to create press releases, informational materials, robo-call messages and a nearly constant Facebook and Twitter presence to get out the word – including walking door to door to deliver notifications to affected customers.

It was a fast moving event as you can see in this summary of our social media efforts assembled by External Affairs: The Boil Water Alert – From the Inside Looking Out.

I also have to give credit to the District government, which quickly convened a sequence of emergency calls to alert relevant agencies and cover a wide variety of important issues: how to provide fire suppression in areas with low pressure; how to provide water to hospitals, dialysis centers and other vulnerable populations; how to support the five schools in the area of potential contamination; and how to minimize the consequences to affected businesses.

I list below the employees who stepped up and helped respond to this crisis. The list is long and diverse, and highlights a final additional lesson. Solving crises requires coordination among many people, many skills and perspectives – and an organized structure and format is necessary to do so in order to succeed. I’ll write more about our emergency-planning program in a future blog.

Charles Kiely Incident Management Team
Jonathan Reeves Incident Management Team
Chuck Sweeney Incident Management Team
Lauren Preston Incident Management Team
Jason Hughes Incident Management Team
Carmen Gibson Command Center, Fielding Customer Calls
Geneva Green Command Center, Fielding Customer Calls
Annette Laughter Command Center, Fielding Customer Calls
Erica Smith Command Center, Fielding Customer Calls
Kim Harrison Command Center, Fielding Customer Calls
Lashawn Jones Command Center, Fielding Customer Calls
Pam Easter Command Center, Fielding Customer Calls
Bunmi Akinyosoye Command Center, Fielding Customer Calls
Sharon Roach Command Center, Fielding Customer Calls
Timothy Cannon Command Center, Fielding Customer Calls
Tyvon Leonard Command Center, Fielding Customer Calls
Tammy Banks Command Center, Fielding Customer Calls
Kim Crawford Command Center, Fielding Customer Calls
Gloria Blair Command Center, Fielding Customer Calls
Greg Collins Command Center, Fielding Customer Calls
Gregory Vinson Command Center, Fielding Customer Calls
Kim Arrington Command Center, Fielding Customer Calls
Mary Larris Command Center, Fielding Customer Calls
June Adams Command Center, Fielding Customer Calls
Edith Lanum Command Center, Fielding Customer Calls
Derrick Myers Customer Notification Deployment
Keith Watts Customer Notification Deployment
Victor Browm Customer Notification Deployment
Eric Stevens Customer Notification Deployment
Gregory Stevens Customer Notification Deployment
Eli Philson Customer Notification Deployment
Jeffery Crosson Customer Notification Deployment
Charles Redd Customer Notification Deployment
Reginald James Customer Notification Deployment
Roy Marshal Customer Notification Deployment
John Herring Customer Notification Deployment
Michael Cooper Customer Notification Deployment
Isidro Carranza Customer Notification Deployment
Renard Blanchard Distribution System Monitoring
Curtis Brown Distribution System Monitoring
Francis Peters Distribution System Monitoring
Chris Coit Distribution System Monitoring
Art Smith Distribution System Monitoring
Ervin Sawyers Distribution System Monitoring
Donald Yearwood Distribution System Monitoring
Sam Bridges Distribution System Monitoring
Mike Jackson Distribution System Monitoring
Tyrone Johnson Distribution System Monitoring
Edward Boney Distribution System Monitoring
Marquette Austin Distribution System Monitoring
Jessica Edwards-Brandt EPA Coordination
David Wall Distribution System Monitoring
Calvin Williams Mobile Command Van
Scott Brown Mobile Command Van
Nicole Condon Public Outreach
Pam Mooring Public Outreach
Tamara Stevenson Public Outreach
Will Pickering Public Outreach
Nija Ali Public Outreach
Jill McClanahan Public Outreach
Danny Ballerini Public Outreach
Louis Desjardins Public Outreach
Omar Javed Public Outreach
Renee Lawrence Pump Station Restoration
Ravi Kammila Pump Station Restoration
Carlos Almeida Pump Station Restoration
Chip Yanager Pump Station Restoration
Larry Hendrickson Pump Station Restoration
Warren Small Pump Station Restoration
Steve Lunsford Pump Station Restoration
Leslie Howard Pump Station Restoration
Tony Winbush Pump Station Restoration
James Jones Pump Station Restoration
Wayne Reed Pump Station Restoration
Ray Thompson Pump Station Restoration
Duane Jones Pump Station Restoration
Eugene White Pump Station Restoration
Hiram Tanner Pump Station Restoration
Tom Dyson Pump Station Restoration
John Kennedy Contractor Liason
Maureen Schmelling Water Quality Monitoring
Pierre Constant Water Quality Monitoring
Silas Obasi WQ Water Quality Monitoring
Join me on Facebook and Twitter

Leave a Reply

Your email address will not be published. Required fields are marked *