Breaking News: Damning Report Into Waitangi Weekend Hospital Blackout
The Wigram breaks the story: a power blackout engulfs the Acute Services Building & A & E; report reveals a fragile system & an error by Connetics sparking a critical failure.
A damning Powell Fenwick report into the February power blackout has revealed fragile hospital power infrastructure, inadequate training, and fissures in its ability to respond to future crises.
In the wake of the crisis, Powell Fenwick found several shortcomings in the way the hospital staff were equipped to operate the power generation system, leaving the hospital vulnerable and struggling to resolve crises of the like that developed that hot Friday night during Waitangi Day weekend earlier this year.
Of critical importance, it found that Connetics’s removal of a “umbilical cord” when working on the first cable fault at the St Asaph Street substation led to the hospital generators being later rendered useless and unable to prevent a blackout.
“The root cause of the maloperation of the generator system is reliance on the umbilical cable being connected.”
In Powell Fenwick’s view, it was luck that saw a Duty Manager make a phone call to a off-duty member of staff who worked out how to put the power back on.
It has made an extensive list of recommendations. They include staff being formally trained, the entering into of a 24/7 service contract, and knowledgeable staff having oversight of work orders before sign-off.
The Wigram has contacted Te Whatu Ora. It has yet to confirm whether it has implemented any of Powell Fenwick’s recommendations.
Connectics too has not responded regarding its role in the crisis.
Connetics is owned by Orion, which, in turn, is owned almost completely by the Christchurch City Council’s corporate arm.
The Mayor, Councillors, nor the Chief Executive Dawn Baxendale would be drawn for comment.
What follows is the behind-the-scenes story, along with an explanation of the core issues and potential solutions.
The Story
On February 3, in the middle of a hot summer night, the power to the Accident and Emergency and Acute Services Building cut out, plunging patients, nurses and doctors into darkness. In the hours that followed, staff valiantly rallied to cope with the unexpected nightmare and care for patients.
Meanwhile, hospital maintenance and Connetics staff worked to understand why the power had gone out and the back-up generators hadn’t gone on, after both high voltage power cables had mal-functioned. The hospital was in critical free-fall; no one knew how long the blackout would last.
In the aftermath, Te Whatu Ora commissioned engineering company Powell Fenwick to work out the cause of the power failure and the hospital’s response.
The Wigram can now reveal the sequence of events leading up to the complete outage, how the outage was resolved and Powell Fenwick’s key findings and recommendations.
At 3:45 in the afternoon “an Orion HV cable running from St Asaph Substation to the Acute services building (Waipapa) faulted and caused the two feeder 11kV circuit breakers, CB12 and CB82, to open.”
The system moved onto a second cable, “the Food Services Substation”
“This meant power was restored to the Acute services block in 10 seconds”
A hospital employee “requested Connetics ltd [to] attend site to diagnose and locate the fault.”
“They met at the St Asaph Steet substation.”
Connetics concluded that both circuit breakers had tripped and that there was a “cable fault”.
“A Connetics Operating Order was prepared on site…and checked”.
An order was made to go ahead.
It was during this work that Connetics unplugged a “controls umbilical cable…to facilitate complete removal of the circuit breaker.”
The decision to remove the umbilical cable would set off the first domino that would lead to the complete shutdown.
“The Operating Order…does not mention the umbilical cable.”
“…There was not any label on the breaker warning about removing the umbilical cable. Connetics staff have had no formal training on the generator system.”
“[T]he unplugging of the umbilical cable is standard operating policy by Connetics and also stipulated in the EEA SM-EI-Safety Manual Electricity Industry.”
In the later investigation, Connetics said “they would not be able to complete cable isolation and testing without removing the breaker and the umbilical cable.”
“Connetics staff undertook cable tests and located the cable fault across the road adjacent Hagley Park.
“Connetics staff, [redacted] and [redacted] then left site for a break. …”
They intended to dig a hole that night and return on the Saturday to repair the cable.
Powell Fenwick later found that “[t]his part of the power outage appears to have been handled well as a normal fault situation. The faulted cable had been isolated and services had largely been put back to normal operation in a timely manner.”
By 6pm “[m]ost major systems were back online”.
At 8:19 that night “the circuit breakers on the Food Services Substation backup supply CB42 and CB92 opened (tripped) automatically.”
The second cable had failed. At this point, the back generators were meant to have kicked in.
“With no 11kV power supply available in the Acute Services Building substation the generators were sent a signal to start, which they did.”
“ Whilst the generators engines started, the circuit breakers connecting each generator to the 11kV busbar would not close.”
Here Connetics’ decision to remove the umbilical cord at the St Asaph Street substation came back to haunt them.
The removal of the cord prevented “the generators supplying power to the [high voltage] network.”
The blackout had begun.
Essential power had been cut to the “Acute Services building (Waipapa)…Riverside, Oncology, Parkside, and School of Medicine.”
The latter four had retained non-essential power.
Powell Fenwick concluded that “[w]hen the Food services feeder failed then then the total power outage to Waipapa was an instant crisis situation.”
“The hospital maintenance and engineering business continuity plan was activated, albeit informally. The Emergency Operations Centre (EOC) was established.”
A particular member of staff “was off site but was called by the hospital operations manager to inform him of the loss of power. Off-site technical staff from Engineering were requested to return to site.”
Misinformation began to circulate.
The Women’s hospital had not been impacted but rumours circulated that it was without power.
A second rumour was circulating that the power outage resulted from everyone turning on their air conditioning units. This, too, was untrue.
“Staff from Maintenance and Engineering responded to the event and called in additional technical assistance…and Connetics whilst en route.”
Connetics were unaware of the power outage.
“Luckily” the Duty Manager called the site maintenance manager. “[A]fter some discussion between” them the manager “was able to determine the issue and advised connecting…umbilical and setting the LS5 controllers to Auto which allowed the generators to connect.”
Powell Fenwick concluded that “[w]ithout intervention the power outage could have been much longer.”
9:10pm
The A & E had now been without power for 51 minutes. At 9:10pm, the umbilical cord was reconnected “was reconnected and the generator circuit breakers closed”.
The generators were restored. The power was back on.
10:10pm
After 1 hour, “the Food Services supply (backup supply) circuit breakers” were restored.
Powell Fenwick described the response “well organised and achieved a result in reasonable time to get firstly the generator’s essential power then mains supply restored.”
The next day, a Saturday, the St Asaph Street 11KV cable was repaired.
At 4:24pm, the hospital generator system returned to normal.
Powell Fenwick could not establish the cause of the failure of the St Asaph Street and Food Services (back up) cables.
Its attention turned to the failure of the generators to connect to the backup cable.
The “Root Cause”
Keep reading with a 7-day free trial
Subscribe to The New Zealand Reporter to keep reading this post and get 7 days of free access to the full post archives.