When faced with a real disaster, how effective is a disaster recovery plan?
It’s a question we all need to prepare for but having a disaster recovery plan typed up on a piece of paper and stored in a file somewhere doesn’t necessarily mean you have a robust strategy to execute when the time comes. Nor does it mean you’re prepared for every eventuality, as we’re about to find out.
Last week Claire Priestley, Director of IT at City, University of London kicked off her first CIO+1 event of the year where we were joined by an extraordinary line up of speakers who shared their tales of leadership through events such as the WannaCry virus, two very different perspectives of the London Bridge attack and a perspective of what it took to restore business during the devastating Cumbrian floods.
The WannaCry attack – David Gaunt, Accident and Emergency Consultant at Watford General Hospital
The infamous WannaCry attack caused widespread disruption to global IT systems on the 12th May 2017, raising serious questions about the preparedness of the NHS to deal with such incidents.
A third of the UK’s hospitals were affected with clinicians not just being locked out of PCs but also MRI scanners and devices for testing blood and tissue samples. In total, more than 1,200 pieces of diagnostic equipment were infected by the ransomware, although further devices were put out of use after being disconnected from IT systems to prevent the infection spreading. This resulted in some A&E departments shutting their doors and refusing patients. The attack caused more than 19,000 appointments to be cancelled, costing the NHS £20m between 12-19th May 2017 and £72m in the subsequent clean up and upgrades to its IT systems. But while many trusts were affected, Watford General Hospital avoided the disaster but was experiencing issues in contacting neighbouring trusts who had no access to their emails. To combat this issue, clinicians turned to WhatsApp to communicate with each other as the ransomware continued to spread.
The bulk of the criticism levelled at NHS England and NHS Digital was centred around the lack of communication between organisations as the attack unfolded. Reports of NHS services being locked out of systems began coming in around midday, yet the NHS didn’t declare that an attack was taking place until several hours later. The Audit office’s report concluded that the NHS could have prevented the WannaCry cyber-attack had it followed basic IT security practices, including migrating computer systems to newer software versions and keeping internet-facing firewalls up-to-date.
The Cumbrian floods – a perspective from Adrian Wakefield Managing Director of Transforming IT
Adrian spent 13 years working as the Business Services Director for James Walker Group, a global manufacturing organisation based in Cockermouth, that was left devastated after the Cumbrian flood in 2009. Unprecedented rainfall caused rivers and streams to spill into the factory causing hundreds of thousands of pounds of water damage and completely halting production for almost two weeks. The business had a disaster recovery plan in place, but nothing to advise them on what to do in the event of extreme flooding. Despite the flood being described as a 1 in a 1000-year event, Adrian and the team explained how they weren’t going to let the same disaster devastate them again and it was just as well, as storm Desmond hit 6 years later and this time, the organisation was ready.
Adrian explained how the company considered moving the factory to Workington to mitigate future flood risk to the business, but the enormous cost of moving the business made it financially unviable. Instead, the company invested in devices capable of literally lifting the machinery and IT servers off the floor and suspending them in mid-air. But they aren’t stopping there. The organisation is now investing in a £2.6m flood defense wall around the firm’s production buildings as pumps and automated floodgates.
The London Bridge terror attack – a perspective from Christina Scott, CTO at News UK
In March 2017, a suspicious vehicle was left abandoned with smashed windows on London Bridge road sparking mass evacuation of local businesses, a nearby railway station and neighbouring areas. Although the bomb threat was later cleared, the incident caused widespread disruption. Staff at News UK’s London Bridge office were told by police not to leave the building. Christina Scott, CTO at News UK recalls how police had senior members of staff to move all employees onto higher floors, but as word broke out, a communication breakdown caused employees to panic – some were trying to leave the building though the main entrance, some were trying to go out onto the roof terrace and others were just in sheer panic trying to move their entire workstations upstairs.
Three months later, staff at News UK found themselves once again at the centre of a major incident – a combined vehicle ramming and stabbing attack by terrorists. As well as having a duty to keep staff safe from harm, the publisher also had an obligation to its readers to continue reporting the news despite potential business interruptions. Christina was in the offices at the time of the attack. She recalls being horrified by reporters desperately trying to get out of the office in attempt to cover the story while in the offices, TV screens lit up with the events unfolding right outside their office – despite the risk to their lives.
The events of 2017 led the organisation to make a number of changes. Christina says the organisation has learnt a lot about the importance of staying calm and communicating effectively to all employees to avoid causing additional stress and panic. Disasters are a social process that require support and participation from a wide variety of responders and staff. The organisation now has since revisited their approach to effective communications and Christina explained that since they’ve already been though disasters employees are now aware of what they need to do in the event of a range of different emergencies, from evacuation procedures, though to lockdown situations.
The London Bridge terror attack – a perspective from PC Charlie Guenigault.
While businesses like News UK were on lockdown, some incredibly brave police officers were running towards danger to protect the public. When on a night out with friends, off-duty police officer, Charlie Guenigault left his mates and ran towards someone who had just been stabbed on the street. Charlie initially thought he was attending a brawl that had got out of hand and was completely unprepared for the reality of what was about to happen. When Charlie ran over to help, putting himself in between the victim and the attacker, he was brutally stabbed 5 times, receiving serious injuries to his head, back and stomach.
Charlie went on to explain that no amount of training could have prepared him for the situation even though as a police officer, you are trained to run towards danger, he explained that you never imagine you might suddenly find yourself face to face with a terrorist when you’re simply attending what you believe is a drunken brawl. Charlie has since been awarded the George Medal in recognition of his outstanding bravery.
All the preparation in the world still won’t always be enough. Sometimes you just need to think of your feet and try to address the situation in the best way you can.
Prepare for the worst and hope for the best
All organisations, regardless of size or location need to have emergency and disaster planning in place. It must include coordinated, co-operative process of preparing to combine urgent needs with available resources. But most importantly, it needs to be a living document, that is periodically adapted to changing circumstances and provides a guide to the protocols, procedures, and division of responsibilities in emergency situations.
That being said, you can never plan for every situation. Situations you never imagined possible may very well hit you one day, but if you’ve got a living document and clear robust processes in place, you will be better positioned to deal with whatever the world throws at you.