There is a saying that you can wait ages for a bus and then three come along at once.
This can be true of disasters as well. The story goes like this…
In an office on a small business estate there once was a company that specialised in installing high availability data replication software into medium and large businesses. All was well, business was good, the software solution was good and the clients of the company were secure in the knowledge that their valuable data was being replicated in real time to crucial ‘hot’ backup business systems in recovery locations. Some of these installations had a mere 20 minutes to live switch-over time. The company provided round the clock on call technical assistance so that in the event of one of the clients invoking a switch to a recovery system, the experts would be able to assist them should any questions or problems arise.
One bright and sunny autumn day, a Tuesday if I recall correctly, some workmen were digging up part of one of the roads on the estate. Now, when I say estate, this was more like a country estate than an industrial one, so the works were shattering the usual peaceful atmosphere. After some time drilling away at the material of the road, a dull ‘’crump’ was to be heard – the birds taking flight at this most odd noise – and subsequently a silence fell. It was suddenly very quiet. Too quiet. No whirring of air conditioning, no hum of computer equipment, just the fading sing of a partly boiled kettle.
Some poor workman had managed to drill into a major power cable that supplied the offices. The ‘crump’ was the sound of the business end of his pneumatic road drill being vaporised.
The loss of power had not been a consideration when this company had set up their offices. The main computer system had an uninterruptible power supply that would allow it to gracefully shut down but that was the extent of the power backup facilities. An employee was dispatched to a local hire shop to bring back a generator. As luck would have it, they were second in a queue of three people for two generators that were available. They arrived back at the offices with the generator and the ‘technical’ team began trying to figure out how to power an office off a 3 pin plug. With much running of extension leads and four way adaptors, power was supplied to the main computer system and the support desk PCs. Getting the computers to stay on was a little more challenging than was expected. Hire shop generators rely solely on the speed of the engine to regulate the voltage and frequency of the power being supplied – quite adequate for electric drills, lights, kettles etc. but no good for sensitive computer equipment. The main system was guarded by the uninterruptible supply in the line between it and the generator, the poor PC had no such protection however and soon fell victim to the generator speeding up when the load from the main system via the UPS ‘disappeared’ when the UPS decided the generator power was too .dirty’. The generator suddenly sped up as the load reduced. The second dull crump noise of the day was then heard. Smoke soon followed and then some fog from the fire extinguisher that was deflty applied to the remains of the PC.
A little refining of the setup – plugging another PC into the clean side of the UPS – and some time to allow the generator to settle soon saw a skeleton system up and running.
After remembering to add the phone system to the power and then the internet connection box, communication to the outside world was re-established.
The workmen who were digging up the road had called in the local power distribution company that were responsible for the main power cable and plans were afoot to cut out the damaged chunk and replace it. This would be done the next day and there would be no power at all overnight. The generator was, understandably, outside the office and so would be put away after work hours and set up again the next day.
Thankfully, no customers had problems or disasters of their own overnight and so the lack of the central computer was not an issue. The next morning, the generator was once again set up and connected up with the benefit of some hindsight from the previous day.
The road work team were expanding the hole around the damaged cable and clearing out space around it so, that the damaged portion could be cut away and a new section spliced in. All very straight forward. One of the reasons that the accident had happened in the first place was that the cable was not marked on the site plan and I can only surmise that the communications cable that ran up the road wasn’t on the map either. The mini digger made very short work of it, and without the tell-tale flash and bang, this went apparently un-noticed, by the road workers at least.
At about the same time, the company found out how quickly a loaded generator goes through petrol! It ran out and gave a little burst of speed before dying. The little burst of speed – which I am given to understand is quite typical when an engine runs out of petrol – upset the UPS quite a bit. So much so, the UPS spat out some angry sparks and smoke.
Someone was sent out for petrol and upon their return the generator was filled and re-started. Surprisingly the UPS hadn’t suffered any ill effects form the sparking and smoking earlier and bleeped and clicked and supplied power to the systems that were plugged in.
Several hours were then spent by the technical staff trying to figure out why the network wasn’t working. It was eventually assumed that the network equipment had been damaged by the little outburst from the UPS and that a replacement would be needed ASAP. The local network company were consulted (in person as they were on the same estate, a little further down the hill) and a new box ordered to be couriered overnight.
A little later in the same afternoon, the power was restored, tea was made and the boss of the company decided an early finish was in order – it had been a very quiet day other than the generator issue – so everyone was sent home.
The next morning – Thursday – it was discovered that there was no telephone service. There was some uncomfortable realisation that this may well have been the reason for the quiet day. The phone hadn’t rung all afternoon.
After some discussion with other businesses on the estate, it seemed that another two offices had a phone problem. BT had already been called by one of the other firms and the severing of the other ‘we didn’t think it was live’ cable was uncovered. The cable was then literally uncovered, very carefully, and BT set to work re-connecting everything. A few hours later phone and network links were re-established.
There then ensued quite a few irate calls from a customer who had suffered a minor problem the previous afternoon. They were most upset at the phone not being answered – they had emailed as well but had no response. The call was taken by a junior member of staff who got very agitated at being shouted at by the upset CEO on the other end of the phone. Feeling ‘got at’, they pleaded it was “not there fault, they hadn’t had any power for two days at the office and that the phone line had got cut and nobody knew and the network was down because they thought the router had been damaged by a dodgy generator that they had got from the hire shop down the road that has run out of petrol and he had to go in his own car and get some more and it wasn’t fair and…..”
This story illustrates the nature of disasters and interruptions to businesses. They sneak around and jump up when least expected. In this case (which may or may not be entirely true) we can see evidence of three things that went wrong. The majority of ills suffered were because of a lack of a plan or any planning and by poor decision making.