One of the most exciting things about working in a large telecommunications company such as Interoute is the unifying effect of people all around the company when faced with a crisis. Such crises are thankfully rare in occurrence, but in the early morning of Friday December 19, significant and substantial damage was caused to three major submarine cables running between Europe and the Middle East and, as a result, telecommunications services were severely compromised. Some people experienced failures of main circuits, backup circuits and even tertiary systems. Internet connectivity was congested and slow-running to and from destinations in the affected region.
At Interoute, there was a controlled chaos in the operations centres in Prague and Geneva as overwhelmed staff dealt with an abundance of lengthy or complicated re-routes for private-wire customers and had to make tough decisions regarding congested packet network links. With the Christmas holiday break looming dangerously close and indications from the submarine cable operators that repair efforts were underway but unlikely to be completed before the New Year, the main objective was to achieve a steady-state for as many customers as possible until full service could be restored.
Emails were urgently exchanged, and conference calls hurriedly convened. For each challenge that presented itself, plans, options, suggestions were solicited from all corners of the company, and even customers and suppliers. They were drafted and re-drafted, considered and critiqued, refined and accepted, or dismissed as necessary, until what was left was hopefully a feasible and workable method to overcome the problems. A collective sense of responsibility had gripped the company and people from all aspects of the business were motivated extraordinarily to work together as much as possible to overcome the problem.
I contributed to the drafting of one plan – of several candidates – to restore network connectivity to an affected IP node in Malta, which had been badly compromised by the damage. I worked into the night along with others to establish the details that would allow the re-stitching together of an IP/MPLS facility over a partner network’s infrastructure. As it happens, my plan was rendered unnecessary by the successful early commission of an alternative SDH transmission system which could restore the original topology with much lower risk of complications, but it was good to know it was an option.
I am sure that Interoute weren’t the only service provider faced with difficult challenges over this episode, but I do know that most of the people I work with felt proud to be part of an organisation that could step up to face extraordinary challenges in such a determined way.
packetsource.net 