At approximately 11:09 BST this morning our monitoring systems and some customers reported a loss of service.
Initial investigations point to a connectivity challenge outside our infrastructure as there is nothing in our logs (bar a complete loss of traffic on our internet connection) to indicate any internal failure. Our automated geographically separate disaster recovery systems fired up as designed. These systems take a few mins to spin up and make sure they have the latest code before becoming fully available. It looks like they were fully up for for a minute or so and taking workload but then the main systems recovered connectivity and all is back to normal.
We are raising a ticket with the engineers at the colo in which our primary servers are located and will update our support site with more information as we get it.
Our DR systems have been left up in a "hot" standby configuration so that if there are anymore glitches they are immediately available.
B2B Internet Solutions apologises for any inconvenience this incident has caused.
Update Mon 2 Oct 10:45
We are still awaiting information from our colo supplier but will publish more information as soon as we have it.
As all system have been stable since this event we are returning our DR servers to a "Pilot Light" condition.
Update Mon 2 Oct 11:10 BST
Our colo supplier has now confirmed that a server (not operated by B2B) located in the same rack grabbed all available data bandwidth to/from the rack. This was addressed as soon as their monitors detected the incident.
0 Comments