4th December from 9.14am GMT
We are currently experiencing connection issues within one of our data centres. The Data Centre staff are currently investigating.
Our back service has automatically switched in but you may experience a degradation in performance.
We will update this notice as soon as we have more information.
Apologies for any inconvenience caused.
UPDATE:
The challenges this morning were seen as a result of problems within our co-location facility supplier's network. The supplier has indicated that they believe all systems are back to normal and as of 12:10 we have switched our processing back to our primary systems.
The supplier has provided full detail of what happened and their report is attached to this article.
B2B System Responses Through This Event
This mornings failure process was unusual in that it did not cause an immediate loss of all service. We have analysed the responses of both our systems and fail over processes to see what we can learn from this event. The following graph shows how the failure progressed and an explanation follows it.
The blue line shows the health of our primary web server.
The orange line shows the health of what is being presented to end users.
The green line shows the health of our secondary web server.
"Health" is measured from a number of different locations spread around the world and plotted as a percentage of those locations reporting an acceptable service.
The challenges started at approximately 09:12. Our logs show that between 09:12 and 09:38 the service health went down and up a number of times. The first complete loss of response from the primary web server did not occur until 09:38, and even then (due to slightly different sampling times) the health of the service as a whole did not drop to zero.
The complete failure of service from the primary server at 09:38 triggered the start up of our secondary web server. On a startup being requested this server not only spins up but also updates itself based on the last full back up of the primary server. This ensures that it is running the same versions of code and has as many customer uploaded images etc as possible when it takes over. During the spin-up period end users can expect to see a "Sorry" page. This processes takes approximately 3 minutes to complete.
By 09:42 the secondary web server was up and ready to take over and the 100% overall service figure (orange line) about 5 minutes later shows when the propagation of updated DNS records completed, however there was a near immediate drop out again. This has been attributed to SQL services having switched back to the primary and then being lost again. During failover from primary to secondary there can be an a few minutes lag as connections from the web server are determined to have failed and are re-established to the alternative.
At approximately 09:51 all processes are running and stable using fail over infrastructure.
Finally at 09:56 we see the primary web server becoming available again.
During this whole period connectivity to the primary SQL server was failing and returning. Our secondary mirror took over SQL duties on a number of occasions and then handed it back to the primary a number of times until at approximately 09:49 after connectivity to it (and the primary web server) had failed completely at 09:48 all processing was transferred 100% to the secondary systems.
B2B Learning Points & Actions
- The secondary web-server did not start up until a complete loss of the primary web server was detected: B2B have re-examined the health metrics used to start the secondary web server and adjusted them to trigger startup as soon as the first point below 100% is detected.
- The active SQL server switched between primary and secondary a number of times over the event. This was partially because the check to see if the primary could take over processing occurred every 2 minutes. If at the instant in time that the test is made the primary is seen as healthy the switch back to primary automatically occurs. This process has now been modified to extend to 45 minutes the interval between tests.
The effects of the changes made should make the secondary web server more immediately available and help smooth out the effect of intermittent availability of connections to our SQL servers.
0 Comments