Between 18:42 UTC and 18:56 UTC on 29 April 2019 a small number of the sites hosted by B2B Internet Solutions were unavailable for new connections and displayed a "Maintenance" message.
B2B would like to apologise for any inconvenience that this event caused.
The cause of this outage is understood and mitigation actions are being taken to minimise the chances of a future event.
A customer provided a large data update which generated considerable and prolonged loading on our database server. This triggered (as designed) an autoscaling function on the database to provide it with additional processing capacity.
During a scaling event there is a period of between 30 and 60 seconds during which no new connections can be made. In terms of customer impact this is normally seen as just a slight delay in page load time if they are unlucky enough to access a page that cannot make use of an existing pooled data connection.
To ensure any challenges within our infrastructure are swiftly detected our server monitors use a new connection every time they test the website and allow a short period for a response. This testing takes place every 30 seconds. Should two consecutive tests fail (as occurred during this event) replacement hardware is immediately started. Once new hardware is running it takes approximately 10 minutes for all the configuration procedures and checks to complete and the replacement server to become fully capable of supporting service. During this 10 minute period new connections to the "failed" server are blocked (existing ones can continue) and the old server is shut down.
The effect of this hardware cycling was to turn an event that would have produced a brief period of slow response in to an outage.
We are as a matter of urgency:
- Reviewing the number of monitor failures that are required before replacement hardware is demanded.
- Reviewing the timeout criteria for the monitor to database connections.
0 Comments