Due to an incident at the Manchester Collocation Facility where our primary servers are housed we are currently running on our secondary servers in Maidenhead. This means that system response may be slightly slower than normal and that reporting will be unavailable.
The incident started at about 05:40 GMT with intermittent connectivity losses with prolonged outages (>5 mins) starting at 06:38. Our DR servers started and were ready to take load but due to the intermittent nature of the problem at Manchester the secondary servers kept handing control back to the primaries only for them to fail again shortly afterwards.
We have turned off all automatic switching and will manually revert to our primary servers once we are confident they are stable.
We apologise for any inconvenience this mornings challenges may be causing, the root cause of which is completely outside our control.
UPDATED 07:46
Our collocation supplier has published an incident report. We continue to monitor our primary servers but will remain on the stable secondary servers for now.
UPDATED 10:10
Our monitoring of the Manchester servers indicated another transient event at approximately 08:30 this morning but they have been stable since. I have spoken with our supplier and they believe they are now fully stable. We will continue to monitor our primary servers and if no further causes for concern are seen we will switch back to them at approximately 12:00 so that reporting can be made available again.
During the switch back it is likely that end users will see a brief interruption in service.
UPDATED 17:25
Since 12:00 we have been running on our primary servers and this incident is now considered over.
0 Comments