The aim of this article is to reassure users that B2B Internet Solutions' systems were not impacted by the Wannacrypt "ransomware" that was widely reported by news organisations around the same time as this outage.
The B2B SaaS Software is served from a distributed network. The basic concept of which is that when everything is working properly the active web server and SQL server are as close physically as possible to optimise performance. When thing go wrong various systems in other geographic locations (100's of miles apart) kick in to maintain site availability. A simplified network diagram is below:
SQL Server Resiliency
- Primary SQL Server sends copies of all updates to Secondary SQL Server
- SQL Witness monitors Primary and Secondary SQL Servers and informs the Web Servers of which one to use.
- On failure of Primary SQL Server the Web Servers should start using the Secondary SQL Server.
Web Server Resiliency
- External monitors check the ability of Primary and Secondary servers to provide web service and modify DNS so that users connect appropriately. If neither are available a third web server (in a fourth geographic location) is used to provide a "Sites Down" message and alert B2B Engineers via SMS.
Sunday's Outage - Failure to Resolution
Event Sequence
- Primary SQL server goes off line to reboot during update process. Witness SQL Server sees this and informs web servers to start using Secondary SQL Server.
- Primary Web Server fails to connect to Secondary SQL Server. External monitors detect this and switch web serving to Secondary Web Server
- Primary SQL server comes back on line. Witness SQL Server sees this and informs web servers to start using Primary SQL Server.
- Both Primary and Secondary Web Servers fail to connect to Primary SQL Server
- Witness SQL Server can see nothing wrong so does not allow Secondary SQL Server to take over
- External monitors detect both Primary and Secondary SQL servers down and divert users to "Sites Down Server"
- B2B Engineers awoken via SMS to investigate and understand causes of events viewed above and bring all services back on line. During this process a number of service ups and downs took place as engineers worked through solutions.
What Went Wrong
Everything worked as designed up until step 4. Failure analysis shows:
- Primary SQL Server was configured to us a "Dynamic TCP Port for Data transmission. This was a configuration error. It should have been configured to use a known fixed port.
- On reboot the Primary SQL Server came up with a different TCP Port number for Data Transmission
- The firewalls blocked Primary and Secondary Web Servers' access to Primary SQL server through the new port. Despite a reboot for updates happening most months this was the first time a change in dynamic port was witnessed on this server.
- Ports used by Primary SQL server for communication with Witness and Secondary SQL Server remained unchanged. All three were able to work together so Witness SQL Server believed Primary SQL Server was fit to provide data to either of the web servers which meant that service was not automatically switched to Secondary SQL Server.
- During troubleshooting B2B Engineers were misled by being able to connect to the Primary SQL Server from both Web servers. It was only when data was required that errors occurred. This caused B2B Engineers to try to switch service back to the Primary Servers before all required corrective actions were in place.
- A second configuration error made troubleshooting more confusing for B2B engineers. A connection from Primary Web Server to Secondary SQL Server failed due to a firewall rule allowing such access having been disabled during a major maintenance event and not having being re-enabled. This had little effect on end-users in this instance as Secondary Web Server was able to connect and take over web serving responsibility but is another learning point from this event. Business continuity testing in the past had always simulated a complete failure of Geographic Location 1 resulting in a failure to detect this configuration error.
Solutions
- Primary SQL server has been configured with fixed data transmission port number.
- All firewall rules have been updated.
- Update B2B Troubleshooting, Recovery and Test procedures.
Conclusion
B2B Internet Solutions apologises for the resulting service disruption and would like to assure our users that lessons have been learnt and operational procedures are being updated to minimise the chances of a similar event in the future.
0 Comments