Weekend outage 13th - 14th June 2009
Posted on June 15, 2009 by Tim
Filed Under Technology |
Some users will have noticed all of our services were unavailable from Saturday late afternoon through to Sunday.
We apologise for this unplanned outage, due to a power failure in one of our cabinets at our primary data centre. Whilst the majority of our hardware has fault tolerant power supplied from two separate circuits, our master nameserver - which broadcasts information to ISPs about where our websites can be found - wasn’t covered in this way.
Due to the extended outage our secondary nameservers, provided by one of the leaders in outsourced DNS solutions, also timed out since they could not get updates from the master server. At this point an increasing number of ISPs would have failed to resolve our web addresses and returned ‘not found’.
Resolution: Power was re-established to the cabinet at 06:00hrs on Sunday morning following which a disk failure became evident on the server in question. The server was rebuilt and services re-established early Sunday afternoon. As is the nature of domain name resolution ‘DNS’ , it would have taken a varying amount of time for ISPs to re-establish their records with us , but for most it would have been instant.
The failure did not warrant a full DR failover to our secondary data centre as this would have created an equal amount of downtime, possibly more, due to IP routing delays.
Prevention of further occurance:
1. We now have a standby master nameserver along with the two primary nameservers at our main data centre.
2. We anticipate replacing the last remaining hardware not offering fault tolerant power supplies in the next two to three months.
3. Futher 3rd party availability monitoring has been added to our DNS services.
Due to the nature of this outage we were unable to provide our transactional customers with links to their trading accounts by way of our usual holding page. These services remained unaffected throughought however unreachable via our site obviously.
Transactional customers are advised to bookmark the relevant links directly to allow continued use if such an occurance should happen again. These links are available via the help page at http://www.iii.co.uk/help
Comments
Leave a Reply
