Amazon Attributes AWS Disruption to Race Condition in DNS Automation

by Justin Erickson

Amazon says a latent race condition in DynamoDB’s DNS management system produced an empty DNS record for the service’s us-east-1 regional endpoint, triggering widespread connection failures and greatly impacting AWS services. The issue began at 11:48 PM PDT on October 19th, and full recovery across affected services was completed by the afternoon of October 20th. Amazon has disabled the faulty automation globally and added safeguards and tests to prevent it from happening again.