Articles in this section

WebSync Cloud Outage

Avatar
Anton Venema
Updated
Follow

An Amazon EC2 network outage today resulted in a brief period of time where the WebSync Cloud service was unavailable. Specifically, both the WebSync 3 and 4 clusters were inaccessible between 1:48pm and 2:01pm UTC (total of 13 minutes). The connectivity issues in the data centre were addressed quickly by Amazon staff, and everything is working well again.

Statement from Amazon:

7:28 AM PDT Between 6:47 AM and 7:10 AM PDT we experienced increased launch failures for EC2 Instances, degraded EBS volume performance and connectivity issues for some instances in a single Availability Zone in the US-EAST-1 Region. The issue has been resolved and the service is operating normally.

The event was logged by our Pingdom monitoring account, and has brought our uptime for the month down to 99.97%:

Sadly, this ends our streak of 20 consecutive 100% uptime months, just shy of 2 years.

Comments

4 comments

Sort by Date Votes
  • Avatar
    Anton Venema

    As of 4:29PM PDT today, August 2, there are additional network health issues in the AWS EC2 data centre. We are monitoring this closely and will provide updates as soon as they arrive.

    0
    Comment actions Permalink
  • Avatar
    Anton Venema

    We are in the process of deploying new servers to a different zone to work around the problem.

    0
    Comment actions Permalink
  • Avatar
    Anton Venema

    Everything is back online as of 4:55 PDT. The affected servers are still offline, but we have restored from backups to new machines. This appears to be the result of a critical hardware failure.

    0
    Comment actions Permalink
  • Avatar
    Anton Venema

    Final status from Amazon:

    • 4:36 PM PDT We are investigating network connectivity issues for some instances in a single Availability Zone in the US-EAST-1 Region.
    • 4:58 PM PDT We can confirm that some instances are unreachable and some EBS volumes are experiencing degraded performance in a single Availability Zone in the US-EAST-1 Region. Engineers are engaged and we are working to resolve the issue.
    • 5:05 PM PDT We have identified the root cause and are beginning to see recovery for instances and EBS volumes in the affected Availability Zone in the US-EAST-1 Region. We continue to work toward full resolution.
    • 5:23 PM PDT Between 4:24 PM and 5:09 PM PDT some instances were unreachable and some EBS volumes experienced degraded performance in a single Availability Zone in the US-EAST-1 Region. The issue has been resolved and the service is operating normally.
    0
    Comment actions Permalink

Please sign in to leave a comment.