Outage Report

On Thursday, December 3, Segment experienced 17 minutes of partial downtime, from 01:49-02:06 UTC. During that period, roughly half the data was rejected by our API, and we saw elevated error rates, along with timeouts from the majority of our client libraries.

The outage was caused by a sudden influx of batch requests. Our monitoring notified the engineers on-call that we were seeing elevated CPU and error rates. Our ops team immediately started booting additional instances to handle the load. The API continued to return errors as the instances booted up, required provisioning, and adjustments to their networking settings a few minutes later.

Any data sent through analytics.js, iOS or Android will be complete in third party tools. Data processed by our server-side integrations, S3, and Redshift will only reflect partial data during the time of the outage.

The API was fully restored by 02:06.

Over the course of this period our CDN and web interface remained unaffected.

Remediations

Keeping your data complete, safe, and secure is Segment’s highest priority. Over the next few days we will begin to focus on the following remediations:

First, we’re increasing the headroom we keep on each instance during normal operation. By increasing the amount of available computing, and adding alerting at much lower CPU thresholds, we’ve created sufficient buffer to handle much larger unanticipated influxes of data. [Dec. 3 Update: Complete]

We plan to implement auto-scaling groups on our API within the next month, triggered at a lower CPU threshold. For the API itself, we’ll make sure all settings are baked exactly into the instance AMI, so that it boots as quickly as possible when on-demand capacity is needed. This will allow us to scale the API far ahead of a high-load scenario without the need for human intervention.

Additionally, we plan to audit our client libraries in failure scenarios, so that they properly queue data, retry, and back off when they get a bad response from the API. The majority of these libraries handle intermittent network failures properly, but don’t handle the failure case of a long-term outage.