From 07:20-08:05 UTC Dec 18, 2015 our Tracking API experienced a partial outage in service. Around 15% of the API calls were rejected due to bad code being pushed and a misconfigured instance which was rejecting calls.

That means:

• any client-side tools will be unaffected, and have received the full end data as normal
• data in S3, Redshift, 3rd-party tools from server-side libraries will also be unaffected for libraries that retry. The exception is for high traffic apps which buffer data internally, they will experience some data loss.
• data in S3, Redshift, and Segment from browser javascript will see a 15% drop for those 45 minutes

Full Post-mortem

At 07:15 we began to deploy a new version of our Tracking API, which includes significant updates to increase the speed and scalability of the system. The updates passed our staging tests, so we began deploying them to prod. After the deploy started, the ECS scheduler didn't boot additional containers due to insufficient resources. In an effort to make more hosts available to schedule those containers, the engineer running the deploy booted additional instances. Unfortunately these instances did not run the full bootstrapping script necessary, resulting in running the Tracking API but continuing to return elevated error rates. By 08:00, the engineers on-call had recognized the problem, properly provisioned these instances, and removed the tracking API from them. During that time, data sent to the affected instances was rejected.

Remediations

We're already in process of adding fixes to our provisioning process so that nodes automatically bootstrap with everything they need. To make sure that ECS can actually schedule new containers, we're going to keep increased capacity on the tracking api. To ensure that instances don't boot but fail to actually work, we'll add additional healthchecks on the instance to make sure everything is fully operational.

Once these fixes are in place we'll deploy the new code along with an updated AMI. We'll run a series of "fire drills" on our staging environment as well to ensure that the scheduling works as anticipated.