Incident Summary

At approximately 12:10am, January 9 UTC, our debugger stopped functioning properly. By 12:28, the debugger returned to processing data. By 12:50, the backlog of data had been successfully processed and we had returned to real-time processing of the data. During this time, no data sent to third-party tools, warehouses, or customer S3 buckets was delayed. The issue was scoped purely to the Segment UI.

Technical Details

One of the redis nodes responsible for streaming data from our 'live' debugger view restarted. Our workers stopped publishing messages, and the app stopped displaying new events coming through in real-time. Incoming analytics messages queued internally as our alerting mechanisms notified us that there was a problem.

To fix the issue, we modified the configuration so that both producer and consumer were reading from the failover cluster endpoint, rather than the endpoints of specific nodes. This means that in the future, if any of the individual nodes die, we'll still be able to process messages through the debugger without any sort of issue.