Multiple Segment features are experiencing intermittent issues
Incident
Report for Segment
Resolved
This incident has been resolved.
Monitoring
AWS has resolved the issues with the usw2-az1 and usw2-az2 Availability Zones in the US-WEST-2 Region.
As the AWS issues is resolved, Tracking API, UI and other APIs are recovered too. We will continue to monitor our sub-systems as It could take some time to work through the lag that has built up in the system.
As the AWS issues is resolved, Tracking API, UI and other APIs are recovered too. We will continue to monitor our sub-systems as It could take some time to work through the lag that has built up in the system.
Update
AWS continues to work towards resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. They have seen network mapping propagation latencies return to normal levels in both usw2-az1 and usw2-az2 Availability Zones. At this stage, all EC2 instance launches and network configuration changes are operating normally. They continue to monitor network mapping latencies and will provide a final update once all has been confirmed to be operating normally.
As AWS issues resolve, Tracking API, UI and other APIs should recover too. It could take some time to work through the lag that has built up in the system. We will continue to provide updates as we get them
As AWS issues resolve, Tracking API, UI and other APIs should recover too. It could take some time to work through the lag that has built up in the system. We will continue to provide updates as we get them
Update
AWS continues to work towards resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. Over the last 30 minutes, they have seen a strong improvement in network mapping latencies in both Availability Zones. They have also seen a reduction in latencies and error rates for other AWS services affected by this event. At this stage, they would expect that the vast majority of EC2 instance launches and network configuration changes would complete within the normal latencies. They continue to monitor until all network propagation latencies have returned to normal levels and will continue to provide updates.
As AWS issues resolve, Tracking API, UI and other APIs should recover too. It could take some time to work through the lag that has built up in the system. We will continue to provide updates as we get them
As AWS issues resolve, Tracking API, UI and other APIs should recover too. It could take some time to work through the lag that has built up in the system. We will continue to provide updates as we get them
Update
AWS continues to work toward resolving the increased networking latencies and errors affecting Availability Zones (usw2-az1 and usw2-az2) in the US-WEST-2 Region. They have now completed all updates to mitigate the resource contention being experienced by the subsystem responsible for network mapping propagation. They're seeing early signs of improvements as the system works through the backlog of network mapping propagations. They expect to see network mapping propagation latencies continue to recover over the next 2 hours.
Soon after Tracking API, UI and any other Segment APIs should recover from the degraded performance
Soon after Tracking API, UI and any other Segment APIs should recover from the degraded performance
Update
AWS has identified the issues as errors affecting Availability Zones. They expect to fully recover in the next 90 to 120 mins. We expect to the 500 errors on Tracking API and any errors on UI & API to resolve at that time. The pipeline latency has improved but would take some time to work through the lag.
Identified
The segment ecosystem is impacted by the AWS US West 2 incident. There will be intermittent UI and API errors until AWS becomes stable again.
We are observing a small number of 500 errors on Tracking API due to the AWS Incident. We observed a lag in our end-to-end processing, which is recovering.
We are observing a small number of 500 errors on Tracking API due to the AWS Incident. We observed a lag in our end-to-end processing, which is recovering.
Investigating
We are currently investigating an issue with multiple features within the Segment ecosystem. This includes Public API, the segment web app, monitoring features, and other interface features. At the time we do not believe this is impacting event delivery or data processing.
This incident affected: Web Interface (Web Interface (US)), Data Processing Pipeline (Data Processing Pipeline (US)), and Public API (Public API (US)).