Segment Status

Apr 21, 2018

No incidents reported today.

Apr 20, 2018

No incidents reported.

Apr 19, 2018

Increased 5xx rate from Segment's Tracking API

Between April 7 and April 19th, we've seen an elevated rate of errors coming from Segment's Tracking API, between the hours of 7:30am-11:30am PDT.

For users of our direct HTTP endpoint, or users of the Java library, you may have seen elevated error rates, and some percentage of data loss. These clients would have seen 5xx errors coming back from the API, and not retried accordingly.

Any client-side integrations, or any other client libraries will have properly retried, and no data will be lost. All other data will have been unaffected.

We've since increased more of the headroom on these servers to reduce the 500 rate, and have released version 2.1.1 of the Java library to properly handle 5xx error codes. We strongly recommend upgrading to the latest version.
Apr 19, 14:27 PDT

Apr 18, 2018

No incidents reported.

Apr 17, 2018

Delayed Delivery for Facebook Ads Sources

Resolved - We have discussed this issue with Facebook engineers. We believe this issue had to do with Facebook authentication token validation and expiry. The resolution is that affected users should re-authenticate their Sources with Facebook. We will be reaching out to affected customers over email.
Apr 17, 17:09 PDT

Investigating - At 5:29 AM April 17 UTC, we began to receive permissions errors from the Facebook Graph API that have interrupted some Source syncs. We’re currently investigating both for a root cause and solution, and a workaround if a root cause solution is not immediately forthcoming. Other Sources are operating normally, and while we are experiencing data delays for syncs, no data loss is expected.
Apr 17, 11:25 PDT

Apr 16, 2018

Delays delivering data to downstream destinations

Resolved - This incident has been resolved.
Apr 16, 21:08 PDT

Investigating - We're seeing delays in delivering data to downstream destinations. No data is being lost, and our on-call team is currently investigating the issue. We will continue to post updates here as the situation progresses.
Apr 16, 20:48 PDT

Delays delivering data to downstream destinations

Resolved - This incident has been resolved.
Apr 16, 20:39 PDT

Investigating - We're seeing delays in delivering data to downstream destinations. No data is being lost, and our on-call team is currently investigating the issue. We will continue to post updates here as the situation progresses.
Apr 16, 20:33 PDT

Scheduled Maintenance

Resolved - This incident has been resolved.
Apr 16, 18:47 PDT

Update - We are on our way to recovering for all blocked data. It should resume processing within the next 15 minutes.
Apr 16, 17:36 PDT

Update - 65% of data is being delivered. The other 35% of data is currently being queued and will be delivered once we resolve issues that arose during maintenance. No data has been lost
Apr 16, 16:58 PDT

Monitoring - Segment is conducting scheduled maintenance at this time. Event delivery and processing may be delayed up to 90 minutes.
Apr 16, 15:08 PDT

Delays delivering data to downstream destinations

Resolved - This incident has been resolved.
Apr 16, 15:37 PDT

Investigating - We're seeing delays in delivering data to downstream destinations. No data is being lost, and our on-call team is currently investigating the issue. We will continue to post updates here as the situation progresses.
Apr 16, 15:29 PDT

Apr 15, 2018

No incidents reported.

Apr 14, 2018

No incidents reported.

Apr 13, 2018

Delays delivering data to downstream destinations

Resolved - This incident has been resolved.
Apr 13, 01:55 PDT

Investigating - We're seeing delays in delivering data to downstream destinations. No data is being lost, and our on-call team is currently investigating the issue. We will continue to post updates here as the situation progresses.
Apr 13, 00:20 PDT

Apr 12, 2018

Delays delivering data to downstream destinations

Resolved - This incident has been resolved.
Apr 12, 12:49 PDT

Investigating - We're seeing delays in delivering data to downstream destinations. No data is being lost, and our on-call team is currently investigating the issue. We will continue to post updates here as the situation progresses.
Apr 12, 12:48 PDT

Delays delivering data to downstream destinations

Resolved - This incident has been resolved.
Apr 12, 12:36 PDT

Identified - We've identified the issue, and our on-call engineers are working to resolve it. No data is currently being lost.
Apr 12, 12:13 PDT

Investigating - We're seeing delays in delivering data to downstream destinations. No data is being lost, and our on-call team is currently investigating the issue. We will continue to post updates here as the situation progresses.
Apr 12, 12:01 PDT

Apr 11, 2018

Data Flow Graphs Displaying Incorrectly

Resolved - Historical data has been restored and the incident has been resolved.

Data volume statistics from April 11, 2018 9:30am UTC to April 12 1:30pm UTC are unavailable. This had no impact on data or message delivery and the counters are now working as expected. We apologize for the inconvenience.
Apr 11, 18:45 PDT

Identified - We've identified the problem. No data has been lost, we are restoring historical data and will soon resume updates to data flow graphs.
Apr 11, 17:46 PDT

Investigating - Graphs of data flow information are incorrectly reporting no data is flowing into Segment from Library, Event, and Object sources despite normal operation otherwise.
Apr 11, 17:11 PDT

Apr 10, 2018

No incidents reported.

Apr 9, 2018

Delays delivering data to downstream destinations

Resolved - This incident has been resolved.
Apr 9, 11:35 PDT

Investigating - We're seeing delays in delivering data to downstream destinations. No data is being lost, and our on-call team is currently investigating the issue. We will continue to post updates here as the situation progresses.
Apr 9, 10:50 PDT

Delays delivering data to downstream destinations

Resolved - This incident has been resolved.
Apr 9, 10:01 PDT

Investigating - We're seeing delays in delivering data to downstream destinations. No data is being lost, and our on-call team is currently investigating the issue. We will continue to post updates here as the situation progresses.
Apr 9, 09:25 PDT

Apr 8, 2018

Delays delivering data to downstream destinations

Resolved - This incident has been resolved.
Apr 8, 12:31 PDT

Investigating - We're seeing delays in delivering data to downstream destinations. No data is being lost, and our on-call team is currently investigating the issue. We will continue to post updates here as the situation progresses.
Apr 8, 09:21 PDT

Apr 7, 2018

Delays delivering data to downstream destinations

Resolved - This incident has been resolved.
Apr 7, 15:58 PDT

Investigating - We're seeing delays in delivering data to downstream destinations. No data is being lost, and our on-call team is currently investigating the issue. We will continue to post updates here as the situation progresses.
Apr 7, 15:53 PDT

Delays delivering data to downstream destinations

Resolved - This incident has been resolved.
Apr 7, 15:48 PDT

Identified - We've identified the backlog in our data, and are currently working to resolve it. We're seeing delays of around an hour when passing through the pipeline, but no data is being lost.
Apr 7, 09:09 PDT

Investigating - We're seeing delays in delivering data to downstream destinations. No data is being lost, and our on-call team is currently investigating the issue. We will continue to post updates here as the situation progresses.
Apr 7, 07:35 PDT