Increased 5xx rate from Segment's Tracking API
Incident
Report for Segment
Resolved
Between April 7 and April 19th, we've seen an elevated rate of errors coming from Segment's Tracking API, between the hours of 7:30am-11:30am PDT.
For users of our direct HTTP endpoint, or users of older versions of our Java, Go, Node, Ruby, PHP and .NET libraries, you may have seen elevated error rates, and some percentage of data loss (upto 0.02% during this period). These clients would have seen 5xx errors coming back from the API, and not retried accordingly.
Any client-side integrations, or any other client libraries will have properly retried, and no data will be lost. All other data will have been unaffected.
We've since increased more of the headroom on these servers to reduce the 500 rate, and have released versions Java, Go, Node, and Ruby libraries to properly handle 5xx error codes. A fix for the PHP and .NET libraries is in progress as well. We recommend upgrading to the latest version.
For users of our direct HTTP endpoint, or users of older versions of our Java, Go, Node, Ruby, PHP and .NET libraries, you may have seen elevated error rates, and some percentage of data loss (upto 0.02% during this period). These clients would have seen 5xx errors coming back from the API, and not retried accordingly.
Any client-side integrations, or any other client libraries will have properly retried, and no data will be lost. All other data will have been unaffected.
We've since increased more of the headroom on these servers to reduce the 500 rate, and have released versions Java, Go, Node, and Ruby libraries to properly handle 5xx error codes. A fix for the PHP and .NET libraries is in progress as well. We recommend upgrading to the latest version.
This incident affected: Data Ingestion (Tracking) API (Data Ingestion (Tracking) API (US)).