We have confirmed that we have fully recovered and expect no further issues. Warehouse and source syncs will now start and complete correctly for all customers. No data was dropped during the extended outage and subsequent sync delays. If you see any problems, please don't hesitate to contact us at http://segment.com/help/contact
This extended outage was caused by a series of cascading failures in our upstream infrastructure provider that was initially triggered by a scaling event on our end. We're actively working closely with them on diagnosing and resolving the underlying performance issue on their end as well as improving our ability to recover more quickly in situations similar to this.
Dec 7, 11:22 PST
We have resolved the underlying issue with our infrastructure provider. Warehouse and Source syncs are now recovering. No data was dropped, and warehouses will catch up during their next sync. We will post an update when we're satisfied that this incident is resolved.
Dec 7, 00:18 PST
We're still working with our infrastructure provider to resolve the underlying issue as quickly as we can.
Dec 6, 22:47 PST
We've identified the root cause with our infrastructure provider and will be restoring service shortly.
Dec 6, 20:50 PST
The underlying cause of the outage is an issue with our infrastructure service provider and we are actively working with them to resolve the issue. We'll update this status incident as soon as we know more.
Dec 6, 19:13 PST
This issue also affects Cloud Source syncs from 4:00 PM PST onwards. As soon as the underlying issue is resolved, Cloud Sources will recover with no data loss.
Dec 6, 17:20 PST
Warehouse syncs scheduled to start on or after 4:00 PM PST are not starting due to an issue in our underlying infrastructure. Engineers are working on the issue now.
No data is being lost and all warehouses will automatically catch up during their next scheduled sync if they missed any sync windows.
Dec 6, 17:00 PST