Warehouse Sync Delays
Incident
Report for Segment
Resolved
All warehouses are now back on schedule. If you continue to see any issues, please contact us at https://segment.com/help/contact and we'll happily get you back on track.
A hardware failure affecting an internal database disk caused our warehouse sync coordinator to go offline at approximately 1 AM PDT. Normally, an engineer would be immediately alerted. But the failure also affected our alerting subsystem, which was unable to page any on-call engineers. We'll be removing all external dependencies form our alerting subsystem, adding new alerting, and auditing existing alerts to ensure that this doesn't happen again.
A hardware failure affecting an internal database disk caused our warehouse sync coordinator to go offline at approximately 1 AM PDT. Normally, an engineer would be immediately alerted. But the failure also affected our alerting subsystem, which was unable to page any on-call engineers. We'll be removing all external dependencies form our alerting subsystem, adding new alerting, and auditing existing alerts to ensure that this doesn't happen again.
Monitoring
Most warehouses are now back on track, but we're monitoring a small percentage of syncs that are still behind schedule. We will update this status incident when we've ensure that all syncs are back on track. If your warehouse is behind schedule and you would like us to get you caught up ASAP, please don't hesitate to let us know at https://segment.com/help/contact
Investigating
Warehouses are significantly delayed due to an underlying infrastructure issue. We're investigating and working to resolve the issue.
This incident affected: Warehouses (Warehouse Destinations (US)).