Resolved - On July 15 we started the process of gradually rolling out a backend service improvement for the Warehouses product to all our customers. This change was tested in our staging environment and the rollout process was also closely monitored to ensure there are no errors. However on July 19 we started receiving reports that data from some of the connected Cloud Object Sources was not getting synced to the corresponding Warehouses, even though it was successfully ingested from the Cloud Source and stored in our local data stores. We immediately stopped the rollout process and started investigating the issue. We identified a bug in the new change that caused certain ingested Cloud Sources objects being excluded while performing Warehouses syncs. We confirmed that the bug impacted all Warehouse syncs for a subset of customers that were triggered between July 15 6:00 PDT and by July 19 8:42 PDT. We rolled back the change on July 19 8:40 PDT and confirmed that subsequent Warehouse syncs were picking up all the Cloud Sources objects available. To ensure previously skipped Cloud Source objects are synced to the Warehouse we executed a replay for all impacted Sources. The replays completed on July 21 16:30 PDT.
Jul 19, 08:00 PDT

Resolved - 6.66% of the syncing of real-time computed traits and audiences had a lag. While the overall throughput was high, there was a delay. Once the issue was resolved, the lag caught up. This incident is resolved.

Timeline in PT:
5:09am incident detected
5:09am issue identified
7:10am issue resolved
7:18am errors stopped
7:20am lag went down
7:25am lag down to 0
Jul 18, 05:00 PDT