tag:status.segment.com,2005:/historySegment Status - Incident History2024-03-19T01:48:47-07:00Segmenttag:status.segment.com,2005:Incident/202390392024-03-14T15:33:22-07:002024-03-14T15:33:23-07:00Realtime audiences: delays syncing to list destinations, incorrect counts<p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>15:33</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>14</var>, <var data-var='time'>08:00</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>22:06</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for the delayed syncs and incorrect counts. No data has been lost.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>18:56</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>18:42</var> PDT</small><br><strong>Identified</strong> - We have identified the issue causing the delay and currently working on fixing it.</p>tag:status.segment.com,2005:Incident/202155132024-03-11T07:29:17-07:002024-03-11T07:33:15-07:00EU traffic delayed for 2 hours<p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>07:29</var> PDT</small><br><strong>Resolved</strong> - As of 12:00 PM UTC on 11.03.2024, our Segment Data Collection Activation -Streaming Sources service was impacted for 2 hours. Support team has received reports that EU customers were not receiving events in their sources since roughly 12:00pm UTC. A traffic surge caused this incident. Platform has finished processing through the surge. There is no data loss. The impact was a 2 hour delay for EU traffic. EU Systems are back to normal. The team continues to monitor it.</p>tag:status.segment.com,2005:Incident/201218352024-03-01T02:58:32-08:002024-03-01T02:58:32-08:00Processing delays for real-time Audiences, Computed Traits, and Journeys<p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>02:58</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 1</var>, <var data-var='time'>02:57</var> PST</small><br><strong>Update</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>22:40</var> PST</small><br><strong>Update</strong> - We have identified the issue and implemented a fix. Processing has resumed and we are monitoring recovery.</p><p><small>Feb <var data-var='date'>29</var>, <var data-var='time'>20:33</var> PST</small><br><strong>Investigating</strong> - We're currently experiencing delays processing Engage Audiences, Computed Traits, and Journeys created between Aug 8th 2022 and Aug 19th 2022, as well as those created between Aug 22 2022 and March 14th 2023. Our engineers are working on identifying a solution. No data has been lost.</p>tag:status.segment.com,2005:Incident/200864812024-02-27T07:54:18-08:002024-02-27T07:55:16-08:00Error accessing customer workspaces - Workspace not found<p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>07:54</var> PST</small><br><strong>Resolved</strong> - This Workspace access issue is now resolved, and there was no data loss as a result of this incident. Thank you again for your patience while we worked through this.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>06:21</var> PST</small><br><strong>Monitoring</strong> - A fix has been deployed, and workspace access has now resumed. We are monitoring the situation for stability before resolving. Thank you for your patience.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>04:36</var> PST</small><br><strong>Update</strong> - We are continuing to make progress on the root cause of the errors. Workspace access is improving, with occasional errors experienced. We will continue to provide updates as more information becomes available. Thank you for your patience.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>03:40</var> PST</small><br><strong>Identified</strong> - We are making progress on restoring access to workspaces and resolving the API errors. We will continue to provide updates as more information becomes available. Thank you for your patience.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>02:59</var> PST</small><br><strong>Update</strong> - We are actively working to restore access to workspaces and resolve the API errors. Please note, there is no impact on data ingestion or delivery to destinations.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>01:59</var> PST</small><br><strong>Investigating</strong> - Customers attempting to access their workspaces within the web app on US and EU are met with an error message. We're investigating the issue and will post updates as soon as we have further information.</p>tag:status.segment.com,2005:Incident/200332782024-02-20T14:47:15-08:002024-02-20T14:47:15-08:00Syncs delayed for Snowflake warehouse destination<p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>14:47</var> PST</small><br><strong>Resolved</strong> - We experienced an intermittent issue while ingesting data into Snowflake destination which caused sync delays for a small subset of customers. The issue is now resolved with no data loss reported.</p>tag:status.segment.com,2005:Incident/199765732024-02-13T10:30:13-08:002024-02-13T10:30:13-08:00Source API Keys page not loading<p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>10:30</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>09:15</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>09:03</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>09:00</var> PST</small><br><strong>Investigating</strong> - We are currently investigating an issue affecting Source API Keys page, which is not loading due to an error</p>tag:status.segment.com,2005:Incident/199718452024-02-12T22:10:03-08:002024-02-12T22:10:03-08:00Processing delays for real-time Audiences, Computed Traits, and Journeys<p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>22:10</var> PST</small><br><strong>Resolved</strong> - This issue is resolved. Audiences, Computed Traits, and Journeys are now all processing in real-time.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>19:33</var> PST</small><br><strong>Monitoring</strong> - We have identified the issue and implemented a fix. Processing has resumed and we are monitoring recovery.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>19:22</var> PST</small><br><strong>Identified</strong> - We're currently experiencing delays processing Engage Audiences, Computed Traits, and Journeys created before February 2021, between May 19 2022 and July 18th 2022, and between July 13 2023 and July 19 2023. Our engineers are working on identifying a solution. No data has been lost.</p>tag:status.segment.com,2005:Incident/198840042024-02-01T04:58:32-08:002024-02-01T04:58:32-08:00Segment Web Interface Issues<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>04:58</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>04:21</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>03:54</var> PST</small><br><strong>Investigating</strong> - We are investigating an issue with the Segment web interface and public api that is affecting all customers. Data ingestion is not affected.</p>tag:status.segment.com,2005:Incident/198787672024-01-31T06:00:00-08:002024-01-31T12:02:31-08:00Unify and Engage pipeline delay<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>06:00</var> PST</small><br><strong>Resolved</strong> - We experienced an delay processing data onto Unify profiles and into Twilio Engage (US). Engineering team has identified and mitigated an underlying issue. Data processing delay recovered after 4.5 hours.</p>tag:status.segment.com,2005:Incident/198684622024-01-30T10:24:34-08:002024-01-30T10:24:34-08:00Segment's Catalog Search is not working<p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>10:24</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>10:21</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>09:28</var> PST</small><br><strong>Identified</strong> - The team has identified the issue and a fix is being implemented. We will provide an update you soon.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>08:50</var> PST</small><br><strong>Investigating</strong> - We are currently investigating an issue affect the Segment Search functionality. Our team is working on identifying a fix and we will update this status as soon as we have more information.</p>tag:status.segment.com,2005:Incident/198225082024-01-24T14:08:13-08:002024-01-24T14:08:13-08:00Intermittence in the loading of actions mapping UI<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>14:08</var> PST</small><br><strong>Resolved</strong> - Issue has been identified and resolved.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>13:46</var> PST</small><br><strong>Investigating</strong> - The actions mapping UI is experiencing issues. We are currently investigating them.</p>tag:status.segment.com,2005:Incident/198133252024-01-24T10:17:25-08:002024-01-24T10:17:25-08:00Unify and Engage pipeline delay (US)<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>10:17</var> PST</small><br><strong>Resolved</strong> - This issue is mitigated. All services are processing in real-time and restored to healthy operations.</p><p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>08:59</var> PST</small><br><strong>Update</strong> - The Unify pipeline is real-time. Audiences, Computed Traits, Journeys created between July 19 2023 and Oct 26 2023 are 1.5 hrs behind real-time. We expect to be fully recovered within the next hour.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>21:28</var> PST</small><br><strong>Monitoring</strong> - The Unify profiles pipeline has caught up to real-time, and our Engage Computed Trait, Audiences, and Journeys pipeline is recovering from previous delays.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>20:13</var> PST</small><br><strong>Identified</strong> - We've identified the problem and are beginning to catch up on the backlog of data. Unify profile data is catching up from the delay, but computed traits and audiences continue to be delayed. Our engineers are working to completely resolve the issue.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>20:09</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>20:01</var> PST</small><br><strong>Investigating</strong> - We’re experiencing a delay processing data onto Unify profiles and into Twilio Engage (US), impacting Profiles, Audiences, Journeys, Computed Traits, and Syncs. Our engineers are investigating the problem. No data has been lost.</p>tag:status.segment.com,2005:Incident/197315312024-01-16T19:07:58-08:002024-01-16T23:46:20-08:00Warehouse destination delays<p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>19:07</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved. During the impact period, any warehouse syncs scheduled to start would have failed, resulting in data delay. Any warehouse syncs already in progress were not affected.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>18:35</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>17:28</var> PST</small><br><strong>Identified</strong> - We have identified the issue with Warehouse destination delays that affects all US customers.</p><p><small>Jan <var data-var='date'>16</var>, <var data-var='time'>16:53</var> PST</small><br><strong>Investigating</strong> - We are investigating an issue with Warehouse destinations delays that is affecting all US customers.</p>tag:status.segment.com,2005:Incident/196040572024-01-03T12:58:59-08:002024-01-03T12:59:00-08:00Internal Error in Destination Function event delivery since 12/14<p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>12:58</var> PST</small><br><strong>Resolved</strong> - We have resolved the issue for events being discarded due to an "Internal" error. Customers that have seen this error can request to have their data replayed by reaching out to Segment support.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>10:59</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'> 3</var>, <var data-var='time'>09:32</var> PST</small><br><strong>Identified</strong> - Since 12/14, a small number of Destination Functions have errantly discarded data. The events will not have been delivered to their source and can be explored in Delivery Overview "filtered at destination" step as "internal" errors.</p>tag:status.segment.com,2005:Incident/194420292023-12-14T13:29:06-08:002023-12-14T13:29:06-08:00Data Processing Delays<p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>13:29</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>12:51</var> PST</small><br><strong>Update</strong> - A fix has been implemented and we are continuing to monitor the result.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>12:22</var> PST</small><br><strong>Monitoring</strong> - We have identified an issue causing streaming destination delivery delay that affects US region customers. A fix has been implemented and we are monitoring the result.</p>tag:status.segment.com,2005:Incident/194281522023-12-13T07:42:17-08:002023-12-13T07:42:18-08:00Intermittent Failures In Tracking API (eu-west-1)<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>07:42</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>06:51</var> PST</small><br><strong>Update</strong> - This incident is resolved.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>06:19</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>06:18</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p>tag:status.segment.com,2005:Incident/193692252023-12-07T15:31:02-08:002023-12-07T15:31:03-08:00Delay for a subset of real-time computed traits, audiences, journeys<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>15:31</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>11:06</var> PST</small><br><strong>Monitoring</strong> - We have identified the issue and implemented a fix. Our processing pipeline is recovering and working through a backlog of data.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>10:22</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>10:22</var> PST</small><br><strong>Investigating</strong> - We are experiencing a delay processing Engage real-time computed traits, audiences, and journeys created between July 19 2023 and Oct 26 2023. Our engineers are working on identifying a solution. No data has been lost.</p>tag:status.segment.com,2005:Incident/192473182023-11-29T21:02:07-08:002023-11-29T21:02:07-08:00Performance Degraded for Extole Platform Integration<p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>21:02</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>29</var>, <var data-var='time'>20:48</var> PST</small><br><strong>Investigating</strong> - We are experiencing a degradation in performance for our Extole Platform integration. We are currently investigating the issue.</p>tag:status.segment.com,2005:Incident/191146272023-11-10T12:00:00-08:002023-11-13T22:39:26-08:00Delay in event processing in profiles pipeline<p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>12:00</var> PST</small><br><strong>Resolved</strong> - We detected an issue with our system that resulted in some events not processed by the profiles pipeline. The issue has been addressed and we are correctly processing all events through the profiles pipeline. We are evaluating the overall impact during the affected period (between 11/10/2023 to 11/13/2023).</p>tag:status.segment.com,2005:Incident/190811882023-11-10T03:39:13-08:002023-11-10T03:39:13-08:00Cloud Sources is unable to sync data<p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>03:39</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>02:35</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'> 9</var>, <var data-var='time'>22:06</var> PST</small><br><strong>Investigating</strong> - We are actively investigating an issue that is causing failures for all cloud sources jobs in US region.</p>tag:status.segment.com,2005:Incident/190026082023-11-02T12:07:19-07:002023-11-02T12:07:20-07:00Tracking API in EU is unable to process ingress traffic.<p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>12:07</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>12:00</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are actively monitoring.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>11:32</var> PDT</small><br><strong>Identified</strong> - We have identified the issue with memory allocations and rolling out a fix.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>11:13</var> PDT</small><br><strong>Investigating</strong> - We are currently having issues with Tracking API in EU region. Customer traffic is not being processed.</p>tag:status.segment.com,2005:Incident/190011902023-11-02T09:57:32-07:002023-11-02T09:57:32-07:00Delays in streaming data delivery in US<p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>09:57</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>08:56</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'> 2</var>, <var data-var='time'>08:27</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating an issue affecting the streaming data delivery pipeline causing delays in delivery of event data.</p>tag:status.segment.com,2005:Incident/189149212023-10-24T19:23:16-07:002023-10-24T19:23:18-07:00We are investigating an issue with streaming destinations delay that is affecting most customers.<p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>19:23</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>17:56</var> PDT</small><br><strong>Update</strong> - P95 Latency trending down and is ~6 mins now and we are actively monitoring. About 5% of traffic is experiencing longer delays.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>16:06</var> PDT</small><br><strong>Update</strong> - P95 E2E latency is recovering and under 10 mins and we are actively monitoring for any issues.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>15:09</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>14:16</var> PDT</small><br><strong>Update</strong> - P95 latencies are recovering and we are actively monitoring.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>12:44</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>12:22</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>11:46</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>11:28</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>10:45</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>10:25</var> PDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>09:52</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.segment.com,2005:Incident/188457662023-10-18T21:44:14-07:002023-10-18T21:44:14-07:00Data processing delays<p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>21:44</var> PDT</small><br><strong>Resolved</strong> - The incident is now completely resolved</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>18:00</var> PDT</small><br><strong>Monitoring</strong> - The systems are coming back to a healthy state and we are actively monitoring the progress.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>17:07</var> PDT</small><br><strong>Identified</strong> - A fix has been implemented and we are actively monitoring the issue as we look at bringing the delay down.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>16:26</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>15:11</var> PDT</small><br><strong>Investigating</strong> - We are actively investigating an issue which is causing data processing delays</p>tag:status.segment.com,2005:Incident/188332892023-10-17T18:35:09-07:002023-10-17T18:35:09-07:00Audience UI failing to list audiences<p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>18:35</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>17:44</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>17:41</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>17:41</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>17:24</var> PDT</small><br><strong>Investigating</strong> - We have identified the issue with the Audience List UI page. We are actively working to mitigate the issue.</p>