tag:status.lexer.io,2005:/historyLexer Status - Incident History2024-03-28T13:52:21+11:00Lexertag:status.lexer.io,2005:Incident/199630022024-02-12T13:32:13+11:002024-02-12T13:32:13+11:00Hub performance degraded<p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>13:32</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>12:51</var> AEDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>11:34</var> AEDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>11:27</var> AEDT</small><br><strong>Investigating</strong> - We're investigating reports of degraded performance for hub.lexer.io.</p>tag:status.lexer.io,2005:Incident/191770972023-11-20T19:24:32+11:002023-11-20T19:24:32+11:00Conversation processing delays<p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>19:24</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>19:18</var> AEDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>19:13</var> AEDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Nov <var data-var='date'>20</var>, <var data-var='time'>19:04</var> AEDT</small><br><strong>Investigating</strong> - We're investigating conversation processing delays for integrated Facebook and Email accounts.</p>tag:status.lexer.io,2005:Incident/191307922023-11-16T14:32:11+11:002023-11-16T14:32:11+11:00Facebook message publishing errors<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>14:32</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>14:16</var> AEDT</small><br><strong>Monitoring</strong> - Meta has indicated they are recovering from disruptions and we're observing a significantly lower rate of errors. We're continuing to monitor results.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>10:31</var> AEDT</small><br><strong>Update</strong> - Meta (Facebook) has acknowledged the issue and has assigned an engineering team to resolve the issue as quickly as possible. See more information here: https://metastatus.com/messenger.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>09:30</var> AEDT</small><br><strong>Identified</strong> - We believe the root cause to be an issue with the Facebook API as this is impacting other platforms. We have raised this issue with Facebook and we're awaiting a response or resolution.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>07:55</var> AEDT</small><br><strong>Investigating</strong> - We're currently investigating a high rate of errors for publishing replies to Facebook messages.</p>tag:status.lexer.io,2005:Incident/189840382023-11-01T14:54:42+11:002023-11-01T14:54:42+11:00Issues with Facebook Integrations: Displaying as disconnected<p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>14:54</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>13:09</var> AEDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we're monitoring results. Facebook pages integrations are now accurately displaying their connection status.</p><p><small>Nov <var data-var='date'> 1</var>, <var data-var='time'>10:29</var> AEDT</small><br><strong>Investigating</strong> - All Facebook pages integrations are displaying as 'disconnected,' while Instagram and Facebook ads integrations remain unaffected. Users are encountering difficulties when attempting re-integration, as they encounter errors in the process. It's crucial for users to refrain from re-integration attempts at this time.</p>tag:status.lexer.io,2005:Incident/188621872023-10-21T07:13:51+11:002023-10-26T14:56:52+11:00Meta experiencing API issues<p><small>Oct <var data-var='date'>21</var>, <var data-var='time'>07:13</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>15:37</var> AEDT</small><br><strong>Monitoring</strong> - We have not observed the error for over 24 hours with reports from clients that Facebook content is being consumed as normal. Although it is unconfirmed by Meta, we believe they have fixed it.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>15:28</var> AEDT</small><br><strong>Identified</strong> - Meta is experiencing an issue with their API that is impacting Respond users ability to reply to Facebook content.</p>tag:status.lexer.io,2005:Incident/186827162023-10-04T15:14:05+11:002023-10-04T15:14:05+11:00Search Performance Degraded<p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>15:14</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>10:33</var> AEDT</small><br><strong>Update</strong> - We have identified the root cause for the issue and a fix is currently being deployed.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>10:27</var> AEDT</small><br><strong>Monitoring</strong> - We're currently investigating degraded performance to search.</p>tag:status.lexer.io,2005:Incident/185404952023-09-19T16:40:55+10:002023-09-19T16:40:55+10:00Data Processing Delays - AWS network connectivity<p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>16:40</var> AEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>16:39</var> AEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>10:34</var> AEST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>08:00</var> AEST</small><br><strong>Investigating</strong> - Our data processing infrastructure is running behind which is causing inaccuracies in the reporting tools. No data has been lost and the system should be caught up shortly. AWS status page is being monitored closely for updates - https://health.aws.amazon.com/health/status</p>tag:status.lexer.io,2005:Incident/184969182023-09-05T13:00:00+10:002023-09-14T14:26:51+10:00Lexer Forms are not displaying properly<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>13:00</var> AEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p>tag:status.lexer.io,2005:Incident/184968912023-09-05T11:00:00+10:002023-09-14T14:21:59+10:00Lexer Forms are not displaying properly<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>11:00</var> AEST</small><br><strong>Resolved</strong> - We're experiencing issue on selected Lexer forms are not loading properly and are currently looking into the issue.</p>tag:status.lexer.io,2005:Incident/178105942023-07-24T08:44:47+10:002023-07-24T08:44:47+10:00Facebook page integration errors<p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>08:44</var> AEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>09:30</var> AEST</small><br><strong>Monitoring</strong> - While Meta has not issued an official resolution, we have not encountered any errors in the last 24 hours. We are continuing to monitor the situation.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>13:22</var> AEST</small><br><strong>Update</strong> - We're continuing to observe publishing errors and we've informed Meta with details to facilitate a resolution.</p><p><small>Jul <var data-var='date'>10</var>, <var data-var='time'>14:53</var> AEST</small><br><strong>Investigating</strong> - Facebook is currently experiencing issues with publishing replies to comments. Although Facebook says that your message has failed to publish, in some cases we can see that comments have in fact been created on your page.<br /><br />We recommend checking facebook.com if you encounter publishing errors for comments while this issue remains active. If your reply has been published successfully, we recommend adding an internal comment in Lexer and closing the comment.<br /><br />We're working with Meta to ensure this can be resolved as soon as possible.</p>tag:status.lexer.io,2005:Incident/178513862023-07-17T13:21:45+10:002023-07-17T13:21:45+10:00Yotpo API error<p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>13:21</var> AEST</small><br><strong>Resolved</strong> - The missing data backload is now complete, and this incident is resolved.</p><p><small>Jul <var data-var='date'>17</var>, <var data-var='time'>12:25</var> AEST</small><br><strong>Identified</strong> - Yotpo have identified the issue, and have applied a fix. We've now begun backloading the data that was missed during the time Yotpo was offline, and will confirm once this is complete.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>16:42</var> AEST</small><br><strong>Investigating</strong> - Yotpo are currently experiencing issues with their APIs, meaning Yotpo integration data for all clients is not up to date.<br />We're working with Yotpo to resolve this, and they're actively investigating.<br />There is currently no ETA on resolution but we'll provide updates from Yotpo as soon as we receive them.</p>tag:status.lexer.io,2005:Incident/176846362023-06-28T17:45:36+10:002023-06-28T17:45:36+10:00Shopify integration service disruption<p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>17:45</var> AEST</small><br><strong>Resolved</strong> - All missing data has been loaded.</p><p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>16:58</var> AEST</small><br><strong>Monitoring</strong> - A solution has been deployed. The missing Shopify data is being processed and scheduled updates will resume within the next 24 hours.<br /><br />Users can monitor their dataset updates via the Dataset feature. Anyone with Admin permission is able to access Dataset. Read more here: https://learn.lexer.io/docs/dataset-management</p><p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>15:48</var> AEST</small><br><strong>Identified</strong> - We are currently experiencing issues with our Shopify integration resulting in failures in processing data. Our team is working to implement a solution and backfill missing data.</p>tag:status.lexer.io,2005:Incident/167318182023-04-05T22:56:30+10:002023-04-05T22:56:30+10:00Scheduled Database Maintenance<p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>22:56</var> AEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>22:40</var> AEST</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>22:25</var> AEST</small><br><strong>Update</strong> - Access to hub.lexer.io is currently unavailable while we perform database maintenance.</p><p><small>Apr <var data-var='date'> 5</var>, <var data-var='time'>22:00</var> AEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>10:26</var> AEST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time. Access to hub.lexer.io will be unavailable for up to 20 minutes. Processing of customer data and conversations will be unaffected. Updates will be shared here at https://status.lexer.io/.</p>tag:status.lexer.io,2005:Incident/159701342023-01-31T16:31:02+11:002023-01-31T18:54:29+11:00Hub performance degraded<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>16:31</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>00:33</var> AEDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>17:38</var> AEDT</small><br><strong>Update</strong> - We will be undergoing scheduled database maintenance this evening to resolve this issue.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>14:55</var> AEDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>13:20</var> AEDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>13:17</var> AEDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>13:16</var> AEDT</small><br><strong>Identified</strong> - We are currently investigating this issue.</p>tag:status.lexer.io,2005:Incident/159714492023-01-31T00:30:57+11:002023-01-31T00:30:57+11:00Scheduled Database Maintenance<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>00:30</var> AEDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>00:00</var> AEDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>17:22</var> AEDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled database maintenance during this time to resolve an open incident (https://status.lexer.io/incidents/ft4qkp2byrkf). <br /><br />We expect this to take no longer than a couple of minutes, however we're suggesting to expect access to hub.lexer.io to be unavailable for up to 15 minutes.<br /><br />Please subscribe for updates before, during, and at the conclusion of this event.</p>tag:status.lexer.io,2005:Incident/159482272023-01-27T16:59:31+11:002023-01-27T16:59:31+11:00Hub performance degraded<p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>16:59</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>15:50</var> AEDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>15:45</var> AEDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.lexer.io,2005:Incident/107504032022-08-03T18:18:09+10:002022-08-03T18:18:09+10:00Attribute searches in Segment are degraded<p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>18:18</var> AEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>18:05</var> AEST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>18:04</var> AEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'> 3</var>, <var data-var='time'>17:32</var> AEST</small><br><strong>Investigating</strong> - We're aware of an issue with Attribute searches in Segment, and we're currently investigating.</p>tag:status.lexer.io,2005:Incident/106903472022-07-26T15:49:18+10:002022-07-26T15:49:18+10:00Segment search performance degraded<p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>15:49</var> AEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>12:54</var> AEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>12:24</var> AEST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>12:23</var> AEST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>12:08</var> AEST</small><br><strong>Investigating</strong> - We are currently seeing some issues in running searches in Segment, and we're looking into the cause.</p>tag:status.lexer.io,2005:Incident/96815432022-04-01T16:47:09+11:002022-04-01T16:47:09+11:00Hub performance issues<p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>16:47</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>13:07</var> AEDT</small><br><strong>Identified</strong> - We're currently experiencing some performance issues in the Hub. We've identified the problem, and are implementing a fix.</p>tag:status.lexer.io,2005:Incident/93850342022-02-24T08:10:27+11:002022-02-24T08:10:27+11:00Respond Automation issue<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>08:10</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>19:37</var> AEDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>11:45</var> AEDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>10:45</var> AEDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>10:45</var> AEDT</small><br><strong>Investigating</strong> - We're investigating an issue with Respond Automation rules, which is impacting the ability to correctly assign objects in Respond queues.</p>tag:status.lexer.io,2005:Incident/70344882021-05-19T11:29:10+10:002021-05-19T11:30:03+10:00Respond and Listen returning errors<p><small>May <var data-var='date'>19</var>, <var data-var='time'>11:29</var> AEST</small><br><strong>Resolved</strong> - This incident has been resolved. We have identified performance degradation in one of our database systems and have redeployed the system onto new hardware with additional resources available to it. Our monitoring now indicates that normal service has been restored. We will be continuing our investigation into the performance degradation to mitigate the impact of future incidents.</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>10:01</var> AEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>09:54</var> AEST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>May <var data-var='date'>19</var>, <var data-var='time'>09:30</var> AEST</small><br><strong>Investigating</strong> - We are currently investigating reports of errors when loading the Respond inbox and Listen results.</p>tag:status.lexer.io,2005:Incident/69798512021-05-14T07:05:16+10:002021-05-14T07:05:16+10:00Mailchimp Integration experiencing errors<p><small>May <var data-var='date'>14</var>, <var data-var='time'>07:05</var> AEST</small><br><strong>Resolved</strong> - This issue is now resolved. We are no longer seeing errors from Mailchimp integrations.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>11:41</var> AEST</small><br><strong>Monitoring</strong> - We have stopped receiving errors from the Mailchimp API, and both Activations and Integrations are succeeding once more. We will continue to monitor this, for the time being, and are confident this was a temporary issue.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>17:59</var> AEST</small><br><strong>Update</strong> - We are continuing to investigate this issue, we have now found that this is impacting some Activations to Mailchimp also.</p><p><small>May <var data-var='date'>12</var>, <var data-var='time'>14:32</var> AEST</small><br><strong>Investigating</strong> - We are currently investigating an issue related to errors being received when collecting data from Mailchimp. We are in contact with Mailchimp to resolve this issue.</p>tag:status.lexer.io,2005:Incident/69344002021-05-06T23:52:17+10:002021-05-06T23:52:17+10:00Facebook posts and comments failing to load<p><small>May <var data-var='date'> 6</var>, <var data-var='time'>23:52</var> AEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 6</var>, <var data-var='time'>23:41</var> AEST</small><br><strong>Monitoring</strong> - A fix has been implemented and Facebook posts and comments are loading into your inbox.</p><p><small>May <var data-var='date'> 6</var>, <var data-var='time'>18:15</var> AEST</small><br><strong>Identified</strong> - We have identified an issue related to Facebook Posts and Facebook Comments not loading into your account and we are working on a fix. Facebook posts stopped loading approximately 10am Wednesday 5th May. Once fixed, missed content will be loaded into your account.</p>tag:status.lexer.io,2005:Incident/64337502021-03-02T19:58:07+11:002021-03-02T19:58:28+11:00Monitoring conversation performance degradation<p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>19:58</var> AEDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>19:55</var> AEDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>19:53</var> AEDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>19:30</var> AEDT</small><br><strong>Investigating</strong> - We are currently investigating slow performance for Respond Inbox and Activity searches.</p>tag:status.lexer.io,2005:Incident/62934032021-03-02T19:22:23+11:002021-03-02T19:22:23+11:00Scheduled Database Maintenance<p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>19:22</var> AEDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>19:16</var> AEDT</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>19:00</var> AEDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>12:41</var> AEDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time. Access to clients.lexer.io will be temporarily disrupted for up to 15 minutes. Please reach out to Support or your Success Manager for more information.</p>