tag:status.paubox.com,2005:/historyPaubox System Status Status - Incident History2024-03-29T05:23:07ZPaubox System Statustag:status.paubox.com,2005:Incident/201584532024-03-04T23:03:15Z2024-03-04T23:03:15ZSMTP Server Solution Shows "No Delivery Status Found"<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>23:03</var> UTC</small><br><strong>Resolved</strong> - Mail is flowing normally and previously delayed emails should all have been delivered. Please contact support for any exceptions.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>20:48</var> UTC</small><br><strong>Monitoring</strong> - Paubox hosted mail and SMTP customers are seeing ~1-2 hour delays for outgoing mail to be delivered currently, with delay time continuing to decrease.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>17:41</var> UTC</small><br><strong>Update</strong> - Customers on m.paubox.com (smtp server solution, hosted webmail) are experiencing mail delays in excess of 10 minutes. While we have identified the root cause, we are still working to correct it.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>16:37</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>16:15</var> UTC</small><br><strong>Identified</strong> - Engineering is working to correct an issue where emails sent via SMTP Server Solution show in the logs as "no delivery status found." We will update with whether emails need to be re-queued once the solution is in place.</p>tag:status.paubox.com,2005:Incident/199950122024-02-15T21:55:49Z2024-02-15T21:55:49ZUsers unable to sign into Paubox Admin App<p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>21:55</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>21:27</var> UTC</small><br><strong>Monitoring</strong> - Access has been restored. We will continue to monitor this issue.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>21:25</var> UTC</small><br><strong>Identified</strong> - Customers are currently unable to log into our admin app and are receiving 504 errors. We have identified the source of the problem and will work to resolve this as quickly as possible.</p>tag:status.paubox.com,2005:Incident/190480112023-11-07T23:11:40Z2023-11-07T23:11:40ZOutbound & Inbound Mail Delays<p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>23:11</var> UTC</small><br><strong>Resolved</strong> - Engineering has confirmed all mail queues have returned to normal levels.<br /><br />We’re no longer seeing any inbound or outbound mail delays and believe the issue to be resolved.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>20:04</var> UTC</small><br><strong>Monitoring</strong> - At this time, mail queues have returned to normal and there are no outbound mail delays.<br /><br />We’re still seeing delays for mx1.paubox.com inbound mail currently.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>19:13</var> UTC</small><br><strong>Identified</strong> - Engineering has identified the cause of delays for inbound and outbound mail and deployed a fix to address this.<br /><br />At this time, there are no delays for inbound mail routing through mx.paubox.com and mx2.paubox.com. <br /><br />We’re still seeing delays for mx1.paubox.com inbound mail and all outbound mail currently.</p><p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>17:07</var> UTC</small><br><strong>Investigating</strong> - We are investigating delays in both outbound and inbound mail servers.</p>tag:status.paubox.com,2005:Incident/184807812023-09-13T05:48:34Z2023-09-13T05:48:34ZMX1 Mail Delay affecting limited group of customers<p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>05:48</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>19:21</var> UTC</small><br><strong>Monitoring</strong> - Changes have been applied to the inbound security pipeline. The engineering team is closely monitoring the results. Delays have been resolved.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>16:46</var> UTC</small><br><strong>Identified</strong> - Engineering team has identified the cause of the bottleneck leading to slower than usual mail processing time. The root cause is currently being confirmed. Customers on MX1 are experiencing mail delays of ~9 min and falling.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>16:03</var> UTC</small><br><strong>Update</strong> - While customers on MX1 may be experiencing mail delays up to ~40min, we are seeing the queue recover. We're continuing to investigate.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>14:55</var> UTC</small><br><strong>Investigating</strong> - We are investigating a mail delay on the MX1 server cluster. Only customers on this cluster have been affected.</p>tag:status.paubox.com,2005:Incident/184248612023-09-07T20:20:06Z2023-09-07T20:20:06ZNew users unable to access their dashboard<p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>20:20</var> UTC</small><br><strong>Resolved</strong> - After monitoring throughout the afternoon with no issues, this issue is resolved.</p><p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>18:39</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>17:36</var> UTC</small><br><strong>Investigating</strong> - New users who have not previously authenticated with two factor authentication (2FA) are unable to log into their dashboard. Existing users are able to log in without issues.</p>tag:status.paubox.com,2005:Incident/182134232023-08-21T19:36:52Z2023-08-21T19:36:52ZMail delays for some customers<p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>19:36</var> UTC</small><br><strong>Resolved</strong> - The mail delay has been resolved.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>19:02</var> UTC</small><br><strong>Monitoring</strong> - We are monitoring a traffic spike on one of our inbound server clusters. The queue is currently unspooling and there is a ~10min mail delay.</p>tag:status.paubox.com,2005:Incident/181865432023-08-18T20:40:39Z2023-08-18T20:40:39ZInbound mail delays for some customers<p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>20:40</var> UTC</small><br><strong>Resolved</strong> - Mail is flowing as expected and all systems are operational.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>19:02</var> UTC</small><br><strong>Monitoring</strong> - Delay has been reduced to ~30 seconds. We'll continue to monitor.</p><p><small>Aug <var data-var='date'>18</var>, <var data-var='time'>16:59</var> UTC</small><br><strong>Investigating</strong> - Engineering is monitoring ~20 inbound mail delays for some customers. We're expecting the queue will clear within the next hour.</p>tag:status.paubox.com,2005:Incident/181644462023-08-16T18:02:49Z2023-08-16T18:02:49ZMail delays for some customers<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>18:02</var> UTC</small><br><strong>Resolved</strong> - No additional issues have been observed. All systems are operational.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>17:10</var> UTC</small><br><strong>Monitoring</strong> - Delays have resolved, we are monitoring server health for the next hour.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>16:43</var> UTC</small><br><strong>Identified</strong> - We have identified a ~20 min mail delay for a limited set of customers. We're in the process of correcting this.</p>tag:status.paubox.com,2005:Incident/181530392023-08-15T23:32:13Z2023-08-15T23:32:13Z504 errors on inbound and outbound email<p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>23:32</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>20:06</var> UTC</small><br><strong>Update</strong> - We're deploying a hotfix to roll back an update that caused email corruption. Emails sent between 10:00am - 10:05am ET were potentially affected.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>14:24</var> UTC</small><br><strong>Monitoring</strong> - Customers reported issues with 504 bad gateway emails this morning - we located and corrected the problem. We'll be monitoring for the next hour.</p>tag:status.paubox.com,2005:Incident/181530222023-08-15T20:03:11Z2023-08-15T20:03:11ZCustomers unable to access admin panel<p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>20:03</var> UTC</small><br><strong>Resolved</strong> - We discovered and resolved an underlying problem with anEC2 instance used to host our database servers. All users are able to access the admin panel as expected.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>14:58</var> UTC</small><br><strong>Identified</strong> - Access has been restored for most users - you may receive 2 working 2fa codes as an iPhone user. Please contact support if you are still unable to access your dashboard.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>14:22</var> UTC</small><br><strong>Investigating</strong> - You may be receiving multiple MFA codes - we are investigating this.</p>tag:status.paubox.com,2005:Incident/181530592023-08-15T15:59:47Z2023-08-15T15:59:47ZInbound mail delays for some customers<p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>15:59</var> UTC</small><br><strong>Resolved</strong> - Delays have been resolved.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>14:59</var> UTC</small><br><strong>Monitoring</strong> - Delays have been reduced to ~5m. We'll continue to monitor.</p><p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>14:26</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.paubox.com,2005:Incident/181418372023-08-15T03:05:45Z2023-08-15T03:05:45ZMail Delays for some customers<p><small>Aug <var data-var='date'>15</var>, <var data-var='time'>03:05</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>23:08</var> UTC</small><br><strong>Monitoring</strong> - While mail servers are operational, the queue is still unspooling and inbound mail flow is still impacted. We'll continue to monitor.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>23:05</var> UTC</small><br><strong>Update</strong> - During today's event contents of received / delivered mail may have been lost between the hours of ~8am and 12pm ET. We recommend resending and/or requesting contacts resend mail from this time period.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>19:55</var> UTC</small><br><strong>Identified</strong> - We've discovered a networking problem with our infrastructure provider (AWS) and have taken the necessary steps to address it.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>17:23</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate the cause of inbound and outbound mail delays. Customers have reported receiving bounce backs and 404 errors. We'll share information on redelivery once we identify the root of the problem.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>16:07</var> UTC</small><br><strong>Update</strong> - Confirming mail delays are affecting our outbound servers. We are continuing to investigate.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>15:37</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>14:23</var> UTC</small><br><strong>Investigating</strong> - We are investigating inbound mail delays.</p>tag:status.paubox.com,2005:Incident/181422852023-08-14T23:09:50Z2023-08-14T23:09:50ZCustomers receiving html "bad gateway" emails<p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>23:09</var> UTC</small><br><strong>Resolved</strong> - During today's event contents of received / delivered mail may have been lost between the hours of ~8am and 12pm ET. We recommend resending and/or requesting contacts resend mail from this time period.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>22:05</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for further issues.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>16:51</var> UTC</small><br><strong>Monitoring</strong> - Customers continue to report 504 bad gateway emails; a reminder that this will likely continue over the next 3 hours as our mail queue unspools.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>15:04</var> UTC</small><br><strong>Identified</strong> - Some customers have reported receiving html emails with "bad gateway" errors. This was caused by database timeouts on our footer microservice. We have temporarily disabled it to investigate the source of those timeouts. However, as the mail queue unspools, you may receive emails like this.</p>tag:status.paubox.com,2005:Incident/181413162023-08-14T20:51:10Z2023-08-14T20:51:10ZPlatform Login Issues<p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>20:51</var> UTC</small><br><strong>Resolved</strong> - Login access is fully restored.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>20:15</var> UTC</small><br><strong>Monitoring</strong> - We've verified admins are able to log into their dashboards but will continue to monitor this issue.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>16:52</var> UTC</small><br><strong>Identified</strong> - We are working on a hot fix for this issue.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>14:49</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate the issue.</p><p><small>Aug <var data-var='date'>14</var>, <var data-var='time'>13:36</var> UTC</small><br><strong>Investigating</strong> - We are investigating issues logging into our admin app.</p>tag:status.paubox.com,2005:Incident/180207832023-08-01T22:36:21Z2023-08-01T22:36:21Z8/1 Inbound Mail Delays<p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>22:36</var> UTC</small><br><strong>Resolved</strong> - Mail queues have returned to normal and the incident is resolved.</p><p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>19:35</var> UTC</small><br><strong>Monitoring</strong> - Our improvement is live. We'll continue to monitor mail queues.</p><p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>18:27</var> UTC</small><br><strong>Identified</strong> - We are releasing a performance update to inbound security. While mail is flowing, a limited number of customers could still be experiencing mail delays. We will continue to monitor the mail queues until this is resolved.</p><p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>16:14</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate the issue.</p><p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>14:09</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating inbound mail delays.</p>tag:status.paubox.com,2005:Incident/179021292023-07-24T14:36:13Z2023-07-24T14:36:13ZIssues with International 2FA Codes<p><small>Jul <var data-var='date'>24</var>, <var data-var='time'>14:36</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>21:25</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>16:29</var> UTC</small><br><strong>Investigating</strong> - Paubox engineering is currently investigating reported issues around a limited set of users unable to receive 2FA codes to international numbers.</p>tag:status.paubox.com,2005:Incident/178587862023-07-15T00:09:59Z2023-07-15T00:09:59ZIntermittent UI Failures in Paubox Marketing<p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>00:09</var> UTC</small><br><strong>Resolved</strong> - Paubox confirms the issue to be resolved</p><p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>00:08</var> UTC</small><br><strong>Monitoring</strong> - The root cause has been identified and the team performed a configuration update to resolve.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>23:26</var> UTC</small><br><strong>Investigating</strong> - The engineering team is currently investigating intermittent failures with Paubox Marketing pages.</p>tag:status.paubox.com,2005:Incident/177721192023-07-06T16:47:31Z2023-07-06T16:47:31ZPaubox Marketing Dashboard Errors<p><small>Jul <var data-var='date'> 6</var>, <var data-var='time'>16:47</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>19:26</var> UTC</small><br><strong>Monitoring</strong> - The hotfix has been deployed, Paubox engineering confirms it's working as expected. The team will continue to monitor for the next few hours.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>17:49</var> UTC</small><br><strong>Update</strong> - The engineering team has confirmed that the issue is limited only to the Paubox Marketing API endpoints as documented at https://docs.paubox.com. Paubox Dashboard users are not affected by this problem. If you are attempting to access the dashboard and are having difficulty, you should log out, clear cache, and retry. A hotfix for the Paubox Marketing API is currently being tested.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>16:08</var> UTC</small><br><strong>Identified</strong> - Paubox engineering team has identified an misconfiguration on the Paubox Marketing backend and are working to resolve.</p>tag:status.paubox.com,2005:Incident/177129142023-07-02T07:00:38Z2023-07-02T07:00:38ZPaubox Email API System Upgrades<p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>07:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'> 2</var>, <var data-var='time'>04:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>20:47</var> UTC</small><br><strong>Scheduled</strong> - On Saturday, July 1 between 9PM-12AM PT we are performing scheduled maintenance for our Paubox Email API and Paubox Marketing solutions that requires a temporary outage of both services. During this time, emails sent via Paubox Email API or Paubox Marketing may not be delivered.<br /><br />Please plan accordingly to ensure that your scheduled emails are not sent during this maintenance period.</p>tag:status.paubox.com,2005:Incident/173715682023-06-25T07:00:09Z2023-06-25T07:00:09ZPlanned Maintenance for Paubox Email API<p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>07:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>25</var>, <var data-var='time'>04:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>26</var>, <var data-var='time'>00:17</var> UTC</small><br><strong>Scheduled</strong> - On Saturday, June 24 between 9PM-12AM PT we are performing scheduled maintenance for our Paubox Email API and Paubox Marketing solutions that requires a temporary outage of both services. During this time, emails sent via Paubox Email API or Paubox Marketing may not be delivered. <br /><br />Please plan accordingly to ensure that your scheduled emails are not sent during this maintenance period.</p>tag:status.paubox.com,2005:Incident/176442022023-06-21T17:43:00Z2023-06-21T17:43:00ZPaubox Admin Dashboard pages not Loading<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>17:43</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>16:26</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>16:07</var> UTC</small><br><strong>Identified</strong> - Paubox engineering team has identified a problem with the Paubox Admin Dashboard (including the Paubox Marketing UI) and is working to resolve.</p>tag:status.paubox.com,2005:Incident/176432982023-06-21T16:05:06Z2023-06-21T16:05:06ZEmail API 500 Errors<p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>16:05</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>21</var>, <var data-var='time'>14:41</var> UTC</small><br><strong>Investigating</strong> - Paubox Email API is currently experiencing temporary 500 errors. Our engineering team is investigating the problem.</p>tag:status.paubox.com,2005:Incident/175654952023-06-13T23:38:23Z2023-06-13T23:38:23ZMail Delays<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>23:38</var> UTC</small><br><strong>Resolved</strong> - Mail delays, including mail caught in the queue, have cleared up and all systems are operational.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>22:39</var> UTC</small><br><strong>Monitoring</strong> - Inbound mail queues have returned to normal levels. We are no longer observing delays. We will continue to monitor.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>22:26</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>22:16</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>21:53</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue with inbound mail delays affecting some customers.</p>tag:status.paubox.com,2005:Incident/174251832023-05-30T23:10:24Z2023-05-30T23:10:24ZImage Uploads on Paubox Marketing<p><small>May <var data-var='date'>30</var>, <var data-var='time'>23:10</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>30</var>, <var data-var='time'>20:39</var> UTC</small><br><strong>Identified</strong> - Paubox engineering has identified an error regarding image uploads with Paubox Marketing and are working to resolve</p>tag:status.paubox.com,2005:Incident/172792642023-05-17T20:40:46Z2023-05-17T20:40:46ZDelivery Delays<p><small>May <var data-var='date'>17</var>, <var data-var='time'>20:40</var> UTC</small><br><strong>Resolved</strong> - Mail delays, including mail caught in the queue, have cleared up and all systems are operational.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>19:32</var> UTC</small><br><strong>Monitoring</strong> - Mail queues have returned to normal levels. We are no longer observing significant delays. We will continue to monitor.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>16:42</var> UTC</small><br><strong>Identified</strong> - Engineering has identified a bottleneck with our inbound security stack and has temporarily disabled the problematic component.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>15:50</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate the issue. More details to follow.</p><p><small>May <var data-var='date'>17</var>, <var data-var='time'>14:37</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing delays with inbound delivery. Our engineering team is investigating the issue.</p>