Early on the 30th September we became aware of an incident that affected timely delivery of inbound email to customers who host their email on a Hosted Microsoft Exchange platform. The delays to inbound email were initially reported to be only an hour or so however most customers reported delays of several hours; only at around 3:30 PM did we receive an update from the supplier which suggested the issue should soon be resolved.
Clearly, the outage has been disruptive. Customers that rely on their email have been particularly affected but regardless of how badly you’ve been impacted we’re sorry that you’ve experienced these problems.
The suppliers note about the incident can be seen at https://status.cloud.market/incident/567063
Update 8:45 AM 1st October 2024
By 8:30 AM on 1st October we had performed testing and the resolution of the fault is a mixed bag. Some emails are getting delivered immediately, some emails continue to be delayed, and some customers with a backlog of inbound email are still waiting for them. We have spoken to the provider who are yet to give us an update. We will be pressing them again for an update at 10AM.
Update 9:00 AM 1st October 2024
Though service is not yet back to normal, the suppliers have confirmed that the backlog of emails will de delivered in due course.
Update 10:15 AM 1st October 2024
We have chased the supplier and although there is no official update, the engineers are reasonably satisfied that all inbound email is still pending delivery, there have been no reports of lost email.
Update 12:20 PM 1st October 2024
The latest statement from the upstream provider (see incident link above) suggests that the situation is returning to normal, and the queued email is now being delivered with consistent reductions in delay. Do note the point about the timestamps on the inbound email being different from the timestamp it arrived at the inbound mail queue, if you need any help exploring the email headers for the actual time an email was sent to you please get in touch.
Update 2:20 PM 1st October 2024
Though there are no further updates on the suppliers status page we have pressed them for an update. The outcome of this was that they will make another announcement before close of business today at 5:30 PM. The engineers we spoke to also said it would be “unusual” if any customers has not started to receive older emails. From our side we’ll be assessing the situation in the next couple if hours and trying to determine where problems remain.
Update 3:30 PM 1st October 2024
Again there are no further updates from the supplier. Our MD has now personally called most affected customers to see what the current situation is, most users are reporting good mail-flow on newer inbound email and are also reporting older emails have arrived, but a few users are saying some expected emails have not yet arrived. We have told the upstream provider we need a clear statement from them as to when the mail queue has returned to normal levels, once we have this we can investigate non deliveries. The supplier is expected to issue a statement before 5:30 today.
Update 3:40 PM 1st October 2024
We finally have a more detailed update from the supplier, see the link above. Our summary is as follows:
- The service is improving, with delays of around 30 minutes for new incoming emails.
- The supplier is still investigating potentially stuck emails. If you’re missing an email, it might be in this category.
- Remember that delayed emails will show the correct send time in the email headers, but the receipt time in your mailbox will be timestamped with the delayed time of arrival.
In some circumstances, the sending server (not the receiving server) may have held onto emails during the issue. If this is the case, you will either receive the delayed emails or the sender will get an NDR (non delivery receipt) otherwise known as a bounce, i.e. the original sender should be notified.
Our next update will be tomorrow morning.
Update 7:30 AM 2nd October 2024
Though there is no update from the supplier today, we have performed our own testing and we believe that normal mail flow has now resumed, with emails arriving in mailboxes within seconds of them being sent. We assume that the inbound mail queue has also reduced to normal levels but we will be confirming this with the supplier today.
Update 11:30 AM 2nd October 2024
The latest update from the supplier suggests that normal service has resumed for inbound mail, but they have identified and ringfenced older email yet to be delivered and are moving this email into delivery queues in a way that does not significantly impact normal service for newer emails.
Update 9:10 AM 3rd October 2024
The latest update from the supplier is more of the same. Our experience is that inbound email is being delivered in a timely manner but there remain a number of older emails that have still not been delivered which the supplier is trying to remediate. Please note we are actively looking at other solutions for Hosted Exchange customers which we’ll be elaborating on once this issue has been fully resolved.
Update 10:00 AM 3rd October 2024
We have pressed the supplier for a more meaningful update who have repeated the last statement. We explicitly asked if there is likely to be any loss of as yet undelivered inbound email and they responded that the behaviour will be as expected – either emails will be delivered or an NDR be sent back to the sender.
Update 11:09 AM 3rd October 2024
The suppliers point about inbound email delays being largely fixed appears to be less than accurate in some situations. Some emails sent yesterday at around noon arrived at 3AM today for example, others are getting delivered almost immediately. Please note we are already looking at options for other mail servers, these will roughly be:
- Migrate to Microsoft 365
- Migrate to Hosted Exchange Elsewhere
- Migrate to on Premise Email Server
We’ll be in touch about these before the end of October.
Update 12:05 PM 3rd October 2024
As per the suppliers last update, customers who are experiencing delays with inbound email should raise a ticket, this will be marked as a Non Chargeable Tracking Ticket. You must tell us who the email is from and what time you received it. We will then inspect the email headers to determine the actual sent time and if the delay is anything more than an hour this will escalated to the supplier as a specific incident.
Update 09:10 AM 4th October 2024
Our experience is more of the same today – all new inbound emails are being delivered as expected but we have had one report that an email sent to our customer late on 30th September has still not arrived, they only know this because the sender has chased / resent the original email. If you have any evidence of emails that have not arrived (and we need hard evidence to include date and time it was sent to you an who from) please let us know so we can get it traced.
Update 10:15 AM 7th October 2024
We are pleased to report that – after nearly a week of updates – the upstream mailbox supplier has confirmed that all of the older queued email has now been delivered. They will continue to monitor the situation. We will be waiting for the supplier to issue the root cause analysis and will contact all customers individually with recommendations as to how this may be avoided in the future.