SYNAQ Cloud Mail Incident - 11/06/2019
Incident Report for SYNAQ
Postmortem

Summary and Impact to Customers

On Monday 10th June from 1:38pm to Tuesday 11th June 4:55pm, SYNAQ Cloud Mail experienced a minor service incident which caused mail delays for a subset of clients.

The resultant impact of the event was that certain users experienced mail delays of up to 2 hours for some of their mail.

Root Cause and Solution

The root cause of this event was due to a failed controller on a backend storage device. As a result of the failure, all data accessing the storage had to failover to a single data path. This failover caused an abnormal level of usage over the single data path, resulting in increased latency times for data reads and writes, which in turn caused mail delivery to users mailboxes to be delayed.

In order to resolve this issue, the controller was replaced and dual paths were restored. As a result, mail delivery performance was restored to the affected users.

Remediation Actions

• Whilst the built-in redundancy on the storage array worked to prevent a complete loss of access to mailbox data, SYNAQ engineers are working with our storage vendor to increase the single data path capacity to handle failover load without impacting users.

Posted 3 months ago. Jul 23, 2019 - 16:01 CAT

Resolved
Dear Clients

Mail flow is optimal again
Posted 4 months ago. Jun 11, 2019 - 16:55 CAT
Monitoring
Dear Clients,

The Cloud Mail incident has been resolved. Mail queues are currently working through the backlog and engineers are monitoring
Posted 4 months ago. Jun 11, 2019 - 15:53 CAT
Investigating
Dear Clients,

SYNAQ Cloud Mail is experiencing mail delays. We are aware of the issue and we are working on the resolution as a matter of urgency.
Posted 4 months ago. Jun 11, 2019 - 08:53 CAT
This incident affected: SYNAQ Cloud Mail.