SYNAQ Cloud Mail Incident 20-09-2018
Incident Report for SYNAQ
Postmortem

Summary and Impact to Customers

On Thursday 20 th September 2018 from 09:29 to 13:33, SYNAQ Cloud Mail experienced a mail

authentication and mail delay incident.

The resultant impact of the event was:

  • The Admin and Webmail Consoles were inaccessible,

  • Mail Clients could not access the platform, and;

  • The resultant delay of inbound and outbound emails

Root cause and Solution

The root cause of this event was due to a scheduled change to increase the authentication

restrictions for the SYNAQ Cloud Mail solution. The addition of these changes to the platform

resulted in an increased amount of queries being sent to the LDAP servers which caused an overload

and an inability to process any authentication and mail delivery requests. More specifically, the

Admin, Webmail Consoles and Mail Clients were unable to query the LDAP servers for

authentication and information purposes which caused these consoles to become inaccessible and

further resulted in the delay of inbound and outbound emails.

In order to resolve this issue, we rolled back the Cloud Mail change, which was completed by

10:53am. This automatically resolved the Admin, Webmail Console and Mail Client authentication

issue. However, due to the large email backlog that accumulated during this time, mail flow only

resumed to normal levels at 13:33pm.

Remediation Actions

We will be developing more comprehensive and stringent LDAP testing procedures as part

of future Cloud Mail changes to ensure that any further incidents of this nature are avoided.

Posted Sep 27, 2018 - 10:45 CAT

Resolved
The issue is now resolved and all systems are fully operational.
Posted Sep 20, 2018 - 13:33 CAT
Update
The Cloud Mail queue backlog is reducing and SYNAQ engineers will continue to monitor the situation.

We will provide further updates in due course.
Posted Sep 20, 2018 - 12:43 CAT
Monitoring
SYNAQ engineers have implemented a fix and are currently monitoring. Mail queues are currently are decreasing but there are still currently delays.
Posted Sep 20, 2018 - 10:53 CAT
Identified
SYNAQ engineers have identified the cause for the partial outage. Engineers are implementing corrective measures to resolve the issue. Further updates to follow.
Posted Sep 20, 2018 - 10:24 CAT
Investigating
We are currently experiencing mail delays and very high load. We are currently investigating the issue and will update clients as soon as possible with further feedback once it becomes available.
Posted Sep 20, 2018 - 09:29 CAT
This incident affected: SYNAQ Cloud Mail.