SYNAQ Securemail Incident - 13/03/2019
Incident Report for SYNAQ
Postmortem

Summary and Impact to Customers

On Wednesday 13th March 2019 from 07:40am – 20:04pm SYNAQ Securemail experienced a major

incident.

The resultant impact of the event was Clients were initially unable to send and receive mail and

thereafter experienced mail delays.

Root Cause and Solution

The root cause of this event was due to exhausted DNS connections on our DNS servers as a result

of two authoritive name servers that were not responding to DNS queries for the domains that they

were the authoritive for. DNS is important to the operation of the Securemail service because it is

used to perform multiple security checks, resolve MX records and to allow connectivity to our

platform via the relevant host names.

This caused connections from our DNS server to the two authoritive name servers to remain open

as the affected servers did not respond to our queries. This caused the SYNAQ DNS servers to reach

their connection limit and as such, prevented all further DNS queries from taking place. As a result,

no mail could be sent or received.

In order to solve this issue, and whilst we were waiting for the name server provider to resolve their

incident, we implemented a temporary work-around where we redirected our DNS queries away

from the root servers to other DNS servers that still had cached results and responded with the

required domain information. This allowed mail to start flowing again and to work through the

existing backlog of mails.

At approximately 15:40pm, the name server provider resolved their incident and we rerouted our

DNS queries back to the root servers. This allowed our mail delivery to resume at a normal rate and

the backlog of mails was completed at 20:04pm.

Remediation Actions

We are investigating new methods to mitigate our connections limits from being reached

should a third party DNS name server provider experience any incidents.

Posted 4 months ago. Mar 18, 2019 - 13:09 CAT

Resolved
The issue has been resolved and no further mail delays are being experienced
Posted 4 months ago. Mar 13, 2019 - 20:04 CAT
Update
Dear Clients. The incident has been resolved. Engineers will continue to monitor the situation whilst the mail queues continue to reduce.
Posted 4 months ago. Mar 13, 2019 - 15:50 CAT
Update
Mail is flowing and the mail queues are reducing whilst our engineers continue to work on the resolution.
Posted 4 months ago. Mar 13, 2019 - 15:32 CAT
Update
Our engineers are still working on the complete resolution as a matter of urgency
Posted 4 months ago. Mar 13, 2019 - 14:38 CAT
Update
Dear Clients,

Status Update: Mail is flowing and we are currently processing a backlog of approximately 40 minutes. Our engineers will continue to monitor closely whilst working on the resolution.
Posted 4 months ago. Mar 13, 2019 - 13:21 CAT
Monitoring
Mail is flowing, processing backlog of ±40 min, engineers monitoring closely
Posted 4 months ago. Mar 13, 2019 - 11:33 CAT
Identified
Engineers have identified the problem and are implementing a fix. Mail is starting to flow again and access to the API slowly being restored
Posted 4 months ago. Mar 13, 2019 - 10:09 CAT
Update
Engineers are still working on the issue. This is being treated with the highest priority.
Posted 4 months ago. Mar 13, 2019 - 09:55 CAT
Investigating
There is currently a connectivity problem through to SYNAQ Securemail services. This will affect all sending and receiving of mail and provisioning to the API
Posted 4 months ago. Mar 13, 2019 - 08:53 CAT
This incident affected: SYNAQ Securemail.