SYNAQ Securemail Incident - 2016-11-01 - Mail Delay
Incident Report for SYNAQ
Postmortem

SYNAQ Root Cause Analysis

SYNAQ Securemail – Inbound and Outbound Mail Delivery Incident

Date: 2016-11-02

Summary and Impact to Customers

On Wednesday 2nd November from 07:59 – 14:31, SYNAQ Securemail experienced an inbound and outbound mail delivery incident.

The resultant impact of the event was the delay of up to 13 hours for certain inbound and outbound mail.

Root cause and Solution

The root cause of this event was due to a DDoS attack, where a large number of black listed IP addresses were attempting to send us mail. As a result, the ensuing reject logs that were sent to the MySQL server utilised all of the existing connections and mails could thus not be processed accordingly.

In order to resolve this issue, our engineers restarted the MySQL server to establish new connections to the MySQL database so that mails could once again be processed efficiently.

Remediation Actions

• We have increased our monitoring measures and controls so that this issue does not recur in the future.

• We have upgraded our Botnet detector to detect and block these attacks at the connection layer.

Posted Nov 07, 2016 - 16:58 CAT

Resolved
All mail flow has returned to normal
Posted Nov 02, 2016 - 14:31 CAT
Update
The mail delay has come down to 3 hours. They backlog of mail is still processing.
Posted Nov 02, 2016 - 12:55 CAT
Monitoring
The current queues are flushing. There may be significant delays on certain mails from last night.
Posted Nov 02, 2016 - 10:09 CAT
Identified
Engineers have Identified and resolved the issues. Please note their is a backlog of mail and you will be experiencing mail delays. We will keep you up to date with the progress of the backlog.
Posted Nov 02, 2016 - 08:53 CAT
Investigating
SYNAQ Securemail is experiencing a Mail Delay for a subset of clients. The team is currently investigating as a matter of urgency and we will feedback as soon as possible
Posted Nov 02, 2016 - 07:59 CAT
This incident affected: SYNAQ Securemail.