SYNAQ Securemail mail delays 11-05-2016 Partial
Incident Report for SYNAQ
Postmortem

Summary and Impact to Customers

On Wednesday 11th May 2016, SYNAQ Secure Mail experience a degraded performance incident from 12h24 until 23h59.

The impact of the event resulted in up to 8 hour delays of mail on the SYNAQ Secure Mail platform between 12h24 to 23h59.

Root cause and Solution

The root cause of this event was a distributed denial of service attack on the SYNAQ Securemail environment combined with routine code changes. At 11h40 the platform was hit with a unique spoofing attack where hundreds of thousands of emails from thousands of different IP addresses were sent to the platform.

The attacker spoofed numerous addresses on the SYNAQ Securemail environment with the from address matching the recipient. The spoofed mails also contained JavaScript attachments which contained Malicious Malware intended to acquire sensitive information from the end users this information was sent to.

The platform blocked these messages from reaching the intended recipients ensuring no compromised spoofed mails with Malicious Malware was received by the end users and ensuring the protection of the SYNAQ Securemail client base.

With these messages being blocked however bounce messages were sent back indicating the message was not delivered. With the continuous attack the bounces served to double the email load on the platform from this particular attack.

This load would have been successfully handled by the platform had a routine code push not been pushed to that database at exactly the same time as the attack was taking place. The code push required the dropping of a table on a test database and with the load on the databases already from the spoofing attack it caused the database to go unresponsive.

Once the database had recovered the platform was able to resume normal operations and work through the backlog that built up in the queues.

Remediation Actions

Over 5000 addresses have been blocked from sending mails to the platform as part of this spoofing attack

Development deployment process to be reviewed to ensure no code changes that can affect production in any way to be deployed during business hours

Posted Nov 07, 2016 - 17:00 CAT

Resolved
All mail delays have recovered
Posted May 12, 2016 - 00:02 CAT
Update
The delay is currently still at 6 hours but the backlog is starting to clear at a faster rate now
Posted May 11, 2016 - 18:34 CAT
Update
Just an update on the current delays. There is currently a 6 hour delay. Further updates to follow as the queue decreases.
Posted May 11, 2016 - 17:43 CAT
Identified
There is a 3 hour delay for a small subset of users on the SYNAQ Securemail platform. The issue has been resolved but there is a back log of mail to work through.
Posted May 11, 2016 - 15:03 CAT
This incident affected: SYNAQ Securemail.