Summary and Impact to Customers
On Wednesday 11th May 2016, SYNAQ Secure Mail experience a degraded performance incident from 12h24 until 23h59.
The impact of the event resulted in up to 8 hour delays of mail on the SYNAQ Secure Mail platform between 12h24 to 23h59.
Root cause and Solution
The root cause of this event was a distributed denial of service attack on the SYNAQ Securemail environment combined with routine code changes. At 11h40 the platform was hit with a unique spoofing attack where hundreds of thousands of emails from thousands of different IP addresses were sent to the platform.
The platform blocked these messages from reaching the intended recipients ensuring no compromised spoofed mails with Malicious Malware was received by the end users and ensuring the protection of the SYNAQ Securemail client base.
With these messages being blocked however bounce messages were sent back indicating the message was not delivered. With the continuous attack the bounces served to double the email load on the platform from this particular attack.
This load would have been successfully handled by the platform had a routine code push not been pushed to that database at exactly the same time as the attack was taking place. The code push required the dropping of a table on a test database and with the load on the databases already from the spoofing attack it caused the database to go unresponsive.
Once the database had recovered the platform was able to resume normal operations and work through the backlog that built up in the queues.
Over 5000 addresses have been blocked from sending mails to the platform as part of this spoofing attack
Development deployment process to be reviewed to ensure no code changes that can affect production in any way to be deployed during business hours