Summary and Impact to Customers

On Thursday 18th May 2016, SYNAQ Secure Mail experience a degraded performance incident from 12h05 until 16h45.

The impact of the event resulted in up to a 5 hour delays of mail on the SYNAQ Secure Mail platform between 12h05 to 16h45.

Root cause and Solution

The root cause of this event was a local Mail server check for Phishing being updated with bad data which caused a huge delay in the processing of mail when those mails were checked against this configuration.

The Mail servers all have their own local copy of this Phishing check and automatically update this information from an upstream provider. The old method used to be that this information was stored locally on one of the database servers and then pushed to the Mail servers. These database servers pushed outdated information to the Mail servers corrupting their data.

This configuration was removed from the databases servers and the local phishing check information on the mail servers then updated from the upstream provider.

Once this updated, mail was able to flow again and the backlog on the server had to be worked through.

Remediation Actions

 All configurations have been removed from the centralized database servers

 A new Mail server has been added to increase capacity to allow for faster processing of