SYNAQ Securemail mail delays 18-05-2016 affecting a subset of users
Incident Report for SYNAQ

Summary and Impact to Customers

On Thursday 18th May 2016, SYNAQ Secure Mail experience a degraded performance incident from 12h05 until 16h45.

The impact of the event resulted in up to a 5 hour delays of mail on the SYNAQ Secure Mail platform between 12h05 to 16h45.

Root cause and Solution

The root cause of this event was a local Mail server check for Phishing being updated with bad data which caused a huge delay in the processing of mail when those mails were checked against this configuration.

The Mail servers all have their own local copy of this Phishing check and automatically update this information from an upstream provider. The old method used to be that this information was stored locally on one of the database servers and then pushed to the Mail servers. These database servers pushed outdated information to the Mail servers corrupting their data.

This configuration was removed from the databases servers and the local phishing check information on the mail servers then updated from the upstream provider.

Once this updated, mail was able to flow again and the backlog on the server had to be worked through.

Remediation Actions

 All configurations have been removed from the centralized database servers

 A new Mail server has been added to increase capacity to allow for faster processing of

Posted about 1 year ago. Nov 07, 2016 - 16:59 CAT

Resolved
All may delays have been restored
Posted over 1 year ago. May 18, 2016 - 16:45 CAT
Identified
SYNAQ Securemail is currently experiencing mail delays on a subset of users. The issue has been resolved and the backlog of mail is clearing. We apologize for any inconvenience this may have caused.
Posted over 1 year ago. May 18, 2016 - 12:05 CAT