SYNAQ Branding Incident - 05/06/2019
Incident Report for SYNAQ
Postmortem

Summary and Impact to Customers

On Wednesday 5th June from 10:55am to 7:27pm, SYNAQ Branding experienced a major service incident which affected all elements of the email branding service.

The resultant impact of the event was that a Branding Clients experienced malformed mails or mails with the body being removed and eventually no branding at all.

Root Cause and Solution

The root cause of this event was due to a code release to correct a line break format issue for mails destined for a certain type of ticket management system, where the branding was not showing correctly. We enforced line breaks on every mail to be a CRLF format type and these in turn caused issues with certain outbound MTA’s that were expecting the line breaks to be in an LF format.

As a result of the above, formatting issues occurred with the message body of numerous mails going through the platform.

In order to resolve this issue, the Branding service was disabled to allow mail to be delivered in its original format. Thereafter, the new code was rolled back to the previous stable version and Branding was then re-enabled for all clients and mail began to be correctly branded.

Remediation Actions

• Additional end-to-end testing methods were introduced to proactively identify this issue for any further releases.

Posted about 1 month ago. Jul 23, 2019 - 15:38 CAT

Resolved
Dear Clients,

All branding services are fully restored and all mails are once again being correctly branded.
Posted 3 months ago. Jun 05, 2019 - 19:27 CAT
Identified
Dear Clients,

The issue has been identified and engineers are currently working on the resolution.
Posted 3 months ago. Jun 05, 2019 - 17:24 CAT
Investigating
Dear Clients,

Please note that SYNAQ Branding has been temporarily disabled to alleviate the mail corruption issues identified at 11:00 this morning. As a result, no branding will be added to mails whilst this issue is being investigated.

Engineers are currently investigating the cause of these issues and are working on a speedy resolution.
Posted 3 months ago. Jun 05, 2019 - 15:07 CAT
This incident affected: SYNAQ Branding.