Summary and Impact to Customers

On Wednesday 5th June from 10:55am to 7:27pm, SYNAQ Branding experienced a major service incident which affected all elements of the email branding service.

The resultant impact of the event was that a Branding Clients experienced malformed mails or mails with the body being removed and eventually no branding at all.

Root Cause and Solution

The root cause of this event was due to a code release to correct a line break format issue for mails destined for a certain type of ticket management system, where the branding was not showing correctly. We enforced line breaks on every mail to be a CRLF format type and these in turn caused issues with certain outbound MTA’s that were expecting the line breaks to be in an LF format.

As a result of the above, formatting issues occurred with the message body of numerous mails going through the platform.

In order to resolve this issue, the Branding service was disabled to allow mail to be delivered in its original format. Thereafter, the new code was rolled back to the previous stable version and Branding was then re-enabled for all clients and mail began to be correctly branded.

Remediation Actions

• Additional end-to-end testing methods were introduced to proactively identify this issue for any further releases.