Summary and Impact to Customers

On Monday 25th May 2020 from 02:53 to 11:27, SYNAQ Securemail experienced a mail timeout incident.

The resultant impact of the event was intermittent connectivity to and from our platform when trying to send outbound mail. This caused the delay of outbound mail.

Root cause and Solution

The root cause of this event was due to the change logged on Sunday the 24th to move the Securemail environment to a new location within the Data Centre. The Securemail outbound servers all have multiple IP addresses assigned to them in order to combat blacklisting. The outbound servers utilise a script that only allows for one IP address to be brought up per server at a time. Unfortunately, this script failed, and multiple IP addresses were brought up at the same time, thus causing a conflict. This caused timeouts when trying to connect to and from these servers on port 25, causing the delay of sending mail.

In order to resolve this issue, the additional IP addresses were removed so that only a single IP address remained on each server and mail returned to normal functionality and began to flow.

Remediation Actions

• The automated scripts that failed have been removed and replaced with improved technology.

• An improved monitoring check is being built to detect this scenario earlier, should it reoccur.