Failure information

Backlog failed to send emails to a portion of users from July 12, 2016 at 5:00am to July 28, 2016 at 3:24am (UTC).

[Summary]

One of Backlog’s mail servers encountered a failure. A user reported that they were no longer receiving emails from Backlog. After our investigation into this issue, we discovered the server had stopped. The server was restarted and is now operating normally.

[Impact Range]

One of Backlog’s mail servers failed to send emails to a portion of users. All forms of emails from the Backlog application was affected.

[Detail]

• July 12, 2016 at 5:00am (UTC): A mail server encountered a failure.
• July 27, 2016 at 7:12am (UTC): A user reported that they were no longer receiving emails from Backlog.
• July 28, 2016 at 2:30am (UTC): We discovered the server failure.
• July 28, 2016 at 3:24am (UTC): We restarted the server and the emails in the server’s mail queue were sent.

[Cause]

The mail server that failed was not included in the regular testing schedule for email queues. This is why the failure was not detected immediately.

[Resolution]

1. Monitor all mail queues across all servers.
2. Regularly restart mail servers and monitor sending logs closely.

 

We sincerely apologize for any delayed emails and email blasts that you may have encountered. We are continuing to work diligently to prevent any mail issues in the future.