Email Queue Processing
Incident Report for Rev.io, LLC
Postmortem

The Email Queue’s design was such that it was single-threaded per client, and as the number of clients has grown, it has become more likely that one client (or many clients on a busy day like the beginning of the month) can slow down processing for others.

In order to address this, we redesigned & segmented this service into our latest design which uses low & high priority queues based on volume and prevents clients from slowing processing for each other.

This new version has completed testing and has been deployed and is live into production, and we are confident this resolves the issue going forward.

Posted Aug 13, 2018 - 10:47 EDT

Resolved
This incident has been resolved.
Posted Aug 01, 2018 - 18:15 EDT
Update
There are a few clients it is still processing but is gaining speed as it clears out the queues.
Posted Aug 01, 2018 - 17:54 EDT
Monitoring
A fix has been implemented and the queues are processing now and should be completely clear within 15 minutes.
Posted Aug 01, 2018 - 17:32 EDT
Identified
The issue has been identified and is being addressed. We are cautiously optimistic it is running smoothly now and the Email Queues will start clearing out in the next few minutes but will update based on observations.
Posted Aug 01, 2018 - 17:18 EDT
Investigating
We are currently investigating reports of the Email Queue not processing and will provide an update shortly.
Posted Aug 01, 2018 - 16:48 EDT
This incident affected: Processes (Email Queue).