ReadCast - Jobs taking longer than usual to process – Incident details

Jobs taking longer than usual to process

Resolved
Operational
Started over 5 years agoLasted about 1 hour

Affected

Website

Operational from 3:51 PM to 4:39 PM

API

Operational from 3:51 PM to 4:39 PM

Voices

Operational from 3:51 PM to 4:39 PM

Updates
  • Resolved
    Resolved

    We just resolved the issue! We flushed everything on our Redis server and it wiped all of our background jobs. We'll monitor things over the next few days but things seem to be back to normal.

  • Monitoring
    Monitoring

    We have stopped Redis, the service we use to run all of our background jobs. We are going to monitor CPU usage on the servers until they are back at a reasonable level, then we will attempt to flush all Redis jobs on our server.

  • Update
    Update

    All of our background jobs seem to be broken at the moment. Our server has decided to reprocess every single job that's ever been processed. We're working to clear the queue to try and resolve the issue.

  • Investigating
    Investigating

    We are currently investigating this incident.