Show current status

[Resolved] Jobs taking longer than usual to process

Started on November 16, 2019 at 3:51 PM. Resolved after about 1 hour

Affected

Website
API
Voices
Legacy Voices
Web Scraper
  • Investigating
    November 16, 2019 at 3:51 PM  –  

    We are currently investigating this incident.

  • Investigating
    November 16, 2019 at 3:59 PM  –  

    All of our background jobs seem to be broken at the moment. Our server has decided to reprocess every single job that's ever been processed. We're working to clear the queue to try and resolve the issue.

  • Monitoring
    November 16, 2019 at 4:21 PM  –  

    We have stopped Redis, the service we use to run all of our background jobs. We are going to monitor CPU usage on the servers until they are back at a reasonable level, then we will attempt to flush all Redis jobs on our server.

  • Resolved
    November 16, 2019 at 4:39 PM  –  

    We just resolved the issue! We flushed everything on our Redis server and it wiped all of our background jobs. We'll monitor things over the next few days but things seem to be back to normal.