Subscribe to our blog by email.
August 09, 2020 — Jonathon Rossi
Between 2020-08-08 04:58:46 UTC and 2020-08-09 12:14:54 UTC Feedrabbit's background job processing services failed to process subscriptions for everyone. The issue was caused by a few feeds that had 6.5MB feed items (yes, individual items) which Feedrabbit's HTML scrubber failed to handle and silently sat there attempting to process the HTML content of these items unsuccessfully. Unfortunately I don't have monitoring for this specific situation, but will be looking at adding that and have plans to rework the job processing pipeline. Feed item content is now truncated to 512KB per item, this fix has been deployed and by 2020-08-09 13:41 UTC Feedrabbit has caught up with all pending jobs.
The effect of this outage means that all emails that should have been sent during this time were not, however no emails have been lost, the backlog has now been processed and subscription processing has returned to normal.
Please get in contact if you've got any questions.