Production slowed by a calendar-PDF crawler

Incident Report for Congregate

Postmortem

On 14 October 2026, from approximately 01:00 to 16:50 UTC, pages on Congregate sites loaded slowly. An automated crawler requested a very large number of calendar download links that did not correspond to real calendar months, and each request also triggered a sign-in redirect. At its peak the crawler accounted for roughly 60 % of all requests. The application servers reached their scaling limit and could not keep up, so response times rose for every site, typically to several seconds and at times above ten.

We identified the crawler from our request logs and applied limits at our edge so its requests are rejected before they reach the application. Response times returned to normal by 16:50 UTC. No data was lost or exposed, and email and text messaging were not affected. We have added alerting for this kind of saturation and are changing the application so these links are rejected outright.

Posted Oct 14, 2026 - 14:07 CDT

Resolved

On 14 October, Congregate websites were slow for most of the day. An automated program was visiting an enormous number of pages that don't really exist, much faster than any person could, and our servers spent most of their effort answering it. We found it, blocked it, and sites were back to normal speed by late morning US Pacific time. No information was lost or exposed, and emails and text messages kept going out. We've added warnings so we notice this sooner, and we're closing the gap it used.

Posted Oct 14, 2026 - 02:00 CDT