Three memory-leak patterns in long-running scrapers (and how I caught them after 968 Trustpilot runs)

A long-running Trustpilot scraping project found three major memory-leak patterns after 968 runs: an unbounded asyncio queue, repeated per-URL regex compilation, and BeautifulSoup objects staying alive through stored result references.

These leaks usually did not crash jobs, but they steadily increased RSS memory, forcing larger instances and higher operating costs over time.

The case highlights common pitfalls in Python web scrapers and why memory profiling matters for long-running async workloads.