Web Crawler System Design: Frontier, Politeness, and Dedup

https://singhajit.com/web-crawler-system-design/