Robotcop: enforcing your robots.txt policies and stopping bots before they reach your website

https://blog.cloudflare.com/ai-audit-enforcing-robots-txt/