The ultimate guide to robots.txt by Yoast

https://staging.yoast.com/ultimate-guide-robots-txt/