The ultimate guide to robots.txt by Yoast

https://yoast.com/ultimate-guide-robots-txt/