What the robots.txt file does

https://map.bnnd.eu.org/user/Carnildo/diary/390115