https://blog.mojeek.com/2020/11/good-bot-bad-bot-how-search-indexing-works.html