Here is another tool you can use from Bing to see how BingBot can or cannot access your web pages.
Where can I access it? You can access the new robots.txt tester within the new Bing Webmaster tools over here. From there you can enter a URL to see how BingBot will fetch the URL.
What does it look like? Below is a screenshot of the tool. Enter a URL, test it with BingBot and Bing’s Ad bot and also see how the robots.txt file differs from HTTP to HTTPS:
More details. The Bing help document says that this “Robots.txt Tester helps Webmasters to analyze their robots.txt file and highlight the issues that would prevent them from getting optimally crawled by Bing and other robots.”
Fabrice Canal from the Bing team told us, “We resurrected and enhanced our old robots.txt tester.” Canal explained that this was a feature they had many years ago that the company brought back to life and made better.
Google also has a similar tool, you can learn more about it over here.
Why we care. Now you can see how BingBot sees your content. If you need to troubleshoot an indexing or ranking issue with Bing, this is one of the tools you can add to your toolkit to see how Bing can access your pages.
