Tools
robots.txt Validator
Fetch and analyse a domain's robots.txt, its directives and the referenced sitemaps.
The fetch is performed from AP Interactive's network, not your browser.
About this tool
The robots.txt Validator downloads a domain's robots.txt, analyzes its directives (User-agent, Allow, Disallow, Crawl-delay) and lists the referenced sitemaps. Useful to make sure you are not accidentally blocking pages you want indexed.
- Review which paths you allow and block for each bot.
- Check that you declare your sitemap inside robots.txt.
- Spot accidental blocks that sink your SEO.
Frequently asked questions
Does a Disallow hide my page from Google?
Not entirely. Disallow prevents crawling, but a blocked URL can still appear without a description if others link to it. To truly exclude it use the noindex tag, not just robots.txt.
Should I link the sitemap from robots.txt?
Yes, it is good practice to add a Sitemap: line with your sitemap URL. It helps search engines discover all your URLs even if you have not submitted it via Search Console.