Website Tools

Robots.txt Checker

Check whether a website has a robots.txt file and view its contents.

Enter a domain to check whether it has a robots.txt file and view its contents.

More Website Tools

Website Response Time Checker Sitemap Checker Redirect Checker Favicon Checker HTTP Headers Checker Website Technology Checker HTTP Status Checker HTML Validator

JustChecker monitors your robots.txt and sitemap continuously.

What Is the Robots.txt Checker?

robots.txt is a plain-text file at the root of a website (/robots.txt) that tells search engine crawlers which paths they may crawl. This tool fetches the file, shows its contents and checks whether it references a sitemap.

When to Use It

When pages are not being crawled or indexed.
After launching or migrating a site - a leftover Disallow: / from staging can block the whole site.
To confirm your sitemap is listed.

How to Use It

Enter the domain.
Read the rules and look for a Sitemap: line.

Understanding the Result

User-agent: * applies rules to all crawlers. Disallow: /admin blocks that path; Disallow: / blocks everything. A missing robots.txt is fine - crawlers then assume everything is allowed.

Limitations

robots.txt controls crawling, not indexing - a blocked page can still appear in search results if other sites link to it. Use a noindex tag for that; check it with the Indexability Checker.

Frequently Asked Questions

Search engine crawlers default to assuming the entire site is allowed to be crawled - a missing robots.txt is not an error, but it also means you have no way to explicitly exclude specific paths from crawling.

Not reliably on its own - it only requests that crawlers not fetch the page; a page can still be indexed (using information from links elsewhere) even if crawling is blocked. Use a noindex meta tag for a genuine guarantee.

It instructs crawlers not to access anything on the entire site - this is occasionally left in place accidentally after a staging/development site goes live, silently blocking the whole site from being crawled.

Yes, it's standard and recommended practice to include a "Sitemap:" line pointing to your sitemap.xml location - this gives crawlers an easy, explicit way to discover it.

Yes - robots.txt supports separate rule blocks per "User-agent", so you can allow Googlebot broader access while restricting a different, less-trusted crawler, for example.

No - it's purely a crawling-behavior request that well-behaved bots choose to honor; it provides no actual access restriction or security, and should never be relied on to hide genuinely sensitive content.