Robots.txt Checker
Paste any page URL to see whether crawlers may index it. The checker reads the robots meta tag, crawler response headers and canonical so blocks are caught before they cost traffic.
Enter a URL to begin
Results appear here — nothing is stored.
What this check covers
- Page status
- Robots meta tag
- X-Robots-Tag header
- Indexable
- Canonical
- Final URL
How it works
01
Paste the page URL
Enter the exact page you want search engines to show, not just the homepage.
02
Read the crawler signals
Robots meta, crawler headers and canonical each get a row with a pass, warning or fail dot.
03
Unblock or confirm
Remove stray noindex rules on pages that should rank; keep them on pages that should stay hidden.
Common questions
Is the robots txt checker free?
Yes. Check any public page with no account and nothing to install.
What is the difference between robots.txt and a robots meta tag?
Robots.txt blocks crawling of paths site-wide; the meta tag blocks indexing of a single page.
What does noindex mean?
It tells search engines to leave the page out of results while still letting the link be followed.
What is the X-Robots-Tag header?
A response header that applies crawling rules, useful for files such as PDFs that have no HTML.
Why is my page not indexed?
Common causes are a noindex tag, a crawler block, or a canonical pointing elsewhere; this check surfaces each one.
Related tools
See all toolsTrack your site visitors with Traffsite Web Analytics
Start free