Robots.txt & Sitemap.xml Validator Community
Questions, answers and tips from people who use the Robots.txt & Sitemap.xml Validator. Check robots.txt syntax and sitemap.xml structure for errors that block Google.
Ask a question about Robots.txt & Sitemap.xml Validator
Anyone can read this page. To ask a question, reply or vote, please log in. It takes under a minute.
Log in to askNo questions yet. Yours can be first.
This page shows only real posts from real people, so it starts empty. The guide below has the questions people most often have about the Robots.txt & Sitemap.xml Validator. Pick one to start a thread, or write your own.
From the tool guide: common questions about Robots.txt & Sitemap.xml Validator
The items below are not posts by community members. They come from the written guide for this tool and are here to get a conversation started. Use “Ask a follow-up” to open a thread about any of them.
Why did fetching my sitemap fail?
Most sites do not send the CORS header that allows browser JavaScript on another domain to read their files. Open the sitemap URL in a new tab, copy its source, and paste it into the box instead.
From the tool guide Ask a follow-upWhat does Disallow: / actually do?
It tells the named crawler not to fetch anything on the site. Under User-agent: * it blocks every search engine from the whole site — almost always a mistake.
From the tool guide Ask a follow-upDo I need a robots.txt at all?
Not strictly, but adding one that at least declares your Sitemap: line helps crawlers find your content faster.
From the tool guide Ask a follow-upWhat is the difference between a sitemap and a sitemap index?
A sitemap lists page URLs directly; a sitemap index lists other sitemap files, used when a site has more than 50,000 URLs.
From the tool guide Ask a follow-upIs my data sent anywhere besides the resolver/site I ask it to fetch?
No. Parsing and validation both run in your browser; a fetch only goes to the exact URL you provide.
From the tool guide Ask a follow-upWhy does a valid-looking lastmod get flagged?
It must be a real, parseable calendar date in YYYY-MM-DD or full ISO 8601 format — text like "today" or a malformed date will be flagged.
From the tool guide Ask a follow-upWhat this tool does
A single stray character in robots.txt — a slash where there shouldn't be one — can hide an entire website from Google. This tool checks your robots.txt line by line: it confirms every directive (User-agent, Disallow, Allow, Sitemap, Crawl-delay) uses valid syntax, flags unknown or misplaced directives, and specifically calls out the most damaging mistake of all — a <code>Disallow: /</code> under <code>User-agent: *</code> that accidentally blocks every search engine from the entire site.
How to use it, step by step
- Choose the robots.txt or sitemap.xml tab.
- Enter a site URL and click Fetch, or paste the file content directly.
- Click Validate and read each error, warning and note.
- Fix the issues in your real file and re-check.
Tips from the guide
- Always test robots.txt again after any deployment that touches it.
- Keep exactly one Sitemap: line pointing to an absolute HTTPS URL.
- Split sitemaps with more than 50,000 URLs into a sitemap index.
- Use ISO 8601 dates (YYYY-MM-DD) for every lastmod value.
Common mistakes to avoid
- Adding Disallow: / to block a staging site and forgetting to remove it after launch.
- Listing relative URLs like /page instead of https://example.com/page inside a sitemap.
- Trusting a sitemap.xml auto-fetch when the site blocks cross-origin requests — pasting the source is the reliable path.
- Assuming robots.txt hides pages from search results — it only blocks crawling; use noindex to remove pages already indexed.
Who uses it
- SEO professionals and site owners. Catch a crawl-blocking mistake before it costs rankings.
- Developers. Validate robots.txt and sitemap.xml as part of a launch checklist.
- Agencies auditing client sites. Quickly spot the most common on-page technical SEO errors.
How to ask a question people can answer
A good thread on the Robots.txt & Sitemap.xml Validator saves everyone time. Start with a title that names the exact problem, such as the input you used, the result you expected and the result you got. A title like “Not working” is hard to answer, while a title with the specifics gets replies quickly.
- Say what you tried. List the steps you followed and any settings you changed, so replies do not repeat what you already did.
- Share a small example. A short sample input and the output you saw is worth more than a long description. Remove anything private first.
- Mention your setup. Your device, browser and whether you are on a phone or a computer can explain many odd results.
- One question per thread. Separate topics get clearer answers and are easier to find later with the search box above.
- Come back and reply. If a suggestion worked, say so and upvote it, and if you found the fix yourself, post it for the next person.
Community rules
Be kind, stay on topic and keep personal data out of your posts. Do not post adult, hateful or illegal content, advertising or repeated links. Every post is shown with its author’s username and date, and posts that break these rules can be hidden by moderators. This space is for help with the Robots.txt & Sitemap.xml Validator and related tasks, and it is separate from the sitewide discussion forum where broader topics live.