robots.txt, sitemap and Search Console without the headaches
· Updated · 2 min read
These three things decide whether Google sees your pages at all. Setup takes half an hour and mistakes here are the most expensive.
What robots.txt does and does not do
robots.txt tells crawlers which parts of the site not to fetch. It controls crawling, not indexing: a page blocked in robots.txt can still appear in results if other sites link to it. To keep a page out, use a noindex tag, and leave the page crawlable so the bot can see it.
A simple correct example:
- User-agent: *
- Disallow: /admin/
- Sitemap: https://your-site.com/sitemap.xml
Never leave "Disallow: /" after launch. It is a classic mistake when a staging site goes live.
What belongs in sitemap.xml
List only URLs you want in search: canonical, returning 200, without noindex. One file holds up to 50,000 URLs or 50 MB; split beyond that.
Google ignores priority and changefreq, and uses lastmod only when it reflects real changes. For multilingual sites add hreflang alternates.
Search Console: first steps
Verify the domain by DNS, submit sitemap.xml and open the Pages report. The most useful tool is URL Inspection: it shows whether Google can see a page, which canonical it chose and any problems.
Common indexing statuses
"Discovered, currently not indexed" often points to thin or duplicate content or missing internal links. "Crawled, currently not indexed" means Google saw the page and did not find it useful enough. Unique content, internal links from strong pages and faster loading help: see /en/blog/website-speed-core-web-vitals/.
Check after every change
After editing robots.txt or the sitemap, open them in a browser and in Search Console. The free audit here also checks robots.txt, sitemap, canonical URLs and noindex. The overall plan is in /en/blog/free-seo-audit-step-by-step/.
Frequently asked questions
Does robots.txt keep a page out of Google?
Not reliably. It blocks crawling, but the URL can still be listed. Use noindex to prevent indexing.
Is a sitemap required?
No, but it helps Google discover pages faster, especially on new or large sites.
How fast will Google index a new page?
From hours to a few weeks. Internal links and a URL Inspection request can speed it up.