seo.bills.lt

robots.txt, sitemap and Search Console without the headaches

· Updated · 2 min read

These three things decide whether Google sees your pages at all. Setup takes half an hour and mistakes here are the most expensive.

What robots.txt does and does not do

robots.txt tells crawlers which parts of the site not to fetch. It controls crawling, not indexing: a page blocked in robots.txt can still appear in results if other sites link to it. To keep a page out, use a noindex tag, and leave the page crawlable so the bot can see it.

A simple correct example:

  • User-agent: *
  • Disallow: /admin/
  • Sitemap: https://your-site.com/sitemap.xml

Never leave "Disallow: /" after launch. It is a classic mistake when a staging site goes live.

What belongs in sitemap.xml

List only URLs you want in search: canonical, returning 200, without noindex. One file holds up to 50,000 URLs or 50 MB; split beyond that.

Google ignores priority and changefreq, and uses lastmod only when it reflects real changes. For multilingual sites add hreflang alternates.

Search Console: first steps

Verify the domain by DNS, submit sitemap.xml and open the Pages report. The most useful tool is URL Inspection: it shows whether Google can see a page, which canonical it chose and any problems.

Common indexing statuses

"Discovered, currently not indexed" often points to thin or duplicate content or missing internal links. "Crawled, currently not indexed" means Google saw the page and did not find it useful enough. Unique content, internal links from strong pages and faster loading help: see /en/blog/website-speed-core-web-vitals/.

Check after every change

After editing robots.txt or the sitemap, open them in a browser and in Search Console. The free audit here also checks robots.txt, sitemap, canonical URLs and noindex. The overall plan is in /en/blog/free-seo-audit-step-by-step/.

Frequently asked questions

Does robots.txt keep a page out of Google?

Not reliably. It blocks crawling, but the URL can still be listed. Use noindex to prevent indexing.

Is a sitemap required?

No, but it helps Google discover pages faster, especially on new or large sites.

How fast will Google index a new page?

From hours to a few weeks. Internal links and a URL Inspection request can speed it up.

Keep reading

  1. 6/9 · 2 min read

    Keyword research: find the queries and write what people expect

    How to do keyword research for free, understand search intent, place keywords on a page and link pages internally. Also tuned for AI answers.

  2. 7/9 · 2 min read

    Redesign your website without losing rankings

    How to redesign a website and keep Google rankings: preserve URLs, set up 301 redirects, use a staging site and check before and after launch.

  3. 8/9 · 2 min read

    The best free SEO tools, and what each one is for

    A short list of free SEO tools that cover indexing, speed, keywords, structured data and site audits, with what each one tells you and when to skip it.

All articles →