This file controls how search engine bots interact with your site.
Robots.txt Explained
Why Robots.txt Matters
Prevents indexing of unwanted pages.
Where You'll Come Across It
At yourdomain.co.uk/robots.txt
Practical Example
Blocking admin pages from search engines.
Frequently Asked Questions
Can robots.txt hide pages from Google?
It can prevent Google from crawling pages, but it doesn't guarantee they won't appear in search results. If other sites link to a blocked page, Google may still list it. For truly private pages, use password protection or a noindex meta tag instead.
Do I need to create a robots.txt file?
It's good practice. Even if you want all pages crawled, a robots.txt file can point search engines to your sitemap and prevent them from wasting time on admin pages or duplicate content.
Can a wrong robots.txt hurt my website?
Yes. If you accidentally block important pages or your entire site, search engines won't crawl or index them. Always check your robots.txt carefully and test it using Google Search Console's robots.txt tester.