What Is robots.txt and Why Does It Affect Your SEO?
First robots.txt problem we ran into at To-The-TOP! was around 2009. Calgary retail client had hired a developer to build a staging site. Developer blocked the entire domain from indexing while finishing the build, then the site launched. Nobody removed the block. Three months of fresh content, zero traffic. No diagnostic signal told the owner anything was wrong. Just a number sitting flat.
That is what a misconfigured robots.txt file does to a website. Worth understanding how it works before that surprise shows up on your site.
What the File Is
Simple concept. A robots.txt file sits at the root directory of your website, always at the same address: yourwebsite.com/robots.txt. Search engines request it automatically before crawling anything else on your site.
The file tells web crawlers which sections to access and which ones to skip. Nothing more complicated than that. No encryption, no special format. Just lines of plain text a crawler reads in a millisecond before deciding where to go next.
What it is not: a security measure. Blocking a URL in your robots.txt file does not hide it from anyone who types the address directly. Polite bots respect the rules. Humans browsing and bad-actor scrapers do not need to comply.
How Crawlers Use the File
The mechanics matter. Googlebot and other search engine crawlers check the rules that apply to their user-agent each time they visit your website, then either access the URLs you have allowed or skip the ones you have disallowed.
Google has stated publicly it treats robots.txt as a strong suggestion, not an absolute command. Most comply. However, blocking a URL does not guarantee it disappears from search results entirely. If other websites link to a blocked URL, search engines may still list it without a full content preview. Worth keeping in mind when the goal is complete exclusion from the index.
Bing, Yahoo, and similar indexers follow the same general protocol. Each crawler checks for its own user-agent name in the file, then applies whichever instructions match.
AI bots are a different matter. GPTBot, CCBot, and similar bots do not share Googlebot’s compliance record. Blocking them from the file is reasonable. Whether they honour it is less certain. Voluntary compliance is the underlying assumption. Newer AI data-harvesting tools treat it as optional.
The Syntax: User-Agent, Disallow, Allow
Worth reviewing the basics. Your robots.txt file uses three main directives.
User-agent specifies which crawler the rule applies to. An asterisk targets all bots. Swap it for a specific name, like Googlebot or Bingbot, and the rule applies only to that one.
Disallow tells crawlers which paths to skip. Allow overrides it for a specific URL within a blocked directory.
Here is a basic example:
User-agent: *
Disallow: /wp-admin/This example blocks every crawler from your admin directory. Change the asterisk to Googlebot and the rule targets only that one.
A more layered example:
User-agent: *
Disallow: /staging/
Allow: /staging/assets/
Sitemap: https://yourwebsite.com/sitemap.xmlThat last line is optional but useful. It points search engines to your XML content index so they discover pages efficiently. Most files should include it.
What to Block in Your robots.txt File
Some categories belong in almost every setup.
Admin areas and login sections. Your login form and dashboard serve no ranking purpose. Blocking them keeps automated visitors out of areas that add nothing to rankings and frees up some bandwidth on larger sites.
Duplicate content paths. Many websites generate multiple URLs for the same content through parameters, sort filters, or pagination. These paths create index bloat. Blocking the URL patterns keeps the focus on actual content rather than variations of it.
Staging directories. Any staging or development area on the same domain needs blocking from automated visitors. Otherwise that content competes directly with your live URLs in the index. Sometimes it outranks them.
Internal search results sections. Search engine crawlers do not benefit from indexing your site’s own search results. Low value by design.
Where the Real Mistakes Happen
This is the conversation that comes up after something has already gone wrong.
CSS and JavaScript directories blocked. Someone decides to limit what gets downloaded, so they add a blocking rule for the assets folder. Pages look broken to the search crawler. Rankings drop gradually and the connection to the robots.txt file takes months to find.
Blocking pages that already rank. This one shows up during redesigns. Old URLs get blocked to force a migration. Somebody forgets to remove the rule after the redirects go live. URLs with years of crawl history lose that signal gradually.
Blocking your entire website by accident. Staging directive, deployed to the live site. We see this at To-The-TOP! several times per year during site launches, particularly on WordPress. That is why our website audit checks for this on day one. Worth building into every migration checklist.
robots.txt and Crawl Budget
Relevant mostly on larger sites. For most small business websites, crawl budget is not a real concern. Search engines will eventually reach everything on your site. On larger sites, thousands of URLs or e-commerce stores with complex filter parameters, the robots.txt file starts to matter more.
Google processes a fixed number of URLs on your website within any given window. Spend that quota on low-value URLs, session ID parameters, filtered product results, or paginated archives, and the content that matters gets crawled less often. A well-configured robots.txt file keeps everything pointed at content that matters.
Google Search Console shows crawl statistics. That is the first place to check if important URLs are not indexing on schedule. The coverage report surfaces URLs that were discovered but not indexed, and sometimes the explanation traces back to robots.txt directives.
Checking the File
Two checks worth doing. Type yourwebsite.com/robots.txt directly into a browser. The file is public. No login required. Search Console also has a robots.txt tester under Legacy Tools that shows exactly which directives apply to any specific URL on your website.
Worth doing after any major change on your site. Migrations. Redesigns. WordPress plugin updates specifically. Several tools modify the file during installation, sometimes without obvious notice. A recurring website SEO audit catches these before they have time to cost traffic.
To-The-TOP! has operated as a local search engine optimization company in Calgary since 2007, and robots.txt problems show up on almost every new client audit. They appear more often than most business owners expect.
Frequently Asked Questions
What is a robots.txt used for?
Controlling which sections automated tools can reach. The file lives at your root directory and lists blocked paths, allowed exceptions, and optionally the location of your sitemap. Mostly used to keep automated tools out of admin sections, staging directories, duplicate content paths, and content that adds nothing to your search results. A basic setup for a WordPress website might block the admin panel, plugin directories, and any staging subdirectory on the same domain. Nothing exotic about it.
Does robots.txt help with SEO?
Indirectly, yes. The file itself does not improve rankings. On large sites with duplicate content or thin-content problems, a well-structured file stops search engines from spending crawl budget on low-value sections and keeps indexation focused on content that matters. Most small business websites, however, see the main value in avoiding the mistakes: blocked asset files, accidentally blocked product URLs, staging directives left active after launch. As part of the SEO work we do, robots.txt is rarely the first thing to fix, though it is the thing that, if broken, undoes everything else.
robots.txt error: how do you fix it?
Depends on the error. If Search Console flags that your robots.txt file is blocking indexation on URLs that should rank, open the file at yourwebsite.com/robots.txt and look for any blocking directive covering the affected path. Remove or adjust it. Test using the tester in Search Console. A full site audit works through these systematically. WordPress sites should also check installed plugins, as several tools add or edit the file during installation without flagging it in the settings. Worth verifying after any plugin install or site migration.
Is robots.txt illegal?
No. robots.txt is a protocol based on voluntary compliance. No law requires automated tools to follow the instructions in the file. For search engines like Google, compliance is essentially universal because their own systems depend on it. Scrapers and AI tools do not operate under the same norms. Blocking them in your robots.txt sends the signal; whether they honour it depends on how the tool was built. For any business focused on Calgary SEO alongside Google Ads management, it belongs on the technical checklist before anything else gets optimised.
