SEO Tools > Robots.txt

Robots.txt Tutorial

How to Create Robots.txt Files

Use our Robots.txt generator to create a robots.txt file.

Analyze Your Robots.txt File

Use our Robots.txt analyzer to analyze your robots.txt file today.

Google also offers a similar tool inside of Google Webmaster Central, and shows Google crawling errors for your site.

Example Robots.txt Format

Allow indexing of everything

User-agent: *
Disallow:

Disallow indexing of everything

User-agent: *
Disallow: /

Disallow Googlebot from indexing of a folder, except for allowing the indexing of one file in that folder

User-agent: Googlebot
Disallow: /folder1/
Allow: /folder1/myfile.html

Background Information on Robots.txt Files

  • Robots.txt files inform search engine spiders how to interact with indexing your content.
    • By default search engines are greedy. They want to index as much high quality information as they can, and they will assume that they can crawl everything unless you tell them otherwise.
    • If you specify data for all bots (*) and data for a specific bot (like GoogleBot) then the specific bot commands will be followed while that engine ignores the global/default bot commands.
      • If you make a global command that you want to apply to a specific bot and you have other specific rules for that bot then you need to put those global commands in the section for that bot as well, as highlighted in this article by Ann Smarty.
    • When you block URLs from being indexed in Google via robots.txt they may still show those pages as URL only listings in their search results. A better solution for completely blocking the index of a particular page is to use a robots noindex meta tag on a per page bases. You can tell them to not index a page, or to not index a page and to not follow outbound links by inserting either of the following code bits in the HTML head of your document that you do not want indexed.
      • <meta name="robots" content="noindex">
      • <meta name="robots" content="noindex,nofollow">
      • Please note that if you block the search engines in robots.txt and via the meta tags then they may never get to crawl the page to see the meta tags, so the URL may still appear in the search results URL only.
  • If you do not have a robots.txt file, your server logs will return 404 errors whenever a bot tries to access your robots.txt file. You can upload a blank text file named robots.txt in the root of your site (ie: seobook.com/robots.txt) if you want to stop getting 404 errors, but do not want to offer any specific commands for bots.
  • Some search engines allow you to specify the address of an XML Sitemap in your robots.txt file, but if your site is well structured with a clean link structure you should not need to create an XML sitemap.

Crawl Delay

  • Search engines allow you to set crawl priorities.
    • Google does not support the crawl delay command directly, but you can lower your crawl priority inside Google Webmaster Central.
      • Google has the highest volume of search market share in most markets, and has one of the most efficient crawling priorities, so you should not need to change your Google crawl priority.
    • You can set Yahoo! Slurp crawl delays in your robots.txt file. Yahoo! offers background information here.
      • Their robots.txt crawl delay code looks like
        User-agent: Slurp
        Crawl-delay: 5
        where the 5 is in seconds.
    • Microsoft's information is located in their Live Help menu here, but is a bit harder to find.
      • Their robots.txt crawl delay code looks like
        User-agent: msnbot
        Crawl-delay: 10
        where the 10 is in seconds.

Robots.txt Wildcard Matching

Google, Yahoo! Search, and Microsoft allow the use of wildcards in robots.txt files.

To block access to all URLs that include a question mark (?), you could use the following entry:

User-agent: *
Disallow: /*?

You can use the $ character to specify matching the end of the URL. For instance, to block an URLs that end with .asp, you could use the following entry:

User-agent: Googlebot
Disallow: /*.asp$

More background on wildcards available from Google and Yahoo! Search.

URL Specific Tips

Part of creating a clean and effective robots.txt file is ensuring that your site structure and filenames are created based on sound strategy. What are some of my favorite tips?

  • Avoid Dates in URLs: If at some point in time you want to filter out date based archives then you do not want dates in your file paths of your regular content pages or it is easy to filter out your regular URLs. There are numerous other reasons to avoid dates in URLs as well.
  • End URLs With a Backslash: If you want to block a short filename and it does not have a backslash at the end if it then you could accidentally end up blocking other important pages.
  • Consider related URLs if you use Robots.txt wildcards: I accidentally cost myself over $10,000 in profit with one robots.txt error!
  • Dynamic URL Rewriting: Yahoo! Search offers dynamic URL rewriting, but since most other search engines do not use it, you are probably better off rewriting your URLs in your .htaccess file rather than creating additional rewrites just for Yahoo! Search.
  • More URL tips in the naming files section of our SEO training program.

Sample Robot Oddities

Google Generating Search Pages on Your Site?

Google has begun entering search phrases into search forms, which may waste PageRank & has caused some duplicate content issues. If you do not have a lot of domain authority you may want to consider blocking Google from indexing your search page URL. If you are unsure of the URL of your search page, you can conduct a search on your site and see what URL appears. For instance,

  • The default Wordpress search URL is usually ?s=
    • Adding
      User-agent: *
      Disallow: /?s=
      to your robots.txt file would prevent Google from generating such pages
  • Drupal powers the SEO Book site, and our search URL is /search/node/

Secured Version of Your Site Getting Indexed?

In this guest post by Tony Spencer about 301 redirects and .htaccess he offers tips on how to prevent your SSL https version of your site from getting indexed.

Have Canonicalization or Hijacking Issues?

Throughout the years some people have tried to hijack other sites using nefarious techniques with web proxies. Google, Yahoo! Search, Microsoft Live Search, and Ask all allow site owners to authenticate their bots.

  • While I believe Google has fixed proxy hijacking right now, a good tip to minimize any hijacking risks is to use absolute links (like <a href="http://www.seobook.com/about.shtml">) rather than relative links (<a href="about.shtml">) .
  • If both the WWW and non WWW versions of your site are getting indexed you should 301 redirect the less authoritative version to the more important version.
    • The version that should be redirected is the one that does not rank as well for most search queries and has fewer inbound links.
    • Back up your old .htaccess file before changing it!

Comparing Robots.txt to...

link rel=nofollow & Meta Robots Noindex/Nofollow Tags

Crawled by Googlebot?
Appears in Index?
Consumes PageRank
Risks? Waste?
Format
robots.txt no If document is linked to, it may appear URL only, or with data from links or trusted third party data sources like the ODP yes

People can look at your robots.txt file to see what content you do not want indexed. Many new launches are discovered by people watching for changes in a robots.txt file.

Using wildcards incorrectly can be expensive!

User-agent: *
Disallow: /folder/

OR

User-agent: *
Disallow: /file.html

Complex wildcards can also be used.

robots meta noindex tag yes no yes, but can pass on much of its PageRank by linking to other pages

Links on a noindex page are still crawled by search spiders even if the page does not appear in the search results (unless they are used in conjunction with nofollow).

Page using robots meta nofollow (1 row below) in conjunction with noindex can accumulate PageRank, but do not pass it on to other pages.

<meta name="robots" content="noindex">

OR can be used with nofollow likeso

<meta name="robots" content="noindex,nofollow">
robots meta nofollow tag destination page only crawled if linked to from other documents destination page only appears if linked to from other documents no, PageRank not passed to destination If you are pushing significant PageRank into a page and do not allow PageRank to flow out from that page you may waste significant link equity.

<meta name="robots" content="nofollow">

OR can be used with noindex likeso

<meta name="robots" content="noindex,nofollow">

link rel=nofollow destination page only crawled if linked to from other documents destination page only appears if linked to from other documents no, PageRank not passed to destination If you are doing something borderline spammy and are using nofollow on internal links to sculpt PageRank then you look more like an SEO and are more likely to be penalized by a Google engineer for "search spam" <a href="http://destination.com/" rel="nofollow">link text</a>
Javascript link generally no, as long as the destination URL is not in the a href part of the link destination page only appears if linked to from other documents no, PageRank not passed to destination This may not be as much of an obvious sign that you are an SEO, like using link rel=nofollow is.

<a href="#" onclick="window.open('http://www.site.com/', '', ''); return false;">link</a>

OR

<a href=" javascript: window.open('http://www.site.com/', '', '');">link</a>

Data comes from this 2007 interview of Google's Matt Cutts by Eric Enge.

More Robots.txt Resources

Gain a Competitive Advantage Today

Want more great SEO insights? Read our SEO blog to keep up with the latest search engine news, and subscribe to our SEO training program to get cutting edge tips we do not share with the general public.