• The Web Robots Pages
  • The /robots.txt checker can check your site's /robots.txt file and meta tags. The IP Lookup can help find out more about what robots are visiting you.


  • Robots exclusion standard - Wikipedia, the free ...
  • The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web spiders and other web robots from ...


  • The Web Robots Pages
  • # robots.txt for http://www.example.com/ User-agent: * Disallow: /cyberworld/map/ # This is an infinite virtual URL space # Cybermapper knows where to go.


  • Robots.txt Generator - McAnerin International Inc.
  • robots.txt generator designed by an SEO for public use. Includes tutorial.


  • Robots.txt and Search Indexing - Search Tools Report
  • Information on using the robots.txt file to keep web crawlers, spiders and robots from indexing certain sections of a site.


  • www.whitehouse.gov
  • User-agent: * Crawl-delay: 10


  • google.com
  • User-agent: * Disallow: /search. Disallow: /groups. Disallow: /images. Disallow: /catalogs. Disallow: /catalogues. Disallow: /news. Allow: /news/directory


  • Introduction to "robots.txt"
  • Learn about the robots.txt, and how it can be used to control how search engines and crawlers do on your site.


  • Block or remove pages using a robots.txt file ...
  • A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site, they check to see if a ...


  • Robots.txt Information
  • Information on the robots.txt and how it effects your website. Also includes a free robots.txt generator