--- title: "Building a Robots.txt File and Implementing It Correctly" description: "Search engines use Web Spiders and bots to reach your website and extract data. The extracted data then goes through a series of transformations before your Web" canonical: "https://www.georanker.com/robots-txt-how-to" last_modified: "2014-05-19T07:23:29" --- # Building a Robots.txt File and Implementing It Correctly Published: 2014-05-19. Updated: 2014-05-19. Author: GeoRanker. > Historical article. Product names, prices, links and third-party behavior may have changed since publication. Use [current products](https://georanker.com/ai/index.md), [pricing](https://georanker.com/ai/pricing.md) and [documentation](https://docs.georanker.com/llms.txt) for current GeoRanker answers. Search engines use Web Spiders and bots to reach your website and extract data. The extracted data then goes through a series of transformations before your Website is actually displayed in the SERP (search engine results page). Robots.txt file will tell a search engine when to index or not index a page. ## What Happens when Web Spiders Visit Your Site? When visiting a landing page, Web Spiders will ask for your website’s _“/robots.txt file_” and look for a “_User-agent:_” line that refers to it specifically. To tell a robot where it cannot search, rules for a user-agent are set up as “_Disallow:_” statements. Here is a preview of how it would look: “Disallow: /test” This command tells a Web Spider to completely ignore the /test and everything inside it. This means a search will not be performed in the specified directory. In this case, our “_/test”_ directory. “Disallow: /” This command tells a Web Spider to ignore the whole site. _Usually, webmasters or site admins use this command in specific cases. For e.g., duplicated content or irrelevant content for the website which should not be displayed in the SERP._ ### “Disallow: “ When you leave a blank field after “_disallow: “,_ the command tells a Web Spider to crawl the entire site with no crawling limitations. ## Examples of a Robots.txt File This example shows how to use Robots.txt commands and implement them correctly. Assume we want all Web Spiders from all Search Engines not to crawl the /uploads folder. Then the command would look like: _User-agent: \*_ _Disallow: /uploads_ This other example specifically tells only to the Google Web search bot (called GoogleBot) not to crawl the /uploads folder: _User-agent: Googlebot_ _Disallow: /uploads_ ### Observations - You can easily write the codes/commands in a simple notepad file. - To check if robots.txt can be accessed by Web crawlers, access yourdomain.com/robots.txt - To check if the rules in robots.txt are correctly written, you can use an online checker like Google Webmaster tools (Crawl -> Blocked URLs) and follow the instructions on the page. [Canonical article]()