Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

A robots.txt file is a plain text file in plain text format. Its primary function is to prevent search engine crawlers such as Google from crawling and indexing content on a website for SEO purposes.

Website crawlers, or robots, are used by search engines such as Bing, Google or Yahoo to review all of the content on your website. You may not want them to crawl certain parts of your website in order to include them in user search results, such as the admin page. These pages can be explicitly ignored by adding them to the file. The Robots Exclusion Protocol is used by robots.txt files.

An incorrect implementation can prevent search engines like Google from crawling properly critical pages on your site or even your entire domain, which can have a significant negative impact on your SEO.

As a result, while SEO Journey's Robots.txt Generator is simple to use, we recommend that you familiarise yourself with Google's instructions before using it.

Create your first robots.txt file using our tool and let us know how it goes.