Include each path before:"/"
Increase limit directory
Google is xml format, Baidu is html format
Increase Sitemap

Default-all search engines:
Retrieval interval:

search engine

Google
Bing
Yahoo
Ask/Teoma
Alexa/Wayback
Cuil
MSN Search
Scrub The Web
DMOZ
GigaBlast
Special search engine (robot)

Google Image
Google Mobile
Yahoo MM
Yahoo Blogs
MSN PicSearch

     

Tip: Please save the following results to Notepad, named robots.txt and upload to the root directory of the website (click copy to copy all the results)
Generated robots results:

Online robots file generation tool: Robots.txt                      Is a plain text file stored in the site root directory, and the file can be accessed through the Internet


What is a robots.txt file?

1,Robots.txt is a plain text file stored in the site root directory. Although it is simple to set up, it is very powerful.
It can specify that search engine spiders only crawl specified content, or it can prevent search engine spiders from crawling some or all of the content on the site.
2,robots.txt (Unified Lowercase) is an ASCII-encoded text file stored in the root directory of a website.
It usually tells web search engine bots (also known as web spiders) what content on this site should not be obtained by search engine bots,
Which ones can be obtained by (robot).

3,If you want to individually define the behavior of search engine robots when accessing subdirectories, you can merge custom settings into robots.txt under the root directory,
Or use robots metadata.
4,Because URLs in some systems are case-sensitive, the file names of robots.txt should be unified to lowercase. Robots.txt should be placed in the root directory of the website.

5,The robots.txt protocol is not a specification, but just a convention, so it does not guarantee the privacy of the website.
Note that robots.txt uses string comparison to determine whether to obtain the URL, so the end of the directory has a different URL than the one with no slash "/". robots.txtallow uses something like "Disallow:                      * .gif ".
6,The Robots protocol is a ethical norm commonly used in the Internet community and is based on the following principles: 1. Search technology should serve humanity,
At the same time respect the wishes of the information providers and maintain their privacy rights; 2. The website has an obligation to protect the personal information and privacy of its users from being violated.

How to use robots.txt file


1,The Robots.txt file should be placed in the root directory of the website, and the file can be accessed via the Internet.
For example: If your website address is http://www.xxx.com/, then the file must be able to open and see the contents at http://www.xxx.com/robots.txt.

Online tool navigation