Robots file generator (Robots.txt is a plain text file placed in the root of the site, which specifies that search engine spiders should crawl specified content, or prohibits search engine spiders from crawling some or all of the site's content)
Generated robots results:
Online robots file generation tool: Robots.txt
Is a plain text file stored in the site root directory, and the file can be accessed through the Internet
What is a robots.txt file?
1,Robots.txt is a plain text file stored in the site root directory. Although it is simple to set up, it is very powerful.
It can specify that search engine spiders only crawl specified content, or it can prevent search engine spiders from crawling some or all of the content on the site.
2,robots.txt (Unified Lowercase) is an ASCII-encoded text file stored in the root directory of a website.
It usually tells web search engine bots (also known as web spiders) what content on this site should not be obtained by search engine bots,
Which ones can be obtained by (robot).
3,If you want to individually define the behavior of search engine robots when accessing subdirectories, you can merge custom settings into robots.txt under the root directory,
Or use robots metadata.
4,Because URLs in some systems are case-sensitive, the file names of robots.txt should be unified to lowercase. Robots.txt should be placed in the root directory of the website.
5,The robots.txt protocol is not a specification, but just a convention, so it does not guarantee the privacy of the website.
Note that robots.txt uses string comparison to determine whether to obtain the URL, so the end of the directory has a different URL than the one with no slash "/". robots.txtallow uses something like "Disallow:
* .gif ".
6,The Robots protocol is a ethical norm commonly used in the Internet community and is based on the following principles: 1. Search technology should serve humanity,
At the same time respect the wishes of the information providers and maintain their privacy rights; 2. The website has an obligation to protect the personal information and privacy of its users from being violated.
How to use robots.txt file
1,The Robots.txt file should be placed in the root directory of the website, and the file can be accessed via the Internet.
For example: If your website address is http://www.xxx.com/, then the file must be able to open and see the contents at http://www.xxx.com/robots.txt.