What is supposed by a person-agent in robots.txt is the precise type of World wide web crawler which you provide the command to crawl. This Website crawler typically differs according to the search engine used. Though Google as well as other important search engines have complied Using the commands inside https://seotoolstube.com/