If you are using a robots.txt file, be sure that you understand what you are excluding from being crawled as it only takes one mistake for your entire site to not be crawled! Limitations of Robots.txt You would also need to have a robots.txt file in place if you are in the process of developing a new site which is LIVE on your server but you don’t want it to be indexed by Google yet. Your robots.txt file should contain the location of your sitemap so it’s easier for spiders, especially search engine spiders, to access all the pages on your site. Some webmasters think that because they want all robots to be able to crawl their entire site that they don’t need a robots.txt file however this is not the case. Blocking the robot’s IP address could be an option but as these spammers usually use different IP addresses it can be a tiresome process. Some could be malicious, even if you create a section in your robots.txt file to exclude it from crawling your site, as these robots usually ignore your robots.txt file it would be unsuccessful. These are normally malicious bots who may harvest information from your site. Note: Some robots will ignore your robots.txt file, as it is only a directive and so will still access pages on your site regardless. When creating your robots.txt file you should be careful of what parameters you set, as if your robots.txt looks like the above example this means that you website won’t be crawled by Google! The ‘Disallow: /’ tells the robots that it is not allowed to visit any pages on this domain. The ‘User-agent: *’ tells the robot that this rule applies to all robots, not only search engine or Google bots. If a robot wants to visit a page on your website, before it does so it checks your robots.txt (placed at – case sensitive, if you call it Robots.TXT it won’t work) and sees that your robots.txt file contains the following exclusion: A typical robots.txt file, placed on your site’s server, should include your sitemap’s URL and any other parameters you wish to put in place. The purpose of a robots.txt file, also known as the robots exclusion protocol, is to give webmasters control over what pages robots (commonly called spiders) can crawl and index on their site.
0 kommentar(er)
