Robots.txt – Keeping web crawlers under control and off your site

In the process of promoting your website, you’ve undoubtedly invited some electronic guests to your website — by submitting your site to Search Engines, Directories, etc. These guests, called Spiders or Crawlers, collect information from your site much like human visitors do. Much like inviting strangers into your home, you’ll get both good guests and [...]