usabilityfirst
usabilityfirst
usabilityfirst

The article is not entirely correct about the effects of robots.txt. Actually, it is more like the other way around. Traditionally, robots.txt could be used to prevent the Internet Archive from displaying your site (the Wayback crawler would still capture copies of the site from time to time, presumably to keep them