TechnoPrimitive

Techdirt: European Newspapers Look To Reinvent Robots.txt

Tis plain that these folks have no grasp on standard usage of robots.txt.  You don’t want the page/site indexed just put it in the damn robots.txt file and forget it.  The spiders respect that file IF it is written right.

This entry was posted on Monday, September 25th, 2006 at 1:03 pm and is filed under GeekStuff, Here's your sign - STUPID. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.