Techdirt: European Newspapers Look To Reinvent Robots.txt

Tis plain that these folks have no grasp on standard usage of robots.txt.  You don’t want the page/site indexed just put it in the damn robots.txt file and forget it.  The spiders respect that file IF it is written right.