| Subject: | Wrong "no robots allowed" messages? |
In the generated full report page after a checksite run I see a lot of
lines like
http://www.bbbike.de returned status no robots allowed
though I am pretty positive that most of the sites do not have a
robots.txt file or do not include Disallow rules for the visited paths.
At least for the URL above you can check that there's no robots.txt
existent. I also did not specify a --disallow option to checksite.
Regards,
Slaven