Subject: | Empty Robot Rules file causes everything to be disallowed |
Many sites reply with an empty robots.txt indicating that robots can go anywhere on the site. This behavior doesn't seem to be supported.
I'm happy to supply a patch but want to check if this is something you are interested in.
Thanks,
Eric