| Subject: | Clearer crawler documentation |
The crawler method looks very handy, however it is not clear from the documentation how one is meant to get past the page_match checks in Scrappy:Scraper since there is insufficient explanation of what patterns are supposed to be. In my case, I am not interacting with a WordPress blog that does lots of URI rewriting, but instead am trying to crawl a site that has complex query-strings as part of the URI. I've had no luck thus far with this in the crawl() mode, and I cannot help but think that a little bit more text explaining the functionality of this method would allow more to benefit from your work.
Cheers!