Combining Spider and CrawlSpider with Middleware and Simulated Login in Scrapy
Mixing Spider and CrawlSpider Behaviors It's possible to combine the extraction logic of CrawlSpider with the manual request handling of a regular Spider. For instance, you might use CrawlSpider rules to follow links and collect intermediate data, then make additional requests manually to scrape det...