scrapy-cloudflare-middleware

Scrapy "CloudFlare" middleware

A Scrapy middleware to bypass the CloudFlare's anti-bot protection, based on cloudflare-scrape.

Installation

pip install scrapy_cloudflare_middleware

Usage

Add the middleware to your DOWNLOADER_MIDDLEWARES settings

DOWNLOADER_MIDDLEWARES = {
    # The priority of 560 is important, because we want this middleware to kick in just before the scrapy built-in `RetryMiddleware`.
    'scrapy_cloudflare_middleware.middlewares.CloudFlareMiddleware': 560
}

Done. Happy scraping !