scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

MIT License

Downloads
1K
Stars
1.2K
Committers
28
View Code on GitHub Visit Website
Ecosystems: Redis, Python

Commit Statistics

Past Year

All Time

Total Commits
657
Total Committers
31
Avg. Commits Per Committer
21.19
Bot Commits
0

Issue Statistics

Past Year

All Time

Total Pull Requests
0
44
Merged Pull Requests
0
19
Total Issues
0
60
Time to Close Issues
N/A
3 months
Package Rankings
Top 4.55% on Pypi.org
Badges
Extracted from project README
Build Status Documentation Join the chat at https://gitter.im/istresearch/scrapy-cluster Coverage Status License Docker Pulls
Related Projects
crawdad

Cross-platform persistent and distributed web crawler

20 Jun 2017 61
sci-pype

A Machine Learning API with native redis caching and export + import using S3. Analyze entire dat...

01 Aug 2016 100
scrapy-redis

Redis-based components for Scrapy.

29 Aug 2011 5,448