scrapy-wayback-middleware

Scrapy middleware for submitting URLs to the Internet Archive Wayback Machine

MIT License

Downloads
531
Stars
10
Committers
2
View Code on GitHub Visit Website
Ecosystems: Python

Commit Statistics

Past Year

All Time

Total Commits
26
Total Committers
2
Avg. Commits Per Committer
13.0
Bot Commits
2

Issue Statistics

Past Year

All Time

Total Pull Requests
0
3
Merged Pull Requests
0
2
Total Issues
0
1
Time to Close Issues
N/A
N/A
Package Rankings
Top 11.45% on Pypi.org
Badges
Extracted from project README
Build status
Related Projects
scrappy

scrapy best practice

02 Mar 2016 37
logparser

A tool for parsing Scrapy log files periodically and incrementally, extending the HTTP JSON API o...

20 Jan 2019 88
waymore

Find way more from the Wayback Machine, Common Crawl, Alien Vault OTX, URLScan & VirusTotal!

24 Jun 2022 1,675
scrapy-scraper

Web crawler and scraper based on Scrapy and Playwright's headless browser.

13 Apr 2023 9
pywb

Core Python Web Archiving Toolkit for replay and recording of web archives

09 Dec 2013 1,366
waybackpack

Download the entire Wayback Machine archive for a given URL.

11 Apr 2016 2,862
scrapy-examples

Multifarious Scrapy examples. Spiders for alexa / amazon / douban / douyu / github / linkedin etc.

11 Jan 2014 3,171
scrapy-wayback-machine

A Scrapy middleware for scraping time series data from Archive.org's Wayback Machine.

05 Apr 2017 109
wayback-machine-scraper

A command-line utility and Scrapy middleware for scraping time series data from Archive.org's Way...

04 Apr 2017 416
get_user_headers

Python module to retrieve identifying request headers from the user's browser for use by local bots

15 Jul 2016 1