scrapy-scraper

v2.1 release notes:

• minor code refactoring,
• fixed the random sleep time between two consecutive requests by adding a new -rs / --random-sleep option,
• fixed few other minor bugs,
• updated the user agents list.

Web crawler and scraper written in Python and based on Scrapy and Playwright's headless browser.

v1.7 release notes:

• removed redundant argparse dependency,
• updated the user agents list,
• version bump.

Web crawler and scraper written in Python and based on Scrapy and Playwright's headless browser.

v1.6 release notes:

• auto throttling bug fix and value changes for the -at option,
• added new -s option for sleeping between two consecutive requests to the same domain,
• updated the user agents list,
• version bump.

Web crawler and scraper written in Python and based on Scrapy and Playwright's headless browser.

v1.5 release notes:

• install instructions update,
• updated the user agents list,
• version bump.

Web crawler and scraper written in Python and based on Scrapy and Playwright's headless browser.

v1.4 release notes:

• argument parsing code rebase and improved few other stuff,
• boolean options / arguments no longer require yes or no value,
• directory -d option has been changed to directory -dir option,

Web crawler and scraper written in Python and based on Scrapy and Playwright's headless browser.

Web crawler and scraper written in Python and based on Scrapy and Playwright's headless browser.