crawlpy

Scrapy python crawler/spider with post/get login (handles CSRF), variable level of recursions and optionally save to disk

MIT License

Stars
55
Committers
1
View Code on GitHub
Ecosystems: Python

Commit Statistics

Past Year

All Time

Total Commits
0
30
Total Committers
0
1
Avg. Commits Per Committer
0.0
30.0
Bot Commits
0
0

Issue Statistics

Past Year

All Time

Total Pull Requests
0
2
Merged Pull Requests
0
2
Total Issues
0
1
Time to Close Issues
N/A
N/A
Badges
Extracted from project README
Tag License
Related Projects
facebook-scraper

Scrape Facebook public pages without an API key

15 Mar 2019 2,393
curl-auth-csrf

Python tool that mimics cURL, but performs a login and handles any Cross-Site Request Forgery (CS...

22 Jan 2016 34
pylinkvalidator

pylinkvalidator is a standalone and pure python link validator and crawler that traverses a web s...

24 Jun 2014 142
OpenScraper

An open source webapp for scraping: towards a public service for webscraping

20 Feb 2018 92
xss-demo

XSS Demo for Python UserGroup Freiburg

09 Mar 2016 6
flask_Captchaify

A DDoS defense system for flask applications, first sends users to a captcha page without a javas...

01 Aug 2023 6
PythonSpiderNotes

Python入门网络爬虫之精华版

19 Aug 2015 6,877
WebCrawler

A web crawler based on requests-html, mainly targets for url validation test.

24 Mar 2017 32
scrapy-scraper

Web crawler and scraper based on Scrapy and Playwright's headless browser.

13 Apr 2023 9
crawlee-python

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extr...

10 Jan 2024 4,020
urlbuster

Powerful mutable web directory fuzzer to bruteforce existing and/or hidden files or directories.

26 Jan 2020 156