crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

APACHE-2.0 License

Downloads

1.9M

Stars

15.2K

Committers

View Code on GitHub Visit Website Twitter

Ecosystems: Puppeteer, npm, JavaScript, Node.js, Playwright, TypeScript, Chrome

Commit Statistics

Past Year

All Time

Total Commits

781

4,146

Total Committers

36

102

Avg. Commits Per Committer

21.69

40.65

Bot Commits

350

677

Issue Statistics

Past Year

All Time

Total Pull Requests

360

609

Merged Pull Requests

275

459

Total Issues

186

903

Time to Close Issues

21 days

4 months

Package Rankings

Top 1.07% on Npmjs.org

Top 4.76% on Proxy.golang.org

Related Projects

puppeteer-webperf

Automating Web Performance testing with Puppeteer 🎪

26 Apr 2020 1,800

automation-extra-interception-proxy

Simple way to play with a site requests and responses by the plugin for puppeteer-extra.

crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In ...

26 Aug 2016 15,235

browserless

Deploy headless browsers in Docker. Run on our cloud or bring your own. Free for non-commercial u...

17 Nov 2017 7,567

spider

The fastest web crawler written in Rust. Maintained by @a11ywatch.

07 Jan 2018 842

rebrowser-patches

Collection of patches for puppeteer and playwright to avoid automation detection and leaks. Helps...

06 Aug 2024 173

fingerprint-suite

Browser fingerprinting tools for anonymizing your scrapers. Developed by Apify.

04 May 2022 703

headless-crawler

A crawler implemented using a headless browser (Chrome).

apify-sdk-js

Apify SDK monorepo

22 Apr 2022 119

web-garden

Building blocks for making progressive and future-proof websites. Components collection features ...

18 Jun 2022 156

headless-chrome-crawler

Distributed crawler powered by Headless Chrome

02 Dec 2017 5,519

crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In ...

26 Aug 2016 15,235

puppeteer-real-browser

This package is designed to bypass puppeteer's bot-detecting captchas such as Cloudflare. It acts...

22 Dec 2023 625

x-crawl

x-crawl is a flexible Node.js AI-assisted crawler library. Making crawler work more efficient, in...

22 Jan 2023 829

htcrawl

Htcrawl is nodejs module for the recursive crawling of single page applications (SPA) using javas...