node-crawler
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
MIT License
Bot releases are visible (Hide)
- Fixed issue where no callback was called when the 'skipDuplicates' option is active and only one URL is to be crawled. Also added the related test case. https://github.com/sylvinus/node-crawler/pull/124
- Fixed issue when
crawler.queue(undefined)https://github.com/sylvinus/node-crawler/issues/126
Published by almost 10 years ago
Crawler use now Cheerio by default.
Avoid compiling C++ when installing crawler by removing Jsdom from deps.
Jsdom is still supported, see the doc.
Published by almost 10 years ago
Published by almost 10 years ago
-
require('crawler')instead ofrequire('crawler').Crawler - New option
rateLimits(see doc) - Re-organized the code structure
Next big release will be for Cheerio Support
Published by about 10 years ago
Load Jquery File in a windows context
Update README to build Contextify https://github.com/sylvinus/node-crawler#contextify
Published by over 10 years ago
Cleaning in package.json
Update packages to new versions
First use of HttpBin for testing
