A selector-based html snapshot tool using Puppeteer or PhantomJS that sources sitemap.xml, sitemap-index, robots.txt, or arbitrary input
MIT License
Bot releases are hidden (Show)
Breaking Changes:
Other updates:
Published by localnerve over 1 year ago
Drops support for node 14 and update dependencies.
Published by localnerve about 2 years ago
Added puppeteer as the default browser.
Added browser
option to allow selection of "phantomjs".
Added debug
option to run puppeteer headed with devtools.
Published by localnerve about 2 years ago
request
in favor of the got
library.robots
input types now search for Sitemap
directives and favor those over any other information in the robots.txt file. If Sitemap
directives are found, those alone are used to drive crawling of the site. If no Sitemap
directives are found, it will fallback to Allow
directives as in previous versions.Published by localnerve over 2 years ago
Supports Node 14+ (dropped 10 & 12)
Select dependency update, copyright 2022.
Published by localnerve over 4 years ago
This release supports Node 10 and greater only.
Published by localnerve over 5 years ago
Change List:
async
dependency used for worker process management to 3.0.1.Published by localnerve over 6 years ago
Published by localnerve over 7 years ago
This is a release with major and breaking changes.
The readme has the scoop, but here's the TL;DR:
run
method now returns a Promiserun
methodrun
Promise will also resolve. The callback argument is deprecated and may be removed at some future date.A Promise failure handler will receive an Error instance that contains all of the errors that have occurred. Also, the Error instance contains additional properties that contain useful information:
completed
- An array of the file output paths that actually completed and were written to storage.notCompleted
- An array of the file output paths that did not complete and were not written to storage.