node-website-scraper

What's Changed

• Fix ENOENT when running from working directory without package.json

Full Changelog: https://github.com/website-scraper/node-website-scraper/compare/v5.3.0...v5.3.1

What's Changed

• Bump cheerio from 1.0.0-rc.11 to 1.0.0-rc.12 by @dependabot in https://github.com/website-scraper/node-website-scraper/pull/499
• Use encoding from resource text by @s0ph1e in https://github.com/website-scraper/node-website-scraper/pull/504

Full Changelog: https://github.com/website-scraper/node-website-scraper/compare/v5.2.0...v5.3.0

What's Changed

• fix: non-english char encoding by @phawxby in https://github.com/website-scraper/node-website-scraper/pull/496

New Contributors

• @phawxby made their first contribution in https://github.com/website-scraper/node-website-scraper/pull/496

Full Changelog: https://github.com/website-scraper/node-website-scraper/compare/v5.1.0...v5.2.0

What's Changed

• Fix encoding issue for non-English websites https://github.com/website-scraper/node-website-scraper/pull/482
• Bump cheerio from 1.0.0-rc.10 to 1.0.0-rc.11 by @dependabot in https://github.com/website-scraper/node-website-scraper/pull/491

⚠️ Version 5.1.0 was deprecated because of bugs introduced by encoding changes https://github.com/website-scraper/node-website-scraper/issues/493, failed tests in website-scraper-puppeteer

Full Changelog: https://github.com/website-scraper/node-website-scraper/compare/v5.0.0...v5.1.0

💥 Breaking changes

• module requires nodejs version >= 14.14
• module is now pure ESM and cannot be required from CommonJS module. Read more about ESM modules
• options.request may need to be changed, underlying http library was updates from request(deprecated) to got
• scrape.defaults now can be imported with import defaultOptions from 'website-scraper/defaultOptions';
• scrape.plugins now can be imported with import * as plugins from 'website-scraper/plugins';
• callback usage support was removed, now only promises and async/await are supported

🐛 Bug fixes

• urlFilter is no longer applied for root resources #460
• issue with wrong quotes in generated html is fixed #355

🌵 Other changes

• cheerio (underlying dependency for html parsing) was updated from version 0.22 to 1.0.0-rc.10

• 529e27c541da6358387455cdc0c502ad6083f944 - fix file save error on windows https://github.com/website-scraper/node-website-scraper/issues/413
• update dependencies

• db206b705341e31c3598b8bf0463028068008697 Scrape resources with query strings when using bySiteStructure

• c565e14fcea79a983a1e1f1557254d764b121a39 fix attributes with html-entity encoded quotes

• 7cc60dbec87912f5d54d6ee3503fff3302b937c3 add responseData to generateFilename action params

• 6988b86b42c441a9348cc0dc21e523c1aa9809b8 Fix issue with transforming attribute names to lowercase #369

• Update dependencies
• Update readme

Breaking changes

• drop support for nodejs versions < 8
• remove options:
  • resourceSaver
  • updateMissingSources
  • updateSources
  • filenameGenerator with function (string with name is supported)
  • httpResponseHandler
  • request property with function (object is supported)
  • onResourceSaved
  • onResourceError
    Each removed option functionality can be implemented using plugins, see Migration guide
• option ignoreErrors is false by default

New functionality

• plugins

• 5afb782e2f43dadf8cb1002f50103bf3091158df - add updateSources option

• e1422032a960ac192e28227b26b93ba06a7e08fa - fix debug breaking change

• 7531dc27903381c27bfaaf04a754fc5c97f13463 encode relative path when update url for downloaded resource

Same as 3.3.4

This version was unpublished, 3.3.5 is the same as 3.3.4

• 5ed0f624a34ff09ffe38f759fe88c8aad8ca1535 - add js type for Resource, resolve it from mime

update dependencies

• 5a1b991 - Fix malformed url error

• 539b48dbca3b1965408681d8d9cafbb6ca1c1de8 - Fix relative resource urls in redirected page