article-extractor

• Stop using purified HTML to extract content (#388)
• Update examples & test with pupperteer (#389)

• Decode content using detected charset
• Update dependencies
  • Update eslint config

Related issues: #386, #320

Thanks to the advices from @martinrotter 🤝

• Update dependencies

Related issue: #382

• Update dependencies
• Update security email

• Fix error while parsing ldjson (#378)
• Update dependencies

• Merge pr #374 by @andremacola (issue #373)
• Update examples
• Update dependencies
• Update CI config
• Fix function call in evaluation script

• Merge pr #369
• Update deno example (#368)
• Stop ci test with node <= 16, this version is in its end-of-life

• Use childNodes instead of children to get the same behaviour as Deno DOM
• Update dependencies

• Build CJS version and export it for the outdated platforms

Try to fix the issues: #359 #360

• Update dependencies
• Fix imports section

Related issues: #345 #357

• Add deno.json & import sections
• Update deps
• Improve README

• Add support to signal
• Stop support Node < 15
• Stop support commonjs version
  • Remove build script
• Update examples code
• Update dependencies

Example with signal

import { extract } from '@extractus/article-extractor'

const url = 'https://www.cnbc.com/2022/09/21/what-another-major-rate-hike-by-the-federal-reserve-means-to-you.html'

const article = await extract(url, null, {
  signal: AbortSignal.timeout(5000),
})
console.log(article)

• Add test for proxy agent
• Fix README issue
• Update dependencies

• Merge pr #350 by @LarchLiu
• Add agent to fetchOptions
• Update CI to test with Node 20
• Update dependencies
• Update README

Example article extraction via proxy server with agent

import { extract } from '@extractus/article-extractor'

import { HttpsProxyAgent } from 'https-proxy-agent'

const proxy = 'http://abc:[email protected]:31113'

const url = 'https://www.cnbc.com/2022/09/21/what-another-major-rate-hike-by-the-federal-reserve-means-to-you.html'

const article = await extract(url, {}, {
  agent: new HttpsProxyAgent(proxy),
})
console.log('Run article-extractor with proxy:', proxy)
console.log(article)

• Fix issue #347
• Update dependencies

• Merge with changes from pr #341
• Fix unsupported package string-similarity
• Update deps

• Add support parsely meta tags

Maybe it comes from Parse.ly. Our users found that serveral websites such as TheVerge start using the strange meta tags that may break the extraction process. With these non-standard resources, this release should be helpful.

• Fix issue while fetching data from some websites (Deno platform only)

• Set default user-agent
• Avoid error if parserOptions is null
• Update dependencies

• Merge pr #333 (thanks to @willwashburn)
• Update dependencies