HTML to Markdown converter and crawler.
APACHE-2.0 License
Clipper is a Node.js command line tool that allows you to easily clip content from web pages and convert it to Markdown. It uses Mozilla's Readability library and Turndown under the hood to parse web page content and convert it to Markdown.
Clipper provides a quick and simple way to save bits of content from the web for personal archival or note taking purposes, similar to browser extensions like Evernote Web Clipper or Notion Web Clipper. However, Clipper runs entirely in the terminal so you don't need to install any extensions or sign up for accounts.
HTML | Markdown |
---|---|
npm install -g @philschmid/clipper
note: for crawling you need playwright
and the browser dependencies.
Options:
-i, --input <file> | <directory>
- Input file (html) or directory to clip content from. If a directory is provided, all files in the directory will be clipped.-u, --url <url>
- URL to clip content from-f, --format <format>
- Output format (markdown, json) (default: markdown)-o, --output <file>
- Output file for clipped content (default: output.md)Examples:
clipper clip -u <url>
clipper clip -i <file>
clipper clip -i <directory> -f json -o dataset.jsonl
[!WARNING] Only use this command if you know what you're doing. Crawling websites can be resource intensive and may cause issues for the website owner. Use at your own risk.
Options:
-u, --url <url>
- URL to crawl-g, --glob <glob>
- Glob pattern to match URLs against-o, --output <file>
- Output file for crawled content (default: dataset.jsonl)Examples:
Crawl a site and clip all pages:
clipper crawl -u <url>
Results will be saved in dataset.jsonl
file in the current directory.
If you want to convert a PDF to Markdown you can use poppler to convert the PDF to HTML and then use Clipper to convert the HTML to Markdown.
pdftohtml -c -s -noframes test.pdf test.html
clipper clip -i test.html
npm install
npm run test -- clip -u https://huggingface.co/docs/transformers/index
to test the CLInpm run test -- clip -i examples/
to test the CLI with directory inputnpm run test -- crawl -u https://awsdocs-neuron.readthedocs-hosted.com/en/v2.14.1/index.html -g https://awsdocs-neuron.readthedocs-hosted.com/en/v2.14.1/\*\*/\*
to crawl the AWS Neuron docsnpm run build
to build for productionnpm install -g .
to symlink the CLI for local testingclipper clip -u https://huggingface.co/docs/transformers/index
to build for developmentnpm r clipper -g
Clipper uses the following open source libraries:
rm -rf dist
package.json
if minor or major versionnpm run build
npm publish --access public
package.json
to the next patch version