Scrape websites for text by CSS selector.
Web Scraper makes it effortless to scrape websites. You provide a URL and CSS selector and it will return you JSON containing the text contents of the matching elements. You can also scrape HTML attribute values by optionally specifying an attribute name.
web.scraper.workers.dev/?url=example.com&selector=h1
{"result":["Example Domain"]}
{
"result": {
".vcard-fullname": [
"Adam Schwartz"
],
".d-md-block [itemprop=worksFor]": [
"@cloudflare"
],
".d-md-block [itemprop=homeLocation]": [
"Boston, MA"
]
}
}
web.scraper.workers.dev/?url=quotes.net/random.php&selector=%23disp-quote-body,.author&pretty=true
{
"result": {
"#disp-quote-body": [
"We are advertis'd by our loving friends."
],
".author": [
"William Shakespeare"
]
}
}
GET
against https://web.scraper.workers.dev
.url
and selector
.attr
, pretty
and spaced
.If at least url
and selector
are set, the response value will always be JSON.
If only one node is found on the page matching the selector
, the result will be a string. If more than one node is found, the result will be an array of strings.
If an attr
is provided, the result will be a string matching only the first node found which has a non-empty value for that HTML attribute.
url
(required)https://
and http://
protocols.http://
is prepended.
https://web.scraper.workers.dev/?url=example.com&selector=p
selector
(required)HTMLRewriter
class
*
– any elementE
– any element of type EE:not(s)
– an E element that does not match either compound selector sE.warning
– an E element belonging to the class warningE#myid
– an E element with ID equal to myid.E[foo]
– an E element with a foo attributeE[foo="bar"]
– an E element whose foo attribute value is exactly equal to barE[foo="bar" i]
– an E element whose foo attribute value is exactly equal to any (ASCII-range) case-permutation of barE[foo="bar" s]
– an E element whose foo attribute value is exactly and case-sensitively equal to barE[foo~="bar"]
– an E element whose foo attribute value is a list of whitespace-separated values, one of which is exactly equal to barE[foo^="bar"]
– an E element whose foo attribute value begins exactly with the string barE[foo$="bar"]
– an E element whose foo attribute value ends exactly with the string barE[foo*="bar"]
– an E element whose foo attribute value contains the substring barE[foo|="en"]
– an E element whose foo attribute value is a hyphen-separated list of values beginning with enE F
– an F element descendant of an E elementE > F
– an F element child of an E elementattr
(optional)attr
is not set, the text contents of all matched nodes are returned.attr
is set, that HTML attribute is scraped from the first matching node with a non-empty value for that attribute.pretty
(optional)false
or not included, JSON is minified.true
, JSON is formatted using JSON.stringify(json, null, 2)
.spaced
(optional)false
or not included, the text nodes of children of the nodes matching selector
will be concatenated raw.true
, a single space character is added after the end tag of each child node found.Consider the following DOM structure:
<div><p>This is the first paragraph.</p><p>This is another paragraph.</p></div>
If the selector
is set to match div
, by default the resulting text will be:
This is the first paragraph.This is another paragraph.
This is because there is no space character between </p>
and <p>
.
With spaced
set to true
, the result is:
This is the first paragraph. This is another paragraph.
Web Scraper is powered by Cloudflare Workers, heavily utilizing HTMLRewriter for parsing the HTML and scraping the text.
To develop Web Scraper locally, pull down the repo, and follow these steps:
npm i @cloudflare/[email protected] -g
wrangler preview --watch
This will open up the Workers preview experience, so you can test and debug the site. The main source can be found in index.js
. As you make changes you’ll see them live in the previewer.
Web Scraper is deployed automatically when changes are pushed to master using a GitHub Action and the Workers CLI.
Web Scraper was created by Adam Schwartz.