shot-scraper

• New --auth-username x --auth-password y options for each shot-scraper command, allowing a username and password to be set for HTTP Basic authentication. #140
• shot-scraper URL --interactive mode now respects the -w and -h arguments setting the size of the browser viewport. Thanks, mhalle. #128
• New --scale-factor option for setting scale factors other than 2 (for retina). Thanks, Niel Thiart. #136
• New --browser-arg option for passing extra browser arguments (such as --browser-args "--font-render-hinting=none") through to the underlying browser. Thanks, Niel Thiart. #137

• New --bypass-csp option for bypassing any Content Security Policy on the page that prevents executing further JavaScript. Thanks, Brenton Cleeland. #116
• Screenshots taken using shot-scraper --interactive $URL - which allows you to interact with the page in a browser window and then hit <enter> to take the screenshot - it no longer reloads the page before taking the shot (which ignored your activity). #125
• Improved accessibility of documentation. Thanks, Paolo Melchiorre. #120

• New --omit-background option to the shot command to optionally create transparent PNGs. Thanks, Ben Welsh. #108
• Fixed bug that caused shot-scraper to fail to take screenshots on Windows. Thanks, Omer Rosenbaum. #104
• New --silent option for the shot, multi, pdf and html commands, to disable the default console output. #107

Full documentation: https://shot-scraper.datasette.io/

• Deprecated the shot-scraper multi --fail-on-error option in favor of the new --fail option. --fail-on-error will continue to work until shot-scraper 2.0 (should that ever be released), but is no longer displayed in the --help menu or documentation. #103

• New --log-console option for logging the output of calls to console.log() to standard error. #101
• New --skip and --fail options to specify what should happen if an HTTP 4xx or 5xx error is encountered while trying to load the page. --skip will ignore the error and either exit cleanly or move on to the next screenshot (in the case of multi). --fail will cause the tool to return a non-zero exit code, useful for running in CI environments. #102

• shot-scraper.datasette.io homepage now includes an example screenshot. #97

• New shot-scraper html URL command (documented here) for outputting the final HTML of a page, after JavaScript has been executed. #96
• shot-scraper javascript has a new -r/--raw option for outputting the result of the JavaScript expression as a raw string rather than JSON encoded (shot-scraper javascript documentation). #95
• Tutorial: Automating screenshots for the Datasette documentation using shot-scraper.

• New shot-scraper multi -o option for specifying a subset of one or more output files to execute that are defined in the YAML. This is useful for testing a larger shots.yml file without re-taking every screenshot every time the command is run. #94

• Fixed error message displayed when attempting to use --log-requests on a page that includes a redirect. #90
• --log-requests output now includes a "status": 200 field indicating the HTTP status code returned by that response.

• Fix error message shown when --log-requests tries to log a request with an unknown response size. #89

• New --log-requests option for logging out newline-delimited JSON showing all requests made while rendering a page, including image and asset loads. #88
• New shot-scraper pdf formatting options: --format, --width, --height, --scale and --print-background. Thanks, Eddie Chapman. #87
• Removed -h shortcut for help - use --help instead. -h was clashing with the shorter version of --height. Thanks, Matthew Bafford. #84

• Improved example workflow in Optimizing PNGs using Oxipng.
• Fixed typos in README and documentation. #83

• Documentation improvements. 0.14.1...0.14.2

• Switched documentation site over to the Furo theme. #77

• Documentation is now hosted at shot-scraper.datasette.io and has been expanded to multiple pages. #73
• shot-scraper $URL --wait-for EXPRESSION can be used to take the screenshot only once the provided JavaScript expression returns true. See Waiting until a specific condition. #72
• The wait_for: key in the YAML format used by shot-scraper multi provides equivalent functionality for scripted multiple screenshots.
• New documentation on Using shot-scraper with GitHub Actions. #74
• New example: Extracting page content with Readability.js

• shot-scraper auth --devtools option for opening DevTools when authenticating with a site. #66
• shot-scraper multi -n/--no-clobber option for skipping screenshots where the output image already exists on disk. #70

• --selector-all option to take a screenshot that encompasses every element matching the specified CSS selector - complements --selector which takes a screenshot of just the first element matching that selector. See Specifying elements using JavaScript filters. #64
• selector_all: and selectors_all: keys in the shot-scraper multi YAML format.
• --js-selector and --js-selector-all options for specifying elements to screenshot using a JavaScript expression, for cases which cannot be handled using CSS selectors #43. The following example takes a screenshot of the first paragraph on the page that mentions shot-scraper:

  shot-scraper https://github.com/simonw/shot-scraper \
    --js-selector 'el.tagName == "P" && el.innerText.includes("shot-scraper")'
  

• js_selector:, js_selectors:, js_selector_all: and js_selectors_all: equivalent keys in the shot-scraper multi YAML format.
• --user-agent option for setting a custom user agent header. #63
• --browser webkit option for running WebKit - thanks, Ryan Murphy. #56

• New shot-scraper accessibility --timeout option, thanks Ben Welsh. #59
• shot-scraper auth --browser option for authentication using a browser other than Chromium. #61
• Using --quality now results in a JPEG file with the correct .jpg extension. Thanks, Ian Wootten. #58
• New --reduced-motion flag for emulating the "prefers-reduced-motion" media feature. Thanks, Ryan Murphy. #49

• New -b/--browser option for the shot-scraper install, shot, multi and javascript commands. This can be used to install and run alternative browsers firefox, chrome or chrome-beta. Thanks, Ben Welsh. #53
• New --timeout option for shot-scraper shot and shot-scraper multi. Thanks, Ben Welsh. #47
• shot-scraper multi now continues to create other shots despite a timeout error, unless --fail-on-error is passed. Thanks, Ryan Cheley. #50
• README now describes the async () => { ... } pattern for shot-scraper javascript, as discussed in Extracting web page content using Readability.js and shot-scraper.
• New demo: shot-scraper-template, which lets you setup shot-scraper running in a GitHub repository in a single click. See also Instantly create a GitHub repository to take screenshots of a web page.
• New scraping demo: scrape-hacker-news-by-domain uses shot-scraper javascript to scrape a web page. See Scraping web pages from the command-line with shot-scraper.

• New shot-scraper javascript command for executing JavaScript against a web page and returning the result to the console as JSON: #38

  % shot-scraper javascript datasette.io document.title
  "Datasette: An open source multi-tool for exploring and publishing data"
  

  This can be used for web scraping and data extraction. Any JavaScript errors will cause the command to return an exit code of 1, so this can also be used to run tests against a website from within a continuous integration environment such as GitHub Actions.

• The shot-scraper pdf and shot-scraper accessibility commands can both now be used with local files in addition to URLs. #37

• The output: key is no longer required in YAML shot configuration: if omitted, an automatic filename will be used instead. #40

• An empty YAML file no longer produces an error. #41