3.11.4 (2024-09-23)

Bug Fixes

• SitemapRequestList.teardown() doesn't break persistState calls (#2673) (fb2c5cd), closes #2672

3.11.3 (2024-09-03)

Bug Fixes

• improve FACEBOOK_REGEX to match older style page URLs (#2650) (a005e69), closes #2216
• RequestQueueV2: reset recently handled cache too if the queue is pending for too long (#2656) (51a69bc)

3.11.2 (2024-08-28)

Bug Fixes

• RequestQueueV2: remove inProgress cache, rely solely on locked states (#2601) (57fcb08)
• use namespace imports for cheerio to be compatible with v1 (#2641) (f48296f)
• Use the correct mutex in memory storage RequestQueueClient (#2623) (2fa8a29)

Features

• globs & regexps for SitemapRequestList (#2631) (b5fd3a9)
• resilient sitemap loading (#2619) (1dd7660)

This release is pinning the dependency on cheerio to the last RC version, we might postpone the official support for v1 to next major, or at least wait for them to fix their stuff. Nice demonstration of how not to maintain popular open source projects 😞

3.11.1 (2024-07-24)

Bug Fixes

• use getHTML in the shadow root expansion (#2587) (a244d62), closes #2583

3.11.0 (2024-07-09)

Features

• add iframe expansion to parseWithCheerio in browsers (#2542) (328d085), closes #2507
• add ignoreIframes opt-out from the Cheerio iframe expansion (#2562) (474a8dc)
• Sitemap-based request list implementation (#2498) (7bf8f0b)

3.10.5 (2024-06-12)

Bug Fixes

• allow creating new adaptive crawler instance without any parameters (9b7f595)
• declare missing peer dependencies in @crawlee/browser package (#2532) (3357c7f)
• fix detection of HTTP site when using the useState in adaptive crawler (#2530) (7e195c1)
• mark context.request.loadedUrl and id as required inside the request handler (#2531) (2b54660)

3.10.4 (2024-06-11)

Bug Fixes

• add waitForAllRequestsToBeAdded option to enqueueLinks helper (925546b), closes #2318
• add missing useState implementation into crawling context (eec4a71)
• make crawler.log publicly accessible (#2526) (3e9e665)
• playwright: allow passing new context options in launchOptions on type level (0519d40), closes #1849
• respect crawler.log when creating child logger for Statistics (0a0d75d), closes #2412

3.10.3 (2024-06-07)

Bug Fixes

• adaptive-crawler: log only once for the committed request handler execution (#2524) (533bd3f)
• increase timeout for retiring inactive browsers (#2523) (195f176)
• respect implicit router when no requestHandler is provided in AdaptiveCrawler (#2518) (31083aa)
• revert the scaling steps back to 5% (5bf32f8)

Features

• add waitForSelector context helper + parseWithCheerio in adaptive crawler (#2522) (6f88e73)
• log desired concurrency in the default status message (9f0b796)

3.10.2 (2024-06-03)

Bug Fixes

• Autodetect sitemap filetype from content (#2497) (62a9f40), closes #2461
• improve fix for double extension in KVS with HTML files (#2505) (157927d), closes #2419

Features

• Loading sitemaps from string (#2496) (38ed0d6), closes #2460

3.10.1 (2024-05-23)

Bug Fixes

• adjust URL_NO_COMMAS_REGEX regexp to allow single character hostnames (#2492) (ec802e8), closes #2487
• investigate and temp fix for possible 0-concurrency bug in RQv2 (#2494) (4ebe820)
• provide URLs to the error snapshot (#2482) (7f64145), closes /github.com/apify/apify-sdk-js/blob/master/packages/apify/src/key_value_store.ts#L25

3.10.0 (2024-05-16)

Bug Fixes

• EnqueueStrategy.All erroring with links using unsupported protocols (#2389) (8db3908)
• conversion between tough cookies and browser pool cookies (#2443) (74f73ab)
• fire local SystemInfo events every second (#2454) (1fa9a66)
• use createSessionFunction when loading Session from persisted state (#2444) (3c56b4c)
• do not drop statistics on migration/resurrection/resume (#2462) (8ce7dd4)
• double tier decrement in tiered proxy (#2468) (3a8204b)
• Fixed double extension for screenshots (#2419) (e8b39c4), closes #1980
• malformed sitemap url when sitemap index child contains querystring (#2430) (e4cd41c)
• return true when robots.isAllowed returns undefined (#2439) (6f541f8), closes #2437
• sitemap content-type check breaks on content-type parameters (#2442) (db7d372)

Features

• add FileDownload "crawler" (#2435) (d73756b)
• implement ErrorSnapshotter for error context capture (#2332) (e861dfd), closes #2280
• make RequestQueue v2 the default queue, see more on Apify blog (#2390) (41ae8ab), closes #2388

Performance Improvements

• improve scaling based on memory (#2459) (2d5d443)
• optimize RequestList memory footprint (#2466) (12210bd)
• optimize adding large amount of requests via crawler.addRequests() (#2456) (6da86a8)

3.9.2 (2024-04-17)

Bug Fixes

• break up growing stack in AutoscaledPool.notify (#2422) (6f2e6b0), closes #2421
• don't call notify in addRequests() (#2425) (c4d5446), closes #2421

Features

• sitemap: Support CDATA in sitemaps (#2424) (635f046)

3.9.1 (2024-04-11)

Features

• browserPerProxy browser launch option (#2418) (df57b29)

3.9.0 (2024-04-10)

Bug Fixes

• include actual key in error message of KVS' setValue (#2411) (9089bf1)
• notify autoscaled pool about newly added requests (#2400) (a90177d)
• puppeteer: allow passing networkidle to waitUntil in gotoExtended (#2399) (5d0030d), closes #2398
• sitemaps support application/xml (#2408) (cbcf47a)

Features

• createAdaptivePlaywrightRouter utility (#2415) (cee4778), closes #2407
• tieredProxyUrls for ProxyConfiguration (#2348) (5408c7f)
• better newUrlFunction for ProxyConfiguration (#2392) (330598b), closes #2348 #2065
• expand #shadow-root elements automatically in parseWithCheerio helper (#2396) (a05b3a9)

3.8.2 (2024-03-21)

Bug Fixes

• core: solve possible dead locks in RequestQueueV2 (#2376) (ffba095)
• correctly report gzip decompression errors (#2368) (84a2f17)
• puppeteer: improve detection of older versions (98d4e86), closes #2370
• use 0 (number) instead of false as default for sessionRotationCount (#2372) (667a3e7)

Features

• implement global storage access checking and use it to prevent unwanted side effects in adaptive crawler (#2371) (fb3b7da), closes #2364

3.8.1 (2024-02-22)

Bug Fixes

• fix crawling context type in router.addHandler() (#2355) (d73c202)

3.8.0 (2024-02-21)

Bug Fixes

• createRequests works correctly with exclude (and nothing else) (#2321) (048db09)
• declare missing dependencies on csv-stringify and fs-extra (#2326) (718959d), closes /github.com/redabacha/crawlee/blob/2f05ed22b203f688095300400bb0e6d03a03283c/.eslintrc.json#L50
• puppeteer: add 'process' to the browser bound methods (#2329) (2750ba6)
• puppeteer: replace page.waitForTimeout() with sleep() (52d7219), closes #2335
• puppeteer: support puppeteer@v22 (#2337) (3cc360a)

Features

• KeyValueStore.recordExists() (#2339) (8507a65)
• accessing crawler state, key-value store and named datasets via crawling context (#2283) (58dd5fc)
• adaptive playwright crawler (#2316) (8e4218a)
• add Sitemap.tryCommonNames to check well known sitemap locations (#2311) (85589f1), closes #2307
• ci: snapshot docs automatically on minor/major publish (#2344) (092f51e)
• core: add userAgent parameter to RobotsFile.isAllowed() + RobotsFile.from() helper (#2338) (343c159)
• Support plain-text sitemap files (sitemap.txt) (#2315) (0bee7da)

3.7.3 (2024-01-30)

Bug Fixes

• enqueueLinks: filter out empty/nullish globs (#2286) (84319b3)
• pass on an invisible CF turnstile (#2277) (d8734e7), closes #2256

3.7.2 (2024-01-09)

Bug Fixes

• RequestQueue: always clear locks when a request is reclaimed (#2263) (0fafe29), closes #2262

3.7.1 (2024-01-02)

Bug Fixes

• ES2022 build compatibility and move to NodeNext for module (#2258) (7fe1e68), closes #2257