feroxbuster

• fixed a bug where --collect-backups wasn't requesting backups from the same directory where the original was found

🎉 Thank you to @gtjamesa for reporting the bug! 🎉

Full Changelog: https://github.com/epi052/feroxbuster/compare/v2.6.0...v2.6.1

What's Changed

• Add --no-state option, filter queries from links, title-case headers by @godylockz in https://github.com/epi052/feroxbuster/pull/474
• added --collect-extensions and --dont-collect
• added --collect-words
• added --collect-backups
• added --burp
• added --burp-replay
• added --smart
• added --thorough
• added --no-state
• added nlp module with html-based TF-IDF implementation
• fixed bug in --resume-from where ScanType::File scans were erroneously kicked off as though they were ScanType::Directory
• fixed bug where some extracted requests may have been requested with redirects allowed, regardless of runtime config
• fixed bug where the first request made with --extract-links was not async
• implemented workaround for bug when --method was POST, PUT, PATCH and --data was empty/not used (awaiting upstream for a real fix)
• query params and fragments are removed from extracted urls before being requested
• updated dependencies
• headers are now Title-Cased
• directory listing detection happens regardless of --extract-links usage, however the links found therein are only requested when --extract-links is used
• numerous code quality improvements

Full Changelog: https://github.com/epi052/feroxbuster/compare/v2.5.0...v2.6.0

What's Changed

• docs: add narkopolo as a contributor for ideas by @allcontributors in https://github.com/epi052/feroxbuster/pull/437
• docs: add justinsteven as a contributor for ideas by @allcontributors in https://github.com/epi052/feroxbuster/pull/436
• Added support for specifying cookies with the -b flag by @7047payloads in https://github.com/epi052/feroxbuster/pull/444
• docs: add 7047payloads as a contributor for code by @allcontributors in https://github.com/epi052/feroxbuster/pull/455
• docs: add unkn0wnsyst3m as a contributor for ideas by @allcontributors in https://github.com/epi052/feroxbuster/pull/456
• Add support of multiple methods during scan #440 by @MD-Levitan in https://github.com/epi052/feroxbuster/pull/441
• added option groups to help output and updated dependencies by @epi052 in https://github.com/epi052/feroxbuster/pull/463
• Directory Listing & Web Scraping Links by @godylockz in https://github.com/epi052/feroxbuster/pull/464
• docs: add its0x08 as a contributor for ideas by @allcontributors in https://github.com/epi052/feroxbuster/pull/468
• docs: add MD-Levitan as a contributor for ideas, code by @allcontributors in https://github.com/epi052/feroxbuster/pull/469
• docs: add godylockz as a contributor for ideas, code by @allcontributors in https://github.com/epi052/feroxbuster/pull/470
• Add redirect messages to normal reports by @epi052 in https://github.com/epi052/feroxbuster/pull/466

New Contributors

• @7047payloads made their first contribution in https://github.com/epi052/feroxbuster/pull/444
• @MD-Levitan made their first contribution in https://github.com/epi052/feroxbuster/pull/441
• @godylockz made their first contribution in https://github.com/epi052/feroxbuster/pull/464

Full Changelog: https://github.com/epi052/feroxbuster/compare/v2.4.1...v2.5.0

• Revised Scan cancel Management Menu to allow for adding a new scan (docs); thanks to @narkopolo for the suggestion! 🥳
• Added original_url entry to json output, suggestion courtesy of @justinsteven 🎉
• updated dependencies to current versions

Full Changelog: https://github.com/epi052/feroxbuster/compare/v2.4.0...v2.4.1

• integrated bugfix; submitted by @cortantief
• added --random-agent feature; submitted by @dsaxton
• added regex support for --dont-scan; idea form @mzpqnxow, implemented by me 🙃

Thanks to everyone involved in this release!!! 🌟 🙏

• fixed wildcard filtering when wildcard response is 0
• fixed total # of urls reported to be scanned in the 'overall' progress bar (was increasing out of sync with reality)

Thank you to @Tib3rius for pointing out the wildcard bug, which led me to see the other bug while fixing the first 🎉

Thank you to @mzpqnxow and @0xdf_ for their suggestions!

• Default status codes now include 500.
• logging while using --parallel now uses the value of -o|--output as a seed to create a directory named OUTPUT_VALUE-TIMESTAMP.logs/. Within the directory, an individual log file is created for each target passed over stdin.

Example Command:

cat large-target-list | ./feroxbuster --stdin --parallel 10 --output super-cool-mega-scan

Resulting directory structure (illustrative):

super-cool-mega-scan-1627865696.logs/
├── ferox-https_target_one_com-1627865696.log
├── ...
└── ferox-https_target_two_net-1627865696.log

Closes #301

feroxbuster now complies with kali's auto package tests

• updated dependencies
• added --dont-scan option that prevents sub-files/folders from being requested (not a filter, applied before the request)
• drastically improved performance when --extract-links is used; should be at or near normal scan performance now

🌮 special thanks to @mzpqnxow and @black-A for their feature requests! 🌮

• bumped multiple dependency versions
• added -o plusdirs to bash completion script as part of the formal build process
• installs that use cargo will now have a config file dropped in ~/.config/feroxbuster (or w/e is OS appropriate) if one doesn't already exist
• --help and -h always result in the long-form help message being printed, even if an erroneous flag/option is placed before it

Thank you to 0xdf, @secure-77, and @hunter0x8 for their suggestions 🎉

• updated scan cancel menu's input to support comma and range delimited values
• updated scan cancel menu's input to support a -f option that will skip asking the user for confirmation

Another superb request from @mzpqnxow 🥳

• fixed [de]serialization of atomics on arm hosts
• added official builds of armv7 and aarch64

Thanks to @tienna-0051 for reporting the build failure!

• bumped serde_json to version 1.0.64
• bumped rlimit to version 0.5.4
• bumped ctrlc to version 3.1.8
• added command completion script install to .deb build
• wordlist order is now preserved after being stored (#233 & #226)
  • previous implementation used a hashset for auto-dedup, with the side effect having an unordered wordlist. Now using a Vec to store the wordlist, which preserves insertion order

thanks to @secure-77 and @Kiblyn11 for bringing wordlist ordering to my attention 🙏

• fixed regression in banner that reintroduced an emoji that threw off alignment (originally fixed in 1.12.3)
• bumped reqwest version to 0.11.1

• added --parallel option
• bumped tokio to version 1.2.0
• bumped serde_json to version 1.0.62

special thanks to Nicolas Krassas (@Dinosn) for the suggestion 🎉

Version 2.2.0 introduces the --parallel option. If you're one of those people who use feroxbuster to scan 100s of hosts at a time, this is the option for you! --parallel spawns a child process per target passed in over stdin (recursive directories are still async within each child).

The number of parallel scans is limited to whatever you pass to --parallel. When one child finishes its scan, the next child will be spawned.

Unfortunately, using --parallel limits terminal output such that only discovered URLs are shown. No amount of -v's will help you here. I imagine this isn't too big of a deal, as folks that need --parallel probably aren't sitting there watching the output... 🙃

Example Command:

cat large-target-list | ./feroxbuster --stdin --parallel 10 --extract-links --auto-bail

Resuling Process List (illustrative):

\_ target/debug/feroxbuster --stdin --parallel 10
    \_ target/debug/feroxbuster --silent --extract-links --auto-bail -u https://target-one
    \_ target/debug/feroxbuster --silent --extract-links --auto-bail -u https://target-two
    \_ target/debug/feroxbuster --silent --extract-links --auto-bail -u https://target-three
    \_ ...
    \_ target/debug/feroxbuster --silent --extract-links --auto-bail -u https://target-ten

• added --auto-tune
• added --auto-bail
• bumped env_logger to version 0.8.3

Thanks to @mzpqnxow and @N0ur5 for their requests/suggestions! 🥳

Version 2.1.0 introduces the --auto-tune and --auto-bail flags. You can think of these flags as Policies. Both actions (tuning and bailing) are triggered by the same criteria (below). Policies are only enforced after at least 50 requests have been made (or # of threads, if that's > 50).

Policy Enforcement Criteria:

number of general errors (timeouts, etc) is higher than half the number of threads (or at least 25 if threads are lower) (per directory scanned)
90% of responses are 403|Forbidden (per directory scanned)
30% of requests are 429|Too Many Requests (per directory scanned)
both demo gifs below use --timeout to overload a single-threaded python web server and elicit timeouts

--auto-tune:
The AutoTune policy enforces a rate limit on individual directory scans when one of the criteria above is met. The rate limit self-adjusts every (timeout / 2) seconds. If the number of errors have increased during that time, the allowed rate of requests is lowered. On the other hand, if the number of errors hasn't moved, the allowed rate of requests is increased. If no additional errors are found after a certain number of checks, the rate limit will be removed completely.

--auto-bail:
The AutoBail policy aborts individual directory scans when one of the criteria above is met. They just stop getting scanned, no muss, no fuss.

• added clarity to error messages resulting from attempting to connect to HTTPS servers w/o a valid certificate

🎉 thanks to @bpsizemore for the issue and PR!

• fixed small bug that prevented requests/second on scan progress bars from displaying correctly

Version 2.0.0 was a large undertaking with the overall goal to address the Focus Areas below.

Focus Areas:

• - long-term project maintainability
• - improve code quality
• - improve error handling
• - reduce memory consumption

Changes incorporated not specifically related to Focus Areas:

• Wildcards now properly filter 0-length responses (thanks to @BitThr3at for the bug report)
• non-fatal errors have mostly been turned into warnings
  • a -v is needed to see non-fatal errors that are handled internally
  • the overall status bar will show errors incrementing
  • the user will need to increase verbosity to see what's going on
  • prevents spammy error messages during timeouts etc
• added rate limiting (--rate-limit) - https://github.com/epi052/feroxbuster/issues/123 (thanks to @mzpqnxow for the suggestion)
• added --silent and added modified behavior of --quiet (thanks to @islanddog and @LaiKash for the report and their help fleshing this out)
  • --quiet - Hide progress bars and banner (good for tmux windows w/ notifications)
  • --silent - Only print URLs + turn off logging (good for piping a list of urls to other commands)
  • closes https://github.com/epi052/feroxbuster/issues/119
• bump tokio-util from 0.6.2 to 0.6.3
• bump predicates from 1.0.6 to 1.0.7
• bumps assert_cmd from 1.0.2 to 1.0.3

• non-utf8 lines in wordlists are now skipped instead of causing an error

In the event that a single line within the given wordlist isn't UTF-8, that line will be skipped. Prior behavior was to exit if ANY line wasn't UTF-8.

Thanks to @sh0reline for the report that sparked this change!