gitleaks

What's new

• Bump to Go 1.14
• ignore leaks in config file by default https://github.com/zricethezav/gitleaks/pull/362

What's fixed

• Object Not Found bug when using --commit=latest https://github.com/zricethezav/gitleaks/pull/365
• --uncommitted now works with empty repos https://github.com/zricethezav/gitleaks/issues/352

What's new

• Added the option to use latest in the parameter --files-at-commit to scan the state of all the files in the latest commit of the repository so you don't need to look for the commit hash: https://github.com/zricethezav/gitleaks/pull/348

Bugs Fixed

• fixed github auth issue where you couldn't clone private repos for an organization/user: https://github.com/zricethezav/gitleaks/pull/350

Also huge shoutout to @NoelAlgora for two big PRs that pushed gitleaks up to 4.0

What's new

• Ope! I forgot to bump the version in the projects packages...

What's new

• Added Gitleaks Action!!! https://github.com/marketplace/actions/gitleaks
• Enhanced Entropy https://github.com/zricethezav/gitleaks/pull/333
• Filename/Filepath Regexes https://github.com/zricethezav/gitleaks/pull/340
• added new logo

Bug fixes

• fixed a bug where gitleaks was deduplicating leaks when using the --redact option https://github.com/zricethezav/gitleaks/pull/345

Breaking changes

• Both https://github.com/zricethezav/gitleaks/pull/333 https://github.com/zricethezav/gitleaks/pull/340 remove the Global table in the regex. It didn't make sense to include global rules. We want to keep the config simple and only rely on rules

What's new

• --files-at-commit= option scans ALL files in the repo at a specific commit.

Bug fixes

• fixed a bug where --commit= had the same behavior as --files-at-commit in that it was scanning all the files, not the patch. --commit= now scans only the patch. See https://github.com/zricethezav/gitleaks/issues/326 for more information

What's new

v3.2.1 did not properly load the version when gitleaks --versioning

What's new

• Fixing module imports https://github.com/zricethezav/gitleaks/pull/323

What's new

• Audit timeouts via timeout option. PR: https://github.com/zricethezav/gitleaks/pull/319
• depth option added: https://github.com/zricethezav/gitleaks/pull/321
• Exclude forks when auditing github/gitlab org/group/user via exclude-forks option: https://github.com/zricethezav/gitleaks/pull/322

Bugs

• Github/Gitlab host audits now correctly log total number of commits: https://github.com/zricethezav/gitleaks/pull/320

What's new

• Commit scanning range options via commit-from and commit-to options. https://github.com/zricethezav/gitleaks/issues/315

git log -10 --pretty=format:"%H"

d3c4342c15be0445f3984a74b758557fa8a44e3d
b4c2f8e69c380af92bf5ed3522e5d094c00d8276
a0f72a4e3595ddb382a77804d2674d54b2b0e880
593edeae2134fd23b58306ea7d3494c913954c52
6df770d2614a7548c6b5efe71bbe764ad8fd6768
4f0c9dcede10ea0705bd5654150c745735355923
52425a8ca9a26c57dcb03f7a8bcf20a324bb24bc
454acebe5ac5961d3acefe96e430d205a19f550c
6fcf91e9343cfc03008a343acd2cebfdfc7ab486
f89b8f2b29d8810e6988bbceb9bd0adce37a7b31

using commit-from=a0f72a4e3595ddb382a77804d2674d54b2b0e880 the audit would start at a0f72... and move backwards to the root commit

d3c4342c15be0445f3984a74b758557fa8a44e3d
b4c2f8e69c380af92bf5ed3522e5d094c00d8276
a0f72a4e3595ddb382a77804d2674d54b2b0e880 FROM
593edeae2134fd23b58306ea7d3494c913954c52    |
6df770d2614a7548c6b5efe71bbe764ad8fd6768    |
4f0c9dcede10ea0705bd5654150c745735355923    |
52425a8ca9a26c57dcb03f7a8bcf20a324bb24bc    |
454acebe5ac5961d3acefe96e430d205a19f550c    |
6fcf91e9343cfc03008a343acd2cebfdfc7ab486    V
f89b8f2b29d8810e6988bbceb9bd0adce37a7b31    ...

using commit-to=a0f72a4e3595ddb382a77804d2674d54b2b0e880 the audit would start at HEAD and stop at a0f72...

d3c4342c15be0445f3984a74b758557fa8a44e3d   |
b4c2f8e69c380af92bf5ed3522e5d094c00d8276   V
a0f72a4e3595ddb382a77804d2674d54b2b0e880 STOP
593edeae2134fd23b58306ea7d3494c913954c52
6df770d2614a7548c6b5efe71bbe764ad8fd6768
4f0c9dcede10ea0705bd5654150c745735355923
52425a8ca9a26c57dcb03f7a8bcf20a324bb24bc
454acebe5ac5961d3acefe96e430d205a19f550c
6fcf91e9343cfc03008a343acd2cebfdfc7ab486
f89b8f2b29d8810e6988bbceb9bd0adce37a7b31

using commit-from=a0f72a4e3595ddb382a77804d2674d54b2b0e880 and commit-to=454acebe5ac5961d3acefe96e430d205a19f550c the audit would start at a0f72... and stop at 454ac...

d3c4342c15be0445f3984a74b758557fa8a44e3d
b4c2f8e69c380af92bf5ed3522e5d094c00d8276
a0f72a4e3595ddb382a77804d2674d54b2b0e880 FROM
593edeae2134fd23b58306ea7d3494c913954c52    |
6df770d2614a7548c6b5efe71bbe764ad8fd6768    |
4f0c9dcede10ea0705bd5654150c745735355923    |
52425a8ca9a26c57dcb03f7a8bcf20a324bb24bc    V
454acebe5ac5961d3acefe96e430d205a19f550c    STOP
6fcf91e9343cfc03008a343acd2cebfdfc7ab486
f89b8f2b29d8810e6988bbceb9bd0adce37a7b31

What's new

• Added some new rule defaults: https://github.com/zricethezav/gitleaks/pull/296 https://github.com/zricethezav/gitleaks/pull/295
• Default config ignores gitleaks.toml files as those will trigger false positive leaks: https://github.com/zricethezav/gitleaks/pull/302
• Running gitleaks on staged, uncommitted changes will now honor the --repo-config flag and use more sensible default commit values rather than using the previous commit for leak values (for things like email, commit hash, date, etc). https://github.com/zricethezav/gitleaks/pull/301

Big thanks to https://github.com/petegallagher for putting all these PRs together 💯 👍 🎉

What's new

• Recover on panic when generating diffs. Issue being resolved here: https://github.com/sergi/go-diff/issues/89
• full file path for "filename" in report https://github.com/zricethezav/gitleaks/issues/288
• version message when not building with make build

What's New

• bug fixed for file rule on commits with no parent
• added --report-format option again so users can export reports to json or csv. Json is the default.
• added an example config for hitting a decent score on https://github.com/Plazmaz/leaky-repo

What's new

• pre-commit scanning (great for pre-commit hooks!)
• whitelists per rule
• faster regex scanning
• more useful debug logging so you can check which regexes are performing poorly
• username/password authentication option
• access-token authentication as an option
• proper unit tests that don't rely on connecting to a git host (no more travis failing for pr's)

pre-commit scanning

This is a feature that has been requested for a while. I finally got around to it and I think it's one of the most useful features going forward as this shifts the secret detection as close as possible to the developer... if a developer attempts to commit keys they get a message stating they cannot commit due to a secret being found in their changes. Below is a demonstration

Here's the pre-commit hook I added for what was seen above:

#!/bin/sh
gitleaksEnabled=$(git config --bool hooks.gitleaks)
cmd="gitleaks --verbose --redact --pretty"
if [ $gitleaksEnabled == "true" ]; then
    $cmd
    if [ $? -eq 1 ]; then
cat <<\EOF
Error: gitleaks has detected sensitive information in your changes.
If you know what you are doing you can disable this check using:

    git config hooks.gitleaks false

EOF
exit 1
    fi
fi

Better Rules:

Take a look at the examples for explanations on sample rules.

Password Authentication

Made possible by go-git, thanks! Supply a --username and --password to clone username/password

More useful debugging

In prior versions changing the log level to debug didn't really yield much benefit to the user trying to gain insight as to
what gitleaks was doing. I added a --debug option that prints some useful information like clone, patch, and individual regex times.
An example output with --debug set looks like:

DEBU[2019-11-12T20:53:50-05:00] -------------------------
DEBU[2019-11-12T20:53:50-05:00] | Times and Commit Counts|
DEBU[2019-11-12T20:53:50-05:00] -------------------------
totalAuditTime:  6 seconds 664 milliseconds 767 microseconds
totalPatchTime:  4 seconds 837 milliseconds 101 microseconds
totalCloneTime:  1 second 666 milliseconds 658 microseconds
totalCommits:  3038
DEBU[2019-11-12T20:53:50-05:00] --------------------------
DEBU[2019-11-12T20:53:50-05:00] | Individual Regex Times |
DEBU[2019-11-12T20:53:50-0h5:00] --------------------------
(?i)linkedin(.{0,20})?(?-i)['\"][0-9a-z]{12}['\"]........................................1 second 711 milliseconds 333 microseconds
xox[baprs]-([0-9a-zA-Z]{10,48})?.........................................................41 milliseconds 468 microseconds
AIza[0-9A-Za-z\\-_]{35}..................................................................46 milliseconds 233 microseconds
access_token\$production\$[0-9a-z]{16}\$[0-9a-f]{32}.....................................62 milliseconds 45 microseconds
sk_live_[0-9a-z]{32}.....................................................................51 milliseconds 633 microseconds
sq0atp-[0-9A-Za-z\-_]{22}................................................................59 milliseconds 711 microseconds
sq0csp-[0-9A-Za-z\\-_]{43}...............................................................53 milliseconds 940 microseconds
(?i)aws(.{0,20})?(?-i)['\"][0-9a-zA-Z\/+]{40}['\"].......................................1 second 777 milliseconds 181 microseconds
amzn\.mws\.[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}..................63 milliseconds 72 microseconds
(?i)(facebook|fb)(.{0,20})?(?-i)['\"][0-9a-f]{32}['\"]...................................2 seconds 277 milliseconds 274 microseconds
(?i)(facebook|fb)(.{0,20})?['\"][0-9]{13,17}['\"]........................................2 seconds 232 milliseconds 74 microseconds
(?i)twitter(.{0,20})?['\"][0-9a-z]{35,44}['\"]...........................................1 second 787 milliseconds 692 microseconds
(?i)twitter(.{0,20})?['\"][0-9a-z]{18,25}['\"]...........................................1 second 803 milliseconds 922 microseconds
(?i)linkedin(.{0,20})?['\"][0-9a-z]{16}['\"].............................................1 second 742 milliseconds 230 microseconds
(?i)(api_key|apikey|secret)(.{0,20})?['|"][0-9a-zA-Z]{16,45}['|"]........................4 seconds 615 milliseconds 233 microseconds
https://hooks.slack.com/services/T[a-zA-Z0-9_]{8}/B[a-zA-Z0-9_]{8}/[a-zA-Z0-9_]{24}......44 milliseconds 51 microseconds
(?i)stripe(.{0,20})?['\"][sk|rk]_live_[0-9a-zA-Z]{24}....................................2 seconds 477 milliseconds 561 microseconds
(A3T[A-Z0-9]|AKIA|AGPA|AIDA|AROA|AIPA|ANPA|ANVA|ASIA)[A-Z0-9]{16}........................104 milliseconds 256 microseconds
-----BEGIN EC PRIVATE KEY-----...........................................................38 milliseconds 285 microseconds
(?i)heroku(.{0,20})?['"][0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}['"].............1 second 736 milliseconds 486 microseconds
(?i)(mailchimp|mc)(.{0,20})?['"][0-9a-f]{32}-us[0-9]{1,2}['"]............................2 seconds 288 milliseconds 332 microseconds
(?i)(mailgun|mg)(.{0,20})?['"][0-9a-z]{32}['"]...........................................2 seconds 236 milliseconds 818 microseconds
(?i)github(.{0,20})?(?-i)['\"][0-9a-zA-Z]{35,40}['\"]....................................1 second 735 milliseconds 952 microseconds
(?i)twilio(.{0,20})?['\"][0-9a-f]{32}['\"]...............................................1 second 796 milliseconds 873 microseconds

v2.1.0 of Gitleaks is dedicated to my brother, Sam, who passed away on July 3rd at the young age of 29. He was a gentle and artistic spirit. I loved him dearly and could not have asked for a better big bro.

Changes

• --commit does not iterate on commit history. Now it directly accesses the commit object. https://github.com/zricethezav/gitleaks/pull/236
• Fixing logging local repo path https://github.com/zricethezav/gitleaks/pull/204
• Better README examples https://github.com/zricethezav/gitleaks/pull/202
• Better logging on Windows https://github.com/zricethezav/gitleaks/pull/230
• Better error messaging
• Bunch new default rules https://github.com/zricethezav/gitleaks/pull/231 https://github.com/zricethezav/gitleaks/pull/241
• --repo-config works on orgs and users now https://github.com/zricethezav/gitleaks/pull/239
• Proper exit codes https://github.com/zricethezav/gitleaks/pull/244
• Gitlab Token support https://github.com/zricethezav/gitleaks/pull/184

Give thanks

If using gitleaks has made you job easier consider donating to one of Sam's favorite places, the Japan House on the University of Illinois at Urbana-Champaign's campus: https://japanhouse.illinois.edu/make-a-gift

Version 2.0.0 of gitleaks introducing a major change to the gitleaks.toml configuration file. This change allows users to define more aggressive filters by combining three techniques: regex, entropy, and file matching. Goodbye [[regexes]], hello [[rules]]. Below is an example rule that combines these three filtering techniques:

[[rules]]
description = "Generic Key"
regex = '''(?i)key(.{0,6})?(:|=|=>|:=)'''
entropies = [
    "4.1-4.3",
    "5.5-6.3",
]
entropyROI = "line"
filetypes = [".go", ".py", ".c"]
tags = ["key"]
severity = "8"

This rule will first attempt to match the regex, then see if the entropy value of either the line or word --depending on entropyROI-- is within the range of entropies, then it will check if the filetype. If all three conditions are met, then voilà, you have a leak.

tags and severity are used for post-audit reporting as per https://github.com/zricethezav/gitleaks/issues/193

Supplemental video https://www.youtube.com/watch?v=e6tqps8MnTY

Bug Fixes

• Fixing https://github.com/zricethezav/gitleaks/issues/188

Features

• Pretty big refactor, see src directory
• Dropping dep for go modules
• Separating email and author
• Readding branch support with --branch=

Made things suck less and hopefully its easier to contribute to this project now that the code is a lil cleaner

Features

• --commit now allows users to target a specific commit to audit
• --commit-stop audit all commits up to and including what is specified at --commit-stop
• Updated go-git version to 4.9.1

Features

• --repo-config allows users to load configs specific to a repo target

Bugs

• panicking while writing large reports due to excessive leaks
• smarter whitelisting when inspecting patches

Features

• context inclusion for redactions
• noise reduction for entropy signals via --noise-reduction option. reduces the number of finds when entropy checks are enabled.

s/o to @camaya for these features