gitleaks

Features

• cleaning up. refactoring some code. functionality/ux stays the same.

Bug-fixes

• Fixing dangling goroutines, removing channel messaging

Bug fixes

• fixed issue which prevented repos specified with --repo -r and --repo-path from being audited

Features

• Exit code == 2 on error
• Cleaning up some logs
• Removing some unreachable code

Bug Fixes

• recovery from panic caused by https://github.com/sergi/go-diff/issues/89. This fix will skip that commit entirely until that issue has been resolved.

Features

• Default go-routines spawn defaults to # cores your cpu runs with (assuming hyperthreading=2threads/core). Value comes from GOMAXPROCS/2. I did some benchmarking and notices a plateau when go-routines=max cores. Makes sense cause each goroutine will be redlining that core.

go test -run=Benchmark -bench=. -benchtime=5s
goos: darwin
goarch: amd64
pkg: github.com/zricethezav/gitleaks
BenchmarkAuditRepo1Proc-8                      1        17250146938 ns/op
BenchmarkAuditRepo2Proc-8                      1        8195122742 ns/op
BenchmarkAuditRepo4Proc-8                      2        5076421992 ns/op
BenchmarkAuditRepo8Proc-8                      2        4598899766 ns/op
BenchmarkAuditRepo10Proc-8                     2        4674707999 ns/op
BenchmarkAuditRepo100Proc-8                    2        4597661354 ns/op
BenchmarkAuditRepo1000Proc-8                   2        4706597867 ns/op
BenchmarkAuditRepo10000Proc-8                  2        4828204609 ns/op
BenchmarkAuditRepo100000Proc-8                 2        5163678699 ns/op
BenchmarkAuditLeakRepo1Proc-8                300          23209156 ns/op
BenchmarkAuditLeakRepo2Proc-8                500          14325588 ns/op
BenchmarkAuditLeakRepo4Proc-8               1000           9686543 ns/op
BenchmarkAuditLeakRepo8Proc-8               1000           8920760 ns/op
BenchmarkAuditLeakRepo10Proc-8              1000           8755894 ns/op
BenchmarkAuditLeakRepo100Proc-8             1000           9072689 ns/op
BenchmarkAuditLeakRepo1000Proc-8            1000           8569304 ns/op
BenchmarkAuditLeakRepo10000Proc-8           1000           8502946 ns/op
BenchmarkAuditLeakRepo100000Proc-8          1000           8685711 ns/op
PASS
ok      github.com/zricethezav/gitleaks 211.480s

• When auditing users/orgs gitleaks will now delete the repo from disk after the repo audit has been completed if running with --disk option. Before v1.6.0, gitleaks would wait until the entire org audit has been complete.... so you were at risk of running out of disk space. This eliminates that fear.

Features

• CSV report formatting. Run gitleaks with --csv option to output your report as a csv.

Example: $ gitleaks --github-user=gitleakstest --report="report.csv" --csv

contents of report.csv:

Bug Fixes

• Redact option will prevent secrets from being logged
• Repos with single commits will be accounted for now

Features

• Branch target support

Bugs

• Error check commit patch

Features

• Added support for providing an alternate GitHub URL to support scanning GitHub Enterprise repositories

• Changelog
• Version option

Bugs:

• Commit patch order

Misc:

• Update logs
• Update Readme

Bugs:

• Twitter Typo
• Example docker command

Features:

• Default clone option to use in-memory
• Added option for cloning to disk with --disk

Misc:

• Updated Makefile

Welcome to gitleaks v1.0.0

Gitleaks v1.0.0 introduces major performance gains, cleaner ui, and some new features.

Installing

go get -u github.com/zricethezav/gitleaks

Or download from release binaries here

Performance

Prior to v1.0.0 Gitleaks relied on shelling out git commands. This meant for every diff you would be reading from disk. Thanks to src-d's go-git package you are now able to process your repos in memory with the --in-memory option. Additionally, Gitleaks v1.0.0 audits just additions and deletions in your commit patches -- before, gitleaks looked at the entire commit patch. Another gain v1.0.0 introduces is support for whitelisting commits, files, regexes, and branches specified in GITLEAKS_CONFIG or --config= options config.

UI

Gitleaks v1.0.0 uses jessevdk's go-flags which offers much cleaner option handling. Let's go through some examples:

# audit a single public repo
$ gitleaks --repo=https://github.com/gitleakstest/gronit

# audit a single private repo
# NOTE: SSH auth only -- this requires your private key (default: ~/.ssh/id_rsa) or set by --ssh-key
$ gitleaks [email protected]:gitleakstest/privaterepo.git

# audit github user's public repos
$ gitleaks --github-user=gitleakstest

# audit github user's private repos
# NOTE: SSH auth only -- this requires your private key (default: ~/.ssh/id_rsa) or set by --ssh-key
# NOTE: `GITHUB_TOKEN` must be set in your env otherwise it will pull public repos
$ gitleaks --github-user=gitleakstest --private

# save report to file
# NOTE: report format is json -- support for more coming in later versions
$ gitleaks --repo=https://github.com/gitleakstest/gronit --report=report.json

New Features

Externalize Regexes

A common request I heard was the ability to externalize regexes so you could search for whatever you wanted in your git history. This has been added in v1.0.0. Of course there are default checks that I have in place that include AWS, Facebook, Twitter, PGP, BGP, RSA, and Heroku but if you want to remove or add regexes all you need to do is supply gitleaks with a config specified in GITLEAKS_CONFIG or pass the path to the config as an option with --config=. The config is a simple toml file that looks like:

[[regexes]]
description = "AWS"
regex = '''AKIA[0-9A-Z]{16}'''
[[regexes]]
description = "Github"
regex = '''(?i)github.*['\"][0-9a-zA-Z]{35,40}['\"]'''
[whitelists]
files = [
  "(.*?)(jpg|gif|doc|pdf|bin)$"
]

This sample config will look for github and aws keys and ignore and jpg, gif, doc, pdf, or bin files it encounters during the audit.

Whitelists

You can whitelist more than just files! In addition to files you can whitelist regexes, branches, and commits. Say for example you want to ignore the leak that occurs for commit cb5599aeed261b2c038aa4729e2d53ca050a4988 in https://github.com/gitleakstest/gronit/commits/master. You can do this by adding including a commits array to your [whitelist] table in your config. So for the example above our config would look like this now:

[[regexes]]
description = "AWS"
regex = '''AKIA[0-9A-Z]{16}'''
[[regexes]]
description = "Github"
regex = '''(?i)github.*['\"][0-9a-zA-Z]{35,40}['\"]'''
[whitelists]
files = [
  "(.*?)(jpg|gif|doc|pdf|bin)$"
]
commits = [
  "cb5599aeed261b2c038aa4729e2d53ca050a4988"
]

Similarly you can include a branch and regex array in your whitelist table. See these tests for more examples.

Table Driven Tests

More robust code (please add to tests if you want to help the project)

Control of concurrency

You can set the maximum concurrent go routines spun up per repo with --max-go. Note: I've been running unbounded and haven't run into any problems... let me know if you do.

Audit ALL refs.

Prior to v1.0.0 Gitleaks was only auditing origin/HEAD. You can now scrub through all refs with --all-refs. NOTE: support for auditing an array of refs will be coming in next version.

Added support for external regexes

Version 0.3.0 of Gitleaks introduces

• local scan
• meaningful exit codes
• timestamped logs
• refactored for some maintainability

Version 0.2.0 of Gitleaks is the first version update since this got relatively popular. Based on the issues raised it seems that folks want better support for integration into their pipelines.

• Additionally regex checking
• $HOME/.gitleaks/ directory for clones and reports
• Clone into temp dir option
• Persistent repos for Orgs and Users (no more re-cloning)
• Pagination for Org/User list... no more partial repo lists
• Since commit option
• Updated README
• Multi-staged Docker build
• Travis CI

Version 0.1.0 of Gitleaks demonstrates:

• full git history search
• regex/entropy checks
• report generation
• speedier than https://github.com/dxa4481/truffleHog