semgrep

1.12.0 - 2023-02-13

Fixed

• CLI: No longer reports rules as being run with a lack of interfile: true when interfile
  analysis was not requested. (pa-2528)
• CLI: The 1.11.0 release started printing log lines to stderr even when --quiet was on,
  making it impossible to get well-formed JSON output when mixing stdout and stderr.
  These lines are now gone, and output is again restricted to just scan results.
• Github actions: Output lines in GitHub Actions logs could appear scrambled,
  due to GitHub Actions mixing together the stdout and stderr streams
  in non-deterministic order. Semgrep will now log everything only to one stream in
  GitHub Actions when using text output mode, which ensures lines no longer appear
  scrambled.

1.11.0 - 2023-02-10

Changed

• Pro: Removed already deprecated flags --deep (now --pro), --interfile (now --pro),
  and --interproc (now --pro-intrafile). Also removed already deprecated command
  install-deep-semgrep (now install-semgrep-pro). (pa-2518)

Fixed

• Go: Fixed a bug where the location of Go raw string literals were not being reported correctly.
  This could cause issues with Playground range highlighting and autofix (pa-2206)
• CLI: Progress bar for Semgrep Pro Engine interfile scans now reflects actual progress more faithfully (pa-2313)
• Pro: We now check the downloaded binary before installing it, this should prevent
  installation of corrupted binaries due to errors during the download. (pa-2492)

1.10.0 - 2023-02-08

Added

• Experimental support for XML (gh-5939)
• Rust: Beta support for Rust. (gh-6545)
• Rule syntax: Metavariable bindings bound within metavariable-pattern now persist to outside of the metavariable-pattern (pa-2490)
• Updated all lockfile parsers (except Cargo.lock) to produce better error messages, at the cost of a couple seconds of lowdown on large (>10k lines) lockfiles (sc-better-parsers)

Changed

• Pro: --pro will now enable all Pro features, including Apex, inter-procedural taint
  analysis, and also inter-file analysis for supported languages. For Apex support only
  (and more languages in the future) now use --pro-languages. For intra-file analysis
  only now use --pro-intrafile. Flags --interproc and --interfile are now
  deprecated. (pa-2488)
• The output formatting of semgrep ci is getting revamped in the coming weeks.
  This release includes the first couple changes to the output. (sc-590)
• Packages from the maven ecosystem are now parsed to include their org slug. This means a log4j rule must now use org.apache.logging.log4j:log4j-core instead of just log4j-core. This change is backwards incompatible, in that any Java Supply Chain rules not taking into account will stop producing any findings, since the packages parsed from lockfiles will include the org, but the old rules will not. (sc-maven-org)

Fixed

• Rust: correctly parse the last expression in blocks (gh-7071)
• Dataflow traces: Findings now always display the separating line with --dataflow-traces in the CLI, to reduce
  confusion over where the findings fall between the dataflow traces. (pa-2471)
• CLI: Added install-semgrep-pro to the list of commands in the semgrep --help help text. (pa-2505)
• Fixed bug where gradle.lockfile files would fail to parse if they contained a trailing newline, and bug where an error on a trailing newline would cause our lockfile parse error pretty printing to fail (sc-trailing-newline)

1.9.0 - 2023-02-01

Added

• Pro: If the "Semgrep Pro Engine" toggle is enabled in App, semgrep ci will add
  support for Apex in all scans (including diff scans). (pa-2462)

Fixed

• Fix incorrectly reformatted Bitbucket repository URLs (app-3279)

1.8.0 - 2023-02-01

Added

• Experimental support for Clojure, Lisp, and Scheme. (gh-3328)
• Experimental support for Jsonnet (writing semgrep rules to find
  bugs/security-isses/... in jsonnet files). (pa-1849)

Fixed

• Regexps within string patterns such as "=~/hello/" now support the
  full PCRE syntax, including backreferences and other advanced features
  not supported by ocaml-re. (gh-6913)
• Removed new lockfile parsers for all lockfiles except requirements.txt and maven_dep_tree.txt, for performance reasons (sc-557)

1.7.0 - 2023-02-01

Added

• Experimental support for Clojure, Lisp, and Scheme. (gh-3328)
• Experimental support for Jsonnet (writing semgrep rules to find
  bugs/security-isses/... in jsonnet files). (pa-1849)

Fixed

• Regexps within string patterns such as "=~/hello/" now support the
  full PCRE syntax, including backreferences and other advanced features
  not supported by ocaml-re. (gh-6913)
• Removed new lockfile parsers for all lockfiles except requirements.txt and maven_dep_tree.txt, for performance reasons (sc-557)

1.6.0 - 2023-01-27

Added

• Added exposure property to sarif output for sca findings (sarif)

Changed

• Tests: Allow --test to process entire file trees rather than single files (gh-5487)
• metavariable-pattern: For performance reasons Generic mode ignores target files
  that look like machine generated. Unfortunately, this also prevented using the
  metavariable-pattern operator on text that may look like (or in fact be)
  machine generated, such as an RSA key contained in a legit file. Now, when the
  analysis is requested within a metavariable-pattern operator, Generic mode
  will always match any text even if it looks like machine generated. (pa-2386)
• Pro: Add experimental flags --pro and --interproc. Using --pro you can
  enable Apex support, and with --fast-deep you can enable intra-file inter-procedural
  taint analysis. Also, --deep has been renamed to --interfile. Note that to use
  any of the Pro features you must first run semgrep install-semgrep-pro while being
  logged in. (pa-2440)
• New lockfile parsers with better error messages for all supported ecosystems except Rust (sc-495)

Fixed

• Solidity: parse correctly 'unchecked' blocks (gh-6055)
• Terraform: Implicit coercions between strings, bools, ints, and floats are now allowed to match. (gh-6898)
• Python: Taint now propagates via the splat operators * and **, thus both
  sink(*tainted) and sink(**tainted) will result in findings. (gh-6920)
• Solidity: support ellipsis in contract inheritance
  Thanks to Raz0r for his contribution! (gh-6935)
• CLI: No longer fails when running tests against a config file with no rules in it (gh-6954)
• Fixed a crash that could occur when using the GitHub API to fetch the merge base rather than computing it locally. (merge-base)
• JS/TS: Object types now support metavariables properly (pa-2414)
• CLI: No longer reports the wrong command if you are using the semgrep-core-proprietary executable. (pa-2417)
• Pro: Findings that occur due to rules run with the Semgrep Pro Engine are now displayed separately in the CLI. (pa-2432)
• Fixed bug where Semgrep would generate reachable findings for transitive dependencies, even if a direct dependency at a safe version exists (sc-547)
• Fixed bug in poetry.lock parser where quoted square brackets could cause failures (sc-548)
• Fixed bug where Semgrep would fail to generate unreachable findings for a particular library if a reachable finding for that library had already been found in the same lockfile, even if there are multiple copies of this library at different vulnerable versions. (sc-549)
• Fixed bug where npm lockfiles with packages that listed multiples hashes would fail to parse (sc-550)

1.5.1 - 2023-01-20

Added

• Python: Constant propagation will now recognize the idiom cond and X or Y,
  as well as True and X and False or X. So e.g. cond and "a" or "b" will
  be identified as a constant string. (gh-6079)
• Julia: Julia is now experimental (pa-2366)

Changed

• DeepSemgrep is now Semgrep PRO! To install the Semgrep PRO engine run:
  semgrep install-semgrep-pro. This engine is still invoked using the
  --deep flag, but please expect changes to the CLI in the near future.
  The new Semgrep PRO engine adds support for Apex! (pa-2389)

Fixed

• New 'transform:' field in extract mode rules, with 'concat_json_string_array'
  option useful to extract python code from jupyter notebooks.
  Thanks to Jose Selvi for his contribution! (gh-4477)
• Java: Fixed regression introduced in 0.123.0 that could cause a private class
  attribute to be incorrectly regarded as a constant. (gh-6793)
• Make $F(x) match eval(x). Previously, eval was special-cased and metavariable function call patterns would not match it. (gh-6877)
• DeepSemgrep: Enabled --dataflow-traces by default when --deep is specified (pa-2274)
• In rare situations, mainly in DeepSemgrep and related to naming bugs, the use of
  symbolic propagation could make Semgrep fall into an infinite loop during matching.
  This has been fixed by bounding the number of times that Semgrep can follow
  symbolically-propagated values. (pa-2324)
• CLI: Made an error message for when two autofix matches overlap have a more helpful message, as well as be displayed as a debug message. (pa-2393)
• CLI: Made the warning message when using Semgrep Pro more friendly when logged in (pa-2396)

1.3.0 - 2023-01-04

Changed

• Removed the poor support for reading dependencies from pom.xml files. Instead semgrep will try to read dependencies from a maven_dep_tree.txt file,
  which can be generated using the command mvn dependency:tree -DoutputFile=maven_dep_tree.txt (sc-pom)

Fixed

• Use the GitHub REST API when possible to compute the merge base for semgrep ci, improving performance on shallow clones of large repositories. (gha-mergebase)

• YAML: Fixed a bug where metavariables matching YAML double-quoted strings would not capture the entire range of the string, and would
  not contain the double-quotes. Also added the ability to properly use patterns like "$FOO", which will unpack the contents of the matched string. (pa-2332)

• Fixed a race condition related to the parsing cache that could lead to internal errors (pa-2335)

• YAML: Fixed a bug where literal or folded blocks would not be parsed properly.

  So for instance, in:

  key: |
    string goes here
  

  A metavariable matching the contents of the string value might not be correct. (pa-2347)

• Julia: Greatly improved parsing support (pa-2362)

1.2.1 - 2022-12-16

Fixed

• Go: fix NoTokenLocation for metavariables matching function type without
  an argument (e.g. func()) (gh-6715)
• typed-metavariables: handle ternary expressions so we can type correctly
  expressions like foo(cond ? new A() : this.a) (pa-2328)
• Reverted a change which caused findings with different sources (but the same sink) to be deduplicated. This would cause findings which
  looked identical in range and data, but had different taint traces. (pa-2336)

1.2.0 - 2022-12-15

Fixed

• Fixed rare crash that could occur due to stale file caches when temp file names overlap (cache-invalidation)
• PHP: support metavariables in string (as in foo("$VAR")) (gh-6311)
• Java: support static block patterns (gh-6366)
• Rust: parse correctly scoped identifier in constructor (gh-6594)
• Java: support super(...) patterns (gh-6638)
• C#: bugfix on bad interaction between -fast and the internal code generated
  for LINQ queries (gh-6666)
• Java: support the Java 10 'var' keyword by not using 'var' as a valid type when
  using typed metavariables. (gh-6672)
• When matching module specifiers in imports, prevent metavariables from capturing quotes. (gh-6674)
• Swift: support complex expression in switch case (gh-6704)
• Constant propagation: Fixed a bug where constant propagation would only run within functions. Now,
  it runs on the top-level of the program as well. (pa-1656)
• DeepSemgrep: Fixed a bug where imports which reached type names (among other things) would not resolve properly (pa-2260)
• DeepSemgrep: Fixed a bug which caused Java functions with interfaces to wipe taint (instead of propagating taint by default) (pa-2265)
• Fix matching of qualified names. For example, new $X.Foo() will now match
  new a.b.Foo(). (pa-2296)
• DeepSemgrep: Fix regression in taint-mode introduced by Semgrep v1.1 that caused
  some findings to be missed. Also, DeepSemgrep will assume, for now, that a method
  call on a tainted object is always tainted. (pa-2304)
• Improved matching behavior on JS require calls (require-match)

1.1.0 - 2022-12-05

Added

• JSON output: Added a max_memory_bytes field to the semgrep --time output which corresponds to the amount of memory allocated during the OCaml phase of Semgrep. This is useful for telemetry purposes. (pa-2075)

Changed

• Don't print out summary of blocking rules unless running with semgrep ci subcommand (gh-6651)

Fixed

• taint-mode: In 0.94.0 we made that when a pattern-source (or pattern-sanitizer)
  matched a variable exactly, this was understood as that variable being tainted
  (sanitized, resp.) by side-effect. For example, given tainted(x) we would taint x
  by side-effect, and subsequent occurrences of x were also considered tainted.
  This allowed to write rules like c.lang.security.use-after-free.use-after-free
  in a very succint way, and it also addressed some limitations of the workarounds that
  were being used to simulate this until then.

  This worked well initially, or so we thought, until in 0.113.0 we added
  field-sensitivity to taint-mode, and in subsequent versions we made sources and
  sanitizers apply by side-effect to more kinds of l-values than just simple variables.
  It was then that we started to see regressions that were fairly unintuitive for users.
  For example, if $_GET['foo'] was a taint source, this would make $_GET itself to
  be tainted by side-effect, and a subsequent expression like $_GET['bar'] was also
  considered tainted.

  We now correct the situation by adding the by-side-effect option to sources and
  sanitizers, and requiring this option to be explicitly enabled
  (that is, by-side-effect: true) in order to apply the source or the sanitizer by
  side-effect. Otherwise, the default is that sources and sanitizers matching l-values
  apply only to the precise occurrences that they match. (pa-1629)

• taint-mode: Fixed matching of pattern-sinks to be more precise, so that e.g.
  it will no longer report sink(ok1 if tainted else ok2) as a tainted sink, as
  the expression passed to the sink is actually not tainted. (pa-2142)

• CLI: Separated experimental rules from normal rules in semgrep --debug output. (pa-2159)

• Taint: Fixed an issue where findings with the same sink would be identified as the same, and cause
  only one of them to be reported, even if they had different sources. (pa-2208)

• DeepSemgrep: When the "DeepSemgrep" setting is enabled in Semgrep App, semgrep ci
  will try to run the analysis using the DeepSemgrep engine. But if this engine was
  not installed, semgrep ci failed. Now semgrep ci will automatically try to
  install DeepSemgrep if it is not already present. Note that, if DeepSemgrep is
  already installed, semgrep ci does not attempt to upgrade it to a newer version. (pa-2226)

• CLI: Made the number of jobs when using semgrep --deep default to 1. (pa-2231)

• Autofix: If multiple autofixes are targeting an overlapping range, then one of them is picked arbitrarily to occur, to prevent autofixes which may produce incorrect code. (pa-2276)

• DeepSemgrep: Time data now outputs properly when running semgrep --deep --time (pa-2280)

• DeepSemgrep: Added a message which suggests that users update their version of DeepSemgrep, if the DeepSemgrep binary crashes (pa-2283)

• Yarn 2 parse failure on versions like @storybook/react-docgen-typescript-plugin@canary. This is only present as some kind special version range specifier and never appears as a concrete version. It would only be used to check if the dependency was in the manifest file, so we just parse the version as "canary"
  Yarn 2 parse failure on versions like @types/ol-ext@npm:@siedlerchr/[email protected]
  Yarn 2 parse failure on versions like resolve@patch:resolve@^1.1.7#~builtin<compat/resolve>. These are now just ignored, as they appear to always come with a non-patch version as well. (sc-406)

1.0.0 - 2022-12-01

Added

• DeepSemgrep: If you have a Team tier account in Semgrep App, and you enable the
  DeepSemgrep setting, then semgrep ci will automatically run the DeepSemgrep
  engine instead of the OSS engine on full scans (but not in PR scans). (pa-2226)

0.123.0 - 2022-11-29

Added

• CLI: Added deep traces to --dataflow-traces (pa-2116)

Changed

• Reachable Supply Chain findings will no longer block pull requests when using semgrep ci.
  Note that unreachable findings have been non-blocking already. (sca-nonblocking)

Fixed

• Fix matching issue related to JS imports with multiple imported values (gh-5305)
• DeepSemgrep: Keep only the shortest trace originating from a taint source. This
  also prevents falling into infinite loops when inferring taint signatures for
  mutually recursive procedures. (pa-2224)
• taint-mode: Improved taint tracking for array-like accesses. Previously, if
  x.a.b[i].c got tainted, Semgrep would track x.a.b as tainted, and thus
  x.a.b[i].d would be incorrectly considered as tainted too. Now Semgrep will
  do the right thing and track x.a.b[*].c as tainted, and x.a.b[i].d will
  not be considered tainted. (pa-2225)
• Java: private, singly-assigned class variables now permit constant propagation (pa-2230)
• JS/TS: Allow dependencies to @Injectable and @Component classes in Angular JS to be visible outside the scope of the constructor. (pa-2233)
• Fix matching issue related to Python imports with multiple imported values (python-imports)
• Supply Chain findings from a yarn.lock lockfile were marked as 'transitive'
  when we couldn't find the matching package.json file.
  These findings will now be marked as having 'unknown' transitivity. (sc-425)
• Make $X(...) match this() and super(). (this-match)

0.122.0 - 2022-11-16

Fixed

• DeepSemgrep: Added installation path for DeepSemgrep on M1 machines (pa-2153)
• Correctly handle optional name field in yaml1 lockfiles (parse_yaml)

0.121.2 - 2022-11-10

Fixed

• Fix yaml excessive mapping captures (gh-5698)
• Rule syntax: Allow pattern-not, pattern-inside, and pattern-not-inside to take in arbitrary patterns (such as patterns, pattern-either, and friends) (pa-1723)
• Kotlin: Fixed bug where constructor invocation with a trailing comma would not parse (pa-1988)
• Constant propagation: Type casts and parenthesized expressions (in Go) can now be symbolically propagated. (pa-2054)
• CLI: Added a fix preventing findings from experimental rules being displayed.
  Experimental rules also now no longer run silently. (pa-2121)

0.121.1 - 2022-11-08

No significant changes.

0.121.0 - 2022-11-07

Changed

• taint-mode: Semgrep can now track taint through l-values of the form this.x.
  It will also be more precise when tracking taint on l-values involving an
  array access, previously if x.a[i] was tainted, then x itself was tainted;
  now only x.a will be considered tainted. (pa-2086)

Fixed

• JS: correctly parse exported function pattern (gh-6218)
• Terraform: Can now analyze constant-propagated values of variables with metavariable-comparison and friends (pa-2088)

0.120.0 - 2022-11-02

Added

• Fail gracefully and print error message when running in unsupported Linux aarch64/arm64 environment (arm-fail)
• Added 'fingerprints' field to results in sarif output (gh-5729)
• Add dataflow traces as 'codeFlows' object in SARIF output (gh-6367)
• Fail immediately if semgrep tries to run a supply chain rule with an invalid version range specifier (ssc-fail-early)
• Added Supply Chain support for requirements.txt lockfiles (with requirement.in manifest files) and Yarn 2/3 lockfiles (ssc-yarn-req)

Changed

• Change default behavior of CircleCI configurations. If a user manually sets their environment variables (i.e. SEMGREP_REPO_NAME, SEMGREP_REPO_URL, SEMGREP_BRANCH, SEMGREP_JOB_URL, SEMGREP_COMMIT), use it before falling back on autodetection. (app-2434)
• Change default behavior of Buildkite CI configurations. If a user manually sets their environment variables (i.e. SEMGREP_REPO_NAME, SEMGREP_REPO_URL, SEMGREP_BRANCH, SEMGREP_JOB_URL, SEMGREP_COMMIT), use it before falling back on autodetection. (app-2435)
• Change default behavior of bitbucket CI configurations. If a user manually sets their environment variables (i.e. SEMGREP_REPO_NAME, SEMGREP_REPO_URL, SEMGREP_BRANCH, SEMGREP_JOB_URL, SEMGREP_COMMIT), use it before falling back on autodetection. (app-2436)

Fixed

• Handle unexpected exceptions when performing AST-based autofix. (autofix-exception)
• Fix an issue that could cause AST-based autofix to fail on autofixes containing statements. (autofix-statement-exception)
• Fix an issue preventing AST-based autofix from running in the presence of focus-metavariable. (focus-metavariable-autofix)
• Implement string literal metavariables in Python AST-based autofix (gh-3648)
• Go: parse correctly generic types from other packages (gh-5799)
• Fix parsing of dot access ellipsis in PHP patterns (e.g. $x-> ... ->bar()). (gh-6183)
• JS/TS: Allowed parsing of patterns using the optional chaining "?." syntax. (gh-6201)
• Dockerfile language: Add support for RUN options such as
  RUN --mount=type=$TYPE,target=$TARGET .... (gh-6353)
• taint-mode: Fixed a bug in the experimental taint-labels feature that caused labels to be incorrectly applied to dot accesses. For instance, if a pattern-source that requires label A and adds label B matches a dot-access expression like x.a, the field a will get the label B even if it does not carry label A as required. (gh-6355)
• Use AST-based autofix when possible for JS autofixes. This is more likely to lead to correct output, especially for complicated fixes. (js-autofix)
• taint-mode: Fixed regression in 0.113.0, after field sensitivity support was added,
  that broke existing behavior when a prefix in a chain of dot-accesses such as
  x.a.b was specified as a source/sanitizer/sink. For example, if x had been
  previously tainted, then we encountered sink(x.a.b) where x.a matched a
  sanitizer, there was a finding reported because x.a.b was incorrectly considered
  tainted. (pa-1928)
• JS/TS: Fixed a parsing bug where special identifiers were parsed differently in patterns (pa-2030)
• Language server now appropriately applies regex fixes (vscode-regex)

0.118.0 - 2022-10-19

Added

• Taint mode will now track taint coming from the default values of function
  parameters. For example, given def test(url = "http://example.com"):,
  if "http://example.com" is a taint source (due to not using TLS), then
  url will be marked as tainted during the analysis of test. (gh-6298)

• taint-mode: Added two new rule options that help minimizing false positives.

  First one is taint_assume_safe_indexes, which makes Semgrep assume that an
  array-access expression is safe even if the index expression is tainted. Otherwise
  Semgrep assumes that e.g. a[i] is tainted if i is tainted, even if a is not.
  Enabling this option is recommended for high-signal rules, whereas disabling it
  may be preferred for audit rules. Currently, it is disabled by default for pure
  backwards compatibility reasons, but this may change in the near future after some
  evaluation.

  The other one is taint_assume_safe_functions, which makes Semgrep assume that
  function calls do NOT propagate taint from their arguments to their output.
  Otherwise, Semgrep always assumes that functions may propagate taint. This is
  intended to replace not conflicting sanitizers (added in v0.69.0) in the future.
  This option is still experimental and needs to be complemented by other changes
  to be made in future releases. (pa-1541)

Changed

• Ignore the .npm/ directory by default in Semgrep scans (gh-6315)
• The --scan-unknown-extensions option is now set to false by default.
  This means that from now on --skip-unknown-extensions is the default.
  This is an important change that prevents many errors when using
  Semgrep in a pre-commit context or in CI. (pa-1932)

Fixed

• Add autodetection for pull request numbers for Azure Pipelines. If SEMGREP_PR_ID is set, override the autodetection. (app-2083)
• Fixed an autofix regression that caused Semgrep to fail to replace metavariables in string literals, e.g. foo("xyz $X"). (autofix-string-metavar)