Added

• HCL (a.k.a Terraform) experimental support

Changed

• Constant propagation now assumes that void methods may update the callee (#3316)
• Add rule message to emacs output (#3851)
• Show stack trace on fatal errors (#3876)
• Various changes to error messages (#3827)

Fixed

• Dataflow: Recognize "concat" method and interpret it in a language-dependent manner (#3316)
• PHP: allows certain keywords as valid field names (#3907)

Changelog

This project adheres to Semantic Versioning.

0.65.0 - 09-13-2021

Added

• Allow autofix using the command line rather than only with the fix: YAML key

Changed

• Grouped semgrep CLI options and added constraints when useful (e.g. cannot use --vim and --emacs at the same time)

Fixed

• Taint detection with ternary ifs (#3778)
• Fixed corner-case crash affecting the pattern: $X optimization ("empty And; no positive terms in And")
• PHP: Added support for parsing labels and goto (#3592)
• PHP: Parse correctly constants named PUBLIC or DEFAULT (#3589)
• Go: Added type inference for struct literals (#3622)
• Fix semgrep-core crash when a cache file exceeds the file size limit
• Sped up Semgrep interface with tree-sitter parsing

Added

• Enable associative matching for string concatenation (#3741)

Changed

• Add logging on failure to git ls-files (#3777)

Fixed

• Java: separate import static from regular imports during matching (#3772)
• Taint mode will now benefit from semgrep-core's -filter_irrelevant_rules
• Taint mode should no longer report duplicate matches (#3742)
• Only change source directory when running in docker context (#3732)

Added

• C#: support ellipsis in declarations (#3720)

Fixed

• Hack: improved support for metavariables (#3716)
• Dataflow: Disregard type arguments but not the entire instruction

Changed

• Optimize ending ... in pattern-insides to simply match anything left

Added

• OCaml: support module aliasing, so looking for List.map will also
  find code that renamed List as L via module L = List.
• Add help text to sarif formatter output if defined in metadata field.
• Update shortDescription in sarif formatter output if defined in metadata field.
• Add tags as defined in metadata field in addition to the existing tags.

Fixed

• core: Fix parsing of numeric literals in rule files
• Java: fix the range and autofix of Cast expressions (#3669)
• Generic mode scanner no longer tries to open submodule folders as files (#3701)
• pattern-regex with completely empty files (#3705)
• --sarif exit code with suppressed findings (#3680)
• Fixed fatal errors when a pattern results in a large number of matches
• Better error message when rule contains empty pattern

Changed

• Add backtrace to fatal errors reported by semgrep-core
• Report errors during rule evaluation to the user
• When anded with other patterns, pattern: $X will not be evaluated on its own, but will look at the context and find $X within the metavariables bound, which should be significantly faster

Added

• Hack: preliminary support for hack-lang
  thanks to David Frankel, Nicholas Lin, and more people at Slack!
• OCaml: support for partial if, match, and try patterns
  (e.g., if $X = $Y)
• OCaml: you can match uppercase identifiers (constructors, module names) by
  using a metavariable with an uppercase letter followed by an underscore,
  followed by uppercase letters or digits (e.g. $X_, $F_OO).
  Instead, $FOO will match everything else (lowercase identifiers,
  full expressions, types, patterns, etc.).
• OCaml: match cases patterns are now matched in any order, and ellipsis are
  handled correctly
• Improved error messages sent to the playground

Changed

• Run version check and print upgrade message after scan instead of before
• OCaml: skip ocamllex and ocamlyacc files. Process only .ml and .mli files.
• Memoize range computation for expressions and speed up taint mode
• Report semgrep-core's message upon a parse error
• Deprecated the following experimental features:
  • pattern-where-python
  • taint-mode
  • equivalences
  • step-by-step evaluation output
• Deduplicate findings that fire on the same line ranges and have the same message.

Fixed

• Go: Match import module paths correctly (#3484)
• OCaml: use latest ocamllsp 1.7.0 for the -lsp option
• OCaml: include parenthesis tokens in the AST for tuples and constructor
  calls for better range matching and autofix
• OCaml: fixed many matching bugs with ellipsis
• core: Do not crash when is not possible to compute range info
• eliminate 6x slowdown when using the '--max-memory' option

Fixed

• C/C++: Fixed stack overflows (segmentation faults) when processing very large
  files (#3538)
• JS: Fixed stack overflows (segmentation faults) when processing very large
  files (#3538)
• JS: Detect numeric object keys 1 and 0x1 as equal (#3579)
• OCaml: improved parsing stats by using tree-sitter-ocaml (from 25% to 88%)
• taint-mode: Check nested functions
• taint-mode: foo.x is now detected as tainted if foo is a source of taint
• taint-mode: Do not crash when is not possible to compute range info
• Rust: recognize ellipsis in macro calls patterns (#3600)
• Ruby: represent correctly a.(b) in the AST (#3603)

Changed

• Added precise error location for the semgrep metachecker, to detect for example
  duplicate patterns in a rule

Added

• A new experimental 'join' mode. This mode runs multiple Semgrep rules
  on a codebase and "joins" the results based on metavariable contents. This
  lets users ask questions of codebases like "do any 3rd party
  libraries use a dangerous function, and do I import that library directly?" or
  "is this variable passed to an HTML template, and is it rendered in that template?"
  with several Semgrep rules.

Fixed

• Improve location reporting of errors
• metavariable-pattern: pattern-not-regex now works (#3503)
• Rust: correctly parse macros (#3513)
• Python: imports are unsugared correctly (#3940)
• Ruby: pattern: $X in the presence of interpolated strings now works (#3560)

Fixed

• Significant speed improvements, but the binary is now 95MB (from 47MB
  in 0.58.1, but it was 170MB in 0.58.0)

Fixed

• The --debug option now displays which files are currently processed incrementally;
  it will not wait until semgrep-core completely finishes.

Changed

• Switch from OCaml 4.10.0 to OCaml 4.10.2 (and later to OCaml 4.12.0) resulted in
  smaller semgrep-core binaries (from 170MB to 47MB) and a smaller docker
  image (from 95MB to 40MB).

Added

• New iteration of taint-mode that allows to specify sources/sanitizers/sinks
  using arbitrary pattern formulas. This provides plenty of flexibility. Note
  that we breaks compatibility with the previous taint-mode format, e.g.
  - source(...) must now be written as - pattern: source(...).
• HTML experimental support. This does not rely on the "generic" mode
  but instead really parses the HTML using tree-sitter-html. This allows
  some semantic matching (e.g., matching attributes in any order).
• Vue.js alpha support (#1751)
• New matching option implicit_ellipsis that allows disabling the implicit
  ... that are added to record patterns, plus allow matching "spread fields"
  (JS ...x) at any position (#3120)
• Support globstar (**) syntax in path include/exclude (#3173)

Fixed

• Apple M1: Semgrep installed from HomeBrew no longer hangs (#2432)
• Ruby command shells are distinguished from strings (#3343)
• Java varargs are now correctly matched (#3455)
• Support for partial statements (e.g., try { ... }) for Java (#3417)
• Java generics are now correctly stored in the AST (#3505)
• Constant propagation now works inside Python with statements (#3402)
• Metavariable value replacement in message/autofix no longer mixes up short and long names like $X vs $X2 (#3458)
• Fixed metavariable name collision during interpolation of message / autofix (#3483)
  Thanks to Justin Timmons for the fix!
• Revert pattern: $X optimization (#3476)
• metavariable-pattern: Allow filtering using a single pattern or
  pattern-regex
• Dataflow: Translate call chains into IL

Changed

• Faster matching times for generic mode

Added

• new options: field in a YAML rule to enable/disable certain features
  (e.g., constant propagation). See https://github.com/returntocorp/semgrep/blob/develop/semgrep-core/src/core/Config_semgrep.atd
  for the list of available features one can enable/disable.
• Capture groups in pattern-regex: in $1, $2, etc. (#3356)
• Support metavariables inside atoms (e.g., foo(:$ATOM))
• Support metavariables and ellipsis inside regexp literals
  (e.g., foo(/.../))
• Associative-commutative matching for bitwise OR, AND, and XOR operations
• Add support for $...MVAR in generic patterns
• Add support for $...MVAR in generic patterns.
• metavariable-pattern: Add support for nested Spacegrep/regex/Comby patterns
• C#: support ellipsis in method parameters (#3289)

Fixed

• C#: parse __makeref, __reftype, __refvalue (#3364)
• Java: parsing of dots inside function annotations with brackets (#3389)
• Do not pretend that short-circuit Boolean AND and OR operators are commutative (#3399)
• metavariable-pattern: Fix crash when nesting a non-generic pattern within
  a generic rule
• metavariable-pattern: Fix parse info when matching content of a metavariable
  under a different language
• generic mode on Markdown files with very long lines will now work (#2987)

Changed

• generic mode: files that don't look like nicely-indented programs
  are no longer ignored, which may cause accidental slowdowns in setups
  where excessively large files are not excluded explicitly (#3418).
• metavariable-comparison: Fix crash when comparing integers and floats
• Do not filter findings with the same range but different metavariable bindings (#3310)
• Set parsing_state.have_timeout when a timeout occurs (#3438)
• Set a timeout of 10s per file (#3434)
• Improvements to contributing documentation (#3353)
• Memoize getting ranges to speed up rules with large ranges
• When anded with other patterns, pattern: $X will not be evaluated on its own, but will look at the context and find $X within the metavariables bound, which should be significantly faster

0.56.0 - 2021-06-15

Added

• Associative-commutative matching for Boolean AND and OR operations
  (#3198)
• Support metavariables inside strings (e.g., foo("$VAR"))
• Support metavariables inside atoms (e.g., foo(:$ATOM))
• metavariable-pattern: Allow matching the content of a metavariable under
  a different language.

Fixed

• C#: Parse attributes for local functions (#3348)
• Go: Recognize other common package naming conventions (#2424)

Changed

• Upgrade TypeScript parser (#3102)

Added

• Add helpUri to sarif output if rule source metadata is defined

Fixed

• JSON: handle correctly metavariables as field (#3279)
• JS: support partial field definitions pattern, like in JSON
• Fixed wrong line numbers for multi-lines match in generic mode (#3315)
• Handle correctly ellipsis inside function types (#3119)
• Taint mode: Allow statement-patterns when these are represented as
  statement-expressions in the Generic AST (#3191)

Added

• Added new metavariable-pattern operator (available only via --optimizations), thanks to Kai Zhong for the feature request (#3257).

Fixed

• Scala: parse correctly symbol literals and interpolated strings containing double dollars (#3271)
• Dataflow: Analyze foreach body even if we do not handle the pattern yet (#3155)
• Python: support ellipsis in try-except (#3233)
• Fall back to no optimizations when using unsupported features: pattern-where-python, taint rules, and --debugging-json (#3265)
• Handle regexp parse errors gracefully when using optimizations (#3266)
• Support equivalences when using optimizations (#3259)
• PHP: Support ellipsis in include/require and echo (#3191, #3245)
• PHP: Prefer expression patterns over statement patterns (#3191)
• C#: Support unsafe block syntax (#3283)

Changed

• Run rules in semgrep-core (rather than patterns) by default (aka optimizations all)

Added

• Per rule parse times and per rule-file parse and match times added to opt-in metrics
• $...MVAR can now match a list of statements (not just a list of arguments) (#3170)

Fixed

• JavaScript parsing: Support decorators on
  properties
• JavaScript parsing: Allow default export for any declaration
• Metavariables in messages are filled in when using --optimizations all
• Python: class variables are matched in any order (#3212)
• Respect --timeout-threshold option in --optimizations all mode

Changed

• Moved some debug logging to verbose logging
• $...ARGS can now match an empty list of arguments, just like ... (#3177)
• JSON and SARIF outputs sort keys for predictable results

Added

• Scala alpha support
• Metrics collection of project_hash in cases where git is not available
• Taint mode now also analyzes top-level statements.

Fixed

• Running with --strict will now return results if there are nosem mismatches. Semgrep will report a nonzero exit code if --strict is set and there are nosem mismathces. #3099
• PHP: parsing correctly ... and metavariables in parameters
• PHP: parsing correctly functions with a single statement in their body
• Evaluate interpolated strings during constant propagation (#3127)
• Fixed #3084 - Semgrep will report an InvalidRuleSchemaError for dictionaries with duplicate key names.
• Basic type inference also for implicit variable declarations (Python, Ruby, PHP, and JS)
• JS/TS: differentiating tagged template literals in the AST (#3187)
• Ruby: storing parenthesis in function calls in the AST (#3178)

0.52.0 - 2021-05-18

Added

• C# alpha support
• Let meta-variables match both a constant variable occurrence and that same
  constant value (#3058)

Fixed

• OCaml: fix useless-else false positives by generating appropriate AST for
  if without an else.
• JS/TS: Propagate constant definitions without declaration

Added

• Keep track of and report rule parse time in addition to file parse time
• v0 of opt-in anonymous aggregate metrics

Fixed

• JS/TS: allow the deep expression operator <... ...> in expression
  statement position, for example:

$ARG = [$V];
...
<... $O[$ARG] ...>; // this works now

• PHP arrays with dots inside parse
• Propagate constants in nested lvalues such as y in x[y]
• C# experimental support

Changed

• Show log messages from semgrep-core when running semgrep with
  --debug.
• By default, targets larger than 1 MB are now excluded from semgrep
  scans. New option --max-target-bytes 0 restores the old behavior.
• Report relative path instead of absolute when using --time

Changed

• Reinstate --debugging-json to avoid stderr output of --debug