semgrep

Added

• Dockerfile language: metavariables and ellipses are now
  supported in most places where it makes sense (#4556, #4577)

Fixed

• Gracefully handle timeout errors with missing rule_id
• Match resources in Java try-with-resources statements (#4228)

Added

• Autocomplete for CLI options
• Dockerfile: add support for metavariables where argument expansion is already supported

Changed

• Ruby: a metavariable matching an atom can also be used to match an identifier
  with the same name (#4550)

Fixed

• Handle missing target files without raising an exception (#4462)

Added

• Add an experimental key for internal team use: r2c-internal-project-depends-on that
  allows rules to filter based on the presence of 3rd-party dependencies at specific
  version ranges.
• Experimental support for Dockerfile syntax.
• Support nosemgrep comments placed on the line before a match,
  causing such match to be ignored (#3521)
• Add experimental semgrep login and semgrep logout to store API token from semgrep.dev
• Add experimenntal config key semgrep --config policy that uses stored API token to
  retrieve configured rule policy on semgrep.dev

Changed

• CLI: parse errors (reported with --verbose) appear once per file,
  not once per rule/file

Fixed

• Solidity: add support for for(...) patterns (#4530)

Added

• Pre-alpha support for Dockerfile as a new target language
• Semgrep is now able to symbolically propagate simple definitions. E.g., given
  an assignment x = foo.bar() followed by a call x.baz(), Semgrep will keep
  track of x's definition, and it will successfully match x.baz() with a
  pattern like foo.bar().baz(). This feature should help writing simple yet
  powerful rules, by letting the dataflow engine take care of any intermediate
  assignments. Symbolic propagation is still experimental and it is disabled by
  default, it must be enabled in a per-rule basis using options: and setting
  symbolic_propagation: true. (#2783, #2859, #3207)
• --verbose outputs a timing and file breakdown summary at the end
• metavariable-comparison now handles metavariables that bind to arbitrary
  constant expressions (instead of just code variables)

Fixed

• Rust: inner attributes are allowed again inside functions (#4444) (#4445)
• Python: return statement can contain tuple expansions (#4461)
• metavariable-comparison: do not throw a Not_found exn anymore (#4469)
• better ordering of match results with respect to captured
  metavariables (#4488)

0.77.0 - 12-16-2021

Added

• New language Solidity with experimental support.
• Scala: Patterns like List(...) now correctly match against patterns in code
• A default set of .semgrepignore patterns (in semgrep/templates/.semgrepignore) are now used if no .semgrepignore file is provided
• Java: Ellipsis metavariables can now be used for parameters (#4420)
• semgrep login and semgrep logout commands to save api token

Fixed

• Go: fixed bug where using an ellipsis to stand for a list of key-value pairs
  would sometimes cause a parse error
• Scala: Translate definitions using patterns like
  val List(x,y,z) = List(1,2,3) to the generic AST
• Allow name resolution on imported packages named just vN, where N is a number
• The -json option in semgrep-core works again when used with -e/-f
• Python: get the correct range when matching comprehension (#4221)
• Python and other languages: allow matches of patterns containing
  non-ascii characters, but still with possibly many false positives (#4336)
• Java: parse correctly constructor method patterns (#4418)
• Address several autofix output issues (#4428, #3577, #3338) by adding per-
  file line/column offset tracking

Changed

• Constant propagation is now a proper must-analysis, if a variable is undefined
  in some path then it will be considered as non-constant
• Dataflow: Only consider reachable nodes, which prevents some FPs/FNs
• Timing output handles errors and reports profiling times
• semgrep-core will log a warning when a worker process is consuming above 400 MiB
  of memory, or reached 80% of the specified memory limit, whatever happens first.
  This is meant to help diagnosing OOM-related crashes.

Added

• New language Solidity with experimental support.

Fixed

• Python: set the right scope for comprehension variables (#4260)
• Fixed bug where the presence of .semgrepignore would cause reported targets
  to have absolute instead of relative file paths

Fixed

• Fixed bug where the presence of .semgrepignore would cause runs to fail on files that
  were not subpaths of the directory where semgrep was being run

Added

• Improved filtering of rules based on file content (important speedup
  for nodejsscan rules notably)
• Semgrep CLI now respects .semgrepignore files
• Java: support ellipsis in generics, e.g., class Foo<...> (#4335)

Fixed

• Java: class patterns not using generics will match classes using generics
  (#4335), e.g., class $X { ...} will now match class Foo<T> { }
• TS: parse correctly type definitions (#4330)
• taint-mode: Findings are now reported when the LHS of an access operator is
  a sink (e.g. as in $SINK->method), and the LHS operand is a tainted
  variable (#4320)
• metavariable-comparison: do not throw a NotHandled exn anymore (#4328)
• semgrep-core: Fix a segmentation fault on Apple M1 when using
  -filter_irrelevant_rules on rules with very large pattern-eithers (#4305)
• Python: generate proper lexical exn for unbalanced braces (#4310)
• YAML: fix off-by-one in location of arrays

Changed

• semgrep-core: Log messages are now tagged with the process id
• Optimization: change bloom filters to use sets, move location of filter
• Reduced the size of --debug dumps
• Given --output Semgrep will no longer print search results to stdout,
  but it will only save/post them to the specified file/URL

Fixed

• semgrep-ci relies on --disable-nosem still tagging findings with is_ignored
  correctly. Reverting optimization in 0.74.0 that left this field None when said
  flag was used

Added

• Support for method chaining patterns in Python, Golang, Ruby,
  and C# (#4300), so all GA languages now have method chaining
• Scala: translate infix operators to generic AST as method calls,
  so $X.map($F) matches xs map f
• PHP: support method patterns (#4262)

Changed

• Add profiling_times object in --time --json output for more fine
  grained visibility into slow parts of semgrep
• Constant propagation: Any kind of Python string (raw, byte, or unicode) is
  now evaluated to a string literal and can be matched by "..." (#3881)

Fixed

• Ruby: blocks are now represented with an extra function call in Generic so that
  both f(...) and f($X) correctly match f(x) in f(x) { |n| puts n } (#3880)
• Apply generic filters excluding large files and binary files to
  'generic' and 'regex' targets as it was already done for the other
  languages.
• Fix some Stack_overflow when using -filter_irrelevant_rules (#4305)
• Dataflow: When a switch had no other statement following it, and the last
  statement of the switch's default case was a statement, such as throw,
  that can exit the execution of the current function, this caused break
  statements within the switch to not be resolved during the construction of
  the CFG. This could led to e.g. constant propagation incorrectly flagging
  variables as constants. (#4265)

Added

• Experimental support for C++

Fixed

• Catch PCRE errors (#4258)
• Constant propagation: Avoid "Impossible" errors due to unhandled cases

Added

• Java: Add partial support for synchronized blocks in the dataflow IL (#4150)
• Dataflow: Add partial support for await, yield, &, and other expressions
• Field-definition-as-assignemnt equivalence that allows matching expression
  patterns against field definitions. It is disabled by default but can be
  enabled via rule options: with flddef_assign: true (#4187)
• Arrows (a.k.a short lambdas) patterns used to match also regular function
  definitions. This can now be disabled via rule options: with
  arrow_is_function: false (#4187)
• Javascript variable patterns using the 'var' keyword used to also
  match variable declarations using 'let' or 'const'. This can now be
  disabled via rule options: with let_is_var: false

Fixed

• Constant propagation: In a method call x.f(y), if x is a constant then
  it will be recognized as such
• Go: match correctly braces in composite literals for autofix (#4210)
• Go: match correctly parens in cast for autofix (#3387)
• Go: support ellipsis in return type parameters (#2746)
• Scala: parse case object within blocks
• Scala: parse typed patterns with variables that begin with an underscore:
  case _x : Int => ...
• Scala: parse unicode identifiers
• semgrep-core accepts sh as an alias for bash
• pattern-regex: Hexadecimal notation of Unicode code points is now
  supported and assumes UTF-8 (#4240)
• pattern-regex: Update documentation, specifying we use PCRE (#3974)
• Scala: parse nullary constructors with no arguments in more positions
• Scala: parse infix type operators with tuple arguments
• Scala: parse nested comments
• Scala: parse case class within blocks
• metavariable-comparison: if a metavariable binds to a code variable that
  is known to be constant, then we use that constant value in the comparison (#3727)
• Expand ~ when resolving config paths

Changed

• C# support is now GA
• cli: Only suggest increasing stack size when semgrep-core segfaults
• Semgrep now scans executable scripts whose shebang interpreter matches the
  rule's language
• CLI output no longer displays severity levels

0.71.0 - 11-01-2021

Added

• Metavariable equality is enforced across sources/sanitizers/sinks in
  taint mode, and these metavariables correctly appear in match messages
• Pre-alpha support for Bash as a new target language
• Pre-alpha support for C++ as a new target language
• Increase soft stack limit when running semgrep-core (#4120)
• semgrep --validate runs metachecks on the rule

Fixed

• text_wrapping defaults to MAX_TEXT_WIDTH if get_terminal_size reports
  width < 1
• Metrics report the error type of semgrep core errors (Timeout,
  MaxMemory, etc.)
• Prevent bad settings files from crashing Semgrep (#4164)
• Constant propagation: Tuple/Array destructuring assignments now correctly
  prevent constant propagation
• JS: Correctly parse metavariables in template strings
• Scala: parse underscore separators in number literals, and parse
  'l'/'L' long suffix on number literals
• Scala: parse by name arguments in arbitary function types,
  like (=> Int) => Int
• Bash: various fixes and improvements
• Kotlin: support ellipsis in class body and parameters (#4141)
• Go: support method interface pattern (#4172)

Changed

• Report CI environment variable in metrics for better environment
  determination
• Bash: a simple expression pattern can now match any command argument rather
  than having to match the whole command

Added

• Preliminary support for bash

Fixed

• Go: support ... in import list (#4067),
  for example import (... "error" ...)
• Java: ... in method chain calls can now match also 0 elements, to be
  consistent with other use of ... (#4082), so o. ... .foo() will now
  also match just o.foo().
• Config files with only a comment give bad error message (#3773)

Changed

• Resolution of rulesets use legacy registry instead of cdn registry
• Benchmark suite is easier to modify

0.69.1 - 2021-10-14

Fixed

• The --enable-metrics flag is now always a flag, does not optionally take an argument

Added

• C: support ... in parameters and sizeof arguments (#4037)
• C: support declaration and function patterns
• Java: support @interface pattern (#4030)

Fixed

• Reverted change to exclude minified files from the scan (see changelog for
  0.66.0)
• Java: Fixed equality of metavariables bounded to imported classes (#3748)
• Python: fix range of tuples (#3832)
• C: fix some wrong typedef inference (#4054)
• Ruby: put back equivalence on old syntax for keyword arguments (#3981)
• OCaml: add body of functor in AST (#3821)

Changed

• taint-mode: Introduce a new kind of not conflicting sanitizer that must be
  declared with not_conflicting: true. This affects the change made in 0.68.0
  that allowed a sanitizer like - pattern: $F(...) to work, but turned out to
  affect our ability to specify sanitization by side-effect. Now the default
  semantics of sanitizers is reverted back to the same as before 0.68.0, and
  - pattern: $F(...) is supported via the new not-conflicting sanitizers.

Fixed

• Respect --skip-unknown-extensions even for files with no extension
  (treat no extension as an unknown extension)

Added

• Added support for raise/throw expressions in the dataflow engine and improved
  existing support for try-catch-finally

Fixed

• Respect rule level path filtering

Added

• Input can be derived from subshells: semgrep --config ... <(...)
• Java: support '...' in catch (#4002)

Changed

• taint-mode: Sanitizers that match exactly a source or a sink are filtered out,
  making it possible to use - pattern: $F(...) for declaring that any other
  function is a sanitizer
• taint-mode: Remove built-in source source(...) and built-in sanitizer
  sanitize(...) used for convenience during early development, this was causing
  some unexpected behavior in real code that e.g. had a function called source!
• Improved Kotlin parsing from 77% to 90% on our Kotlin corpus.
• Resolution of rulesets (i.e. p/ci) use new rule cdn and do client-side hydration
• Set pcre recursion limit so it will not vary with different installations of pcre
• Better pcre error handling in semgrep-core

Fixed

• taint-mode: Fixed bug where a tainted sink could go unreported when the sink is
  a specific argument in a function call
• PHP: allows more keywords as valid field names (#3954)

Added

• Added support for break and continue in the dataflow engine
• Added support for switch statements in the dataflow engine

Changed

• Taint no longer analyzes dead/unreachable code
• Improve error message for segmentation faults/stack overflows
• Attribute-expression equivalence that allows matching expression patterns against
  attributes, it is enabled by default but can be disabled via rule options: with
  attr_expr: false (#3489)
• Improved Kotlin parsing from 35% to 77% on our Kotlin corpus.

Fixed

• Fix CFG dummy nodes to always connect to exit node
• Deep ellipsis <... x ...> now matches sub-expressions of statements
• Ruby: treat 'foo' as a function call when alone on its line (#3811)
• Fixed bug in semgrep-core's -filter_irrelevant_rules causing Semgrep to
  incorrectly skip a file (#3755)
• PHP: allows more keywords as valid field names (#3954)