semgrep

Fixed

• Regression in 0.31.0 where only a single file was being used when --config
  was given a directory with multiple rules (#2019).

Fixed

• Gracefully handle empty configuration file.
• Gracefully handle LexicalErrors from semgrep-core.
• Fix stack overflows in spacegrep on large input files (#1944).
• Fix extension-based file selection when the language is generic (#1968).
• Fix semgrep error when no valid config on path provided (#1912).
• Fix NO_FILE_INFO_YET error which causes the python wrapper to crash (#1925).
• Fix usage of '...' in special builtin arguments for PHP (#1963).
• Fix automatic semicolon insertion parse error in javascript (#1960).

Added

• kotlin-tree-sitter integration into semgrep-core. Can now call
  dump-tree-sitter-cst on kotlin files.
• c++ tree-sitter integration into semgrep-core (#1952).
• More documents for language porting.
• Error handling in spacegrep to print stderr when CalledProcessError occurs.

0.30.0 - 2020-11-03

Added

• Better examples for the generic mode aka spacegrep (#1951).

Fixed

• Fix matching of trailing dots in spacegrep (#1939).
• Allow matching on one-line files with spacegrep (#1929).
• Fix incorrect number of lines matched by dots with spacegrep (#1918).
• Other subtle spacegrep matching bugs (#1913).
• Metavariable for method call should be matched against corresponding
  metavariable in method definition (#1861).
• Typescript class properties/declarations not recognized (#1846).
• Can't match inside Python try/except clause (#1902).

0.29.0 - 2020-10-27

Added

• Semgrep will now partially parse files with parse errors and report findings detected before the parse errors was encountered.
• Allow user to specify registry path without having to add semgrep.dev url
  i.e.: instead of --config https://semgrep.dev/p/r2c-ci users can use --config p/r2c-ci
• Allow user to specify snippet id withouth having to add semgrep.dev url
  i.e.: instead of --config https://semgrep.dev/s/username:snippetname
  users can use --config username:snippetname
• --test will now error out if ruleid or ok is not in reported IDs
• Semgrep will run JavaScript rules on TypeScript files automatically.

Fixed

• More off by one fixes in autofix
• Support for matching dynamic class names in Ruby
• Removed nosem findings from the final findings count
• Matching nested JSX elements works properly. See https://semgrep.dev/s/erlE?version=0.29.0.
• Can now match partial class definitions with annotations in Java. See https://github.com/returntocorp/semgrep/issues/1877.
• Fixed errors in TypeScript "implements" keyword. See https://github.com/returntocorp/semgrep/issues/1850.

Added

• A metavariable-comparison operator for evaluating numeric comparisons on metavariable values, such as comparison: $KEY_SIZE < 2048. This is a safe alternative to pattern-where-python snippets. Check the full documentation of this feature!
• Matching 1-to-N attributes with a ... wildcard in JSX tags' attribute lists, such as <$TAG attr="1" ... />
• Matching only the function signature without the function body, such as function foo(...). This is useful to have cleaner match output when the body content doesn't matter in a rule. This works on JavaScript, TypeScript, and Java code currently.
• SARIF output now includes the exact CWE and OWASP categories as tags. Thanks @hunt3rkillerz!
• Matching of annotation patterns for Java (like @SomeAnnot(...)) in any context.

Fixed

• PHP superglobals such as $_GET, which start with a dollar sign just like Semgrep metavariables, are now correctly interpreted as PHP code instead of Semgrep pattern code.
• Calls to isset(...) in PHP look like function calls, but technically are not functions calls. Now you can match them anyway!
• It's now possible to write unit tests for OCaml rules.
• JavaScript's special identifiers, such as this, can now be captured into a metavariable.
• A Java pattern for implements B will now also match code that does implements A, B, C.
• Addressed off by one errors when applying autofixes
• Missing characters in metavariable interpolation in messages
• And many more minor code parser fixes!

Added

• Added a --debug flag and moved most of the output under --verbose to it.
• Can run multiple rule configs by repeating --config option
• Jenkins information added to integrations
• Added matching with partial patterns for function signatures.

Changed

• Parse and other errors are mentioned at final output, but not individually displayed unless --verbose is passed
• tree-sitter parse error exceptions converted to parsing_error, improving error location

Fixed

• Dislayed types using the message key are more complete.
• Triple token repeat for EncodedString in semgrep messages fixed.
• Crashes on 3 or more layered jsonschema errors fixed.

Fixed

• Metavariables are able to match empty tuples
• Correctly parse optional chaining (?.) in Typescript
• Correctly parse logical assignment operators (&&=, ||=, ??=) in Typescript
• Some type constraing matching in Typescript

Changed

• Added default timeout of 5 seconds to javascript parsing (related to ?. on large minified files stalling)

0.25.0 - 2020-09-23

Added

• Added support for the JUnit XML report format (--junit-xml)
• C now supports the deep expression operator: <... $X ...>. See this example.
• Added support for ellipses ... in PHP. (https://github.com/returntocorp/semgrep/issues/1715). See this example.

Fixed

• JavaScript will parse empty yields (https://github.com/returntocorp/semgrep/issues/1688).
• In JavaScript, arrow functions are now considered lambdas (https://github.com/returntocorp/semgrep/issues/1691). This allows matching arrow functions in var assignments.
• tsx and typescript are now properly recognized in the languages key. (https://github.com/returntocorp/semgrep/issues/1705)

Added

• The --test functionality now supports the --json flag
• Alpha support for TypeScript
• Alpha support for PHP
• PyPI artifacts are now compatible with Alpine Linux

Fixed

• Can now parse ECMAScript object patterns with ellipses in place of fields

Added

• Experimental support for Typescript (with -lang ts). You can currently
  mainly use the Javascript subset of Typescript in patterns, as well
  as type annotations in variable declarations or parameters.
• Ability to read target contents from stdin by specifying "-" target.

Changed

• You can now specify timeouts using floats instead of integers
  (e.g., semgrep -timeout 0.5 will timeout after half a second)

Fixed

• We now respect the -timeout when analyzing languages which have
  both a Tree-sitter and pfff parser (e.g., Javascript, Go).

Added

• The 'languages' key now supports 'none' for running pattern-regex on arbitrary files. See this file for an example.
• You can now use the '...' ellipsis operator in OCaml.
• True negatives to '--test' functionality via the 'ok:' annotation.

Changed

• A groups of rules are now called "Rulesets" in the Semgrep ecosystem,
  instead of their previous name, "Packs".
• We now use also the tree-sitter-javascript Javascript parser, which
  can parse quickly minified files. Thus, we also removed the 5 seconds
  parsing timeout we were using for Javascript.
• We should correctly report ranges when matching array access expressions
  (e.g., 'foo[$X]').
• Breaking: regular expressions in semgrep string patterns (e.g., '"=~/foo/"')
  are now using the PCRE (Perl Compatible Regular Expressions) syntax instead of
  the OCaml syntax. This means you should not escape parenthesis for grouping
  or escape pipes for dijunctions (e.g., use simply '"=~/foo|bar/"' instead of
  '"=~/foo\|bar/"'). You can also use more advanced regexp features available
  in PCRE such as case-insensitive regexps with '/i' (e.g., "=~/foo/i").
  The semantic of matching changes also to look for the regexp anywhere
  in the string, not just at the beginning, which means if you want to
  enforce a format for the whole string, you will now need to use the '^' anchor
  character (e.g., "=~/^o+$/" to check if a string contains only a sequence
  of 'o').

Removed

• Breaking: install script installation procedure (semgrep--ubuntu-generic.sh).
  Please use 'pip install' for equivalent Linux installation.

Added

• Parsing JSX (JavaScript React) files is now supported as a beta feature!
  In this release, you need to target .jsx files one by one explicitly to have them be scanned.
  We're planning to scan all .jsx files in targeted directories in our next release
• We now bundle a json-schema spec for rules YAML syntax.

Changed

• Our custom-made rules YAML validator has been replaced with a jsonschema standard one.
  This results in more reliable and comprehensive error messages
  to help you get back on track when bumping into validation issues.
• Calling semgrep --validate now includes more information,
  such as the number of rules validation ran on.

Fixed

• Fixed a bug where multiple assignment,
  also known as tuple unpacking assignment in Python,
  such as a, b = foo,
  could be misinterpreted by semgrep.
• Fixed a bug that would cause a crash when trying to get debug steps output as JSON.
• .mly and .mll files are no longer targeted implicitly by OCaml scans.
• Fixed the --skip-unknown-extensions flag skipping files even with recognized extensions.
• Fixed JavaScript conditionals without braces,
  such as if (true) return;,
  not being matched by patterns such as if (true) { return; }.

0.20.0 - 2020-08-18

Added

• Support for JSX tag metavariables (e.g., <$TAG />) and ellipsis inside
  JSX attributes (e.g., )
• By default Semgrep treats explicitly passed files with unknown extension as possibly any language and so runs all rules on said files. Add a flag --skip-unknown-extensions so that Semgrep will treat these files as if they matched no language and will so run no rules on them. Link: PR

Fixed

• Python patterns do not have to end with a newline anymore.
• Pattern $X = '...'; in JavaScript matches var $X = '...'. Additionally, semicolon is no longer required to match. Link: Issue; Link: Example
• In JavaScript, can now match destructured object properties inside functions. Link: Issue; Link: Example
• Java annotations can be matched with fully qualified names. Link: Issue; Link: Example
• Ensure /src exists in Dockerfile; Link: PR

Fixed

• Update Docker container to run successfully without special volume
  permissions

Added

• --timeout-threshold option to set the maximum number of times a file can timeout before it is skipped
• Alpha support for C#

Fixed

• Match against JavaScript unparameterized catch blocks
• Parse and match against Java generics
• Add ability to match against JSX attributes using ellipses
• Add ability to use ellipses in Go struct definitions
• No longer convert Go expressions with a newline to a statement

Changed

Added

Match arbitrary content with f"..."
Performance improvements by filtering rules if file doesn't contain string needed for match
Match "OtherAttribute" attributes in any order
Support Python 3.8 self-documenting fstrings
--max-memory flag to set a maximum amount of memory that can be used to apply a rule to a file

Changed

Added

• The metavariable-regex operator, which filters finding's by metavariable
  value against a Python re.match compatible expression.
• --timeout flag to set maximum time a rule is applied to a file
• Typed metavariables moved to official support. See docs

Changed

• Improved pattern-where-python error messages

Added

• Match file-name imports against metavariables using import "$X" (most
  useful in Go)
• Support for taint-tracking rules on CLI using the key-value pair 'mode: taint'
  (defaults to 'mode: search')

Changed

• Don't print out parse errors to stdout when using structured output formats

Fixed

• Parse nested object properties in parameter destructuring in JavaScript
• Parse binding patterns in ECMAScript 2021 catch expressions
• Was mistakenly reporting only one of each type of issue even if multiple issues exist

Added

• Alpha level support for Ruby

Changed

• Show semgrep rule matches even with --quiet flag

Fixed

• Fixed a crash when running over a directory with binary files in it.
• Fix SARIF output format
• Parse nested destructured parameters in JavaScript
• True and False are not keywords in Python2
• Send informative error message when user tries to use semgrep on missing files

Changed

• Default Docker code mount point from /home/repo to /src - this is also
  configurable via the SEMGREP_SRC_DIRECTORY environment variable

Removed

• --precommit flag - this is no longer necessary after defaulting to
  pre-commit's code mount point /src

Fixed

• Parse python files with trailing whitespace
• Parse python2 tuple as parameter in function/lambda definition
• Parse python3.8 positional only parameters (PEP 570)
• Parse python2 implicit array in comprehension
• Cache timeout errors in semgrep-core so running multiple rules does not
  retry parsing