syntect

Improvements

• Fix how syntect::util::split_at() handles multi-byte characters
• Allow case insensitive search for .tmtheme paths
• impl PartialEq for Theme

Dependencies

• Upgrade fancy-regex to 0.11
• Upgrade regex-syntax to 0.7
• Replace lazy_static with once_cell

Other

• Change MSRV policy to "last three stable versions"
• Make Debug impl of syntect::highlighting::Color less verbose

Thanks to @Enselic for basically all the work to make this release happen!

Breaking changes

• Lazy-load syntaxes to significantly improve startup time. This changes the binary format of syntax dump files.
• Remove ContextId::new() from public API to support lazy-loading of syntaxes
• Rename HighlightLines::highlight() to HighlightLines::highlight_line() to make it clear that the function takes one line at a time
• Make plist dependency (used for loading themes) optional via new plist-load feature
• Remove obsolete dump-load-rs and dump-create-rs features that has been identical to dump-load and dump-create for two years
• Remove deprecated items ThemeSettings::highlight_foreground, ThemeSettings::selection_background, ClassedHTMLGenerator::new, ClassedHTMLGenerator::parse_html_for_line, html::css_for_theme, html::tokens_to_classed_html and html::tokens_to_classed_spans
• Mark all error enums as #[non_exhaustive]
• These functions have been changed to return a Result to allow propagation of errors:
  • html::ClassedHTMLGenerator::parse_html_for_line_which_includes_newline
  • html::append_highlighted_html_for_styled_line
  • html::css_for_theme_with_class_style
  • html::highlighted_html_for_string
  • html::line_tokens_to_classed_spans
  • html::styled_line_to_highlighted_html
  • parsing::ParseState::parse_line
  • parsing::ScopeStack::apply
  • parsing::ScopeStack::apply_with_hook
  • parsing::syntax_definition::Context::match_at
  • parsing::syntax_definition::ContextReference::id
  • parsing::syntax_definition::ContextReference::resolve

Other changes

• Fall back to Plain Text if a referenced syntax is missing
• Add support for hidden_file_extensions key in syntaxes.
• Implement Error and Display for all error enums by using thiserror
• Replace lazycell with once_cell to fix crash on lazy initialization
• Add ScopeRangeIterator
• Add CI check for Minimum Supported Rust Version. This is currently Rust 1.53.
• Make looking up a syntax by extension use case-insensitive comparison
• Make from_dump_file() ~15% faster
• Blend alpha value on converting colors to ANSI color sequences
• Fix sample code in documentation to avoid double newlines
• Fix lots of build warnings and lints
• Add Criterion benchmarks for a whole syntect pipeline and for from_dump_file()

See the release notes for v4.7.0, this release removes a new Cargo feature which constituted a semver violation:

• Remove 'plist-load' feature again due to semver violation. #403

Big release this time thanks to tons of fantastic contributions from @Enselic, this release was basically all him! The headline feature is much faster startup time due to lazy-loading at the syntax level.

• Lazy-load syntaxes to significantly improve startup time
• Remove ContextId::new() from public API to support lazy-loading of syntaxes. This is technically a breaking change but I have no idea why I made this API public and I'm pretty sure zero people have used it.
• Add ScopeRangeIterator
• Add CI check for Minimum Supported Rust Version. This is currently Rust 1.51.
• Make 'plist' dependency (used for loading themes) optional via new 'plist-load' feature
• Make looking up a syntax by extension use case-insensitive comparison
• Make from_dump_file() ~15% faster
• Blend alpha value on converting colors to ANSI color sequences
• Fix sample code in documentation to avoid double newlines
• Fix lots of build warnings and lints
• Add Criterion benchmarks for a whole syntect pipeline and for from_dump_file()
• Replace lazycell with once_cell to fix crash on lazy initialization

• Add html::line_tokens_to_classed_spans to also take a mutable ScopeStack, deprecate tokens_to_classed_spans, to avoid panics and incorrect highlighting.
• Derive Hash for Color and Style
• Add find_unlinked_contexts to SyntaxSet
• Add syntaxes method to SyntaxSetBuilder
• Bump fancy-regex to v0.7 and yaml-rust to v0.4.5

• Added a new function for producing classed HTML which handles newlines correctly and deprecated old one. #307

• Errors are now Send + Sync + 'static #304

I also forgot to make a Github release for v4.3.0 but released it on crates.io so belatedly in v4.3.0:

• Fixes unnecesary dependency of the html feature on the assets feature. #300
• Adds ability to add prefixes to html module CSS class names. #296

• Updates to new versions of onig and plist. The new onig version doesn't require bindgen thus making compilation easier. #293

• Properly handle backreferences in included contexts #288

• Make sure errors implement Send #285
• Fix errors to not use the deprecated description() #286

Thanks @sharkdp for the bug fixes! Bumping second part of semver since Send is adding functionality (back).

Headline feature: pure-Rust fancy-regex engine option

Users can now opt in to a pure-Rust regex engine using Cargo features, making
compilation easier in general. People experiencing difficulty compiling for
Windows and Wasm should try switching to fancy-regex. Note this currently
approximately halves highlighting speed.

See the Readme and #270 for details.
Thanks to @robinst for implementing this!

Other changes

• Ability to generate CSS for a theme for use with classed HTML generation (won't always be correct) #274
• Don't generate empty spans in classed HTML #276
• Miscellaneous dependency bumps and cleanup

Breaking changes and upgrading

Upgrading should cause no errors for nearly all users. Users using more unusual APIs may have a small amount of tweaking to do.

• Edit March 30: If you generate custom pack files and want to use fancy-regex you need to regenerate them. The binary format is the same but at YAML loading time regex rewrites get applied that make fancy-regex work properly.
• If you use default-features = false you may need to update your features to choose a regex engine
• A bunch of technically public APIs that I don't know if anyone uses changed due to the regex engine refactor, common uses shouldn't break

• Fixes multiple bugs
• Add RangedHighlightIterator
• Add as_latex_escaped util

Hopefully no behavior changes.

Really a bug fix but the bug was not having an API be public so hence SemVer demands a minor version bump.

• Add support for loading metadata (#223 #225 #230)
• Improve support for generating classed HTML and fix a bug, old function is deprecated because it's impossible to use correctly (#235)
• Update plist to v0.4 and pretty_assertions to v0.6 (#232 #236)

• Fix application of multiple with_prototypes (#220, fixes #160, #178, ASP highlighting)
• Fix prototype marking logic (#221, fixes #219)

• Fix a bug with syntaxes that used captures in lookarounds (#176 #215)
• Fix the precedence order of syntaxes to match Sublime (#217 #216)

See previous release for major breaking changes and new things.

This is a major release with multiple breaking API changes, although upgrading shouldn't be too difficult. It fixes bugs and comes with some nice new features.

Breaking changes and upgrading

• The SyntaxSet API has been revamped to use a builder and an arena of contexts. See example usage.
• Many functions now need to be passed the SyntaxSet that goes with the rest of their arguments because of this new arena.
• Filename added to LoadingError::ParseSyntax
• Many functions in the html module now take the newlines version of syntaxes.
  • These methods have also been renamed, partially so that code that needs updating doesn't break without a compile error.
  • The HTML they output also treats newlines slightly differently and I think more correctly but uglier when you look at the HTML.

Breaking rename upgrade guide

• SyntaxSet::add_syntax -> SyntaxSetBuilder::add
• SyntaxSet::load_syntaxes -> SyntaxSetBuilder::add_from_folder
• SyntaxSet::load_plain_text_syntax -> SyntaxSetBuilder::add_plain_text_syntax
• html::highlighted_snippet_for_string -> html::highlighted_html_for_string: also change to newlines SyntaxSet
• html::highlighted_snippet_for_file -> html::highlighted_html_for_file: also change to newlines SyntaxSet
• html::styles_to_coloured_html -> html::styled_line_to_highlighted_html: also change to newlines SyntaxSet
• html::start_coloured_html_snippet -> html::start_highlighted_html_snippet: return type also changed

Major changes and new features

• Use arena for contexts (#182 #186 #187 #190 #195): This makes the code cleaner, enables use of syntaxes from multiple threads, and prevents accidental misuse.
  • This involves a new SyntaxSetBuilder API for constructing new SyntaxSets
  • See the revamped parsyncat example.
• Encourage use of newlines (#197 #207 #196): The nonewlines mode is often buggy so we made it easier to use the newlines mode.
  • Added a LinesWithEndings utility for iterating over the lines of a string with \n characters.
  • Reengineer the html module to use newlines syntaxes.
• Add helpers for modifying highlighted lines (#198): For use cases like highlighting a piece of text in a blog code snippet or debugger. This allows you to reach into the highlighted spans and add styles.
  • Check out split_at and modify_range in the util module.
• New ThemeSet::add_from_folder function (#200): For modifying existing theme sets.

Bug Fixes

• Improve nonewlines regex rewriting: #212 #211
• Reengineer theme application to match Sublime: #209
• Also mark contexts referenced by name as "no prototype" (same as ST): #180
• keep with_prototype when switching contexts with set: #177 #166
• Fix unused import warning: #174
• Ignore trailing dots in selectors: #173
• Fix embed to not include prototypes: #172 #160

Upgraded dependencies

• plist: 0.2 -> 0.3
• regex: 0.2 -> 1.0
• onig: 3.2.1 -> 4.1

• Check regexes compile upon loading from YAML (There's technically a small breaking change here if you match on the previously unused regex error, but I don't think anyone does)
• Can detect the correct syntax on full file names like CMakeLists.txt
• Make nonewlines mode marginally less buggy (still prefer using newlines mode)
• Better error types
• Better examples and tests

• Parsing now abandons a regex after reaching a recursion depth limit instead of taking forever
• Loop detection better matches Sublime Text
• Parsing is faster!
• Dependency upgrades
• Other minor tweaks

Thanks to @robinst for the headline features of this release!