✨ New features and improvements

• Sanitize direct download for spacy download (#13313).
• Convert Cython properties to decorator syntax (#13390).
• Bump Weasel pin to allow v0.4.x (#13409).
• Improvements to the test suite (#13469, #13470).
• Bump Typer pin to allow v0.10.0 and above (#13471).
• Allow typing-extensions<5.0.0 for Python < 3.8 (#13516).

🔴 Bug fixes

• #13400: Fix use_gold_ents behaviour for EntityLinker.

📖 Documentation and examples

• Make the file name for code listings stick to the top (#13379).
• Update the documentation of MorphAnalysis (#13433).
• Typo fixes in the documentation (#13466).

👥 Contributors

@danieldk, @honnibal, @ines, @JoeSchiff, @nokados, @Paillat-dev, @rmitsch, @schorfma, @strickvl, @svlandeg, @ynx0

✨ New features and improvements

• Improve NumPy 2.0 compatibility (#13103).
• Added language extensions for Faroese and Norwegian Nynorsk (#13116).
• Add new TextCatReduce.v1 layer for text classification (#13181).
• Add new TextCatParametricAttention.v1 layer for text classification (#13201).
• Use build module for creating model packages by default (#13109).
• Add support for code loading to the benchmark speed command (#13247).
• Extend lexical attributes for English with more numericals (#13106).
• Warn about reloading dependencies after downloading models (#13081).

🔴 Bug fixes

• #13259, #13304, #13321: Correctness fixes for multiprocessing support in Language.pipe.
• #13187: Typing and documentation fixes for Doc.
• #13086: Update Tokenizer.explain for special cases with whitespace.
• #13068: Fix displaCy span stacking.
• #13149: Add spacy.TextCatBOW.v3 to use the fixed SparseLinear layer.

📖 Documentation and examples

• Many improvements and updates to the LLM documentation.
• Update trf_data examples and the transformer pipeline design section.

👥 Contributors

@adrianeboyd, @danieldk, @evornov, @honnibal, @ines, @lise-brinck, @ridge-kimani, @rmitsch, @shadeMe, @svlandeg

✨ New features and improvements

• Update __all__ fields (#13063).

🔴 Bug fixes

• #13035: Remove Pathy requirement.
• #13053: Restore spacy.cli.project API.
• #13057: Support Any comparisons for Token and Span.

📖 Documentation and examples

• Many updates for spacy-llm including Azure OpenAI, PaLM, and Mistral support.
• Various documentation corrections.

👥 Contributors

@adrianeboyd, @honnibal, @ines, @rmitsch, @svlandeg

🔴 Bug fixes

• Revert lazy loading of CLI module for spacy.info to fix availability of spacy.cli following import spacy (#13040).

👥 Contributors

@adrianeboyd, @honnibal, @ines, @svlandeg

This release drops support for Python 3.6 and adds support for Python 3.12.

✨ New features and improvements

• Add support for Python 3.12 (#12979).
• Use the new library Weasel for spaCy projects functionality (#12769).
  • All spacy project commands should run as before, just now they're using Weasel under the hood.
  • ⚠️ Remote storage is not yet supported for Python 3.12. Use Python 3.11 or earlier for remote storage.
• Extend to Thinc v8.2 (#12897).
• Extend transformers extra to spacy-transformers v1.3 (#13025).
• Support registered vectors (#12492).
• Add --spans-key option for CLI evaluation with spacy benchmark accuracy (#12981).
• Load the CLI module lazily for spacy.info (#12962).
• Add type stubs for spacy.training.example (#12801).
• Warn for unsupported pattern keys in dependency matcher (#12928).
• Language.replace_listeners: Pass the replaced listener and the tok2vec pipe to the callback in order to support spacy-curated-transformers (#12785).
• Always use tqdm with disable=None to disable output in non-interactive environments (#12979).
• Language updates:
  • Add left and right pointing angle brackets as punctuation to ancient Greek (#12829).
  • Update example sentences for Turkish (#12895).
• Package setup updates:
  • Update NumPy build constraints for NumPy 1.25+ (#12839). For Python 3.9+, it is no longer necessary to set build constraints while building binary wheels.
  • Refactor Cython profiling in order to disable profiling for Python 3.12 in the package setup, since Cython does not currently support profiling for Python 3.12 (#12979).

📦 Trained pipelines updates

The transformer-based trf pipelines have been updated to use our new Curated Transformers library through the Thinc model wrappers and pipeline component from spaCy Curated Transformers.

⚠️ Backwards incompatibilities

• Drop support for Python 3.6.
• Drop mypy checks for Python 3.7.
• Remove ray extra.
• spacy project has a few backwards incompatibilities due to the transition to the standalone library Weasel, which is not as tightly coupled to spaCy. Weasel produces warnings when it detects older spaCy-specific settings in your environment or project config.
  • Support for the spacy_version configuration key has been dropped.
  • Support for the check_requirements configuration key has been dropped due to the deprecation of pkg_resources.
  • The SPACY_CONFIG_OVERRIDES environment variable is no longer checked. You can set configuration overrides using WEASEL_CONFIG_OVERRIDES.
  • Support for SPACY_PROJECT_USE_GIT_VERSION environment variable has been dropped.
  • Error codes are now Weasel-specific and do not follow spaCy error codes.

📖 Documentation and examples

• New and updated documentation for large language models and spaCy Curated Transformers.
• Various documentation corrections and updates.
• New additions to the spaCy Universe:
  • Hobbit spaCy: NLP for Middle Earth
  • rolegal: a spaCy Package for Noisy Romanian Legal Document Processing

👥 Contributors

@adrianeboyd, @bdura, @connorbrinton, @danieldk, @davidberenstein1957, @denizcodeyaa, @eltociear, @evornov, @honnibal, @ines, @jmyerston, @koaning, @magdaaniol, @pdhall99, @ringohoffman, @rmitsch, @senisioi, @shadeMe, @svlandeg, @vinbo8, @wjbmattingly

✨ New features and improvements

• Allow Pydantic v2 using transitional v1 support (#12888).
• Add find-function CLI for finding locations of registered functions (#12757).
• Add extra spacy[cuda12x] for cupy-cuda12x (#12890).
• Extend tests for init config and train CLI (#12173).
• Switch from distutils to setuptools/sysconfig (#12853).

🔴 Bug fixes

• #12817: Escape annotated HTML tags in displaCy span renderer.
• #12857: Display model's full base version string in incompatibility warning.
• #12882: Update <br> tags in displaCy.

📖 Documentation and examples

• Various documentation corrections and updates.
• New additions to spaCy Universe:
  • OdyCy
  • SaysWho

👥 Contributors

@adrianeboyd, @afriedman412, @arplusman, @bdura, @connorbrinton, @honnibal, @ines, @it176131, @pmbaumgartner, @rmitsch, @shadeMe, @svlandeg, @thomashacker, @victorialslocum, @x-tabdeveloping

✨ New features and improvements

• NEW: span_finder pipeline component to identify overlapping, unlabeled spans (#12507).
• Language updates:
  • Add initial support for Malay (#12602).
  • Update Latin defaults to support noun chunks, update lexical/tokenizer defaults and add example sentences (#12538).
• Add option to return scores separately keyed by component name with spacy evaluate --per-component, Language.evaluate(per_component=True) and Scorer.score(per_component=True) (#12540).
• Support custom token/lexeme attribute for vectors (#12625).
• Support spancat_singlelabel in spacy debug data CLI (#12749).
• Typing updates for PhraseMatcher and SpanGroup (#12642, #12714).

🔴 Bug fixes

• #12569: Require that all SpanGroup spans come from the current doc.

📦 Trained pipelines updates

We have added new pipelines for Slovenian that use the trainable lemmatizer and floret vectors.

• 🙏 Special thanks to @orglce for help with the new pipelines!

The English pipelines have been updated to improve handling of contractions with various apostrophes and to lemmatize "get" as a passive auxiliary.

The Danish pipeline da_core_news_trf has been updated to use vesteinn/DanskBERT with performance improvements across the board.

⚠️ Backwards incompatibilities

• SpanGroup spans are now required to be from the same doc. When initializing a SpanGroup, there is a new check to verify that all added spans refer to the current doc. Without this check, it was possible to run into string store or other errors.

📖 Documentation and examples

• Various documentation corrections and updates.
• New additions to spaCy Universe:
  • spaCy-SetFit
  • spacy-vscode
  • SpanMarker
  • vetiver

👥 Contributors

@adrianeboyd, @bdura, @danieldk, @davidberenstein1957, @diyclassics, @essenmitsosse, @honnibal, @ines, @isabelizimm, @jmyerston, @kadarakos, @KennethEnevoldsen, @khursani8, @ljvmiranda921, @rmitsch, @shadeMe, @svlandeg, @tomaarsen, @victorialslocum, @vin-ivar, @ZiadAmerr

✨ New features and improvements

• Extend Typer support to v0.9 (#12631).

🔴 Bug fixes

• #12701: Fix issues with component names and listeners for sourced components.
• #12623: Support overrides for registered functions in configs.

👥 Contributors

@adrianeboyd, @bdura, @honnibal, @ines, @svlandeg

This bug fix release is primarily to address Pydantic incompatibility with typing_extensions>=4.6.0.

✨ New features and improvements

• Huge speed improvements for spancat, in particular on GPU (~10x-30x faster) (#12577).

🔴 Bug fixes

• Add typing_extensions requirement due to Pydantic incompatibility with typing_extensions>=4.6.0.
• Remove #egg from download URLs due to future deprecation in pip.

👥 Contributors

@adrianeboyd, @honnibal, @ines, @kadarakos, @svlandeg

This bug fix release is primarily to address Pydantic incompatibility with typing_extensions>=4.6.0.

✨ New features and improvements

• Huge speed improvements for spancat, in particular on GPU (~10x-30x faster) (#12577).

🔴 Bug fixes

• Add typing_extensions requirement due to Pydantic incompatibility with typing_extensions>=4.6.0.
• Remove #egg from download URLs due to future deprecation in pip.

👥 Contributors

@adrianeboyd, @honnibal, @ines, @kadarakos, @svlandeg

✨ New features and improvements

• Huge speed improvements for spancat, in particular on GPU (~10x-30x faster) (#12577).
• Improve speed for child operators (>+, >-, >++, >--) for the dependency matcher (#12528).
• Improve loading speed for tokenizers with a large number of exceptions (#12553).
• Support doc.spans for displaCy output in spacy benchmark accuracy / spacy evaluate (#12575).
• Add MorphAnalysis.get(default=) argument for user-provided default values similar to dict (#12545).
• Only perform vectors checks during initialization if there are sourced components (#12607).

🔴 Bug fixes

• #12567: Remove #egg from download URLs due to future deprecation in pip.

📖 Documentation and examples

• Various documentation corrections and updates.
• New additions to spaCy Universe:
  • LatinCy
  • parsigs
  • spaCysee
  • spacy-wasm

👥 Contributors

@adrianeboyd, @andyjessen, @bdura, @davidberenstein1957, @diyclassics, @honnibal, @ines, @kadarakos, @KennethEnevoldsen, @ljvmiranda921, @moxley01, @royashcenazi, @svlandeg, @tanloong, @victorialslocum

✨ New features and improvements

• Add support for floret vectors in spacy pretrain (#12435).
• Save final model as model-last.bin for spacy pretrain (#12459).
• Support Span input for displacy.parse_deps (#12477).
• Extend support to CuPy 12.0 for cupy install extras.

🔴 Bug fixes

• #12398: Fix entity linker failure on sentence-crossing entities.
• #12405: Fix sentence indexing bug in Span.sents.
• #12469: Fix scores attribute for spancat_singlelabel.
• #12484: Fix Span.sents when the final sentence is the last token in a Doc.
• #12486: Fix pickle for the ngram suggester.
• #12493: Include Span.kb_id and Span.id strings in Doc and DocBin serialization.

📖 Documentation and examples

• Various documentation corrections and updates.
• New addition to spaCy Universe:
  • Sentimental Onix

👥 Contributors

@adrianeboyd, @BLKSerene, @honnibal, @ines, @kadarakos, @prajakta-1527, @rmitsch, @shadeMe, @sloev, @svlandeg, @thomashacker, @willfrey

💥 We'd love to hear more about your experience with spaCy! Take our survey here.

✨ New features and improvements

• NEW: spancat_singlelabel pipeline component for multi-class and non-overlapping span classification. The spancat_singlelabel component predicts at most one label for each suggested span and adds a new setting allow_overlap to restrict the output to non-overlapping spans (#11365).
• Extend to mypy v1.0 (#12245).
• Use transformer + CNN for efficient GPU textcat with spacy init config (#11900).
• Support trainable lemmatizer in spacy debug data (#11419).
• Add new operators to dependency matcher for left/right immediate child/parent nodes (>+, >-, <+, <-) (#12334).
• Add spacy.PlainTextCorpusReader.v1 for plain text input (#12122).
• Add alignment_mode and span_id to Span.char_span() (#12145, #12196).
• Use string formatting types in logging calls (#12215).

🔴 Bug fixes

• #12017: Improve speed for top_k>1 in trainable lemmatizer.
• #12048: Make test_cli_find_threshold() test more robust.
• #12227: Fix return type of registry.find().
• #12272: Fix speed regression for Matcher patterns with extension attributes.
• #12287: Add grc to languages with lexeme norms in spacy-lookups-data.
• #12320: Make generation of empty KnowledgeBase instances configurable.
• #12343: Fix error message for displacy auto_select_port.
• #12347: Fix length check for knowledge base in entity linker, add InMemoryLookupKB.is_empty.
• #12365: Fix types for Lexeme.orth and Lexeme.lower.
• #12366: Raise error for non-default vectors with PretrainVectors.
• #12368: Partially address pending deprecation of pkg_resources.
• Various improvements and fixes for the test suite (#12148, #12157, #12210, #12303, #12372).

📖 Documentation and examples

• Many website updates to improve accessibility.
• Various documentation corrections and updates.
• New projects:
  • Span labeling datasets
  • Comparing embedding layers in spaCy from the technical report Multi hash embeddings in spaCy

👥 Contributors

@adrianeboyd, @andyjessen, @danieldk, @essenmitsosse, @honnibal, @ines, @itssimon, @kadarakos, @kwhumphreys, @ljvmiranda921, @pmbaumgartner, @polm, @richardpaulhudson, @rmitsch, @shadeMe, @svlandeg, @tanloong, @thomashacker, @victorialslocum

✨ New features and improvements

• NEW: New apply CLI command to annotate new documents with a trained pipeline (#11376).
• NEW: New benchmark CLI command to benchmark pipelines. The new benchmark speed subcommand measures the speed of a pipeline, the benchmark accuracy subcommand is a new alias for evaluate (#11902).
• NEW: New find-threshold CLI command to identify an optimal threshold for classification models (#11280).
• NEW: New FUZZY Matcher operator for fuzzy matches based on Levenshtein edit distance. In addition, the FUZZY and REGEX operators are now supported in combination with IN/NOT_IN. (#11359).
• Language updates for Ancient Greek, Dutch, Russian, Slovenian and Ukrainian (#11345, #11162, #11426, #11753, #11811, #11997, more details below).
• Allow up to typer v0.7.x (#11720), mypy 0.990 (#11801) and typing_extensions v4.4.x (#12036).
• New spacy.ConsoleLogger.v3 with expanded progress tracking (#11972).
• Improved scoring behavior for textcat with spacy.textcat_scorer.v2 (#11696 and #11971) and spacy.textcat_multilabel_scorer.v2 (#11820).
• Improved customizability of the knowledge base used for entity linking, with the default implementation being the new InMemoryLookupKB (#11268).
• Optional before_update callback that is invoked at the start of each training step (#11739).
• Improve performance of SpanGroup (#11380).
• Improve UX around displacy.serve when the default port is in use (#11948).
• Patch a security vulnerability in extracting tar files (#11746).
• Add equality definition for vectors (#11806).
• Allow interpolation of variables in directory names in projects (#11235).
• Update default component configs to use the latest tok2vec version (#11618).

🔴 Bug fixes

• #11382: Fix lookup behavior for the French and Catalan lemmatizers.
• #11385: Ensure that downstream components can train properly on a frozen tok2vec or transformer layer.
• #11762: Support local file system remotes for projects.
• #11763: Raise an error when unsupported values are used for textcat.
• #11834: Ensure Vocab.to_disk respects the exclude setting for lookups and vectors.
• #12009: Fix a few typing issues for SpanGroup and Span objects.
• #12098: Correctly handle missing annotations in the edit tree lemmatizer.

⚠️ Backwards incompatibilities and model updates

The following changes may require you to update code that is using the relevant functionality:

• An error is now raised when unsupported values are given as input to train a textcat or textcat_multilabel model - ensure that values are 0.0 or 1.0 as explained in the docs.
• As KnowledgeBase is now an abstract class, you should call the constructor of the new InMemoryLookupKB instead when you want to use spaCy's default KB implementation. If you've written a custom KB that inherits from KnowledgeBase, you'll need to implement its abstract methods, or alternatively inherit from InMemoryLookupKB instead.

The following changes may influence the output of your language pipeline or trained models:

• Updates to language defaults:
  • Extended support for Slovenian (#11162).
  • Switch Russian and Ukrainian lemmatizers to pymorphy3 (#11345, #11811).
  • Support for editorial punctuation in Ancient Greek (#11426).
  • Update to Russian tokenizer exceptions (#11753).
  • Small fix in the list of Dutch stop words (#11997).
• Updates to model defaults:
  • Use the latest tok2vec defaults in all components (#11618).
  • Improve the default attributes used for the textcat and textcat_multilabel components (#11698).
  • Update the default scorer for textcat and textcat_multilabel to fix a bug related to threshold for textcat and to make it possible to score multiple textcat/textcat_multilabel components in a single pipeline with custom scorers. If no custom scorers are used, the cat_p/r/f scores will now only reflect the final component's labels and performance (#11696, #11820).
  • Correct the token_acc score to report the intended measure (# correct tokens / # predicted tokens, the same as in spaCy v2). The token_acc scores for v3.5 will be lower for the same performance because they were incorrectly inflated in v3.0-v3.4. The token_p/r/f scores should remain unchanged (#12073).

The following functionality will be changed in the near future - so it's best to start updating your scripts now to make them more generic:

• From v4 onwards, we'll rename the master branch to main.

📦 Trained pipelines updates

• The CNN pipelines add IS_SPACE as a tok2vec feature for tagger and morphologizer components to improve tagging of non-whitespace vs. whitespace tokens.
• The transformer pipelines require spacy-transformers v1.2, which uses the exact alignment from tokenizers for fast tokenizers instead of the heuristic alignment from spacy-alignments. For all trained pipelines except ja_core_news_trf, the alignments between spaCy tokens and transformer tokens may be slightly different. More details about the spacy-transformers changes in the v1.2.0 release notes.

📖 Documentation and examples

• We've ported our website from Gatsby to Next 🥳
• Updated the documentation on supported languages.
• Added a note about experimental M1 GPU support to the installation quickstart.
• Included documentation for the biluo_to_iob and iob_to_biluo functions.
• Fixed model links in the v3.4 usage documentation.
• Removed "new" tags of functionality from spaCy v2.x.
• Various small additions, spelling and typo fixes.
• spaCy Universe additions:
  • greCy: Providing Ancient Greek models
  • spacy-pythainlp: Add Thai support for spaCy
• New projects:
  • Accelerate NER with Speedster (experimental)

👥 Contributors

@aaronzipp, @adrianeboyd, @albertvillanova, @ArchiDevil, @cfuerbachersparks, @damian-romero, @danieldk, @darigovresearch, @DSLituiev, @essenmitsosse, @gremur, @honnibal, @ines, @jmyerston, @JosPolfliet, @kadarakos, @koaning, @kwhumphreys, @ljvmiranda921, @MarcoGorelli, @orglce, @pmbaumgartner, @polm, @richardpaulhudson, @rmitsch, @ryndaniels, @shadeMe, @svlandeg, @thomashacker, @TrellixVulnTeam, @wannaphong, @zhiiw, @zrpxx

This release addresses future compatibility with NumPy v1.24+.

🔴 Bug fixes

• #11940: Update for compatibility with NumPy v1.24+ integer conversions.

👥 Contributors

@adrianeboyd, @honnibal, @ines, @svlandeg

This bug fix release is primarily to avoid deprecation warnings and future incompatibility with NumPy v1.24+.

🔴 Bug fixes

• #11331, #11701: Clean up warnings in spaCy and its test suite.
• #11845: Don't raise an error in displaCy for unset spans keys.
• #11864: Add smart_open requirement and update deprecated options.
• #11899: Fix spacy init config --gpu for environments without spacy-transformers.
• #11933: Update for compatibility with NumPy v1.24+ integer conversions.
• #11935: Restore missing error messages for beam search.

👥 Contributors

@adrianeboyd, @honnibal, @ines, @polm, @svlandeg

This bug fix release is primarily to avoid deprecation warnings and future incompatibility with NumPy v1.24+.

🔴 Bug fixes

• #10573: Remove Click pin following Typer updates.
• #11331, #11701: Clean up warnings in spaCy and its test suite.
• #11845: Don't raise an error in displaCy for unset spans keys.
• #11860: Fix spancat for docs with zero suggestions.
• #11864: Add smart_open requirement and update deprecated options.
• #11899: Fix spacy init config --gpu for environments without spacy-transformers.
• #11933: Update for compatibility with NumPy v1.24+ integer conversions.
• #11935: Restore missing error messages for beam search.

👥 Contributors

@adrianeboyd, @honnibal, @ines, @polm, @svlandeg

This bug fix release is primarily to avoid deprecation warnings and future incompatibility with NumPy v1.24+.

🔴 Bug fixes

• #10573: Remove Click pin following Typer updates.
• #11331, #11701: Clean up warnings in spaCy and its test suite.
• #11845: Don't raise an error in displaCy for unset spans keys.
• #11860: Fix spancat for docs with zero suggestions.
• #11864: Add smart_open requirement and update deprecated options.
• #11899: Fix spacy init config --gpu for environments without spacy-transformers.
• #11933: Update for compatibility with NumPy v1.24+ integer conversions.
• #11935: Restore missing error messages for beam search.

👥 Contributors

@adrianeboyd, @honnibal, @ines, @polm, @svlandeg

This bug fix release is primarily to avoid deprecation warnings and future incompatibility with NumPy v1.24+.

🔴 Bug fixes

• #10911, #11194: Improve speed in precomputable_biaffine by avoiding concatenation.
• #11276, #11331, #11701: Clean up warnings in spaCy and its test suite.
• #11845: Don't raise an error in displaCy for unset spans keys.
• #11860: Fix spancat for docs with zero suggestions.
• #11864: Add smart_open requirement and update deprecated options.
• #11899: Fix spacy init config --gpu for environments without spacy-transformers.
• #11933: Update for compatibility with NumPy v1.24+ integer conversions.
• #11934: Add strings when initializing from labels in EditTreeLemmatizer.
• #11935: Restore missing error messages for beam search.

👥 Contributors

@adrianeboyd, @danieldk, @honnibal, @ines, @polm, @svlandeg

This bug fix release is primarily to avoid deprecation warnings and future incompatibility with NumPy v1.24+.

🔴 Bug fixes

• #11845: Don't raise an error in displaCy for unset spans keys.
• #11860: Fix spancat for docs with zero suggestions.
• #11864: Add smart_open requirement and update deprecated options.
• #11899: Fix spacy init config --gpu for environments without spacy-transformers.
• #11933: Update for compatibility with NumPy v1.24+ integer conversions.
• #11934: Add strings when initializing from labels in EditTreeLemmatizer.
• #11935: Restore missing error messages for beam search.

👥 Contributors

@adrianeboyd, @danieldk, @honnibal, @ines, @polm, @svlandeg