This is a patch release with two notable fixes and a feature:

• Training logs now correctly list the number of training examples (now called "unique pairs")
• The warmup steps is now based on the number of steps rather than args.max_steps if args.max_steps > the number of steps. This prevents accidentally being in warm-up for longer than the desired warmup proportion.
• When training with string labels, the model now tries to automatically set the string labels to SetFitModel.labels if this variable hasn't been defined yet.

The PRs:

• Set labels based on head classes, if possible by @tomaarsen in https://github.com/huggingface/setfit/pull/476
• Refactor training logs & fix warmup_proportion by @tomaarsen in https://github.com/huggingface/setfit/pull/475

Full Changelog: https://github.com/huggingface/setfit/compare/v1.0.2...v1.0.3

What's Changed

• Fix: Python-ify evaluation results before writing model card by @tomaarsen in https://github.com/huggingface/setfit/pull/460
• Resolve crash with predict_proba & multi-output by @tomaarsen in https://github.com/huggingface/setfit/pull/466
• Remove breaking shuffle DataLoader option by @tomaarsen in https://github.com/huggingface/setfit/pull/470
• Predict for ABSA models with a gold aspect dataset by @tomaarsen in https://github.com/huggingface/setfit/pull/469
• Prepare SetFit for upcoming 2.3.0 release of SentenceTransformers by @tomaarsen in https://github.com/huggingface/setfit/pull/463

Full Changelog: https://github.com/huggingface/setfit/compare/v1.0.1...v1.0.2

v1.0.1 Patch Release

• Fixes ConstructorError when saving a SetFitModel that was trained with a custom evaluation metrics (#460)

v1.0.0 Full SetFit Release

This release heavily refactors the SetFit trainer and introduces some much requested features, such as:

• New Trainer, new TrainingArguments with many, many new arguments.
• Configurable logging, automatic logging to Weights & Biases and Tensorboard if installed.
• Evaluation during training, early stopping support to combat overfitting.
• Checkpointing + loading the best model at the end.
• SetFit for Aspect Based Sentiment Analysis in collaboration with Intel Labs.
• Heavily improved automatic model card generation.
• Extensive callbacks support based on transformers.
• Full, extensive documentation: http://hf.co/docs/setfit
• and more!

v1.0.0 Migration Guide

Read the v1.0.0 Migration Guide in the documentation: https://hf.co/docs/setfit/how_to/v1.0.0_migration_guide

v1.0.0 Detailed Release Notes

Read the more detailed release notes in the documentation: https://huggingface.co/docs/setfit/how_to/v1.0.0_migration_guide#v100-changelog

What's Changed

• Preserve dataset features in sample_dataset by @grofte in https://github.com/huggingface/setfit/pull/396
• Allow other datasets in trainer.evaluate() by @grofte in https://github.com/huggingface/setfit/pull/402
• Normalize device to CPU when evaluating by @tomaarsen in https://github.com/huggingface/setfit/pull/363
• show_progress_bar as parameter on predict and predict_prob by @davidsbatista in https://github.com/huggingface/setfit/pull/429
• Refactor to introduce Trainer & TrainingArguments, add SetFit ABSA by @tomaarsen in https://github.com/huggingface/setfit/pull/265
• fix: make sampling more reproducible by @yahiaelgamal in https://github.com/huggingface/setfit/pull/441
• Allow setting batch size in SetFitModel.predict by @tomaarsen in https://github.com/huggingface/setfit/pull/443
• Save differentiable model head on CPU by @tomaarsen in https://github.com/huggingface/setfit/pull/444
• Allow 'device' on SetFitModel.from_pretrained() by @tomaarsen in https://github.com/huggingface/setfit/pull/445
• Add notebook to demonstrate how efficiently running SetFit with ONNX by @MosheWasserb in https://github.com/huggingface/setfit/pull/435
• Add "labels" to SetFitModel, store/load from configuration file by @tomaarsen in https://github.com/huggingface/setfit/pull/447
• Allow passing strings to model.predict by @tomaarsen in https://github.com/huggingface/setfit/pull/448
• Allow partial column mappings by @tomaarsen in https://github.com/huggingface/setfit/pull/449
• Allow normalize_embeddings with a differentiable head by @tomaarsen in https://github.com/huggingface/setfit/pull/450
• Heavily improve automatic model card generation by @tomaarsen in https://github.com/huggingface/setfit/pull/452
• Also pass metric_kwargs to custom metric callable by @tomaarsen in https://github.com/huggingface/setfit/pull/456
• Prepare v1.0.0 release - Trainer, TrainingArguments, SetFitABSA, logging, evaluation during training, callbacks, docs by @tomaarsen in https://github.com/huggingface/setfit/pull/439

New Contributors

• @rhelmeczi made their first contribution in https://github.com/huggingface/setfit/pull/362
• @bofenghuang made their first contribution in https://github.com/huggingface/setfit/pull/366
• @davidberenstein1957 made their first contribution in https://github.com/huggingface/setfit/pull/384
• @alvarobartt made their first contribution in https://github.com/huggingface/setfit/pull/397
• @bogedy made their first contribution in https://github.com/huggingface/setfit/pull/361
• @grofte made their first contribution in https://github.com/huggingface/setfit/pull/396
• @davidsbatista made their first contribution in https://github.com/huggingface/setfit/pull/429
• @rtrompier made their first contribution in https://github.com/huggingface/setfit/pull/433
• @yahiaelgamal made their first contribution in https://github.com/huggingface/setfit/pull/441

Full Changelog: https://github.com/huggingface/setfit/compare/v0.7.0...v1.0.0

v0.7.0 Bug Fixes Galore

This release introduces numerous bug fixes, including critical ones for push_to_hub, save_pretrained and distillation training.

Bug fixes and improvements

• Add a warning if an unsplit dataset is passed to SetFitTrainer by @jaalu in #299
• Improve dataset pre-processing speeds for large datasets by @logan-markewich in #309
• Add Path support to _save_pretrained, resolve TypeError: unsupported operand type(s) for +: 'PosixPath' and 'str' by @tomaarsen in #332
• Add Hallmarks of Cancer notebook by @MosheWasserb in #333
• Initialize SetFitModel with cls instead by @kobiche in #341
• Allow distillation training with models using differentiable heads by @tomaarsen in #343
• Prevent TypeError on model.predict when using string labels by @tomaarsen in #331
• Restrict pandas to <2 for compatibility tests by @tomaarsen in #350
• Update Trainer.push_to_hub to use **kwargs by @tomaarsen in #351
• Add metric keyword arguments, e.g. add "average" strategy to f1 by @tomaarsen in #353

Significant community contributions

The following contributors have made significant changes to the library over the last release:

• @jaalu
  • Add a warning if an unsplit dataset is passed to SetFitTrainer (#299)
• @tomaarsen
  • Add comparison plotting script (#319)
  • Resolve IndexError if there is just one K-shot scenario
  • Reintroduce Usage in README until docs are ready
  • Add Path support to _save_pretrained (#332)
  • Allow distillation training with models using differentiable heads (#343)
  • Prevent TypeError on model.predict when using string labels (#331)
  • Restrict pandas to <2 for compatibility tests (#350)
  • Update Trainer.push_to_hub to use **kwargs (#351)
  • Add metric keyword arguments, e.g. add "average" strategy to f1 (#353)
• @EdAbati
  • Add cache for 🤗 Hub models in the CI (#312)
  • Rerun hyperparameter search notebook (#321)
• @MosheWasserb
  • Add Hallmarks of Cancer notebook (#333)

v0.6.0 OpenVINO exporter, model cards, and various quality of life improvements 🔥

To bring in the new year, this release comes with many bug fixes and quality of life improvements around using SetFit models. It also provides:

• an OpenVINO exporter that you can optimise your models for inference with. Check out the notebooks for an example.
• a dedicated model card with metadata and usage instructions. See here for an example output from push_to_hub(): https://huggingface.co/lewtun/setfit-new-model-card

Bug fixes and improvements

• Always install the checked-out setfit by @tomaarsen in #235
• Add SetFitModel.to by @tomaarsen in #229)
• Add distillation trainer example by @lewtun in #202
• Prevent overriding the sample size in sample_dataset by @tomaarsen in #231
• add related work in readme by @Yongtae723 in #239
• Fix seed in trainer.py by @danielkorat in #243
• Always display test coverage; add tests by @tomaarsen in #240
• Add Tom to list of maintainers by @lewtun in #253
• Add proper model card by @lewtun in #252
• Added support of OpenVINO export by @AlexKoff88 in #214
• Add has_differentiable_head property to SetFitModel by @zachschillaci27 in #257
• Resolve numpy.ndarray type error with predict_proba by @jegork in #207
• Refactor model_head initialization in SetFitModel by @zachschillaci27 in #263
• Feature/deprecate binary cross entropy loss by @blakechi in #203
• Fix type hints by @Yongtae723 in #266
• pass auth token to sentence transformer by @ken-myers in #277
• Add multi-target support to SetFitHead by @Yongtae723 and @OskarLiew in #272
• Automatically create summary table after scripts/setfit/run_fewshot.py by @tomaarsen in #262
• Fix squared optimization steps bug by @twerkmeister in #280
• Fix squared optimization steps bug in distillation trainer by @tomaarsen in #284
• Dynamic features in datasets based on model input names by @AleksanderObuchowski in #288
• Resolve SentenceTransformer resetting devices after moving a SetFitModel by @tomaarsen in #283
• add run_zeroshot.py; add functionality to data.get_templated_dataset() (formerly add_templated_examples()) by @danielkorat in #292
• Exclude compatibility versions from dev setup by @tomaarsen in #286

Significant community contributions

The following contributors have made significant changes to the library over the last release:

• @tomaarsen
  • Always install the checked-out setfit (#235)
  • Add SetFitModel.to (#229) (#236)
  • Prevent overriding the sample size in sample_dataset (#231)
  • Always display test coverage; add tests (#240)
  • Automatically create summary table after scripts/setfit/run_fewshot.py (#262)
  • Fix squared optimization steps bug in distillation trainer (#284)
  • Resolve SentenceTransformer resetting devices after moving a SetFitModel (#283)
  • Reformat according to the newest black version
  • Remove doubled space in warning message
  • Exclude compatibility versions from dev setup (#286)
• @Yongtae723
  • add related work in readme (#239)
  • Fix type hints (#266)
  • Add multi-target support to SetFitHead (#272)
• @danielkorat
  • Fix seed in trainer.py (#243)
  • add run_zeroshot.py; add functionality to data.get_templated_dataset() (formerly add_templated_examples()) (#292)
• @AlexKoff88
  • Added support of OpenVINO export (#214)

This release comes with two main features:

• A DistillationSetFitTrainer class that allows users to use unlabeled data to significantly boost the performance of small models like MiniLM. See this workshop for an end-to-end example.
• An ONNX exporter that converts the SetFit model instances into ONNX graphs for downstream inference + optimisation. Checkout the notebooks folder for an end-to-end example.

Kudos to @orenpereg and @nbertagnolli for implementing both of these features 🔥

Bug fixes and improvements

• Tidy up Makefile & create notebook table by @lewtun in #163
• Fix by @lewtun in #164
• Fixed typo in model head from predict_prob to predict_proba by @nbertagnolli in #171
• Distill trainer by @orenpereg in #166
• Update evaluate by @lvwerra in #194
• Use scikit-learn rather than sklearn in requirements files by @lesteve in #200
• Bugfix/body and head on different devices by @blakechi in #175
• add option to normalize embeddings by @PhilipMay in #177
• delete duplicated code by @Yongtae723 in #183
• Throw clear ValueError when neglecting to pass train_dataset to DistillationSetFitTrainer by @tomaarsen in #190
• add option to set samples_per_label by @PhilipMay in #196
• Resolve typo: sklean -> sklearn, #220 by @tomaarsen in #221
• Allow setting max length by @blakechi in #176
• add doc for num_iterations by @PhilipMay in #215
• Allow training progress bars to be disabled by @tomaarsen in #218
• Added initial onnx export function by @nbertagnolli in #156
• Fix/input type hint by @Yongtae723 in #184
• fixed spell errors in code example by @Gladiator07 in #210
• For scripts/setfit/run_fewshot.py, add warning for class imbalance w. accuracy by @tomaarsen in #204
• No longer needlessly deepcopy the original model state by @tomaarsen in #201
• Various cleanups; type hint fixes incl. corresponding to PEP 484 by @tomaarsen in #185
• Expand CI tests using matrix; make dependencies less restrictive; fix ONNX tests by @tomaarsen in #233
• Add SetFitModel.to by @jegork in #229
• Revert "Add SetFitModel.to by @lewtun in #229)"

Significant community contributions

The following contributors have made significant changes to the library over the last release:

• @nbertagnolli
  • Fixed typo in model head from predict_prob to predict_proba (#171)
  • Added initial onnx export function (#156)
• @orenpereg
  • Distill trainer (#166)

Fixes an issue on Google Colab, where the default version of Python 3.7 is incompatible with the Literal type. See #162 for more details.

Differentiable heads for SetFitModel

@blakechi has implemented a differentiable head in PyTorch for SetFitModel that enables the model to be trained end-to-end. The implementation is backwards compatible with the scikit-learn heads and can be activated by setting use_differentiable_head=True when loading SetFitModel. Here's a full example:

from datasets import load_dataset
from sentence_transformers.losses import CosineSimilarityLoss

from setfit import SetFitModel, SetFitTrainer


# Load a dataset from the Hugging Face Hub
dataset = load_dataset("sst2")

# Simulate the few-shot regime by sampling 8 examples per class
num_classes = 2
train_dataset = dataset["train"].shuffle(seed=42).select(range(8 * num_classes))
eval_dataset = dataset["validation"]

# Load a SetFit model from Hub
model = SetFitModel.from_pretrained(
    "sentence-transformers/paraphrase-mpnet-base-v2",
    use_differentiable_head=True,
    head_params={"out_features": num_classes},
)

# Create trainer
trainer = SetFitTrainer(
    model=model,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    loss_class=CosineSimilarityLoss,
    metric="accuracy",
    batch_size=16,
    num_iterations=20, # The number of text pairs to generate for contrastive learning
    num_epochs=1, # The number of epochs to use for constrastive learning
    column_mapping={"sentence": "text", "label": "label"} # Map dataset columns to text/label expected by trainer
)

# Train and evaluate
trainer.freeze() # Freeze the head
trainer.train() # Train only the body

# Unfreeze the head and freeze the body -> head-only training
trainer.unfreeze(keep_body_frozen=True)
# or
# Unfreeze the head and unfreeze the body -> end-to-end training
trainer.unfreeze(keep_body_frozen=False)

trainer.train(
    num_epochs=25, # The number of epochs to train the head or the whole model (body and head)
    batch_size=16,
    body_learning_rate=1e-5, # The body's learning rate
    learning_rate=1e-2, # The head's learning rate
    l2_weight=0.0, # Weight decay on **both** the body and head. If `None`, will use 0.01.
)
metrics = trainer.evaluate()

# Push model to the Hub
trainer.push_to_hub("my-awesome-setfit-model")

# Download from Hub and run inference
model = SetFitModel.from_pretrained("lewtun/my-awesome-setfit-model")
# Run inference
preds = model(["i loved the spiderman movie!", "pineapple on pizza is the worst 🤮"]) 

Bug fixes and improvements

• add num_epochs to train_step calculation by @PhilipMay in #139
• Support for the differentiable head by @blakechi in #112
• redirect call to predict by @PhilipMay in #142
• fix: templated examples copy empty vector by @pdhall99 in #148
• Add support to kwargs in compute() method called by trainer.evaluate() by @mpangrazzi in #125
• Small fix on hyperparameter search by @Mouhanedg56 in #150
• Fix typo: temerature => temperature by @tomaarsen in #155
• Add the usage and relevant info. of the differentiable head to README by @blakechi in #149
• Fix non default loss_class issue by @PhilipMay in #154
• Add sampling function & update notebooks by @lewtun in #146
• Fix typos: image(s) -> sentence(s) by @victorjmarin in #160
• Add more loss function options by @PhilipMay in #159

Significant community contributions

The following contributors have made significant changes to the library over the last release:

• @pdhall99
  • fix: allow load of pretrained model without head
  • fix: templated examples copy empty vector (#148)
• @PhilipMay
  • add num_epochs to train_step calculation (#139)
  • redirect call to predict (#142)
  • Fix non default loss_class issue (#154)
  • Add more loss function options (#159)
• @blakechi
  • Support for the differentiable head (#112)
  • Add the usage and relevant info. of the differentiable head to README (#149)
• @mpangrazzi
  • Add support to kwargs in compute() method called by trainer.evaluate() (#125)

This release includes improvements to the hyperparameter_search() function of SetFitTrainer, along with several small fixes in saving fine-tuned models.

Thanks to @sanderland @bradleyfowler123 @Mouhanedg56 for their contributions 🤗 !

This release comes with two main features:

• Support to train models on multilabel text classification datasets
• An optuna integration to run hyperparameter search on both the SetFitModel head and the hyperparameters used during training.

Significant community contributions

The following contributors have made significant changes to the library over the last release:

• @mpangrazzi #69
• @Mouhanedg56 #93

Fixes a bug where the column mapping checks threw an error when a column mapping wasn't provided for datasets with valid column names.

See #82 for more details.

Column mapping for SetFitTrainer

The SetFitTrainer assumes that the training and evaluation datasets contain text and label columns. Previously, this required users to manually rename their dataset columns before creating the trainer. In #75 we added support for users to specify the column mapping directly in the trainer.