Machine Learning with a Reject Option
BSD-3-CLAUSE License
scikit-fallback is a scikit-learn-compatible Python package for machine learning with a reject option.
To allow your probabilistic pipeline to fallback—i.e., abstain from predictions—you can
wrap it with a skfb
rejector. Training a rejector means both fitting your model and
learning to accept or reject predictions. Evaluation of a rejector depends
on fallback mode (inference with or without fallback labels) and measures the ability
of the rejector to both accept correct predictions and reject ambiguous ones.
For example, skfb.estimators.ThresholdFallbackClassifierCV
fits the base estimator and then
finds the best confidence threshold via cross-validation. If fallback_mode == "store"
, then the
rejector returns skfb.core.array.FBNDArray
of predictions and a sparse fallback-mask property,
which lets us summarize the accuracy of both predictions and rejections.
from skfb.estimators import ThresholdFallbackClassifierCV
from sklearn.linear_model import LogisticRegressionCV
rejector = ThresholdFallbackClassifierCV(
LogisticRegressionCV(cv=4, random_state=0),
thresholds=10,
ambiguity_threshold=0.05,
cv=5,
fallback_label=-1,
fallback_mode="store",
)
rejector.fit(X_train, y_train) # Train base estimator and learn best threshold
rejector.score(X_test, y_test) # Compute acceptance-correctness accuracy score
For more information, see the project's Wiki.
scikit-fallback
requires:
If you already have scikit-learn
installed and it's scikit-learn<=1.2
, make sure that numpy<2.0
to prevent incompatibility issues.
pip install -U scikit-fallback
See the examples/
directory for various applications of fallback estimators
and scorers to scikit-learn-compatible pipelines.