A library that contains a rich collection of performant PyTorch model metrics, a simple interface to create new metrics, a toolkit to facilitate metric computation in distributed training and tools for PyTorch model evaluations.
OTHER License
Bot releases are hidden (Show)
We're looking forward to building tooling for metric collections. The first important feature towards this end is collective syncing of groups of metrics. In the example below, we show how easy it is to sync all your metrics at the same time with sync_and_compute_collection
.
This method is not only for convenience, on the backend we only use one torch distributed sync collective for the entire group of metrics, meaning that the overhead from repeated network directives is maximally reduced.
import torch
from torcheval.metrics import BinaryAUPRC, BinaryAUROC, BinaryAccuracy
from torcheval.metrics.toolkit import sync_and_compute_collection, reset_metrics
# Collections should be Dict[str, Metric]
train_metrics = {
"train_auprc": BinaryAUPRC(),
"train_auroc": BinaryAUROC(),
"train_accuracy": BinaryAccuracy(),
}
# Hydrate metrics with some random data
preds = torch.rand(size=(100,))
targets = torch.randint(low=0, high=2, size=(100,))
for name, metric in train_metrics.items():
metric.update(preds, targets)
# Sync the whole group with a single gather
print(sync_and_compute_collection(train_metrics))
>>> {'train_auprc': tensor(0.5913), 'train_auroc': tensor(0.5161, dtype=torch.float64), 'train_accuracy': tensor(0.5100)}
# reset all metrics in collection
reset_metrics(train_metrics.values())
Be on the lookout for more metric collection code coming in future releases.
We're grateful for our community, which helps us improving torcheval by highlighting issues and contributing code. In addition to the core TorchEval team (@ananthsub, @bobakfb, @JKSenthil, @ninginthecloud), the following persons have contributed patches for this release: Rohit Alekar, @lindawangg, Julia Reinspach, @jingchi-wang, Ekta Sardana, @williamhufb, Andreas Floros, @ErikaLal, @samiwilf
Published by edward-io almost 2 years ago
Installation from pypi is recommended:
pip install torcheval
A library with simple and straightforward tooling for model evaluations and a delightful user experience. At a high level TorchEval:
Contains a rich collection of high performance metric calculations out of the box. We utilize vectorization and GPU acceleration where possible via PyTorch.
Integrates seamlessly with distributed training and tools using torch.distributed
Is designed with extensibility in mind: you have the freedom to easily create your own metrics and leverage our toolkit.
Provides tools for profiling memory and compute requirements for PyTorch based models.