A unified evaluation library for multiple machine learning libraries
APACHE-2.0 License
English | ็ฎไฝไธญๆ
MMEval is a machine learning evaluation library that supports efficient and accurate distributed evaluation on a variety of machine learning frameworks.
Major features:
MPI4Py | torch.distributed | Horovod | paddle.distributed | oneflow.comm |
---|---|---|---|---|
MPI4PyDist | TorchCPUDist TorchCUDADist | TFHorovodDist | PaddleDist | OneFlowDist |
NOTE: MMEval tested with PyTorch 1.6+, TensorFlow 2.4+, Paddle 2.2+ and OneFlow 0.8+.
Metric | numpy.ndarray | torch.Tensor | tensorflow.Tensor | paddle.Tensor | oneflow.Tensor |
---|---|---|---|---|---|
Accuracy | โ | โ | โ | โ | โ |
SingleLabelMetric | โ | โ | โ | ||
MultiLabelMetric | โ | โ | โ | ||
AveragePrecision | โ | โ | โ | ||
MeanIoU | โ | โ | โ | โ | โ |
VOCMeanAP | โ | ||||
OIDMeanAP | โ | ||||
COCODetection | โ | ||||
ProposalRecall | โ | ||||
F1Score | โ | โ | โ | ||
HmeanIoU | โ | ||||
PCKAccuracy | โ | ||||
MpiiPCKAccuracy | โ | ||||
JhmdbPCKAccuracy | โ | ||||
EndPointError | โ | โ | โ | ||
AVAMeanAP | โ | ||||
StructuralSimilarity | โ | ||||
SignalNoiseRatio | โ | ||||
PeakSignalNoiseRatio | โ | ||||
MeanAbsoluteError | โ | ||||
MeanSquaredError | โ |
MMEval
requires Python 3.6+ and can be installed via pip.
pip install mmeval
To install the dependencies required for all the metrics provided in MMEval
, you can install them with the following command.
pip install 'mmeval[all]'
There are two ways to use MMEval
's metrics, using Accuracy
as an example:
from mmeval import Accuracy
import numpy as np
accuracy = Accuracy()
The first way is to directly call the instantiated Accuracy
object to calculate the metric.
labels = np.asarray([0, 1, 2, 3])
preds = np.asarray([0, 2, 1, 3])
accuracy(preds, labels)
# {'top1': 0.5}
The second way is to calculate the metric after accumulating data from multiple batches.
for i in range(10):
labels = np.random.randint(0, 4, size=(100, ))
predicts = np.random.randint(0, 4, size=(100, ))
accuracy.add(predicts, labels)
accuracy.compute()
# {'top1': ...}
We appreciate all contributions to improve MMEval. Please refer to CONTRIBUTING.md for the contributing guideline.
This project is released under the Apache 2.0 license.