Tensorflow's Fairness Evaluation and Visualization Toolkit
APACHE-2.0 License
Fairness Indicators is designed to support teams in evaluating, improving, and comparing models for fairness concerns in partnership with the broader Tensorflow toolkit.
The tool is currently actively used internally by many of our products. We would love to partner with you to understand where Fairness Indicators is most useful, and where added functionality would be valuable. Please reach out at [email protected]. You can provide feedback and feature requests here.
Fairness Indicators enables easy computation of commonly-identified fairness metrics for binary and multiclass classifiers.
Many existing tools for evaluating fairness concerns donโt work well on large-scale datasets and models. At Google, it is important for us to have tools that can work on billion-user systems. Fairness Indicators will allow you to evaluate fairenss metrics across any size of use case.
In particular, Fairness Indicators includes the ability to:
This case study, complete with videos and programming exercises, demonstrates how Fairness Indicators can be used on one of your own products to evaluate fairness concerns over time.
pip install fairness-indicators
The pip package includes:
Fairness Indicators also hosts nightly packages at https://pypi-nightly.tensorflow.org on Google Cloud. To install the latest nightly package, please use the following command:
pip install --extra-index-url https://pypi-nightly.tensorflow.org/simple fairness-indicators
This will install the nightly packages for the major dependencies of Fairness Indicators such as TensorFlow Data Validation (TFDV), TensorFlow Model Analysis (TFMA).
Tensorflow Models
Not using existing Tensorflow tools? No worries!
For more information on how to think about fairness evaluation in the context of your use case, see this link.
If you have found a bug in Fairness Indicators, please file a GitHub issue with as much supporting information as you can provide.
The following table shows the package versions that are compatible with each other. This is determined by our testing framework, but other untested combinations may also work.
fairness-indicators | tensorflow | tensorflow-data-validation | tensorflow-model-analysis |
---|---|---|---|
GitHub master | nightly (1.x/2.x) | 1.15.1 | 0.46.0 |
v0.46.0 | 2.15 | 1.15.1 | 0.46.0 |
v0.44.0 | 2.12 | 1.13.0 | 0.44.0 |
v0.43.0 | 2.11 | 1.12.0 | 0.43.0 |
v0.42.0 | 1.15.5 / 2.10 | 1.11.0 | 0.42.0 |
v0.41.0 | 1.15.5 / 2.9 | 1.10.0 | 0.41.0 |
v0.40.0 | 1.15.5 / 2.9 | 1.9.0 | 0.40.0 |
v0.39.0 | 1.15.5 / 2.8 | 1.8.0 | 0.39.0 |
v0.38.0 | 1.15.5 / 2.8 | 1.7.0 | 0.38.0 |
v0.37.0 | 1.15.5 / 2.7 | 1.6.0 | 0.37.0 |
v0.36.0 | 1.15.2 / 2.7 | 1.5.0 | 0.36.0 |
v0.35.0 | 1.15.2 / 2.6 | 1.4.0 | 0.35.0 |
v0.34.0 | 1.15.2 / 2.6 | 1.3.0 | 0.34.0 |
v0.33.0 | 1.15.2 / 2.5 | 1.2.0 | 0.33.0 |
v0.30.0 | 1.15.2 / 2.4 | 0.30.0 | 0.30.0 |
v0.29.0 | 1.15.2 / 2.4 | 0.29.0 | 0.29.0 |
v0.28.0 | 1.15.2 / 2.4 | 0.28.0 | 0.28.0 |
v0.27.0 | 1.15.2 / 2.4 | 0.27.0 | 0.27.0 |
v0.26.0 | 1.15.2 / 2.3 | 0.26.0 | 0.26.0 |
v0.25.0 | 1.15.2 / 2.3 | 0.25.0 | 0.25.0 |
v0.24.0 | 1.15.2 / 2.3 | 0.24.0 | 0.24.0 |
v0.23.0 | 1.15.2 / 2.3 | 0.23.0 | 0.23.0 |