neural-compressor

Features

• Knowledge distillation
  • Supported knowledge distillation on TensorFlow
• Pruning
  • Support Multi-node training on TensorFlow
• Acceleration library
  • Supported Hugging Face minilm_l6_h384_uncased_sst2, bert_base_cased_mrpc, and bert_base_nli_mean_tokens_stsb models

Validated Configurations

• Python 3.6 & 3.7 & 3.8 & 3.9
• Centos 8.3 & Ubuntu 18.04
• TensorFlow 2.6.2 & 2.7
• Intel TensorFlow 2.4.0, 2.5.0 and 1.15.0 UP3
• PyTorch 1.8.0+cpu, 1.9.0+cpu, IPEX 1.8.0
• MxNet 1.6.0, 1.7.0, 1.8.0
• ONNX Runtime 1.6.0, 1.7.0, 1.8.0

Distribution:

Contact:

Please feel free to contact [email protected], if you get any questions.

Features

• Knowledge distillation
  • Implemented the algorithms of paper “Pruning Once For All” accepted by NeurIPS 2021 ENLSP workshop
  • Supported optimization pipelines (knowledge distillation & quantization-aware training) on PyTorch
• Quantization
  • Added the support of ONNX RT 1.7
  • Added the support of TensorFlow 2.6.2 and 2.7
  • Added the support of PyTorch 1.10
• Pruning
  • Supported magnitude pruning on TensorFlow
• Acceleration library
  • Supported Hugging Face top 10 downloaded NLP models

Productivity

• Added performance profiling feature to INC UI service.
• Improved ease-of-use user interface for quantization with few clicks

Ecosystem

• Added notebook of using HuggingFace optimization library (Optimum) to Transformers
• Enabled top 20 downloaded Hugging Face NLP models with Optimum
• Upstreamed more INC quantized models to ONNX Model Zoo

Validated Configurations

• Python 3.6 & 3.7 & 3.8 & 3.9
• Centos 8.3 & Ubuntu 18.04
• TensorFlow 2.6.2 & 2.7
• Intel TensorFlow 2.4.0, 2.5.0 and 1.15.0 UP3
• PyTorch 1.8.0+cpu, 1.9.0+cpu, IPEX 1.8.0
• MxNet 1.6.0, 1.7.0, 1.8.0
• ONNX Runtime 1.6.0, 1.7.0, 1.8.0

Distribution:

Contact:

Please feel free to contact [email protected], if you get any questions.

Intel® Neural Compressor(formerly known as Intel® Low Precision Optimization Tool) v1.7 release is featured by:

Features

• Acceleration library
  • Support unified buffer memory allocation policy

Ecosystem

• Upstreamed INC quantized models (alexnet/caffenet/googlenet/squeezenet) to ONNX Model Zoo

Documentation

• Performance and accuracy data update

Validated Configurations

• Python 3.6 & 3.7 & 3.8 & 3.9
• Centos 8.3 & Ubuntu 18.04
• TensorFlow 2.6.0
• Intel TensorFlow 2.4.0, 2.5.0 and 1.15.0 UP3
• PyTorch 1.8.0+cpu, 1.9.0+cpu, IPEX 1.8.0
• MxNet 1.6.0, 1.7.0, 1.8.0
• ONNX Runtime 1.6.0, 1.7.0, 1.8.0

Distribution:

Contact:

Please feel free to contact INC Maintainers, if you get any questions.

Intel® Neural Compressor(formerly known as Intel® Low Precision Optimization Tool) v1.7 release is featured by:

Features

• Quantization
  • Improved quantization accuracy in SSD-Reset34 and MobileNet v3 on TensorFlow
• Pruning
  • Supported magnitude pruning on TensorFlow
• Knowledge distillation
  • Supported knowledge distillation on PyTorch
• Multi-node support
  • Supported multi-node pruning with distributed dataloader on PyTorch
  • Supported multi-node inference for benchmark on PyTorch
• Acceleration library
  • Added a domain-specific acceleration library for NLP models

Productivity

• Supported the configuration-free (pure Python) quantization
• Improved ease-of-use user interface for quantization with few clicks

Ecosystem

• Integrated into HuggingFace optimization library (Optimum)
• Upstreamed INC quantized models (RN50, VGG16) to ONNX Model Zoo

Documentation

• Add tutorial and examples for knowledge distillation
• Add tutorial and examples for multi-node training
• Add tutorial and examples for acceleration library

Validated Configurations

• Python 3.6 & 3.7 & 3.8 & 3.9
• Centos 8.3 & Ubuntu 18.04
• TensorFlow 2.6.0
• Intel TensorFlow 2.4.0, 2.5.0 and 1.15.0 UP3
• PyTorch 1.8.0+cpu, 1.9.0+cpu, IPEX 1.8.0
• MxNet 1.6.0, 1.7.0, 1.8.0
• ONNX Runtime 1.6.0, 1.7.0, 1.8.0

Distribution:

Contact:

Please feel free to contact [email protected], if you get any questions.

Intel® Low Precision Optimization Tool v1.6 release is featured by:

Pruning:

• Support pruning and post-training quantization pipeline on PyTorch
• Support pruning during quantization-aware training on PyTorch

Quantization:

• Support post-training quantization on TensorFlow 2.6.0, PyTorch 1.9.0, IPEX 1.8.0, and MXNet 1.8.0
• Support quantization-aware training on TensorFlow 2.x (Keras API)

User Experience:

• Improve quantization productivity with new UI
• Support quantized model recovery from tuning history

New Models:

• Support ResNet50 on ONNX model zoo

Documentation:

• Add pruned models
• Add quantized MLPerf models

Validated Configurations:

• Python 3.6 & 3.7 & 3.8 & 3.9
• Centos 8.3 & Ubuntu 18.04
• TensorFlow 2.6.0
• Intel TensorFlow 2.4.0, 2.5.0 and 1.15.0 UP3
• PyTorch 1.8.0+cpu, 1.9.0+cpu, IPEX 1.8.0
• MxNet 1.6.0, 1.7.0, 1.8.0
• ONNX Runtime 1.6.0, 1.7.0, 1.8.0

Distribution:

Contact:

Please feel free to contact [email protected], if you get any questions.

Intel® Low Precision Optimization Tool v1.5.1 release is featured by:

• Gradient-sensitivity pruning for CNN model
• Static quantization support for ONNX NLP model
• Dynamic seq length support in NLP dataloader
• Enrich quantization statistics

Validated Configurations:

• Python 3.6 & 3.7 & 3.8 & 3.9
• Centos 8.3 & Ubuntu 18.04
• Intel TensorFlow 1.15.2, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0 and 1.15.0 UP1 & UP2 & UP3
• PyTorch 1.5.0+cpu, 1.6.0+cpu, 1.8.0+cpu, ipex
• MxNet 1.6.0, 1.7.0
• ONNX Runtime 1.6.0, 1.7.0, 1.8.0

Distribution:

Contact:

Please feel free to contact [email protected], if you get any questions.

Intel® Low Precision Optimization Tool v1.5 release is featured by:

• Add pattern-lock sparsity algorithm for NLP fine-tuning tasks
  • Up to 70% unstructured sparsity and 50% structured sparsity with <2% accuracy loss on 5 Bert finetuning tasks
• Add NLP head pruning algorithm for HuggingFace models
  • Performance speedup up to 3.0X within 1.5% accuracy loss on HuggingFace BERT SST-2
• Support model optimization pipeline
• Integrate SigOPT with multi-metrics optimization
  • Complementary as basic strategy to speed up the tuning
• Support TensorFlow 2.5, PyTorch 1.8, and ONNX Runtime 1.8

Validated Configurations:

• Python 3.6 & 3.7 & 3.8 & 3.9
• Centos 8.3 & Ubuntu 18.04
• Intel TensorFlow 1.15.2, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0 and 1.15.0 UP1 & UP2 & UP3
• PyTorch 1.5.0+cpu, 1.6.0+cpu, 1.8.0+cpu, ipex
• MxNet 1.6.0, 1.7.0
• ONNX Runtime 1.6.0, 1.7.0, 1.8.0

Distribution:

Contact:

Please feel free to contact [email protected], if you get any questions.

Intel® Low Precision Optimization Tool v1.4.1 release is featured by:

1. Support TensorFlow 2.5.0
2. Support PyTorch 1.8.0
3. Support TensorFlow Object Detection YOLO-V3 model

Validated Configurations:

• Python 3.6 & 3.7 & 3.8
• Centos 7 & Ubuntu 18.04
• Intel TensorFlow 1.15.2, 2.1.0, 2.2.0, 2.3.0, 2.4.0, 2.5.0 and 1.15.0 UP1 & UP2
• PyTorch 1.5.0+cpu, 1.6.0+cpu, ipex
• MxNet 1.7.0
• ONNX Runtime 1.6.0, 1.7.0

Distribution:

Contact:

Please feel free to contact [email protected], if you get any questions.

Intel® Low Precision Optimization Tool v1.4 release is featured by:

Quantization

1. PyTorch FX-based quantization support
2. TensorFlow & ONNX RT quantization enhancement

Pruning

1. Pruning/sparsity API refinement
2. Magnitude-based pruning on PyTorch

Model Zoo

1. INT8 key models updated (BERT on TensorFlow, DLRM on PyTorch, etc.)
2. 20+ HuggingFace model quantization

User Experience

1. More comprehensive logging message
2. UI enhancement with FP32 optimization, auto-mixed precision (BF16/FP32), and graph visualization
3. Online document: https://intel.github.io/lpot

Extended Capabilities

1. Model conversion from QAT to Intel Optimized TensorFlow model

Validated Configurations:

• Python 3.6 & 3.7 & 3.8
• Centos 7 & Ubuntu 18.04
• Intel TensorFlow 1.15.2, 2.1.0, 2.2.0, 2.3.0, 2.4.0 and 1.15.0 UP1 & UP2
• PyTorch 1.5.0+cpu, 1.6.0+cpu, ipex
• MxNet 1.7.0
• ONNX Runtime 1.6.0, 1.7.0

Distribution:

Contact:

Please feel free to contact [email protected], if you get any questions.

Intel® Low Precision Optimization Tool v1.3 release is featured by:

1. Improve graph optimization without explicit input/output setting

Validated Configurations:

• Python 3.6 & 3.7 & 3.8
• Centos 7 & Ubuntu 18.04
• Intel TensorFlow 1.15.2, 2.1.0, 2.2.0, 2.3.0, 2.4.0 and 1.15.0 UP1 & UP2
• PyTorch 1.5.0+cpu, 1.6.0+cpu, ipex
• MxNet 1.7.0
• ONNX Runtime 1.6.0, 1.7.0

Distribution:

Contact:

Please feel free to contact [email protected], if you get any questions.

Intel® Low Precision Optimization Tool v1.3 release is featured by:

1. FP32 optimization & auto-mixed precision (BF16/FP32) for TensorFlow
2. Dynamic quantization support for PyTorch
3. ONNX Runtime v1.7 support
4. Configurable benchmarking support (multi-instances, warmup, etc.)
5. Multiple batch size calibration & mAP metrics for object detection models
6. Experimental user facing APIs for better usability
7. Various HuggingFace models support

Validated Configurations:

• Python 3.6 & 3.7 & 3.8
• Centos 7 & Ubuntu 18.04
• Intel TensorFlow 1.15.2, 2.1.0, 2.2.0, 2.3.0, 2.4.0 and 1.15.0 UP1 & UP2
• PyTorch 1.5.0+cpu, 1.6.0+cpu, ipex
• MxNet 1.7.0
• ONNX Runtime 1.6.0, 1.7.0

Distribution:

Contact:

Please feel free to contact [email protected], if you get any questions.

Intel® Low Precision Optimization Tool v1.2.1 release is featured by:

1. user-facing APIs backward compatibility with v1.1 and v1.0.
2. refined experimental user-facing APIs for better out-of-box experience.

Validated Configurations:

• Python 3.6 & 3.7 & 3.8
• Centos 7 & Ubuntu 18.04
• Intel TensorFlow 1.15.2, 2.1.0, 2.2.0, 2.3.0, 2.4.0 and 1.15.0 UP1 & UP2
• PyTorch 1.5.0+cpu, 1.6.0+cpu, ipex
• MxNet 1.7.0
• ONNX Runtime 1.6.0

Distribution:

Contact:

Please feel free to contact [email protected], if you get any questions.

Intel® Low Precision Optimization Tool v1.2 release is featured by:

• Broad TensorFlow model type support
• operator-wise quantization scheme for ONNX RT
• MSE driven tuning for metric-free use cases
• UX improvement, including UI web server preview support
• More key model supports

Validated Configurations:

• Python 3.6 & 3.7 & 3.8
• Centos 7 & Ubuntu 18.04
• Intel TensorFlow 1.15.2, 2.1.0, 2.2.0, 2.3.0, 2.4.0 and 1.15.0 UP1 & UP2
• PyTorch 1.5.0+cpu, 1.6.0+cpu, ipex
• MxNet 1.7.0
• ONNX Runtime 1.6.0

Distribution:

Contact:

Please feel free to contact [email protected], if you get any questions.

Intel® Low Precision Optimization Tool v1.1 release is featured by:

• New backends (PyTorch/IPEX, ONNX Runtime) backend preview support
• Add built-in industry dataset/metric and custom registration
• Preliminary input/output node auto-detection on TensorFlow models
• New INT8 quantization recipes: bias correction and label balance

Validated Configurations:

• Python 3.6 & 3.7
• Centos 7
• Intel TensorFlow 1.15.2, 2.1.0, 2.2.0, 2.3.0 and 1.15.0 UP1 & UP2
• PyTorch 1.5.0+cpu
• MxNet 1.7.0
• ONNX Runtime 1.6.0

Distribution:

Contact:

Please feel free to contact [email protected], if you get any questions.

Intel® Low Precision Optimization Tool v1.0 release is featured by:

• Refined user facing APIs for best OOB.
• Add TPE tuning strategies (Experimental).
• Pruning POC support on PyTorch
• TensorBoard POC support for tuning analysis.
• Built-in INT8/Dummy dataloader Support.
• Built-in Benchmarking support.
• Tuning history for strategy finetune.
• Support TF Keras and checkpoint model type as input.

Validated Configurations:

• Python 3.6 & 3.7
• Centos 7
• Intel TensorFlow 1.15.2, 2.1.0, 2.2.0, 2.3.0 and 1.15UP1
• PyTorch 1.5.0+cpu
• MxNet 1.7.0

Distribution:

Contact:

Please feel free to contact [email protected], if you get any questions.

Intel® Low Precision Optimization Tool v1.0 beta release is featured by:

• Built-in dataloaders and evaluators
• Add random and exhaustive tuning strategies
• Mix precision tuning support on TensorFlow (INT8/BF16/FP32)
• Quantization-aware training POC support on Pytorch
• TensorFlow mainstream version support, including 1.15.2, 1.15UP1 and 2.1.0
• 50+ models validated

Supported Models:

Known Issues:

• TensorFlow ResNet50 v1.5 int8 model will crash on TensorFlow 1.15 UP1 branch

Validated Configurations:

• Python 3.6 & 3.7
• Centos 7
• Intel TensorFlow 1.15.2, 2.1.0 and 1.15UP1
• PyTorch 1.5
• MxNet 1.6

Distribution:

Contact:

Please feel free to contact [email protected], if you get any questions.

Intel® Low Precision Optimization Tool (iLiT) is an open-sourced python library which is intended to deliver a unified low-precision inference solution cross multiple Intel optimized DL frameworks on both CPU and GPU. It supports automatic accuracy-driven tuning strategies, along with additional objectives like performance, model size, or memory footprint. It also provides the easy extension capability for new backends, tuning strategies, metrics and objectives.

Feature List:

• Unified low precision quantization interface cross multiple Intel optimized frameworks (TensorFlow, PyTorch, and MXNet)
• Built-in tuning strategies, including Basic, Bayesian, and MSE
• Built-in evaluation metrics, including TopK (image classification), F1 (NLP), and CocoMAP (object detection)
• Built-in tuning objectives, including Performance, ModelSize, and Footprint
• Extensible API design to add new strategy, framework backend, metric, and objective
• KL-divergence calibration for TensorFlow and MXNet
• Tuning process resume from certain checkpoint

Supported Models:

Known Issues:

• Statistics collection for KL algorithm is slow in TensorFlow due to lack of tensor inspector APIs
• MSE tuning strategy is not supported in PyTorch

Validated Configurations:

• Python 3.6 & 3.7
• Centos 7
• TensorFlow 1.15, 2.0 and 2.1
• PyTorch 1.5
• MxNet 1.6

Distribution:

Contact:

Please feel free to contact [email protected], if you get any questions.