esp-sr | ESP32 Ecosystem Directory

Bot releases are hidden (Show)

esp-sr - ESP-SR Release V1.6 Latest Release

Published by feizi 10 months ago

We are delighted to announce the release of our latest models, MultiNet7 and nsnet1, as well as more wake word models trained by TTS samples.

1. MultiNet7: Speech commands recognition model

We are proud to introduce our new MultiNet7 model. This new model is optimized for efficiency, using less memory and reducing compute time while maintaining high accuracy. You can upgrade MultiNet7 from MultiNet6 smoothly.

2. nsnet1 - The first deep noise suppression model

We are also introducing nsnet1, our first deep noise suppression model. This model is designed to enhance speech quality in noisy environments, making it perfect for real-world applications like voice assistants or telephony systems.

nsnet1 uses a deep learning approach to suppress background noise while preserving the original speech signal. It is trained on a large dataset to learn the patterns of noise and effectively cancel them out without distorting the speech.

This model is available for ESP32-S3 chip. You can enable it by setting afe_config.afe_ns_mode = NS_MODE_NET; . Please refer to esp-skainet/examples/voice_communication for more details.

3. Wake Word Models Trained by TTS

We have expanded our wake word models trained by TTS (Text-to-Speech) samples to include more wake word for our users. With the combination of TTS and LLM methods, the TTS model can be trained on a large amount of unlabeled audio data by self-supervised learning. The zero-shot performance of the TTS model is significantly improved, which allows us to clone voices based on a large number of short audio clips (less than 10 seconds). Now, we can train a wake word model only by TTS samples.

These wake word models are designed to recognize specific keywords or phrases that trigger an action or response from your device or application.

esp-sr - ESP-SR Release V1.2

Published by feizi over 1 year ago

Documentation for release ESP-SR is available at https://docs.espressif.com/projects/esp-sr/en/latest/esp32s3/index.html

We're excited to announce the release of ESP-SR v1.2.0, an advanced version of Espressif's Speech Recognition Library. This update brings 5 significant enhancements to the previous version and is designed specifically for the ESP32-S3 microcontroller platforms.

Improved accuracy: The speech recognition accuracy has been substantially increased, ensuring a more reliable and efficient user experience.
Support for custom keywords: You can now easily add your own keywords for customized voice command recognition.
Enhanced noise reduction: Our improved noise reduction algorithms help filter out unwanted background noise, enabling the system to perform better in a variety of environments.
Lower memory usage: Optimizations in the library have resulted in reduced memory usage, ensuring efficient performance even on resource-constrained devices.
Expanded documentation: The updated library comes with comprehensive documentation, making it easier for developers to integrate the library and understand its functionalities.

With ESP-SR v1.2.0, you can develop voice-controlled applications on Espressif microcontroller platforms with increased accuracy and efficiency. Download the latest release and explore the full potential of our Speech Recognition Library.

Socs support

ESP32
ESP32-S3
ESP32-S2, ESP32-C3(only support Chinese TTS)

MultiNet6

Methods

Training Framework

The RNN-Transducer (RNN-T) framework is used to train MultiNet6. RNN-T combines high accuracy with naturally streaming recognition, which is a good choice for embedding system.
To accelerate RNN-T inference further, MultiNet6 skip some decoding frames based on CTC-based guidance. Please refer to "Accelerating RNN-T Training and Inference Using CTC guidance" for details of this method.

Model

The model of MultiNet6 is based on Emformer. The parameters of MultiNet6 is about 3.5M. The CPU and memory consumption is as shown below(tested on ESP32S3):

	parameters	CPU(one core)	PSRAM	SRAM
MultiNet5	2.2 M	44%	2310 KB	16 KB
MultiNet6	3.5 M	40%	4000 KB	48 KB

Compared to MultiNet5, MultiNet6 has more parameters and less computation. The main reason is that the encoder model of MultiNet6 has a 4x downsampling factor.
Same as MultiNet5, the weight of Multinet6 is quantized by 8-bit.

Decoding

MultiNet6 uses the Finite-State Transducer (FST) to build language model instead of the Trie used in MultiNet5. The beam search based on FST is more efficient and robust. The different units are used for different language:

For English language, MultiNet6 uses subword units to replace phonemes (used in MultiNet5). The subword units are trained by SentencePiece. Please refer to mn6_en.vocab for subword units.
For Chinese language, MultiNet6 uses Pinyin syllables without tone. Please refer to mn6_cn.vocab for Pinyin syllables.