Deepspeech ASR Model for the Catalan Language
MIT License
An ASR model created with the Mozilla DeepSpeech engine. For a comparison with other catalan ASR models check the Catalan Speech Recognition Benchmark
Model de reconeixement de la parla creat amb el motor DeepSpeech de Mozilla. Us podeu descarregar l'ltima versi aqu.
La motivaci principal s la d'aprendre, pel que el model evoluciona constantment a mida que vaig fent proves, per tamb la de contribur a millorar la presncia del catal en les tecnologies de la parla lliures i obertes.
Descarregueu-vos el model i l'scorer i feu servir el motor d'inferncia deepspeech per a inferir el text d'un arxiu audio (16Hz mono WAV)
$ pip install deepspeech
$ deepspeech --model deepspeech-catala.pbmm --scorer kenlm.scorer --audio file.wav
En la taula comparativa de models es fa referncia als segents corpus de veu en catal. Alguns s'han fet servir per entrenar models mentre que altres exclusivament per l'avaluaci.
Tamb anomenat "Scorer" al DeepSpeech, ja que "puntua" la probabilitat que una paraula vingui desprs d'una altra. Els models de llenguatge que es fan servir habitualment en el reconeixement de la parla sn N-Grames que representen la probabilitat de subcadenes de paraules de mida n on 1 n N.
Un mateix model acstic donar diferents resultats segons el model de llenguatge que fem servir i s aconsellable adaptar el model de llenguatge al domini
lingstic de la nostra aplicaci. Durant l'entrenament i l'avaluaci dels diferents models he anat provant diferents models de llenguatge basats en conjunts de dades que podeu trobar al directori lm
d'aquest repositori.
A continuaci una comparativa de les diferents versions del model, el corpus i scorer emprats i el resultats de l'avaluaci (WER).
Les versions anteriors a la 0.4.0 feien servir un alfabet sense vocals accentuades pel que no es consideren en la comparativa.
El dataset test de cada model s diferent pel que no es poden comparar entre s per s'afegeix a mode de documentaci.
Model | Model Base | Dropped layers | Versi DeepSpeech | Corpus | Scorer | WER |
---|---|---|---|---|---|---|
0.4 | Angls 0.7.0 | 1 | 0.7.0 | CV4 | Oscar | 30,16% |
0.5 | Angls 0.7.0 | 1 | 0.7.0 | CV4 | Oscar | 29,66% |
0.6 | Angls 0.7.0 | 1 | 0.7.0 | CV4 + PPC | Oscar | 13,85% |
0.7 | Angls 0.7.2 | 1 | 0.7.0 | CV4 + PPC + FC | TV3 | 16,95% |
0.8 | Angls 0.8.0 | 1 | 0.8.0 | CV5.1 + PPC + FC | TV3 | 19,35% |
0.9 | cap | - | 0.8.0 | CV5.1 + PPC + FC | TV3 | 20,12% |
0.10 | Angls 0.8.0 | 3 | 0.8.0 | CV5.1 + PPC + FC | TV3 | 19,07% |
0.11 | Angls 0.8.0 | 1 | 0.8.0 | CV5.1 + PPC + FC | Oscar | 15,81% |
0.12 | Angls 0.8.0 | 1 | 0.8.0 | CV5.1 + PPC | Oscar | 14,06% |
0.13 | Catal 0.12 | 0 | 0.9.2 | CV6.1 + PPC | Oscar | 12,44% |
0.14 | Angls 0.9.2 | 1 | 0.9.2 | CV6.1 + PPC | Oscar | 13,29% |
Model | Model Base | Dropped layers | Versi DeepSpeech | Corpus | Scorer | WER |
---|---|---|---|---|---|---|
0.6 | Angls 0.7.0 | 1 | 0.7.0 | CV4 + PPC | Oscar* | 12,75% |
0.7 | Angls 0.7.2 | 1 | 0.7.0 | CV4 + PPC + FC | TV3 | 21,69% |
0.8 | Angls 0.8.0 | 1 | 0.8.0 | CV5.1 + PPC + FC | TV3 | 14,47% |
0.9 | cap | - | 0.8.0 | CV5.1 + PPC + FC | TV3 | 31,88% |
0.10 | Angls 0.8.0 | 3 | 0.8.0 | CV5.1 + PPC + FC | TV3 | 16,05% |
0.11 | Angls 0.8.0 | 1 | 0.8.0 | CV5.1 + PPC + FC | Oscar* | 29,93% |
0.12 | Angls 0.8.0 | 1 | 0.8.0 | CV5.1 + PPC | Oscar | 17,34% |
0.13 | Catal 0.12 | 0 | 0.9.2 | CV6.1 + PPC | Oscar* | 9,07% |
0.14 | Angls 0.9.2 | 1 | 0.9.2 | CV6.1 + PPC | Oscar* | 9,05% |
(*) L'scorer Oscar cont les probabilitats extretes de les transcripcions del dataset pel que la WER est esbiaixada.
Model | Model Base | Dropped layers | Versi DeepSpeech | Corpus | Scorer | WER |
---|---|---|---|---|---|---|
0.6 | Angls 0.7.0 | 1 | 0.7.0 | CV4 + PPC | Oscar | 28,45% |
0.7 | Angls 0.7.2 | 1 | 0.7.0 | CV4 + PPC + FC | TV3 | 44,88% |
0.8 | Angls 0.8.0 | 1 | 0.8.0 | CV5.1 + PPC + FC | TV3 | 54,31% |
0.9 | cap | - | 0.8.0 | CV5.1 + PPC + FC | TV3 | 50,10% |
0.10 | Angls 0.8.0 | 3 | 0.8.0 | CV5.1 + PPC + FC | TV3 | 46,89% |
0.11 | Angls 0.8.0 | 1 | 0.8.0 | CV5.1 + PPC + FC | Oscar | 45,89% |
0.12 | Angls 0.8.0 | 1 | 0.8.0 | CV5.1 + PPC | Oscar | 22,65% |
0.13 | Catal 0.12 | 0 | 0.9.2 | CV6.1 + PPC | Oscar | 20,04% |
0.14 | Angls 0.9.2 | 1 | 0.9.2 | CV6.1 + PPC | Oscar | 18,84% |
The main motivation of this project is to learn how to creat ASR models using Mozilla's DeepSpeech engine so the model is constantly evolving. Moreover I wanted to see what was possible with the currently released CommonVoice catalan language dataset.
Download the model and the scorer and use the deepspeech engine to infer text from an audio file (16Hz mono WAV)
$ pip install [email protected]
$ deepspeech --model deepspeech-catala-0.6.0.pbmm --scorer kenlm.scorer --audio file.wav
What follows is a comparison of the different published model versions, the dataset used and the accuracy of each model.
Note: For version 0.6.0 the whole CommonVoice dataset (train, dev and test files) was combined with the clean dataset of ParlamentParla, shuffled and split in train/dev/test files using a 75/20/5 ratio. Due to this fact, a comparison between the models can only be made by using 1713 sentences from the ParlamentParla dataset not seen by any model during training.
Model | Corpus | Augmentation | WER | CER | Loss |
---|---|---|---|---|---|
[email protected] | CommonVoice | No | 30,16% | 13,79% | 112,96 |
[email protected] | CommonVoice | S | 29,66% | 13,84% | 108,52 |
[email protected] | CommonVoice + ParlamentParlaClean | No | 13,85% | 5,62% | 50,49 |
stashify@deepspeech_cat | CommonVoice? | S | 22,62% | 13,59% | 80,45 |
Model | Corpus | Augmentation | WER | CER | Loss |
---|---|---|---|---|---|
[email protected] | CommonVoice | No | 77,60% | 65,62% | 243,25 |
[email protected] | CommonVoice | S | 78,12% | 65,61% | 235,60 |
[email protected] | CommonVoice + ParlamentParlaClean | No | 76,10% | 65,16% | 240,69 |
stashify@deepspeech_cat | CommonVoice? | S | 80,58% | 66,82% | 180,81 |
Validating the models against the FestCat dataset shows that the models do not generalize well. This corpus has a higer variability in the word count of the test sentences, with 90% of the sentences containing an evenly distributed amount of words between 2 and 23, whilst most of the sentences in the CommonVoice corpus contain between 3 and 16 words.
As expected, validating the models against a test set containing only sentences with 4 or more words improves accuracy:
Model | Corpus | Augmentation | WER | CER | Loss |
---|---|---|---|---|---|
[email protected] | CommonVoice | No | 58,78% | 46,61% | 193,85 |
[email protected] | CommonVoice | S | 58,94% | 46,47% | 188,42 |
[email protected] | CommonVoice + ParlamentParlaClean | No | 56,68% | 46,00% | 189,03 |
stashify@deepspeech_cat | CommonVoice? | S | 61,11% | 48,16% | 144,78 |