Python Kaldi speech recognition with grammars that can be set active/inactive dynamically at decode-time

[GitHub is matching (only) my GitHub Sponsors donations.]

Python package developed to enable context-based command & control of computer applications, as in the Dragonfly speech recognition framework, using the Kaldi automatic speech recognition engine.

Normally, Kaldi decoding graphs are monolithic, require expensive up-front off-line compilation, and are static during decoding. Kaldi's new grammar framework allows multiple independent grammars with nonterminals, to be compiled separately and stitched together dynamically at decode-time, but all the grammars are always active and capable of being recognized.

This project extends that to allow each grammar/rule to be independently marked as active/inactive dynamically on a per-utterance basis (set at the beginning of each utterance). Dragonfly is then capable of activating only the appropriate grammars for the current environment, resulting in increased accuracy due to fewer possible recognitions. Furthermore, the dictation grammar can be shared between all the command grammars, which can be compiled quickly without needing to include large-vocabulary dictation directly.

• Binaries: The Python package includes all necessary binaries for decoding on Windows/Linux/MacOS. Available on PyPI.
  • Binaries are generated from my fork of Kaldi, which is only intended to be used by kaldi-active-grammar directly, and not as a stand-alone library.
• Pre-trained model: A compatible general English Kaldi nnet3 chain model is trained on ~3000 hours of open audio. Available under project releases.
  • Comparison of models
  • Improved models are under development.
• Plain dictation: Do you just want to recognize plain dictation? Seems kind of boring, but okay! There is an interface for plain dictation (see below), using either your specified HCLG.fst file, or KaldiAG's included pre-trained dictation model.
• Dragonfly/Caster: A compatible backend for Dragonfly is under development in the kaldi branch of my fork, and has been merged as of Dragonfly v0.15.0.
  • See its documentation, try out a demo, or use the loader to run all normal dragonfly scripts.
  • You can try it out easily on Windows using a simple no-install package: see Getting Started below.
  • Caster is supported as of KaldiAG v0.6.0 and Dragonfly v0.16.1.
  • Support for KaldiAG v1.2.0 has been merged as of Dragonfly v0.20.0! Improvements include Improved Recognition, Weights on Any Elements, Pluggable Alternative Dictation, Stand-alone Plain Dictation Interface, and various bug fixes & optimizations. For details and previous versions' improvements, see project releases.

[GitHub is currently matching all my donations $-for-$.]

• daanzu/kaldi-grammar-simple
• daanzu/speech-training-recorder
• daanzu/dragonfly_daanzu_tools

Want to get started quickly & easily on Windows? Available under project releases:

• kaldi-dragonfly-winpython: A self-contained, portable, batteries-included (python & libraries & model) distribution of kaldi-active-grammar + dragonfly2. Just unzip and run!
• kaldi-dragonfly-winpython-dev: [more recent development version] A self-contained, portable, batteries-included (python & libraries & model) distribution of kaldi-active-grammar + dragonfly2. Just unzip and run!
• kaldi-caster-winpython-dev: [more recent development version] A self-contained, portable, batteries-included (python & libraries & model) distribution of kaldi-active-grammar + dragonfly2 + caster. Just unzip and run!

Otherwise...

Requirements:

• Python 2.7 or 3.6+; 64-bit required!
• OS: Windows/Linux/MacOS
• Only supports Kaldi left-biphone models, specifically nnet3 chain models, with specific modifications
• ~1GB+ disk space for model plus temporary storage and cache, depending on your grammar complexity
• ~1GB+ RAM for model and grammars, depending on your model and grammar complexity

Install Python package, which includes necessary Kaldi binaries:

pip install kaldi-active-grammar

Download compatible generic English Kaldi nnet3 chain model from project releases. Unzip the model and pass the directory path to kaldi-active-grammar constructor.

Or use your own model. Standard Kaldi models must be converted to be usable. Conversion can be performed automatically, but this hasn't been fully implemented yet.

• Errors installing
  • Make sure you're using a 64-bit Python.
  • You must install via pip install kaldi-active-grammar (directly or indirectly), not python setup.py install, in order to get the required binaries.
  • Update your pip (to at least 19.0+) by executing python -m pip install --upgrade pip, to support the required python binary wheel package.
• Try deleting the Kaldi model *.tmp directory and rerunning.
• For reporting issues, try running with import logging; logging.basicConfig(level=1) at the top of your main file to enable full debugging logging.

Documentation is sorely lacking currently. To see example usage, examine the backend for Dragonfly.

import sys, wave
from kaldi_active_grammar import PlainDictationRecognizer
recognizer = PlainDictationRecognizer()  # Or supply non-default model_dir, tmp_dir, or fst_file
filename = sys.argv[1] if len(sys.argv) > 1 else 'test.wav'
wave_file = wave.open(filename, 'rb')
data = wave_file.readframes(wave_file.getnframes())
output_str, likelihood = recognizer.decode_utterance(data)
print(repr(output_str), likelihood)  # -> 'it depends on the context' 2.1386399269104004

Issues, suggestions, and feature requests are welcome & encouraged. Pull requests are considered, but project structure is in flux.

Building:

• Linux/MacOS: python setup.py bdist_wheel (see CMakeLists.txt for details)
• Windows: currently quite complicated (see my fork of Kaldi, then similar to Linux/MacOS)

Donations are appreciated to encourage development.

[GitHub is currently matching all my donations $-for-$.]

• David Zurow (@daanzu)

This project is licensed under the GNU Affero General Public License v3 (AGPL-3.0-or-later). See the LICENSE.txt file for details. If this license is problematic for you, please contact me.

• Based on and including code from Kaldi ASR, under the Apache-2.0 license.
• Code from OpenFST and OpenFST port for Windows, under the Apache-2.0 license.
• Intel Math Kernel Library, copyright (c) 2018 Intel Corporation, under the Intel Simplified Software License.
• Modified generic English Kaldi nnet3 chain model from Zamia Speech, under the LGPL-3.0 license.