Pareto Frontier Speech Recognition for CPU, Mobile and Edge
Automatic speech recognition (ASR / speech to text) has become a ubiquitous method of interfacing with phones and computers. Currently, there are many high quality models to choose from, such as Whisper and Parakeet. But there are very few open models that have architectures which are optimized for non-accelerated (CPU) targets. While the research field is mostly pursuing state of the art accuracy by building larger and larger models, and even fusing LLMs into the decoder, significantly less attention is being paid to developing new architectures that can run locally on devices. We wanted to find out if we could train models that would run efficiently on any hardware environment so that anybody could use them, anywhere.
Local First, Privacy Respecting
At FUTO, not only do we not want your data, we often go to great lengths to ensure we can't collect it. In order to keep your data from leaving your phone or computer, we have to work within the constraint that our software does not need to talk to external APIs backed by accelerated compute in order to perform voice transcription.
Users of FUTO Keyboard and FUTO Voice report being very happy with having access to high quality ASR models that they can own and run on their own hardware. In order to better support our users, we felt we should try and build models that specifically target those needs. To us, this means providing ASR models that leverage ordinary CPU instructions, do not require significant memory bandwidth, are fast and low power, and can operate in streaming mode at or above realtime speeds.
ASR4ALL Models
We've released models in 3 sizes, small (28M), medium (56M) and large (97M). These models all operate in streaming mode, meaning that they can output text while audio is being received. For offline transcription, the models can be used in a high latency streaming mode that commits larger audio chunks at a time for better throughput and accuracy. While offline decoding is typically more accurate, we found that training at both high and low latency streaming settings concurrently enables the model to operate over a range of settings.
Additional Features
We have also bundled a few additional smaller models that enable some unique features that we think are useful.
Punctuation, Capitalization & Error Correction (PCEC)
In addition to the acoustic decoder, we have added a small language model we call PCEC. This small module (5.3M params) is trained to add basic punctuation, capitalization or all caps, by pretraining over a text corpus, and then in a second stage, adapting to the hidden features of the acoustic network. However, the model also has another trick up its sleeve. We added synthetic noise to the text to help the model also learn how to fix mistakes that could be easily identified from a stronger understanding of language. This includes synthetic errors modeled on the outputs of the acoustic network as well as homophone disambiguation (two words that sound the same but are spelled differently). This provides a meaningful improvement in word error rate, at a low additional inference cost.
Voice Activity Detection (VAD)
There are many applications where it is useful to keep a voice decoder running continuously. Some examples of this are home assistants or local conversational LLM pipelines (speech to text -> LLM -> text to speech). However, this adds unnecessary computation load while the majority of the time is spent on decoding silence. Rather than run the decoder continuously, it is helpful to have a "gate" that only sends speech through the decoder and skips it when there is a large window of silence. That's where voice activity detection can be helpful. Distinguishing speech from not-speech is a much simpler task than ASR, and can be performed with a much smaller model. Recent research has shown that good quality VAD models can be trained and then aggressively pruned to very small sizes [1]. We followed the architecture of "kiloVAD" and trained a model that reproduces their results, and adds 2.5k parameters! For very little additional cost to the model size, we felt this would be a useful inclusion to the series.
ASR4ALL "Plus" Models
In order to support pipelines that need to identify speakers, we have developed additional features and packaged them into models designated as "plus" models. These models include a more accurate, slightly slower VAD (fsmn-vad), a speaker segmentation "powerset" head and a speaker embedding (verification) head. These features are trained as add-ons, using adapters on the main acoustic decoder. This allows developers to use them for additional purposes, such as speaker identification or voice authorization, counting numbers of turns and duration in a dialogue, or pipelines that apply combinations of these, such as diarization.
Model Formats and Quantization
Because we want to enable people to use ASR on many platforms and environments, we provide model exports for a number of runtimes. We've uploaded safetensors (PyTorch, transformers), ONNX, ExecuTorch (XNNPACK for CPU and mobile), as well as GGUF format. In general, there is little impact on model accuracy for int8 quantizations, and some GGUF quantizations may be viable down to 5 or 6 bits.
Training
We train on about 40k hours of audio, across many domains such as YouTube, European Parliament proceedings, meeting rooms, audio books, accented speech and voice commands. In order to build robustness to noise, we apply augmentations that introduce background noise, simulate cheap microphone bandwidth compression and mess with the volume. Because audio domains are so diverse, we also rely on benchmarks to expose domain areas of weakness. Improvements to speech recognition benchmarks in recent years have enabled researchers to see the areas where models are collapsing and broaden their data sources and training methods to improve them. It also enables us to make sure our results are strong in domains that overlap with our product concerns.
Performance on the Pareto Frontier
If you look at a chart of word error rate (WER) vs RTFx (seconds of audio per second of compute), you will see a large cluster of models predominantly competing to claim the lowest WER spots. As you look toward higher RTFx, the Pareto frontier flattens, with the fastest models still about 500M parameters. This is quite large for mobile devices, and very few of these models support streaming, which is harder still. Streaming typically trades some WER and RTFx for low latency operation. And notably, RTFx is measured on H200 GPUs that bear little resemblance to a mobile phone.
Our models use an efficient architecture that is designed with CPU inference in mind, but also extend the GPU Pareto frontier in throughput. Our fastest model reaches 73k RTFx on GPU benchmarks, over 3x faster than previous models. According to our tests, you can even run the medium model on a single core of a Raspberry Pi 4 using the more strenuous low latency setting, or other fun places like the integrated GPU of your Intel CPU using OpenVINO through ONNX.
Conclusion
If you're interested in trying these out for yourself, head over to HuggingFace[2] and check them out. These models are permissively licensed for commercial use, except for cases violating our ethical carveouts (not for use in mass surveillance or intelligence gathering). And of course, if you are a current user of FUTO Keyboard or FUTO Voice, these models will become available in our upcoming release.