Speech to Text

Transcribe and translate audio in PHP with local Whisper and wav2vec2 models, including timestamps for subtitles. No FFmpeg or Python required.

Transcribe speech from audio and video files, translate it into English and get timestamps for subtitles, with a local transformers speech recognition model such as Whisper.

Setup#

Terminal

vendor/bin/loves-ai setup speech-to-text
vendor/bin/loves-ai pull openai/whisper-tiny

Supported models#

Whisper-style models, e.g. openai/whisper-tiny, openai/whisper-base or openai/whisper-small (larger is more accurate and slower). They are multilingual: they detect the spoken language, can be told it, and can translate the speech into English.
CTC models, e.g. facebook/wav2vec2-base-960h. They transcribe the one language they were trained on.

The runner decodes audio itself, with no FFmpeg installation needed: WAV, MP3, M4A/AAC, FLAC, OGG/Opus and the audio track of video files work, at any length.

It rejects models it cannot load before starting, with an explanation, including whisper.cpp (GGML) and faster-whisper (CTranslate2) conversions, which are common on Hugging Face.

From the command line#

Terminal

vendor/bin/loves-ai speech-to-text openai/whisper-tiny interview.m4a
vendor/bin/loves-ai speech-to-text openai/whisper-tiny interview.m4a --timestamps

Output

🎧 Listening carefully with openai/whisper-tiny… Perfect time for a cup of tea and a cookie 🍪
If you wish to see all logs, re-run the command with the "--debug" option.
🎉 Transcript by openai/whisper-tiny:
[00:00.00 → 00:05.56] He hoped there would be stew for dinner, turnips and carrots and bruised potatoes and fat
[00:05.56 → 00:11.04] mutton pieces to be ladled out in thick peppered flower fatten sauce.

Options#

Option	Meaning
`--language=LANGUAGE`	Spoken language for Whisper-style models, e.g. `en` or `french` (default: detected)
`--translate`	Translate the speech into English (Whisper-style models)
`--timestamps`	Print when each segment is spoken: phrases for Whisper-style models, words for CTC models
`--device=DEVICE`	`cpu`, `cuda`, `mps`… (default: the best available)
`--log-file=PATH`	Append the runner's output to this file
`--debug`	Show the runner's output while transcribing

Defaults come from config/speech-to-text.php (log_file).

From PHP#

PHP

use PhpLovesAi\Runner\SpeechToText;

// Finds the runner and the pulled model in the project's .local directory by itself.
$speechToText = new SpeechToText();

$transcript = $speechToText->transcribe('openai/whisper-tiny', storage_path('app/interview.m4a'));

$english = $speechToText->transcribe(
    model: 'openai/whisper-small',
    audioPath: storage_path('app/interview-uk.mp3'),
    language: 'uk',
    translate: true,
);

Timestamps and subtitles#

transcribeWithTimestamps() returns segments with their start and end, in seconds from the start of the audio. The last segment may have no end.

PHP

$segments = $speechToText->transcribeWithTimestamps('openai/whisper-tiny', storage_path('app/interview.m4a'));
// [['start' => 0.0, 'end' => 5.56, 'text' => 'He hoped there would be stew for dinner, …'], ...]

Errors#

AudioNotFoundException when the file does not exist
BinaryNotInstalledException when setup speech-to-text has not been run
ModelNotFoundException when the model was not pulled yet
UnsupportedModelException when the model cannot transcribe speech, or cannot be told a language or translate
RunFailedException, with the runner's error output, when transcription fails, e.g. because the file has no audio

Performance#

Small models (whisper-tiny, whisper-base) are quick. Larger models and long recordings take a while, especially on CPU, and each run loads the model from disk again, so run transcription in a queue job.