Aleksei Kokorev

Production GPU service · Python backend engineering

Matching audiobook narration to text

My contribution. I built the Python alignment pipeline in Apache Airflow and deployed a dedicated FastAPI service for GPU inference to production.

Production result. The pipeline produces a validated SMIL synchronization map that links ebook passages to the corresponding time intervals in the audiobook.

The problem

Given a passage in an ebook, where does it begin and end in the audiobook? The book gives us the words, and the recording gives us a timeline. Connecting the two gets tricky when the narrator repeats a line or reads a heading differently from the printed version.

How the pipeline works

The pipeline prepares the book text and uses FFmpeg to split the narration into short, overlapping audio windows. The GPU service runs Wav2Vec2 on one window at a time and returns a score for each possible speech token at each time step. The processor uses those scores to align the text with the recording and place each match on the original audiobook timeline.

From audiobook recording to synchronization map
Python processor · Airflow Prepare overlapping audio windows with FFmpeg.
↓ One audio window per request
GPU service · FastAPI + Wav2Vec2 Run inference on the audio window.
↓ Frame-level token scores and timing metadata
Python processor + ebook text Match the scores to passages, recover source timestamps, and validate the SMIL map.
↓ Validated map or failure details
Calling system

Handling overlapping audio windows

A passage can span two audio windows. Simply joining their results would count part of the recording twice. The processor uses the model’s frame timing to trim the overlap and map the remaining frames back to their positions in the source audio. That gives both windows a common clock.