Optimizing Whisper for Serbian
Overview
While large-scale AI models offer impressive general capabilities, they often fail in the “long tail” of cultural heritage, specifically for low-resource languages. The Serbian language falls into this category, having a very low percentage of available digital resources.
The Challenge: The Data Gap
The original OpenAI Whisper model was trained on a massive corpus of 680,000 hours of multilingual data. However, the representation of Serbian within this dataset was disproportionately small:
- 28 hours of transcription material.
- 136 hours of translation material.
This lack of data resulted in a high Word Error Rate (WER), making the base models impractical for professional archival work in cultural institutions. For example, the Whisper Medium model had a baseline WER of 44.9%, and Whisper Large-v2 was 33.9%.
The Methodology: Fine-Tuning & Optimization
To resolve the high error rates we faced with limits in the available data.
1. Targeted Fine-Tuning
We moved the model from a generalist to a specialist by introducing new, high-quality datasets. Specifically, we utilized the JuzneVesti-SR v1.0 dataset HF Link combined with the Google/Fleurs and Mozilla Common Voice datasets to retrain the model.
2. Hardware Democratization
Standard Large models require significant GPU memory (VRAM). We optimized the inference process by implementing a successive loading technique (loading Encoder → clearing memory → loading Decoder), reducing the VRAM requirement for the Large model to under 10GB of VRAM.
3. Architectural Efficiency
To further decentralize access, we converted the models into Faster-Whisper (CTranslate2) and ONNX formats, enabling direct browser inference via WebGPU.
Impact & Results
These results present the fine-tuned Serbian Whisper values (using validation dataset with corrected text)
| Model Version | Original WER | Optimized WER | Improvement |
|---|---|---|---|
| Whisper Medium | 44.9% | 6.58% | ~85% Reduction |
| Whisper Large-v2 | 33.9% | 6.23% | ~81% Reduction |
| Whisper Large-v3 | 11.6% | 4.15% | ~64% Reduction |
Real-World Application
These optimized models were successfully deployed to process the “Bojićevci” Digital Collection, automating the transcription of historical audio recordings from 1944.
Interactive Data & Demo
Explore the specialized dataset used for fine-tuning or test the model directly in your browser.
Dataset Preview: Audio-LMB-DS
Live Inference
Test the optimized Serbian Whisper model directly via our hosted Gradio Space.
Also available in the [Workbench] →
- WebGPU: Hugging Face WebGPU version
Artifact Resources
- Models: Hugging Face
- Implementation: GitHub Repository: whisper-webui-vad
- Case Study: Library “Milutin Bojić” Digital Collection - “Bojićevci”
- HF Dataset: ASR training dataset for Serbian JuzneVesti-SR v1.0