Umer Naeem

Summary

What I did

• Fine-tuned OpenAI Whisper models on specialized medical and Punjabi vocabulary with local accents.
• Implemented data augmentation (noise variation, playback speeds) to handle realistic clinic recording conditions.
• Developed a post-processing pipeline using GPT-4 for transcript normalization and contextual error correction of low-probability predictions.
• Deployed models for inference on Google Cloud Run and integrated them into the main web server.
• Created an evaluation dataset with thoroughly reviewed transcriptions to measure word error rate for medical jargon and local accents.

Results

• Fine-tuned Whisper models using PEFT (Parameter-Efficient Fine-Tuning) on 20+ hours of diverse clinical voice data, covering medical and Punjabi vocabulary.
• Engineered a post-processing pipeline using GPT-4 to contextually correct low-probability Whisper tokens, significantly improving transcription reliability for code-switching speech.
• Implemented a robust data augmentation strategy (noise variation, playback speeds) to ensure model resilience in realistic, high-noise clinic environments.
• Deployed the speech-to-structured-data pipeline on Google Cloud Run, enabling real-time inference for healthcare workers.
• Reduced the word error rate on a custom clinical dataset from 60% to 25-30% by fine-tuning Whisper models.
• Processed 20+ hours of clinical voice data with high speaker diversity and vocabulary coverage
• Automated form-filling for healthcare workers, reducing operational overhead through voice-first data entry