Mohammad Umair Farooqui

Summary

What I did

• Engineered the backend for an internal AI query chatbot using a vector store with hybrid semantic and keyword search
• Paired BullMQ and Kubernetes to decouple and scale the document ingestion and embedding pipeline.
• Used BullMQ backed by Redis to manage a multi-stage ingestion queue including document parsing, chunking, embedding generation, and vector upsert.
• Implemented KEDA to scale worker pods horizontally based on Redis queue depth during daily delta syncs or bulk re-indexing.
• Fine-tuned Llama-3-8B-Instruct using QLoRA to serve as a fast, deterministic intent-classification and structured query-generation router.
• Built an automated evaluation pipeline to curate and clean roughly 12,000 prompt-completion pairs from production logs and synthetic edge cases.
• Applied deduplication, strict JSON schema validation, and human-in-the-loop verification for ambiguous queries during data preparation.
• Evaluated model checkpoints against holdout sets using JSON schema conformity rates, exact-match tool invocation, and LLM-as-a-judge comparisons.
• Implemented Direct Preference Optimization (DPO) using paired 'chosen' and 'rejected' responses to optimize implicit rewards and reduce hallucinations.
• Utilized bfloat16 base weights with DeepSpeed ZeRO-2 across multi-GPU setups to preserve logical reasoning capabilities by removing quantization loss.
• Performed synthetic negative mining and data filtering using automated heuristics and an ensemble judge to over-index on hard negatives.

Results

• Achieved a 98.4% schema valid rate on structured output generation, up from approximately 72% zero-shot.
• Reduced inference latency by 65% compared to external API-based models by fine-tuning and self-hosting a smaller model.
• Significantly dropped per-query compute cost while gaining complete control over deployment and latency targets.
• Reduced edge-case failure rates by roughly 38% compared to the SFT QLoRA baseline by shifting to Direct Preference Optimization (DPO).
• Improved model calibration through DPO alignment, enabling the model to reliably defer or trigger fallback tools when uncertain.
• Serves around 1,200 to 1,500 internal employees across global teams
• Handles 8,000 to 10,000 queries daily
• Peak concurrency of roughly 60–80 requests per second during business hours
• Indexes and chunks over 500,000 internal documents, technical manuals, and policy records
• Processes scheduled daily delta syncs of tens of thousands of modified records
• Sub-second context retrieval of under 200 ms before hitting the LLM pipeline