Rafay Khokhar

Rafay Khokhar

AI Engineer specializing in Agentic Systems, Edge ML, and Local LLM Inference.

Based in: Lahore, PakistanCurrently: AI Engineer, SLAIExperience: 3 years

Hire me

Experienced AI Engineer with a strong background in building agentic workspaces, medical coding pipelines, and edge-deployed skeletal gait-analysis systems. Proven expertise in deploying production-grade LLM applications using FastAPI, Rust, and Kubernetes across healthcare and enterprise sectors.

Experience

AI Engineer

SLAILahore, Pakistanwork2026 – Present

Summary

Core engineer on Ayra Agent, an agentic AI workspace adopted company-wide as the primary platform for development, documentation, research, meetings, and chat, replacing commercial AI assistant subscriptions. Built FastAPI backend services, tool-use orchestration, and multi-turn conversation state; shipped to production on Azure AKS through GitOps (Argo CD) with Grafana observability. Built Ayra Browser, a cross-platform desktop agent browser in Rust/Tauri, plus LLM systems for medical coding and fine-tuned VLMs for local inference.

What I did

  • Built a meta agent orchestrator with context of every product and code editing capabilities similar to Claude Code
  • Managed context across thousands of files for the orchestrator
  • Built the Ayra agent using FastAPI to handle long-running inference tasks as background jobs rather than blocking HTTP requests.
  • Implemented a system where FastAPI returns a run ID and streams tokens and tool calls back to the client.
  • Managed session state outside of requests to ensure runs persisted through dropped connections or pod restarts.
  • Configured tool calls with timeouts, retries, and structured error handling for agent recovery.
  • Deployed the agentic workspace on Azure AKS using Argo CD and Grafana for monitoring.
  • Built a custom orchestration harness to manage multi-step reasoning and tool routing for the Ayra meta-agent.
  • Developed AyraTest, a testing framework that runs nightly golden test suites and scenario-based regression tests.
  • Defined tool schemas as Pydantic models to provide JSON schemas for LLM consumption and validate arguments before execution.
  • Implemented a standardized response envelope for all runtimes and APIs including status, output, and error fields.
  • Managed handoffs between runtimes by persisting state in shared file workspaces and session summaries.

Results

  • Cut approximately 3000 USD per month in overhead by replacing commercial subscriptions
  • Transitioned about 25 active users to the internal Ayra platform
FastAPIAzure AKSGitOpsArgo CDGrafanaRustTauriLLMVLMClaude CodeCodexPydanticJSON SchemaValidation

ML Engineer

Xavor CorporationLahore, PakistanworkMay 2024 – Present

Summary

Engineered StepSense, an edge-deployed skeletal gait-analysis system on NVIDIA Jetson for fall-risk stratification, reaching 96% accuracy against clinical ground truth across 25 subjects. Built the core AI for Navi, a care-companion robot live in senior-living facilities: action recognition, face ReID, VLM scene understanding, and a LangGraph voice agent with long-term memory. Built an internal agentic knowledge-to-software platform for client Edwards Lifesciences.

Results

  • Reached 96% accuracy against clinical ground truth across 25 subjects for StepSense
NVIDIA JetsonLangGraphVLMFace ReIDAction Recognition

Research Collaborator (Machine Learning)

University of California, Irvine - Sue & Bill Gross School of NursingIrvine, CAwork2024 – 2026

Summary

Co-researcher on a UC President's award-funded project on empathetic AI for care robots, with Case Western Reserve and Notre Dame. Designed Temporal Transition Trigger models detecting behavioral shifts in dementia patients (94% success, 25 subjects); co-author of paper under review.

What I did

  • Conducted research on Temporal Transition Trigger models for dementia care.

Results

  • 94% success rate in detecting behavioral shifts in dementia patients across 25 subjects
Machine LearningTemporal Transition Trigger models

Skills

Technical

Agentic Systems
PythonPython
Quantization (GGUF/4-bit)
Embedding Pipelines
Pydantic
Hybrid Search
LoRA/QLoRA
HF Transformers
Tool-Use Orchestration
LangGraph
FastAPIFastAPI
NVIDIA Jetson
llama.cppllama.cpp
VLMs
Ollama
Schema Validation
RustRust
TypeScript/JavaScriptTypeScript/JavaScript
SQLSQL
Tauri
Structured Decoding
Golden-Set Evals
KV-Cache
TensorRT
Qdrant
Neo4j
Knowledge Graphs
YOLO
OpenCVOpenCV
Azure (AKS, IoT Hub)
DockerDocker
KubernetesKubernetes
Argo CD
Grafana
PostgreSQLPostgreSQL
Azure Kinect
Node.jsNode.js
AWS
TerraformTerraform
C/C++C/C++
TRT-Pose
PoseC3D
Ablations
ROS2
FlaskFlask
ONNX
Speculative Decoding
VLM Fine-tuning
Orchestration
Custom Agent Orchestration
Automated Regression Testing
Reinforcement Learning
Tool Use Orchestration

Projects

Hybrid RAG Memory for AI Robot Companion

Developerprofessional
Built a hybrid retrieval memory that combines vector search with knowledge-graph traversal, using a three-tier fallback and semantic deduplication.
Neo4jKnowledge GraphsEmbeddings

Ayra Browser: Desktop Agent Browser

Developerprofessional
Built a cross-platform desktop browser with Tauri and a Rust core on Obscura, letting the agent navigate, read, and act on web pages. Exposed browser actions to the agent as schema-defined tools, with permission controls and recovery from failed page actions.
RustTauriObscura

Medical Coding Agent

Developerprofessional
Built a multi-step LLM pipeline that assigns ICD-10/CPT codes from clinical documentation, combining vector and full-text (BM25) retrieval over code descriptions. Enforced correctness with structured decoding and schema validation, so every output is a valid code with cited supporting evidence.
LLMsStructured OutputsHybrid SearchPydanticBM25

Vessel Viz: Vascular Visualization

Developerprofessional
Built a pipeline that segments and renders blood-vessel structures from medical scans as interactive visualizations for clinical review.
PythonMedical Imaging3D VisualizationVLMs

VLM Fine-Tuning & Local Inference

ML Engineerprofessional
Fine-tuned Vision-Language Models with LoRA/QLoRA on domain-specific data using HF Transformers and PEFT, evaluated with golden sets and ablations. Fit models into strict RAM budgets with quantization (GGUF/4-bit) and KV-cache management for llama.cpp/Ollama and NVIDIA Jetson (Nemotron, MoonDream).
HF TransformersLoRA/QLoRAllama.cppOllamaTensorRTPEFTGGUFNVIDIA Jetson

Enterprise Knowledge-to-Software Agent Platform

Developerprofessional
Built a knowledge corpus with scheduled sync and source-cited answers over documents, requirements, and existing code. Developed an agentic software-building pipeline that takes a business request through research, planning, multi-agent build, review, and verification.
Multi-Agent OrchestrationRAG

Multi-Agent Text-to-SQL System

Developerprofessional
Built a multi-agent system (schema linking, planning, generation, validation) with embedding-based schema retrieval, which reduced hallucinated columns on large enterprise schemas.
LangGraphPostgreSQL

Ayra Agent: Agentic AI Workspace

Lead Designer/Engineerprofessional
Designed an agent harness that integrates external tools and coding-agent runtimes (Claude Code, Codex), with per-user persistent sessions and failure handling across multi-step workflows. Implemented multi-turn conversation state and context handling for chat, plus meetings (Ayra Meet), a shared file workspace over WebDAV, and live dictation. Built AyraTest evaluation infrastructure: golden test suites run nightly, scenario-based regression testing, and backend/frontend test automation. Deployed on Azure AKS with Argo CD GitOps, CI/CD promotion pipelines, and Grafana monitoring.
PythonFastAPILLM AgentsTool UseKubernetesClaude CodeCodexWebDAVAzure AKSArgo CDGrafanaPydantic

Temporal logic-guided vision for monitoring mobility transitions: A feasibility study on healthy volunteers

personal
A research paper on dementia behavioral shifts and empathetic AI for care robots, currently under review for a healthcare-related journal.
Temporal Transition Trigger models

Education

Bachelor of Science

University of Engineering and Technology (UET)Computer Science2024

Thesis. End-to-End IoT-Integrated Communication System for Ambulance and Hospital Emergency Response

Recognition

Awards

Honorary Mention, 2024 IEEE ComSoc Student Global Competition

IEEE Communications Society2024

For Final Thesis: End-to-End IoT-Integrated Communication System for Ambulance and Hospital Emergency Response

Patents & publications

Empathetic AI for care robots

publication

Paper under review regarding Temporal Transition Trigger models detecting behavioral shifts in dementia patients.

With Case Western Reserve, Notre Dame

Hire RafayReach out about a role, a contract or a conversation.For recruiters

Rafay's twin is AI, it can make mistakes.

More people on Kanvis

Browse more peopleCreate your own Kanvis