Senior Azure Data Engineer
Design and deliver metadata-driven batch, streaming, and event-triggered solutions across Azure Data Factory, Databricks, Event Hubs, Kafka, Delta Lake, and Unity Catalog.
Azure · Databricks · AI / ML
Pramod KumarSenior Azure Data Engineer
Building scalable data systems and applied AI that deliver business value.
I design reliable cloud data platforms and applied AI systems—from streaming ingestion and governed lakehouses to machine learning, RAG, model deployment, and production-ready agents.
- 68data assets migrated
- 14intraday orchestration windows
- 3architecture modes
- 34applied AI / ML notebooks
Open to senior data and GenAI engineering roles, and available for select Azure, Databricks and applied-AI engagements.
68tables in a migration program
14intraday orchestration windows
34hands-on AI / ML notebooks
2public engineering repositories
About me
I design reliable data platforms, with LLM applications on top.
I’m a Lead data engineer with about 10 years of experience designing production pipelines on Azure and Databricks, and I’ve led and mentored a team of 5+ engineers. I also build Generative AI applications, covering both the data platform and the LLM application on top of it, end to end.
Experience
Enterprise scale. Production ownership.
More than a decade across enterprise ETL, Azure modernization, Databricks lakehouse engineering, real-time integration, and applied AI—with current work centered on retail and supply-chain platforms.
Azure Data Engineer
Migrated on-premises data workloads to Azure and built parameterized ingestion, transformation, orchestration, quality, and operational-recovery patterns.
ETL Developer
Delivered Teradata, Informatica, Oracle, SQL, and UNIX data integration across analysis, development, tuning, testing, and production implementation.
Featured engineering
Proof, not a skills list.
Two public repositories show both depth and breadth: a production-style Databricks LLMOps system and a hands-on portfolio spanning classical ML, deep learning, computer vision, NLP, RAG, and deployment.
Public GitHub repository
Arxiv Curator
An AI research assistant that ingests the latest arXiv papers, builds governed retrieval, serves a stateful agent, and continuously evaluates production traces. It uses the Databricks platform end to end rather than stopping at a notebook demo.
System pathmain / monitored
01
IngestarXiv API · PDFs · Delta tables
02
RetrieveParsed chunks · Vector Search
03
ReasonMCP tools · Genie · memory
04
ServeUC model · AI Gateway · endpoint
05
EvaluateMLflow traces · judges · dashboard
AI-ML-Projects · Public GitHub repository
Applied AI & Machine Learning Portfolio
A completed body of notebook-based work that follows the full ML lifecycle: problem framing, data preparation, exploration, feature engineering, model training, evaluation, interpretation, and application deployment.
34notebooks
8skill areas
68tracked files
ML / 01
Medical Knowledge RAG
Builds document ingestion, chunking, embeddings, vector retrieval, prompt engineering, and grounded question answering over a medical reference corpus.
LangChain · Chroma · GTE embeddings · Llama.cpp ML / 02EasyVisa Approval Prediction
Explores 25K+ cases, treats class imbalance, compares ensemble classifiers, tunes models, and evaluates the business tradeoff between recall and precision.
XGBoost · Random Forest · SMOTE · model tuning ML / 03COVID-19 Image Classification
Applies deep-learning image classification to labeled medical imagery and demonstrates the computer-vision workflow from data preparation through evaluation.
Computer vision · TensorFlow · Keras · CNN ML / 04ReneWind Failure Prediction
Uses neural networks for predictive-maintenance classification, with iterative experiments that connect model behavior to an operational decision.
Neural networks · optimization · predictive maintenance ML / 05Rental Price Model Deployment
Takes a serialized prediction model beyond the notebook with a REST backend, interactive frontend, container setup, and online and batch inference.
Flask · Streamlit · Docker · Hugging Face Spaces ML / 06End-to-End ML Practice
Covers EDA, regression, classification, clustering, ensembles, feature engineering, NLP sentiment, article categorization, and model evaluation.
Pandas · scikit-learn · XGBoost · NLP · EDAProduction systems
Built for the messy middle.
The difficult part is rarely moving bytes. It is proving completeness, making retries safe, and giving operators a clear answer when a dependency fails.
SYS / 01Event driven
Batch completion detector
Moves orchestration from schedule-only assumptions to an event signal that validates the end record before downstream processing begins.
SYS / 02Streaming
Warehouse event ingestion
Standardizes Kafka-compatible ingestion, durable checkpoints, schema evolution, partitioning, and recovery for high-frequency operational events.
SYS / 03Governance
Lakehouse migration
Maps legacy dependencies and moves data products toward Unity Catalog with repeatable development, validation, and production promotion.
Architecture explorer
Follow the decision path.
Switch views to see how the same engineering principles—clear contracts, idempotency, governed state, and observable failure—apply across different systems.
DESIGN NOTE
Expertise
From source signal to trusted product.
I work across the full path, with a bias toward designs that stay understandable when the system is under pressure.
Technical range
StreamingEvent Hubs, Kafka, Auto Loader, Structured Streaming, checkpoints, replay, schema evolution
LakehouseDatabricks, PySpark, Delta Lake, Unity Catalog, medallion design, performance tuning
OrchestrationAzure Data Factory, Control-M, Azure Functions, metadata-driven pipelines, dependency handling
AI engineeringRAG, Vector Search, prompt engineering, MLflow tracing and evaluation, Model Serving, Genie, MCP, Lakebase
Machine learningscikit-learn, TensorFlow, Keras, XGBoost, feature engineering, ensembles, computer vision, NLP, model deployment
IntegrationService Bus, REST APIs, managed identity, Key Vault, private networking, operational alerts
Engineering principles
- 01Check the data is complete before moving it on.
- 02Make reruns safe, so a retry never creates duplicates.
- 03Keep settings out of the code, so it runs the same everywhere.
- 04Plan for failure: set aside bad records, alert, and replay.
- 05Turn lessons into reusable templates and checklists.
Writing
Notes on building data and AI systems.
Practical write-ups on Azure, Databricks, streaming and applied AI, drawn from real platform work.
ARTICLESComing soon
The first articles are on the way.
I’m writing up lessons from event-driven pipelines, lakehouse reliability and applied AI. Check back soon.
Connect
Let’s build data and AI systems people can trust.
I’m open to senior data and GenAI engineering roles, and available for select architecture, data-platform, Databricks, streaming, and applied-AI engagements. Start with the project assistant to turn your idea into a focused brief, or email me directly.