Akshay Thakare
Senior Data Engineer | PySpark & Databricks | AWS Cloud | GenAI & LLM Systems
Professional Summary
Results-driven Senior Data Engineer with 5+ years of experience designing and delivering scalable data platforms, real-time streaming pipelines, and production-grade AI systems. Expert in PySpark, Databricks Lakehouse, Delta Lake, Apache Kafka, and AWS cloud services. Currently leading a 50,000-line SAS-to-PySpark migration at Capgemini targeting 60% pipeline runtime reduction. Strong background in Generative AI and LLM engineering, including fine-tuning LLaMA 2/3 and Mistral-7B with LoRA/QLoRA and building LangChain + FAISS RAG pipelines that improved NL-to-SQL accuracy by 45%. Creator of Datainteg.io, an open-source data engineering lab featured in KDnuggets, GenAI Weekly, and AssemblyAI.
Key Achievements
- Led migration of 50,000+ lines of SAS ETL to PySpark on Databricks, targeting 60% runtime reduction with zero data discrepancy.
- Reduced cloud infrastructure costs by 35% at Tata Technologies through Spark job profiling, right-sizing, and compute optimization.
- Architected a real-time streaming platform processing 1.5 TB/day from 500+ EV vehicles using Kafka and PySpark on AWS EMR with sub-second latency.
- Fine-tuned LLaMA 2/3 and Mistral-7B using LoRA/QLoRA; deployed LangChain + FAISS RAG pipeline delivering 45% NL-to-SQL accuracy improvement and 70% reduction in ad-hoc query dependency.
- Built and published Datainteg.io open-source engineering lab; featured in GenAI Weekly, Analytics Insight, KDnuggets, and AssemblyAI.
Technical Skills
- Data Processing
- PySpark, Apache Kafka, Spark Structured Streaming, Delta Lake, Hadoop (Cloudera, EMR), Hive
- Cloud Platforms
- AWS (Glue, EMR, Athena, S3, Lambda, SageMaker, EC2), Azure Data Factory, Databricks, GCP BigQuery
- Programming
- Python, SQL, HQL, SAS (migration), Bash, REST APIs, FastAPI, Flask, Node.js
- AI / GenAI / LLM
- LLaMA 2/3, Mistral-7B, GPT-4, LangChain, LoRA, QLoRA, PEFT, RAG, FAISS, Ollama, NL-to-SQL
- MLOps
- MLflow, Model Registry, Experiment Tracking, Feature Stores, Hugging Face Hub
- Databases
- Snowflake, PostgreSQL, MySQL, MongoDB, Hive, HBase, Elasticsearch, Amazon Redshift
- Orchestration
- Apache Airflow, Oozie, Azure Data Factory, GitHub Actions CI/CD
- Data Engineering
- ETL/ELT, Data Modeling, Schema Evolution, SCD Type 1/2, Star Schema, Data Lineage
- Tools
- Docker, Git, GitHub, JIRA, Linux, Parquet, Delta, Avro, ORC, Agile/Scrum
Work Experience
Consultant – Senior Data Engineer (Lead)
May 2025 – PresentCapgemini | Mumbai, India | Financial Services Domain
- Lead a large-scale SAS-to-PySpark migration, converting 50,000+ lines of legacy ETL to Databricks Lakehouse targeting 60% runtime reduction with zero data discrepancy.
- Define and drive technical direction for the migration workstream, coordinating across onshore business stakeholders, solution architects, and cross-functional delivery teams.
- Architect Delta Lake lakehouse with ACID transactions, SCD Type 1/2, schema enforcement, and time-travel queries for full regulatory audit compliance.
- Developed automated row-by-row validation frameworks and reusable Python migration templates, improving team onboarding velocity by 30% across concurrent workstreams.
- Designed FastAPI and Flask REST APIs for pipeline orchestration, data ingestion, and microservices integration with downstream consumers.
- Implemented data lineage tracking, audit logging, and schema evolution strategies to ensure end-to-end regulatory traceability.
Senior Data Engineer
Dec 2022 – May 2025Tata Technologies | Mumbai, India | Automotive & Manufacturing Domain
Career Progression: Solution Developer → Data Engineer → Senior Data Engineer
- Delivered 35% cloud infrastructure cost reduction by profiling Spark resource consumption, identifying over-provisioned clusters, and implementing right-sizing across AWS and Databricks environments.
- Fine-tuned LLaMA 2 and Mistral-7B using LoRA/QLoRA (PEFT) on SQL datasets; built a LangChain + FAISS RAG pipeline improving NL-to-SQL accuracy by 45% and reducing ad-hoc query dependency by 70%.
- Architected a real-time IoT streaming platform processing 1.5 TB/day from 500+ EV vehicles using Apache Kafka, PySpark on AWS EMR, and MongoDB, achieving sub-second processing latency.
- Reduced ETL pipeline execution time by 40% through adaptive partitioning, broadcast joins, caching strategies, and shuffle reduction across core PySpark jobs.
- Managed enterprise Snowflake and Hive data warehouses including table design, partitioning, bucketing, UDF development, and complex SQL query optimization.
- Leveraged AWS services (S3, Glue, Athena, RDS, Lambda, SageMaker, EC2) for cloud data integration and deployed containerized applications using Docker and GitHub Actions CI/CD.
Data Engineer
Nov 2019 – Dec 2022Persistent Systems Limited | Mumbai, India | Retail Domain
- Built end-to-end data transformation pipelines using PySpark on Cloudera Hadoop and AWS EMR, executing Spark SQL queries for large-scale data processing across retail datasets.
- Created AWS Glue crawlers to catalog RDBMS databases and tables, designed Glue workflows to automate ETL job execution, and stored processed data in Amazon S3 for downstream analytics.
- Performed data pre-processing operations including data validation, cleansing, and schema evolution on AWS EMR clusters to maintain data quality and consistency across ingestion pipelines.
- Automated transient AWS EMR cluster provisioning via Lambda functions for on-demand Spark job execution, reducing idle compute costs and improving resource efficiency.
- Migrated and re-architected legacy ETL code to PySpark on AWS Glue, improving pipeline performance and reducing maintenance overhead.
- Monitored and analysed Spark job logs for performance diagnostics; used Apache Airflow to schedule and orchestrate data pipelines in alignment with business SLAs.
Technologies: PySpark, AWS S3, AWS Glue, AWS Lambda, AWS EMR, Amazon Redshift, Apache Airflow, Cloudera Hadoop, Spark SQL, Python
Key Projects
Agentic AI Platform – BMAD Method POC | Capgemini
2025 – Present- Architected a production-grade multi-agent AI system using the BMAD (Breakthrough Method of Agile AI-Driven Development) framework for autonomous workflow orchestration.
- Built specialized AI agents for planning, code generation, peer review, and data pipeline automation using role-based personas (Analyst, Architect, Developer, QA) with shared memory.
- Integrated agents with LangChain, LlamaIndex, and FastAPI for tool use, RAG-based retrieval, and real-time feedback loops within the Capgemini enterprise environment.
Impact: Reduces repetitive engineering effort through autonomous multi-agent collaboration; applicable to data engineering and cloud migration workflows.
Technologies: BMAD Framework, LangChain, LlamaIndex, FastAPI, Python, RAG, GPT-4, Ollama, Docker
LLM Fine-Tuning & RAG Pipeline – NL-to-SQL System | Tata Technologies
2023 – 2025- Fine-tuned LLaMA 2 and Mistral-7B using LoRA/QLoRA on curated SQL corpora; built a LangChain + FAISS RAG pipeline with vector embeddings for semantic retrieval over internal data dictionaries.
- Deployed model inference via Ollama (Docker) with FastAPI endpoints; built a Streamlit dashboard for real-time natural language querying across PostgreSQL and Elasticsearch.
- Tracked all model experiments and versions using MLflow Model Registry, enabling full reproducibility, evaluation, and rollback capabilities.
Impact: Cut ad-hoc query turnaround from hours to under 10 seconds. Eliminated 70% dependency on data engineers. Improved query accuracy by 45%.
Technologies: LLaMA 2/3, Mistral-7B, LoRA, QLoRA, LangChain, FAISS, MLflow, Ollama, FastAPI, Streamlit, PostgreSQL, Docker
EV Fleet Real-Time Streaming Platform | Tata Technologies
2022 – 2023- Designed a Kafka-to-MongoDB ingestion pipeline handling millions of vehicle telemetry events per hour with schema validation and deduplication logic.
- Implemented distributed PySpark transformations on AWS EMR with Apache Airflow orchestration, SLA monitoring, retry logic, and automated alerting.
- Deployed Elasticsearch and Kibana dashboards for real-time fleet performance monitoring and anomaly detection across 500+ vehicles.
Impact: Reduced ETL execution time by 40%; achieved sub-second processing latency at scale.
Technologies: Apache Kafka, PySpark, AWS EMR, MongoDB, Apache Airflow, Elasticsearch, Kibana, REST APIs
Cloud Infrastructure Optimization | Tata Technologies
2022 – 2023- Profiled Spark resource consumption across production pipelines, identified over-provisioned clusters, and applied right-sizing to Spark and AWS compute resources.
- Tuned PySpark jobs using partitioning, caching, and broadcast joins; set up CloudWatch dashboards to continuously monitor pipeline cost and performance metrics.
Impact: Reduced cloud infrastructure costs by 35% while maintaining all pipeline SLAs and improving overall execution performance.
Technologies: PySpark, AWS EMR, S3, EC2, CloudWatch, Databricks, Apache Airflow, Python
Open-Source Projects
Creator & Maintainer – Datainteg.io
Oct 2025 – Present- Airflow DAG Studio: Visual Apache Airflow DAG builder with drag-and-drop interface and one-click Python DAG export for rapid workflow prototyping.
- HELIX: AI-powered code review platform with 37 specialized agents, shared memory architecture, and automated engineering feedback workflows.
- Featured in GenAI Weekly, Analytics Insight, KDnuggets, and AssemblyAI; active open-source contributor with strong developer community engagement.
Technical Writing
Technical Author – Medium (@akshay03)
Feb 2023 – Present- Apache Airflow Drag-and-Drop Plugin — Visual DAG design without code
- Apache Spark Architecture and Memory Management — Deep dive into distributed execution
- How to Install Apache Airflow In Docker on AWS EC2 — Production-grade deployment guide
- TaskGroup in Apache Airflow — Building reusable modular workflow units
Education
B.E. in Electronics & Telecommunication
Solapur University | Solapur, Maharashtra, India