Nilanjan Deb
Architecting Intelligent Data Solutions & Scalable Software
Data Engineer & Software Developer specializing in Big Data, AI-driven analytics, and full-stack application development. Passionate about leveraging big data technologies and AI to drive insights, optimize performance, and build secure, scalable systems.
About Me
My journey into data engineering and software development began at BITS Pilani, where I built a strong foundation in Computer Science fundamentals. The rigorous curriculum in Data Structures, Algorithms, Operating Systems, Databases, and Networks has been instrumental in shaping my analytical approach to problem-solving and system design.
I believe that understanding core CS concepts is crucial for building robust, scalable systems. Whether I'm optimizing a Spark Streaming pipeline or designing a full-stack application, I leverage these fundamentals to create efficient algorithms, design scalable architectures, and ensure optimal performance.
My passion lies in transforming complex data challenges into actionable solutions. I thrive on data integration, ETL processes, real-time streaming, and performance optimization while maintaining the highest standards of security and reliability.
Data Engineering
Big Data, ETL, Streaming Pipelines
AI & Analytics
LLM Integration, AI-driven Workflows
Full-Stack Development
APIs, Microservices, Web Platforms
System Design
LLD, HLD, Performance Optimization
Professional Experience
InMobi Group
From 1 hour to 1 minute
Data onboarding efficiency
Infrastructure optimization
Key Achievements:
- • Architected real-time data streaming pipelines using Spark Streaming and Kafka
- • Implemented CDC pipeline with Apache Hudi for efficient data lake management
- • Developed Trino gateway with Apache Iceberg for optimized query performance
- • Integrated AI-powered analytics tools (Vanna.AI) with LLM for content generation
- • Designed and deployed infrastructure using Helm charts and Kubernetes
- • Implemented OPA (Open Policy Agent) with Trino for enhanced data security
Helioweb
Operational processes
Administrative workflows
Key Projects:
- • Developed P2P delivery platform with real-time order tracking
- • Built comprehensive order management system for manufacturing
- • Created student management portal with advanced analytics
- • Implemented manufacturing web application with workflow automation
Featured Projects
Data Engineering & AI
Integrated Vanna.AI for AI analytics: natural language DB queries & insights
Key Highlights:
- • Real-time analytics
- • AI-driven insights
- • Automated reporting
Architected high-performance streaming pipeline reducing data latency by 98%
Key Highlights:
- • 98% latency reduction
- • Real-time processing
- • Scalable architecture
Software Engineering & Full-Stack Development
Secure, scalable platform for executing code remotely with advanced concurrency management
Key Features:
- • Concurrent execution
- • Security sandboxing
- • Real-time results
Real-time multiplayer game with state synchronization and low-latency communication
Key Features:
- • Real-time sync
- • State management
- • Low latency
High-performance music streaming and download service with intelligent caching
Key Features:
- • API optimization
- • Intelligent caching
- • High throughput
Comprehensive academic scheduling application with smart notifications and analytics
Key Features:
- • Smart scheduling
- • Analytics dashboard
- • User-friendly interface
Full-featured e-commerce platform with real-time order tracking and payment integration
Key Features:
- • E-commerce features
- • Real-time tracking
- • Payment integration
Dynamic polling application with real-time results visualization and analytics
Key Features:
- • Real-time visualization
- • Interactive charts
- • Live polling
Technical Skills
Core Competencies
Data Architecture
Designing scalable data pipelines and lake architectures
AI Integration
Implementing AI-powered analytics and LLM solutions
System Design
Low-level and high-level system architecture design
Performance Optimization
Optimizing systems for speed, efficiency, and cost
Get In Touch
Let's Connect
I'm always interested in discussing data engineering challenges, AI innovations, and opportunities to build impactful solutions. Feel free to reach out!