Shardul A.

Architected a comprehensive Snowflake data platform that handled over 500K daily transactions from more than 10 source tables. Implemented external S3 stages, COPY commands, clustering keys, time-based partitioning, and materialized views to facilitate real-time sales analytics while achieving a 35% reduction in compute costs. Developed a recommendation engine utilizing Python and AWS services to analyze customer purchasing behavior across 10k+ SKUs, effectively generating personalized product suggestions. Enhanced the recommendation engine with a GenAI-powered explanation module that translated product suggestions into natural language for sales representatives. Executed dbt gold layer transformations and established a production star schema fact_deviation with 4 dimension tables, ensuring standardized naming conventions and data quality checks for over 200K deviation records. Collaborated with pharma quality teams and data scientists to build a data pipeline for pharma deviation analysis, integrating UI-triggered REST APIs with backend Snowflake processing, thus enabling timely time series analysis across 200K records over a 2-year span. Streamlined time-series deviation data from 3 source systems into Snowflake, reducing data preparation time by 60% for downstream modeling and reporting.

Maintained and improved critical Airflow ETL data pipelines that processed over 50M daily payment transactions using various payment methods, supporting substantial annual payment volume. Designed and deployed production-grade Scala applications for payment data processing across distributed Spark clusters implemented on AWS, managing over 10TB of data daily. Enhanced Spark job performance by optimizing executor memory allocation, shuffle partitions, and serialization techniques, resulting in a 35% improvement in job completion times. Executed SQL performance tuning strategies to reduce query execution time by 45%, including partition pruning and index optimization for billions of rows in distributed data warehouses. Developed a centralized logging and alerting infrastructure for over 20 production pipelines, significantly reducing pipeline failure detection time from 2 hours to 15 minutes and enhancing incident response capabilities.

Created and managed ETL pipelines that transferred data from AWS S3 to a data warehouse, integrating more than 5 diverse data sources including REST APIs and financial databases. Developed ETL processes to calculate and aggregate financial metrics, achieving a reduction in manual calculation time by 55%. Implemented event-driven automation on AWS, which led to a 20% decrease in monthly infrastructure costs and a 15% enhancement in pipeline response times.

Developed a machine learning-driven churn prediction pipeline that integrated CRM systems with external API data, engineering customer behavioral features based on transaction history and product usage patterns for downstream model training. Maintained alignment with project objectives by updating stakeholders on project status through effective written and verbal communication and technical documentation while delivering solutions within an Agile framework.