Post a Job Free
0 viewsjobseeker
Ayaz S. — Senior Data Engineer from India

Ayaz S.

Senior Data Engineer

India 3-6 years
Open to offersNew to Platform
Languages
English
Video Introduction
No video introduction yet
The candidate has not added a video.
Contact information and social networks are private. Connect to unlock.
Hidden

About

Ayaz S. is a seasoned Data Engineer with over five years of experience crafting and refining scalable ETL pipelines and cloud-based data platforms, primarily on AWS environments. His technical expertise spans a spectrum of advanced tools and technologies, including PySpark, SQL, and Snowflake, which he has adeptly utilized in optimizing data warehousing and transformation processes. At IntechFY Solutions Pvt Ltd, Ayaz spearheaded projects across various sectors, notably developing a cloud-based data migration platform and a comprehensive healthcare and insurance data platform. His work involved utilizing AWS Glue, Lambda, and EventBridge to ensure seamless data handling and automated execution. He has also delved into emerging technologies by constructing AI applications with LangChain and RAG for enhanced document search and interactive interfaces. Ayaz’s vigilant focus on data quality and performance optimization is underscored by his extensive use of CDC, SCD Type 2, and Apache Iceberg in maintaining robust data management. His proficiency in deploying CI/CD practices further enhances his ability to deliver reliable solutions.

Experience

  • Data Engineer

    INTECHFY SOLUTIONS PVT LTD · 2021 — Present
    Designed and developed a cloud-based data migration platform for ingesting data from MySQL, PostgreSQL, CSV files, and REST APIs into Amazon Redshift. Constructed scalable ETL pipelines using AWS Glue and PySpark, processed structured and semi-structured data from multiple source systems, and implemented CDC logic to capture incremental data changes. Developed SCD Type 2 logic in PySpark for maintaining historical data records, created Apache Iceberg tables for ACID-compliant data management, and automated the deployment of AWS Glue ETL jobs using GitHub Action CI/CD. Automated pipeline execution through S3 event triggers, AWS Lambda, and EventBridge. Executed data quality checks and optimized ETL job performance through various techniques. Configured CloudWatch monitoring and SNS alerts for proactive failure detection and SLA tracking.

  • Data Engineer

    INTECHFY SOLUTIONS PVT LTD · 2021 — Present
    Developed a healthcare and insurance data platform to process structured and semi-structured data using Databricks, Snowflake, and AWS. Created data ingestion pipelines to load data from relational databases, flat files, and Amazon S3 into Snowflake, and constructed ETL pipelines for data cleansing and transformation. Maintained Databricks Notebooks for collaborative development and implemented incremental data loading into Snowflake using Snowflake Streams and Tasks. Managed Snowflake stages and automated data loading from Amazon S3. Optimized Spark jobs and Snowflake SQL queries and developed AWS Glue jobs for data movement. Configured IAM roles for secure access to AWS resources and implemented data validation and quality checks before loading data.

  • Data Engineer

    INTECHFY SOLUTIONS PVT LTD · 2021 — Present
    Created an Enterprise Data Warehouse using Amazon Redshift to unify data from various source systems to meet reporting and analytical needs. Designed end-to-end ETL pipelines with AWS Glue, PySpark, and Amazon S3 for data ingestion. Developed AWS Glue jobs for cleansing and aggregation, loading curated datasets into Amazon Redshift for reporting. Implemented incremental loading strategies, performed Redshift performance tuning, and developed validation processes to ensure accuracy and consistency of data. Automated ETL execution with AWS Lambda and event-driven triggers, while configuring IAM roles for security and operational monitoring.

  • Data Engineer

    INTECHFY SOLUTIONS PVT LTD · 2021 — Present
    Engineered a Retrieval-Augmented Generation (RAG) based AI Knowledge Assistant for advanced document search and contextual Q&A. Assembled an end-to-end RAG pipeline using LangChain for document retrieval and question-answering. Implemented text chunking and embedding generation with Hugging Face models, and facilitated semantic search through FAISS. Integrated local LLMs for privacy-preserving response generation and developed FastAPI REST APIs for document retrieval and AI query processing. Employed prompt engineering techniques to enhance response relevance and managed the codebase with Git and GitHub following CI/CD practices.

Skills & Expertise

AI and Data ScienceVisual DesignerData EngineeringData ArchitectData Engineer

Education

  • Bachelor of Science in Computer Science
    Mumbai · 2019 — 2022