Elmer C.

Engineered a high-performance exporter to transfer data from PostgreSQL to ClickHouse, replacing token-based implementations with batched bulk transfers for enhanced migration speed. Built an automated reconciliation system to monitor discrepancies between PostgreSQL and ClickHouse schemas and row-level data. Developed data ingestion processes from Amazon S3 into PostgreSQL utilizing performance-tuned batch processing and transformation logic. Maintained and managed end-to-end ETL pipelines, including tasks for data extraction, transformation, validation, and export aimed at automated reporting and dashboards. Diagnosed and resolved issues within production data pipelines, contributing to root cause analysis, SQL optimizations, and performance enhancements across the ETL codebase.

Constructed a data ingestion layer using FastAPI and AWS Lambda to facilitate the loading of embeddings and metadata into vector and relational databases for RAG systems. Implemented a pgvector-based vector store, which involved schema design, embedding ingestion pipelines, and optimized similarity search queries. Integrated RDBMS (PostgreSQL) for AI pipelines, overseeing connection management, metadata persistence, and embedding index references. Enhanced vector search performance by creating and tuning multiple pgvector ANN indexes. Developed image text extraction capabilities utilizing EasyOCR and python-doctr[torch], focusing on auto-preprocessing and robust text extraction from images. Produced an image captioning pipeline with Salesforce/blip-image-captioning-large, optimizing it for device-aware loading and model caching. Integrated entity–relationship extraction using google/flan-t5-base into document chunking to identify semantic relationships, applying prompt engineering and rule-based fallbacks. Automated RAG orchestration through AWS Step Functions, facilitating reliable workflow execution with event-driven triggers and concurrency controls. Collaborated with data and AI engineers.

Employed requests, Selenium, and BeautifulSoup alongside various locator strategies to develop a web scraper. Analyzed local network data and packet information using tshark, scapy, and libraries such as sklearn, numpy, and pandas. Leveraged hibp, CVSS, and MITRE ATT&CK frameworks to assess security breaches concerning email addresses stored in a database. Integrated OpenAI (gpt-4o-mini) to generate digestible insights from data, utilizing the Naive Bayes algorithm for classification and creating summaries. Managed version control practices using Git (Github).

Developed Python scripts for web scraping and data processing using requests, BeautifulSoup, pandas, and CSV processing to track the delivery and pickup status of retail websites, outputting data to CSV files. Utilized EC2 for general scraping tasks and automated scraping processes through Mozenda. Employed QGIS for visual verification of quarterly data discrepancies. Engaged in troubleshooting and debugging in collaboration with developers, while managing version control with Git (Bitbucket).

Developed a web application using the WAMP stack and Javascript to monitor client warehouse and office transactions, while also maintaining data integrity and generating reports. Created a mobile web app on the WAMP stack and Javascript to assist drivers in tracking dispatches and proof of deliveries. Built a desktop application with Python that gathered POS files from local directories using FTP, offering both GUI and CLI versions. Designed data visualizations in Crystal Reports and optimized SQL performance with MSSQL by creating views and stored procedures.

Developed a web application using the WAMP stack, HTML, CSS, and Javascript to manage inventory for IT-related hardware and software assets, facilitating user logins through Microsoft 365 accounts. Enabled report generation for each application module.