Senior Data Engineer & Applied AI Engineer
Problems solved
Explore concrete team challenges, my contribution and the results delivered.
A platform too heavy and expensive
About 95% lower monthly cost for comparable day-to-day workloads.
Enyquant
My approach
Choose architecture around workloads, team size and cost. Built the cloud and data foundation from zero, moving comparable daily workloads from Databricks + ADF to DuckDB, Polars and DuckLake on a single VM, with responsibility for implementation, deployment and operations.
Outcome
About 95% lower monthly cost for comparable daily workloads; delivered five years of country-wide electricity operations and market data in 60 days.
- Azure
- Alibaba Cloud
- Airflow
- DuckDB
- Polars
- DuckLake
A job failed. Can it be rerun safely?
Config-driven, idempotent ETL avoids duplicate writes on reruns.
PVH
My approach
Built config-driven, idempotent ETL on PVH’s 500+TB AWS data platform, making safe reruns a pipeline design requirement. Idempotence means processing the same input again leaves target data consistent rather than accumulating duplicates. Also improved data quality, observability and cross-timezone scheduling.
Outcome
Delivered safely replayable production ETL with repeatable processing paths for recovery and daily operations. Configuration reuse reduced duplicate implementation while supporting the large platform’s continued operation and evolution.
- Python
- SQL
- Spark
- Airflow
- AWS
- Idempotent ETL
A dashboard takes 4–6 weeks to deliver
A representative delivery path reduced to 1–2 hours.
PVH
My approach
Analyst notebooks were handed to data engineers for a PySpark rewrite. Made analyst-authored SQL part of a governed delivery path using YAML contracts, validation and a DAG factory. Compared old and new outputs in parallel before adoption.
Outcome
A representative end-to-end delivery path fell from 4–6 weeks to about 1–2 hours; the platform supported 60+ dashboards.
- SQL
- YAML
- Airflow
- Data contracts
Spark runs for hours, then runs out of memory
All 175 business cases completed in under five minutes.
FedEx
My approach
Restructured a Spark job that could not finish on the full dataset, turning analytical code into a working production job and validating every business case against complete data.
Outcome
The old job ran out of memory after three hours. The new version completed all 175 business cases in under five minutes.
- Spark
- PySpark
- EMR
- GKE
The file arrived. The business waits two hours.
Typical latency down to 5–10 minutes; related resource cost down about 90%.
PVH
My approach
An approved PII re-identification workflow relied on hourly polling, always-on EMR and Aurora, and manual result transfers. Replaced it with S3 events, Step Functions, approval checks in DynamoDB, EMR Serverless and automatic delivery to approved destinations, with stage receipts.
Outcome
Typical end-to-end latency fell from about two hours to 5–10 minutes. Identifiable AWS resource costs for this workflow fell about 90%.
- AWS
- Step Functions
- DynamoDB
- EMR Serverless
- Terraform
ETL feedback arrives hours or days later
Feedback in minutes for 40+ developers.
VodafoneZiggo
My approach
Integrated PowerCenter conversion and preflight validation into GitLab CI/CD so developers received results earlier, with less waiting and repeated manual work.
Outcome
Covered 40+ developers; feedback fell from hours or days to minutes, conservatively saving 40+ hours of manual work per week.
- GitLab CI/CD
- PowerCenter
- ETL validation
The platform must move. The business keeps running.
Full-load and CDC support for hundreds of TB from Oracle to Snowflake.
VodafoneZiggo
My approach
Ran AWS DMS Full Load + CDC to connect historical loads with ongoing changes, while evolving the Scala/Spark ETL framework and supporting migration issues.
Outcome
Provided full-load and incremental sync support for hundreds of TB in the Oracle-to-Snowflake migration. My scope covered migration support, pipelines and engineering tools.
- AWS DMS
- CDC
- Oracle
- Snowflake
- Scala
- Spark
Analytical code needs a path to production
Data science environments, production jobs and training-data time semantics.
VodafoneZiggo · FedEx · Enyquant
My approach
Built JupyterHub + MLflow services for 4–5 data scientists at VodafoneZiggo. Operated JupyterHub and productionised Spark code at FedEx. Defined point-in-time visibility for historical training data at Enyquant.
Outcome
Delivered data science services, production Spark workloads and time semantics for model-training data.
- JupyterHub
- MLflow
- Python
- Spark
- Point-in-time data
More agent rules, less clarity
Clear capability ownership; scoped Skill content reduced by 51.7%.
AI collaboration repository · agent-skills
My approach
Reorganised a company-wide AI collaboration repository, separating team contracts, shared interfaces and optional methods. Defined a discover, adopt, contribute and release lifecycle, consolidated duplicate rules and clarified permissions, resource coordination and delivery evidence.
Outcome
Scoped Skill content fell from 2,975 to 1,437 lines; all 18 retained capabilities passed independent validation. Productivity gains have not been quantified.
- AI Agents
- Skills
- Plugins
- Context Design
- Resource Coordination
Green tests do not prove the agent finished the task
Deterministic checks separate mechanism tests, candidate coverage and actual data.
Enyquant
My approach
Built a Docker-distributed validation CLI connecting development contracts, test selection, source-sampling plans and structured reports. Bound checks to code versions, contracts and data sources, rejecting wrong intervals, units, members, old contracts and missing evidence. Designed dual-architecture checks and release gates for final images.
Outcome
Merged tooling rejects samples with incorrect intervals, units or members. Passing mechanism tests cannot override failing actual data; unexecuted steps and missing candidate coverage remain explicit. Real data delivery uses snapshot readback and replay checks, with code, deployment, data and consumer acceptance recorded separately. Formal candidate artifact verification remains a separate execution step.
- Python CLI
- Docker
- Contract Validation
- Structured Reports
- CI/CD
Personal agent workflows are hard to share
Three shared Skills connect development rules, versioned tools and real trials.
Enyquant
My approach
Packaged data development and source understanding into three shared Skills with explicit inputs, tool versions, permissions, evidence and stop conditions. Kept deterministic implementation in the business repository CLI. Used another member’s real Windows/WSL agent trials to improve environment discovery, UTF-8 handling and blocked-execution handoffs, with one maintained source.
Outcome
Source for all three shared Skills was merged, turning personal workflows into maintainable team entry points. Trial feedback informed corrections; missing tools, environment blocks and failed business checks are handled separately. Source acceptance, formal release and member installation are recorded separately.
- AI Agents
- Skills
- Plugins
- Workflow Engineering
- Behavioral Validation