Engineering notes
Project retrospectives and practical work on data platforms, automation and AI.
2026-10-05
Is a code index worth adding to a coding agent? My AOCI test
A comparison in a 360,000-line production repository: the same answer score, with 2.93× the time and 3.40× the uncached input tokens.
2026-09-03
Changing the Wheels While the Car Is Moving: Looking Back at My Oracle-to-Snowflake Migration
How automation reshaped an Oracle-to-Snowflake migration process, reduced disruption to ongoing development and manual drift, and helped the programme finish on time.
2026-09-02
Stop Telling Agents How to Do Every Step
A task that should have taken half an hour took nearly three, forcing me to rethink repositories, Skills, and CI/CD: give agents stable facts and boundaries, and let them choose the execution path.
2026-09-01
Keep or Sweep: a structured way to remove redundancy
Use evidence, the deletion counterfactual, and retention duties to decide whether code, files, documents, or mechanisms should stay or go.
2026-08-26
A GDPR data workflow: from hourly polling to event-driven PII re-identification
How a GDPR-controlled PII workflow moved from always-on EMR and Aurora to S3 events, Step Functions, DynamoDB, and EMR Serverless—cutting typical latency from about two hours to 5–10 minutes and related AWS cost by about 90%.
2026-08-26
When an AI collaboration repo corrupts its own source of truth
I restructured a company AI-collaboration repo, clarified capability ownership and admission gates, and reduced Skill instructions by 51.7% while preserving behavior that genuinely matters.
2026-08-20
Three skills that keep agentic work from turning into entropy
AI can produce code faster, but it can also produce redundancy, collisions, and false explanations faster. I use three small skills to decide what should exist, who may touch it, and what the system actually means.
2026-07-27
I moved a data platform from Databricks to one VM and cut its monthly cost by 95%
A personal data platform migration retrospective: AI agents increased the cost risk of exploratory work, so I built a smaller runtime with a faster feedback loop.
2026-06-08
Transformers are not just about language
Financial transactions are records of real events. Transaction foundation models may learn the hidden structure inside those events.
2026-06-07
LLMs are relation spaces, not knowledge warehouses
A working mental model for GPT: not a database of facts, but a high-dimensional map of relationships.
2026-05-21
Turning a loader that OOMed 6 times into a 52-second job
A Parquet-to-Postgres rewrite: from batch to streaming execution, from Python to Rust.
2026-05-15
Secrets in the Keychain, not in your .env
A 230-line Bash wrapper around the macOS Keychain that lets me inject API keys into one command at a time — without ever exporting them to my shell or committing them to .env.
2026-05-13
CLI, not MCP
Swapped a Python Outlook MCP for a 250-line Rust CLI. The email body the LLM sees dropped from 6,200 to ~500 bytes, and 700 tokens of tool schemas left the context.
2026-05-13
Cutting new-dashboard delivery from 4–6 weeks to 1–2 hours
At a well-known fashion company, Team C analysts developed dashboard logic in notebooks, then DE rewrote it in PySpark for Airflow. After the redesign, end-to-end delivery took about 1–2 hours and daily data was ready before business users started work.
2026-05-10
Small budget, serious build: how this site and its Agent came together
A small Cloudflare and DeepSeek stack, with deterministic context, editable behaviour, live evaluations, and an in-chat human handoff.