I moved a data platform from Databricks to one VM and cut its monthly cost by 95%

This is a technical retrospective. Absolute costs, data volumes, environment identities, and exact resource configurations have been omitted to protect business information.

Over the past few months, I moved the day-to-day runtime of a data platform from Databricks and ADF to a single VM. The new runtime consists of Airflow, DuckDB/Polars, and DuckLake. Data files remain in object storage.

The platform’s monthly operating cost fell by about 95% after the migration.

That number is striking, but the reason I started the work emerged during development. As I began using AI agents more extensively, I could write code and explore ideas much faster. Exploratory work also became a larger part of day-to-day engineering. The platform’s feedback loop and billing model gradually fell out of step with this new pace.

The problem: AI agents accelerated exploration and increased cost risk

Data engineering involves repetition. An engineer takes a sample, changes a transformation, reruns a job, checks edge cases, and adjusts the code again. With an AI agent involved, the same loop can happen much faster and may continue for several rounds.

That created a practical risk. Prompts and code review alone could not guarantee that an agent would correctly estimate scan volume, compute size, and retry behaviour every time. An ordinary data exploration task could start expensive Databricks compute. Automatic retries after a failure could continue adding to the bill. Development accelerated, and monthly spending became less predictable.

At the same time, I reviewed actual job windows and resource utilisation. The daily ETL load was relatively stable, and a suitably sized machine could finish it within the required window. The existing platform offered powerful elasticity and multi-tenant capabilities, but the platform rarely needed them at its current scale.

The runtime path looked roughly like this:

ADF → Databricks / Unity Catalog → object storage

Every component had a clear purpose. Yet one routine code change still had to cross the orchestrator, compute platform, permissions, container images, repositories, and delivery pipelines before a developer knew whether it worked in the real environment.

The problem: CI/CD made each validation cycle too long

A small change often followed this path:

change code → commit → wait for CI/CD → deploy remotely → run job → inspect logs

Even after local tests passed, the cloud run could still fail because of a missing package, service identity, image version, or permission. The developer would fix the issue and repeat the entire path.

Once code generation became faster, waiting occupied a larger share of the cycle. An AI agent could produce a change in minutes, while the person and the agent still had to wait for a remote CI/CD run to return the next useful piece of information. The number of real validations I could complete in an hour became a major constraint on development speed.

Adding CI concurrency did not remove this delay. The CI/CD pipeline was still carrying too much exploratory validation, and each experiment travelled through the full remote path.

The response: consolidate the daily runtime on one VM

I first used historical job data to confirm that one machine could support the platform’s day-to-day workload. I then changed the main path to:

Airflow + DuckDB/Polars (VM)
              ↓
DuckLake catalog: relational metadata database + Parquet in object storage

Airflow handles orchestration, and DuckDB/Polars performs transformations. DuckLake keeps catalog and snapshot metadata in a relational database while Parquet files continue to live in object storage. The migration changed where compute, scheduling, and development feedback happen; it did not tie the data to the VM’s local disk.

Docker Compose describes the runtime. Local development and the VM use the same container definitions, dependencies, and service relationships. Engineers and AI agents can inspect the Compose file, Dockerfiles, requirements, DAGs, and tests to understand where the code will run.

This shortened many debugging paths. Missing dependencies can be reproduced locally, image changes can be tested directly, and job logs are available in a runtime I can inspect directly. Validation with real data still happens in a controlled environment. Host, network, and RBAC changes remain managed through infrastructure code.

The response: give exploration and delivery different rhythms

I kept pull requests, CI, review, and authorised deployments to preserve the audit trail for formal delivery. I also introduced a controlled path for fast validation:

validate with local Compose
        ↓
run real input in a shared validation environment
        ↓
fix and repeat
        ↓
PR / CI / review
        ↓
authorised deployment

The fast path can operate applications and jobs, but it cannot change the host, network, or permission boundaries. It reuses the same containers and deployment logic as formal delivery so that the two paths do not evolve into separate environments.

Exploratory work can now receive real feedback quickly, while formal delivery retains traceable records. This also makes AI agents more useful: after making a change, an agent can see the runtime result sooner and continue with the next correction.

The result: predictable cost and a shorter feedback loop

After the migration, the platform’s monthly operating cost fell by about 95%. Its largest variable compute expense became the fixed capacity cost of one VM. A few extra exploration or validation runs no longer start a group of high-specification clusters.

The engineering effects are equally concrete:

• the local environment reproduces more runtime failures;
• a change can receive real feedback before going through the full CI/CD path;
• dependencies, images, jobs, and logs sit in one inspectable runtime;
• AI agents can perform more validation while the cost ceiling remains clear.

The migration also reduced guesswork during incidents. Previously, a cloud failure sent me through code, image, identity, permissions, and platform state one by one. Most issues can now be located directly from container definitions, job logs, and deployment records.

The boundaries are explicit

A single VM is one failure domain. We keep Parquet files in object storage, back up catalog and runtime metadata, and ensure that containers and configuration can be rebuilt from versioned artifacts. Jobs are designed to be idempotent where practical so that they can be rerun safely after a failure.

These measures support recovery; they do not provide uninterrupted service. When the business requires very low RTO/RPO, continuous availability, or more concurrency, the platform will need additional instances, failover, and tested operational procedures.

The scaling path remains open. The data uses open formats and the runtime is containerised. When job windows, resource utilisation, or concurrency consistently approach the limit of one machine, the next steps can be resizing the VM, separating workers, and later moving to multiple VMs or Kubernetes. Each step can be triggered by observed workload data.

What I took away

This migration left me with a practical decision rule: examine job windows, resource utilisation, concurrency, recovery targets, and the team’s operational capacity before deciding how much platform capability is required.

For this data platform, one VM provides enough compute for the current workload and sets a clear cost ceiling for exploratory development. That ceiling becomes more valuable as I work faster with AI agents.

The 95% reduction is the easiest result to see. The lasting change is a short, clear development loop: make a change, run it, observe the result, and continue.