Cutting new-dashboard delivery from 4–6 weeks to about one working day

Pain 1: every change was implemented twice, but neither team could close an incident alone

The ticket looked routine:

“Some dashboards are showing wrong numbers. Can you fix them?”

The visible problem was a reporting discrepancy. Following the data path upstream revealed a cross-department development process built on duplication.

At this fashion company, DA did not sit inside the DE team. Analysts worked within specific business teams such as Team C. When Team C raised a requirement, its DA developed the business logic in a notebook. That stage usually occupied one sprint in the delivery plan, although pure new-development time was not measured separately from notebook operations and maintenance. The completed notebook was then handed to DE. Engineering rebuilt the logic in PySpark, added it to the Airflow- governed ETL, and spent another one or two sprints doing so.

Team C business requirement
→ Team C DA notebook: development and maintenance stage, scheduled ~1 sprint
→ manual cross-department handoff
→ DE PySpark engineering file: runtime version, 1–2 sprints
→ Airflow-governed ETL
→ dashboard

The same business rule therefore existed as a development version and a runtime version. A new dashboard passed through two team backlogs, producing a typical end-to-end lead time of four to six weeks.

The full handoff applied to every change. A new dashboard went through it, and so did debugging, a filter adjustment, or a small bug fix. DA changed the notebook, explained the change to DE, and waited for DE to update the PySpark engineering file. When numbers disagreed, the teams compared the notebook, the PySpark version, the source data, and the dashboard query. Meetings became part of the debugging process.

Production incidents also blurred ownership. DA understood the business logic but could not read the PySpark runtime code that DE had rewritten, or directly fix the production version. DE could change the PySpark code, but needed DA to explain the business rules, confirm the expected result, and complete another handoff. Both positions were reasonable: DE maintained the runtime code, while DA owned its business meaning. The system had split the knowledge and access needed to solve one problem across two teams, leaving neither able to diagnose, change, and validate it end to end.

The process became too slow for the business. Team C analysts eventually scheduled their notebooks with cron and fed the results directly into BI. This shadow pipeline reduced the wait and moved production outside the ETL that DE governed through Airflow. Tests, lineage, alerting, monitoring, and shared operations still existed on the formal platform, while the data used by the business came from another path.

Pain 2: business users started the day without fresh data

Before I became involved, daily dashboards also refreshed late. The team could not reliably put fresh data in front of business users when they started work. Monday mornings were especially vulnerable: the first analysis after the weekend could begin while dashboards were still waiting for upstream data or a notebook run to finish.

Two structural layers reinforced each other. The notebook was the development version maintained by Team C; the PySpark engineering file was the runtime version maintained by DE. No executable interface synchronized them. Tickets, meetings, and human explanation carried changes across the department boundary.

1. Development and runtime versions evolved separately

DA implemented the requirement in a notebook. DE maintained the PySpark version from that notebook and its handoff notes. A filter, join, time window, or edge case changed on one side without updating the other automatically. Two codebases represented one business intent under different owners, backlogs, and release cadences.

2. Every change crossed the department wall

After DA finished development, it handed inputs, outputs, business rules, and expected results to DE. Engineering reconstructed the requirement, scheduled the work, developed it, and validated it. A small bug followed the same route. The code change could be tiny while the organizational wait still occupied one or more sprints.

3. The cost of the formal path created a shadow pipeline

The Airflow ETL provided shared scheduling, tests, lineage, alerting, and operations, but every requirement waited for a DE rewrite. Cron-scheduled notebooks let Team C avoid that queue and obtain data faster, so they gradually became the production path that the business actually depended on. The governed path and the real delivery path separated.

4. The shadow pipeline could not reliably detect input readiness

Cron notebooks had no automated way to verify that upstream tables had refreshed. DA scheduled them later to reduce the risk of running on stale or missing input. The notebook server also had less compute than the DE cluster. Once cron jobs were scattered across the business team, DE could not see the complete DAG or optimize bottlenecks along the end-to-end critical path.

There was also an ownership constraint. Team C owned the dashboards and its analysts owned the notebooks. I had raised the rework loop during my first engagement with this client, but engineering had neither the mandate nor the trust required to change another team’s working method.

During my second engagement, I spent the first months resolving the problems already in front of the teams: reporting defects, Spark performance incidents, and IAM issues. Those deliveries established enough credibility to test a new workflow when the next reporting discrepancy arrived.

I spent one week building a parallel production-like path for one dashboard. It used the same inputs and produced the same output as the existing route, so the teams could compare the two workflows with real evidence.

DE defined eight constraints for this delivery path. Analysts kept the speed of authoring business logic, while engineering retained control of data boundaries, runtime behavior, and final publication.

The day-to-day development workflow after the redesign

DA continued to develop, change, and debug in the familiar notebook, first running the logic through to a verified output in the experiments layer. The versioning happened behind the scenes: notebooks synchronized automatically to a DE-managed repository. Every change had a history, while DA did not need to operate Git or rewrite the business logic in another language.

Once the result was ready, DA only had to tell DE that it could be deployed. DE pushed the synchronized notebook to a branch and opened a PR, usually a 10–20-minute task. CI/CD then checked SQL statement types, input and output boundaries, idempotency, the YAML contract, and dependencies. A passing pipeline showed that the change stayed within the guardrails defined by DE. It automatically extracted the SQL, built the DAG, and completed the production deployment in about ten minutes. DE reviewed the same notebook that DA had already executed successfully; engineering no longer reimplemented its business logic.

The PR work and pipeline execution themselves totalled about 20–30 minutes. Including notification, review wait time, and deployment confirmation, production onboarding after SQL was ready usually took one to two hours.

DA develops or debugs in a versioned notebook
→ experiments output verified
→ DA tells DE it is ready to deploy
→ DE pushes the synchronized version and opens a PR (10–20 minutes)
→ CI/CD checks the DE guardrails
→ SQL and DAG built automatically; production deployed (~10 minutes)

Incidents followed an equally explicit route. When Airflow raised an alert, DE first checked the governed boundaries: whether an upstream input was missing and whether the final output-table schema had drifted from its contract. DE fixed input, output, or platform-runtime failures. If those boundaries were healthy, the issue returned to the business logic. DA debugged the same notebook in the experiments layer and sent the fix through the automated path above. The teams no longer handed off and translated two implementations; they used observable evidence to route the problem to the right side.

DA SQL notebook
→ CI/CD: SQL-only, idempotent, governed inputs, experiments-only outputs
→ ordered SQL files extracted automatically
→ YAML + DAG factory + upstream refresh gates
→ experiments data layer
→ DQ validation
→ DE-managed Gold table
→ BI publication

The side-by-side pilot made the proposal concrete. Stakeholders could inspect the same input, the same output, and a much shorter path between them. Approval followed from a working example rather than an architecture presentation.

The outcome can be summarized in three wins:

• Business: daily dashboard data was ready between 5 and 7 AM, before users started work at 9.
• DA: analysts recovered time previously spent maintaining old notebooks, resolving runtime failures, and checking refreshes, and refocused on new requirements.
• DE: engineering removed the repeated PySpark implementation and the time spent explaining, handing off, and aligning changes, redirecting that capacity to data quality, performance, and platform capabilities.

The redesign did not directly reduce the thinking and development time needed to write new business logic. We had no stable baseline for that work before the change; after the workflow settled, most new logic took a few hours to one day. The larger gain was team capacity. DA no longer spent so much time keeping cron notebooks alive, handling runtime failures, and manually checking refreshes. More of the team’s time became available for new requirements. Once SQL was ready, production onboarding took another 1–2 hours, bringing the usual end-to-end delivery time to about one working day.

Using an eight-hour working day, the four DAs had spent four hours together every Monday on troubleshooting, or 16 hours across the team. Each DA also spent another 1–1.5 days per week resolving failures in the notebooks they owned, adding 32–48 hours. Assuming those two blocks did not overlap, keeping the existing notebooks running consumed 48–64 team hours each week: roughly 6–8 person-days. The redesign removed most of that maintenance burden, making most of the recovered capacity available for new requirements.

Removing the second implementation and its queue recovered about five engineering person-days per sprint. DE redirected that time to source-data quality, performance bottlenecks, lineage tooling, and compute and storage efficiency.

Dashboard data that had usually been ready between 10 and 11 in the morning became available between 5 and 7. Three changes produced that improvement:

1. Airflow checked whether input tables were ready, so DAGs could be scheduled early and run as soon as upstream refreshes completed.
2. SQL ran on the DE cluster, which was faster than the former notebook server.
3. DE could see the complete DAG graph, identify cross-task bottlenecks, and optimize the critical path.

Business users could start the morning with current data, including the more fragile Monday-morning cycle. The governed route became faster and more predictable, removing the need for DA to maintain the cron shadow pipeline.