A GDPR data workflow: from hourly polling to event-driven PII re-identification

This is a familiar enterprise-data scenario: an analytics team asks the data team to restore hashed personal IDs to their original values for an approved downstream use case, while remaining within the GDPR approval process.

I took over and redesigned this controlled re-identification workflow. The existing solution completed the task, but it had two practical problems: it was slow and expensive.

Problem

Personal IDs in the data lake were hashed by default. Users could join records through stable identifiers without seeing the originals; a small number of approved cases needed to restore the original IDs through a separately stored mapping. This article calls that operation controlled re-identification.

An analysis team first obtained approval, then uploaded a file containing the identifiers it needed restored. Airflow scanned configured upload locations once per hour. When it found a file, the workflow checked whether the request had already been approved and launched a Spark job on a dedicated EMR cluster.

The infrastructure included an always-on EMR cluster and an always-on Amazon Aurora database. After processing, the requester still downloaded the output and manually uploaded it to the downstream system.

approved request
→ file uploaded
→ wait for hourly Airflow scan
→ approval lookup
→ Spark job on always-on EMR
→ user downloads result
→ user uploads it to the destination

A typical end-to-end request took about two hours. Polling, batch processing, and the manual handoff all added delay. That was the slow part.

At the same time, this small, intermittent workload paid continuously for an always-on EMR cluster and Aurora database. That was the expensive part.

Root-cause analysis

1. The AWS migration carried forward an on-premises data-lake mindset.

After the platform moved from local servers to AWS, its architecture still defaulted to permanently running servers. Long-lived services and occasional jobs were not treated differently. Aurora remained online to hold workflow state, while an EMR cluster stayed on standby for files that arrived only intermittently. AWS offered event-driven, pay-per-use services, but the workload still operated like an on-premises data centre.

2. The team fitted new requirements into its existing solution.

The actual requirement was: “When an approved file arrives, check its authorisation, process it, and deliver the result.” Instead, the design started with what the team already had—Airflow, EMR, and a relational database—and fitted the requirement into those components. A process naturally triggered by file arrival became an Airflow polling schedule; a short computation became a permanent cluster waiting for work.

Together, these choices made the architecture follow the existing tools rather than the workload’s trigger, frequency, and lifecycle.

Design approach

1. Replace pull with push to reduce latency.

The original Airflow schedule repeatedly asked whether a file had arrived: a pull model. Its polling interval became unavoidable waiting time. I reversed the direction. As soon as a file reached S3, an event pushed it into the processing workflow, so work started on arrival rather than at the next scan.

2. Use serverless services to remove fixed cost from a low-volume workload.

The business model was simple: requests were infrequent and each run was short, so usage-based cost was already small. Most cost came from fixed, always-on Aurora and standby EMR capacity. With Lambda, Step Functions, DynamoDB, and EMR Serverless, almost no resources ran while there was no request. At this low volume, most serverless calls remained within AWS free-tier allowances.

Solution

I rebuilt the path around the arrival of the file itself:

S3 object-created event
→ Lambda starts Step Functions
→ Lambda checks the prior approval in DynamoDB
→ EMR Serverless performs the approved re-identification
→ Lambda delivers the result to the approved destination
→ owner receives status at every stage

The S3 event removed the hourly wait. Step Functions made the state transitions and failure points explicit. DynamoDB replaced the always-on relational database for this narrow access pattern. EMR Serverless replaced the dedicated cluster, so compute existed only while a file was being processed. The final Lambda delivered the output directly to an approved target such as SFTP.

Every material transition notified the responsible owner: file detected, approval accepted or rejected, processing completed, and downstream delivery completed or failed. Faster processing did not mean weaker control; it made the control path more visible.

Small tip: send a receipt email for every step. At each important state transition, I used Amazon SNS to send stakeholders a receipt. Users no longer had to guess whether the file was detected, approved, processed, or delivered. When something failed, the last successful receipt immediately narrowed the fault to the gap between two steps. It was cheap to implement and materially improved both user experience and troubleshooting.

Results

Typical latency fell from about two hours to 5–10 minutes. This is an observed operating range, not a formal SLA backed by a dedicated measurement system.

The relevant AWS resource costs identifiable within the wider bill fell by about 90%. That comparison was supported by billing, but it was an allocation inside a larger AWS account—not a claim that the total account bill fell by 90%.

The workflow itself also changed:

• no hourly polling delay;
• no always-on EMR cluster;
• no always-on Aurora database for a narrow lookup pattern;
• no manual downstream handoff;
• explicit approval checks and per-stage notifications;
• pay only when an approved request actually runs.

The central change was to align resource lifetime with the lifetime of one approved request: an occasional file no longer waited for a permanent system, but triggered an explicit processing path with governed states, permissions, and notifications.