Healthy ingestion
A scientific device creates records, source storage preserves them, and the live pipeline delivers each record to the electronic lab notebook.
Data is being generated, transferred, and confirmed downstream.
Data · Operational resilience
When healthy is not recovered.
A guided simulation of what happens after live data delivery fails—and why restarting the pipeline is only half the recovery.
02 / THE QUESTION
The pipeline is running again. Why is the data still incomplete?
Follow one outage from healthy ingestion through detection, reconciliation, and a safe historical backfill.
Start the recovery Read the recovery overviewGuided system recovery
Follow the records, not only the service.
Advance the scenario at your own pace. Counts and identifiers remain deterministic so every operational decision can be checked.
Step 1 of 10
Healthy ingestion. Source and destination each contain 108 records.
Vendor-neutral system modelLive ingestion path
Desktop system viewScientific device
Scientific measurements
Capture
Source storage
108 source records
Transform
Live ingestion
Validate and transform
Deliver
Electronic lab notebook
108 destination records
Monitoring & alerts
No checks or alerts
Without observability, the failure can remain invisible while records diverge.
Separate recovery routeReconciliation & historical load
Reconciliation
Available when history diverges
Missing only
Historical loader
Safe, retryable replay
Mobile recovery overview
The solution, without the simulation.
Reliable recovery preserves source data, detects delivery failures, reconciles the historical gap, and safely backfills only the records that never reached their destination.
120 source113 after live restore120 after backfill
- 01
Preserve the source
The device keeps writing records to durable source storage during the outage.
- 02
Detect the gap
Health checks and alerts expose the failed delivery path and seven missing records.
- 03
Reconcile history
Compare source and destination identifiers to isolate records 109–115.
- 04
Backfill safely
Replay only those seven records through a separate, retry-safe historical path.
Live delivery is healthy, the seven-record historical gap is repaired, and source and destination both contain 120 records.
What reliable recovery requires
Seven ideas working as one system
- Resilience
Design for interruption, not only the happy path.
- Observability
Make health, lag, and failure visible enough to act.
- Data integrity
Measure whether the complete record set arrived.
- Reconciliation
Compare independent manifests to find exact gaps.
- Backfilling
Replay historical records through a separate recovery path.
- Idempotency
Make retries safe without creating duplicate records.
- Data lineage
Retain enough context to trace where each record came from.
Why I built this
Recovery is more than restarting a service.
This project is inspired by a recovery pipeline I worked on as a Data Engineer. A historical-loading capability later became useful after a real outage interrupted live delivery. The experience distills the lesson: reliable data engineering includes detection, preservation, reconciliation, and recovery.
This is a simplified educational simulation—not a reconstruction of confidential client architecture. It contains no employer names, proprietary details, or real records.