HL7 ingestion cut from 40 minutes to 90 seconds across 10M patient records

The constraint

Same-day reporting was impossible because the HL7 feed took roughly forty minutes to land, and the batch window could not be moved without renegotiating interfaces with four downstream clinical systems.

What we changed

We left every downstream contract intact and replaced the batch loader with a streaming consumer that writes continuously. The old interface still exists; it now reads from a materialised view rather than the ingest job.

• Message-level idempotency so a replay is always safe
• Schema drift detection on the wire rather than at query time
• Cutover run in parallel for three weeks before the batch job was retired

Where it landed

Ingestion settled at roughly ninety seconds end to end. The number that mattered more to the client was zero: no PHI incidents, and no downstream system that needed changing.