02 / HEALTHCARE · CDC

MNV SaaS

A real-time synchronization layer for healthcare systems that could not afford stale data or repeated polling.

Role Senior Software Engineer
Period 2023 — 2025
Outcome ~50% less replication lag

The problem

Keep distributed data fresh.

Context

Healthcare systems needed to remain synchronized across distributed services and external integrations, where delayed changes reduce the usefulness of real-time data.

Approach

Architected event-driven microservices and implemented Kafka plus Debezium CDC pipelines to move database changes through an event stream instead of repeated polling.

Why this pattern

Change data capture preserves the database as the source of truth while allowing downstream services to react to changes without tightly coupling every system to the same request path.

Flow

Database change → Debezium → Kafka topic → distributed services.

OUTCOME

The implementation reduced replication lag by approximately 50% and supported a scalable, real-time synchronization layer for healthcare data.

Constraints

Keep the database authoritative.

  • Healthcare data loses value when it's stale; downstream services and external integrations need changes promptly.
  • The database had to stay the source of truth; services couldn't all be rewritten to publish their own events.
  • Repeated polling added load to the source database and still left gaps between polls.
  • Consumers had to be able to fall behind, restart or be added later without losing changes.

Architecture

Committed changes become durable events.

DATABASEcommitted changeDEBEZIUMtransaction logKAFKA TOPICSconsumer offsetsSERVICESstores / integrations

Offsets let each consumer resume where it left off.

Key decisions

Capture commits, not guesses.

Decision 1

Log-based CDC instead of polling

Alternatives considered

Scheduled polling queries; timestamp-column diffing.

Why

Polling runs repeatedly whether or not anything changed, and can miss intermediate states between runs. Reading committed changes from the database log captures every change, in commit order, without extra query load on the tables.

Decision 2

CDC instead of dual writes

Alternatives considered

Have each service write to the database and publish an event.

Why

A dual write can succeed in one place and fail in the other, leaving the database and the event stream disagreeing. Capturing from the log means an event exists only if the change actually committed.

Decision 3

Kafka between producers and consumers

Why

Consumers are decoupled from the source and from each other. Each tracks its own offset, so a slow or restarted service catches up instead of losing data, and new consumers can be added without touching the source.

Failure modes & recovery

Recovery is built into the flow.

What goes wrongHow the design handles it
A consumer goes downChanges stay in Kafka; on restart it resumes from its committed offset.
A change is delivered twice after a restartConsumers apply changes idempotently, keyed on the record's identity.
A consumer falls behindLag is visible and recoverable; it catches up without affecting the source database.
The connector restartsDebezium resumes from its recorded log position.

What I'd change next

Next evolution.

  • Track end-to-end freshness (commit time → consumer applied) as a first-class metric, not just broker lag.
  • Add a schema registry and compatibility rules so schema changes can't break consumers.