Exactly-Once Is a Myth; Idempotency Is the Answer
Why distributed systems deliver at-least-once, what 'exactly-once' really means in Kafka and Spark, and how idempotent writes give you correct results.
Data Engineering System DesignAdvanced2 min readUpdated 11 Oct 2026
Delivery guarantees
| Guarantee | Meaning | Risk |
|---|---|---|
| At-most-once | Each message processed 0 or 1 times | Data loss |
| At-least-once | Processed 1 or more times | Duplicates |
| Exactly-once | Effect applied once | Hard; only within boundaries |
Networks fail after a write succeeds but before the acknowledgement arrives. The sender cannot tell whether to retry, so it retries. That makes at-least-once the realistic default.
What "exactly-once" products actually mean
- Kafka transactions give exactly-once from Kafka topic to Kafka topic.
- Spark Structured Streaming gives exactly-once into sinks that support it (checkpointed offsets plus idempotent or transactional sinks such as Delta).
Once data crosses into a system outside that boundary (an API, a database without transactions), the guarantee no longer holds.
Effectively-once = at-least-once + idempotent writes
| Technique | Example |
|---|---|
| Upsert on a natural key | MERGE ... ON order_id |
| Deterministic batch ids + ledger | Replaying batch 42 returns "already published" |
| Partition overwrite | Rewrite business_date=2026-01-05 completely |
| Dedup table of processed message ids | Skip ids already seen |
Watch the side effects
Emails, payments and API calls cannot be rolled back. Give each one an idempotency key, so a retry is recognised and ignored.
Finished this note?