Engineering
Service Maps Need Bounded Correlation, Not Unbounded Joins
The ingest-time watermark, late-arrival paths, and idempotent rollups behind a service map that stays accurate without rescanning the world.
· Vight engineering · 7 min read
Architecture notes
Correlate within limits.
A service map looks like a graph, but building one is a stream-correlation problem. Parent and child spans arrive independently, sometimes late and sometimes in different batches.
The tempting implementation is a historical join on every flush. It is also a query whose cost grows faster than the new information it produces.
01
Ingest time closes the window
Event time tells us when a request happened. Ingest time tells us when Vight became able to process it. A durable ingest-time watermark gives the correlator a monotonically advancing boundary even when clocks differ or spans arrive late.
Each pass waits briefly for in-flight inserts, then processes a fixed five-minute chunk. After downtime, the worker catches up one chunk at a time rather than issuing one enormous recovery query.
02
Late arrivals require two deliberate paths
One query matches newly ingested server spans to older parents. A second matches newly ingested parents to server children that arrived before the current window.
The second path excludes children from the same ingest window. That small condition is what prevents a normal parent-child pair from being counted by both passes.
03
Exactly once is a retry property
The checkpoint moves only after both passes succeed. If either insert fails, the window remains pending and the retry uses the same deterministic token.
A database lease coordinates worker replicas. The design does not depend on one process staying alive; it depends on every process agreeing which window is owned and which token represents its output.
Accurate correlation does not require unlimited history. It requires explicit arrival paths, a bounded horizon, and retries that preserve identity.