Engineering
How to Cut ClickHouse Costs Without Sampling Customer Traces
Self-instrumentation, unbounded joins, and small inserts can turn an observability pipeline into its own largest workload. Here’s how to fix that without losing customer evidence.
· Vight engineering · 9 min read
ClickHouse cost guide
Keep the evidence. Cut the bill.
When an observability bill climbs, sampling customer traces is an obvious lever. It is also the wrong first move when the expensive work comes from the observability pipeline itself.
Self-instrumentation can create a feedback loop: every query creates spans, every span becomes another insert, and every insert creates more work to observe. Add joins that rescan history on every flush and a useful diagnostic system can quietly become its own largest workload.
We found exactly that pattern inside Vight. The changes below preserve customer telemetry and service-map accuracy while replacing open-ended work with bounded, measurable operations.
01
Find the workload your product creates
Start by separating customer work from control-plane work in ClickHouse query logs. Stable query comments, service identities, and query hashes make it possible to rank CPU time and bytes read by feature instead of guessing from aggregate utilization.
In Vight’s 27-hour snapshot, the worker used 104,010 ClickHouse CPU-seconds and read 12.96 TiB. Two service-correlation joins accounted for 101,712 CPU-seconds and 12.91 TiB. Separately, 84% of recent spans described Vight talking to ClickHouse, Redis, and its own telemetry endpoint.
Those numbers changed the decision. Customer sampling would have reduced valuable evidence while leaving the dominant query pattern and self-telemetry loop intact.
02
Remove unbounded work before reducing fidelity
Vight’s original service-map path joined each arriving trace batch against historical spans. The output was accurate, but the work was not proportional to the new data: every small flush could trigger another large scan.
The immediate fix was to remove correlation from the ingest hot path. The durable design processes committed spans in five-minute ingest-time windows, waits briefly for in-flight data, and restricts both sides of each join to the same tenant and a 60-minute event-time horizon.
Bound new work
Advance from a durable ingest-time watermark through one closed window at a time.
Bound history
Keep both join sides inside one tenant scope and an explicit trace horizon.
Bound retries
Reuse deterministic tokens so a failed window can repeat without duplicating an edge.
03
Batch the writes that remain
Frequent small inserts create parts and merge pressure even after expensive queries are gone. A real microbatch window should keep polling a briefly empty queue until it reaches a row limit or a short deadline; it should not interpret the first empty poll as a reason to flush.
For Vight, that means up to 10,000 rows or 15 seconds. Raw span inserts remain the source for materialized views, avoiding a second round of worker-side serialization and rollup uploads. Empty streams still return immediately, so batching does not manufacture idle traffic.
04
Keep useful self-observability, not every self-signal
Turning off raw internal traces does not mean operating blind. Aggregated runtime and pipeline metrics still expose throughput, failures, queue depth, checkpoint lag, and correlation duration without sending every ClickHouse insert and Redis read back through the same ingestion path.
Vight suppresses its own trace and log exports before the customer queues, with an ingest-side discard for the self-telemetry key as defense in depth. Customer scopes and their sampling policies are unchanged.
05
Prove the optimization like a product change
Measure CPU-seconds, bytes read, insert count, and new-part creation before and after the cutover. Then verify the customer-facing invariants separately: continuous rollups, no missing telemetry, one service edge per trace, and a correlation lag users can tolerate.
The important design test is simple: can every recurring query explain its maximum input window? If the answer depends on how much history happens to exist, the cost is still waiting to surprise you.
Before sampling customer telemetry, remove self-generated noise, batch small writes, and make every background scan explain its maximum cost. The cheapest byte is often the one your own pipeline never needed to create.