The 200x Indexing Tax: Why Storing Telemetry Costs Cents but Searching It Costs Thousands

Storing 1 GB of raw telemetry in Amazon S3 or Cloudflare R2 costs roughly two cents per month.

Making that exact same 1 GB searchable in a traditional observability platform? Between $3.00 and $5.00+.

When you ingest tens of terabytes of logs and traces across modern distributed microservices and autonomous AI agents, that math breaks down completely.

This 200x price discrepancy is what I call the Observability Indexing Tax.

It is the single greatest reason why engineering leaders are forced into an unacceptable trade-off: throwing away 90% of their operational data just to survive the monthly observability bill.

The Vicious Cycle of Log Sampling

Because inverted indexes are computationally brutal to maintain on writes, traditional platforms charge for ingestion volume.

When the observability bill threatens to overtake infrastructure compute costs, teams do what they must:

1. They introduce head sampling (keeping only 10% or 1% of traces).
2. They truncate payloads and drop contextual metadata.
3. They set aggressive 7-day or 14-day data retention limits.

Full Runtime Reality (100%)
  ├── 90% Discarded at Ingest Gate (Cost Control)
  └── 10% Stored in Costly Inverted Index
        └── Result: Production incident occurs in the 90% missing void

This creates the Sampling Paradox: you pay millions of dollars for an observability suite, yet whenever a subtle edge-case incident, memory leak, or rogue agent session hits production, your engineers are still flying blind.

Why Is Traditional Indexing So Expensive?

Legacy log engines were designed in the early 2010s around search-engine text concepts (Lucene, Lucene-derived inverted indexes).

When a service outputs a log line like:

{"level":"info","msg":"User checkout completed","userId":"usr_98124","latency_ms":42}

An inverted indexer tokenizes every word: "User", "checkout", "completed", builds posting lists, merges segments, and maintains massive B-trees in memory and NVMe SSDs.

For unstructured web search, inverted indexes are wonderful. For machine telemetry, where 99% of queries are scoped by tenant, session, time window, and status, full-text indexing every token is massive, wasteful overkill.

The Alternative: The Session Graph & Lake Architecture

At Softprobe, we asked a simple architectural question:

What if we stopped indexing every word on the write path, and instead organized telemetry as a Session Graph written directly to columnar Parquet in an object store?

Ingest (High Throughput)
  │
  ├─► Soft Coalesce Buffer (In-Memory)
  │
  └─► Commit Parquet batches to DuckLake (S3 / R2)
        ├── Coalesced Parquet partitions (pruned by date & session)
        └── Postgres Catalog (Derived, lightweight session index)

1. Vectorized Columnar Storage Over Inverted Indexes

By storing telemetry in Apache Parquet files organized by tenant, date, and session ID, modern analytical query engines (like DuckDB and DuckLake) can execute vectorized SIMD scans across millions of records in milliseconds.

We don’t need a multi-million-dollar Elastic cluster sitting idle burning RAM. When you query a session, the engine reads only the exact byte offsets from object storage.

2. A Session Graph, Not Flat Lines

Autonomous AI agents don’t work in isolated spans. An agent execution is a complex loop:

• User prompt
• LLM reasoning turn
• Tool calls (file search, shell execution, database query)
• Error output and retry

When you capture this as a Directed Acyclic Graph (DAG) tied to an explicit session, you no longer need full-text search to find what broke. You traverse the graph directly from intent to error.

Comparing the Real-World FinOps Numbers

Consider a mid-sized engineering organization generating 1 TB of telemetry per day with a requirement for 90-day retention:

The goal of observability is not to generate real-time vanity dashboards for every span ever emitted. The goal is to provide uncompromised forensic evidence when an incident misbehaves. Boring lake storage makes that evidence affordable forever.

Stop Paying the Tax

Engineering teams shouldn’t have to apologize for logging too much context.

If your team is currently cutting logs to avoid price shocks, it’s time to move from the 2012 inverted-index model to a modern lake-backed session graph.

• Keep the evidence.
• Stop the sampling.
• Escape the indexing tax.