# Bill Zuo — Full Knowledge Corpus for LLMs
Author: Bill Zuo (Founder & CEO at Softprobe)
Website: https://billzuo.com
Generated: 2026-09-24T23:49:15.085Z

================================================================================

---
Title: Context Over Index: What AI Coding Agents Actually Need to Operate Software
URL: https://billzuo.com/blog/context-over-index-what-ai-agents-need
Date: 2026-09-24
Category: Agent QA
Tags: agent-qa, ai-infrastructure, observability, context-engineering, ducklake
Description: Why traditional APMs and prompt stacking fail autonomous coding agents, and why full runtime context graphs are the true fuel for production AI.
Key Takeaways:
  * Static Code vs. Runtime Reality: Code repositories only show how software is supposed to run; agents need the full runtime trace of variables, tool calls, and state transitions to solve complex production bugs.
  * Context Engineering Over Prompt Stacking: Stacking longer prompts into model context windows causes token bloat and hallucinations. Feeding structured, high-density session graphs produces reliable autonomous fixes.
  * The Observability Gap: Enterprise backends were built for human eyes scrolling through charts, not for AI agents that require machine-readable, deterministic event graphs.
  * Reproducible Agent QA: Validating whether an AI coding agent truly fixed a bug requires replaying the historical session evidence against the new patch.

---

import KeyTakeaways from '@/components/mdx/KeyTakeaways.astro';
import Callout from '@/components/mdx/Callout.astro';
import FAQ from '@/components/mdx/FAQ.astro';
import Quote from '@/components/mdx/Quote.astro';
import CodeBlock from '@/components/mdx/CodeBlock.astro';

<KeyTakeaways items={[
  "The Missing Fuel: AI agents cannot operate or debug software using static code and human dashboards alone.",
  "Graph over Text: Agents reason with highest precision when telemetry is structured as a Directed Acyclic Graph (DAG) of cause and effect.",
  "Context Engineering: The value is not in raw model intelligence, but in the wrapper: historical decisions, environmental constraints, and evidence.",
  "Agent QA: Autonomous agents must be evaluated against real captured production sessions, not synthetic mock benchmarks."
]} />

Over the past two years, the software engineering industry has poured billions of dollars into foundation models, agent harnesses, and autonomous coding assistants.

Yet, ask any engineering leader what happens when they task an autonomous agent with diagnosing and fixing a real, complex production incident:

The agent loops. It reads the source code. It guesses. It stacks prompts. And eventually, it hallucinates a plausible-looking patch that fails to address the root cause.

Why?

Because **AI cannot operate software without real runtime context**.

---

## The Static Blindness Problem

When a human senior engineer is called to resolve a high-severity production outage, what is the first thing they do?

They do not simply open GitHub and read the codebase from line 1 to 10,000.

They ask:
1. *What specific payload triggered the exception?*
2. *What state was the user session in immediately before the database timeout?*
3. *Did a downstream third-party payment API return an undocumented HTTP 429 response?*

They look for **runtime evidence**.

```text
Static Code (The Plan)         Runtime Telemetry (The Reality)
  "function processOrder()"      Input: { id: "ord_99", amount: null }
  "db.save()"                    DB Error: Violates NOT NULL constraint
  "sendNotification()"           Result: Unhandled Promise Rejection
```

Source code only tells you how software was *intended* to behave. Runtime context tells you how it *actually* behaved in the wild.

When we expect AI agents to fix bugs using only repository files and snippets of flat text logs, we are asking them to perform brain surgery with their eyes closed.

---

## Context Engineering vs. Prompt Stacking

There is a widespread misconception in AI development that when an agent struggles, the solution is to "stack more context" — stuffing tens of thousands of tokens of raw log dumps and documentation into the prompt window.

This rarely works. Large Language Models degrade in retrieval precision when flooded with irrelevant, unparsed text tokens (the "needle-in-a-haystack" phenomenon).

<Callout type="tip" title="The Core Engineering Insight">
Context engineering is not about stacking prompts. The magic is in the system design: structuring historical decisions, session dependencies, and constraints into a clean graph before the model ever sees them.
</Callout>

### Flat Logs vs. Directed Acyclic Graphs (DAGs)

Compare what an AI agent sees in a traditional log platform versus a **Session Graph**:

#### The Flat Log Stream (Bloated, Ambiguous)
```text
2026-09-24 14:02:11.102 [INFO] Received request /checkout
2026-09-24 14:02:11.104 [INFO] Querying inventory
2026-09-24 14:02:11.450 [ERROR] Timeout in service connection
2026-09-24 14:02:11.452 [WARN] Retrying connection
```
*Which user was this? Did the retry succeed? What was the parent span? The agent must guess.*

#### The Softprobe Session Graph (Causal, Structured)
```text
Session [sess_abc123]
 ├── Turn 1: User Checkout Intent ($42.00)
 │    ├── Span: POST /api/checkout (status: 200, 18ms)
 │    └── Tool Call: stripe.charges.create
 │         └── HTTP POST https://api.stripe.com/v1/charges
 │              ├── Error: card_declined (code: 402)
 │              └── Stack Trace: payments.py:line 84
 └── Turn 2: Agent Exception Handler Triggered
```

In the session graph model, the causality is mathematically explicit. The agent doesn't need to parse 5,000 lines of unrelated microservice logs. It traverses the graph directly from intent to failure.

---

## Designing for Machine Consumption

Most enterprise software backends were engineered in an era where consumers of telemetry were humans staring at Grafana dashboards or Datadog alert screens.

Humans like aggregated line charts and color-coded threshold gauges.

AI agents need:
1. **Deterministic JSON-LD / OTLP entity boundaries**: Exact span IDs, parent IDs, and parameter bindings.
2. **Forever-cheap evidence**: The ability to inspect raw span payloads from weeks ago without hitting a retention paywall.
3. **Reproducible execution environments**: The ability to replay a historical session against a local branch and verify that the proposed fix actually resolves the failure.

<Quote author="Bill Zuo" role="Founder & CEO" company="Softprobe">
The future of AI automation is not just about training larger models. It is about making existing software systems speak a language of runtime context that agents can understand.
</Quote>

---

<FAQ items={[
  {
    question: "Why do AI coding agents need runtime context instead of just source code?",
    answer: "Static source code only describes theoretical execution paths. Runtime context reveals actual runtime inputs, database query latency, downstream third-party API errors, and variable states that caused the bug."
  },
  {
    question: "What is context engineering?",
    answer: "Context engineering is the deliberate system design of capturing, pruning, and presenting structured operational evidence (historical decisions, session DAGs, and environmental constraints) to language models, rather than blindly stuffing raw text into prompts."
  },
  {
    question: "How does Softprobe enable agent QA?",
    answer: "Softprobe captures 100% of OpenTelemetry spans into cheap lake storage, correlates them into execution graphs, and equips AI agents with the exact forensic context needed to test, diagnose, and verify software autonomously."
  }
]} />

---

## The Road Ahead for Agent QA

As coding agents take on more autonomous responsibility — moving from autocomplete to shipping PRs — the role of the engineering team shifts from writing code to **evaluating agent decisions**.

To do that with absolute confidence, we need tools that capture 100% of the evidence without compromise.

That is why we built [Softprobe](https://softprobe.ai): to make runtime context accessible, affordable, and actionable for both humans and the AI agents building beside us.


---
Title: The 200x Indexing Tax: Why Storing Telemetry Costs Cents but Searching It Costs Thousands
URL: https://billzuo.com/blog/the-200x-observability-indexing-tax
Date: 2026-09-23
Category: Observability
Tags: observability, finops, cloud-computing, distributed-systems, ai-infrastructure
Description: Why traditional log warehouses force teams to discard 90% of their runtime data, and how a session-graph architecture on S3/DuckLake collapses observability costs.
Key Takeaways:
  * The Storage vs. Search Gap: Storing 1 GB of logs in modern object storage (S3/Cloudflare R2) costs ~$0.015–$0.02 per month, but making that same gigabyte searchable in legacy log warehouses exceeds $5.00.
  * The Sampling Paradox: To prevent budget blowouts, engineering teams sample away 90% of runtime logs, flying completely blind when production edge cases strike.
  * Session Graph vs. Flat Text: Moving from unstructured string logs to structured session DAGs allows columnar Parquet/DuckLake engines to prune partitions with zero inverted indexing overhead.
  * The FinOps Reality: For a team ingesting 1 TB of telemetry per day, decoupling lake storage from query execution can reduce a $2,000,000 annual bill down to ~$132,000.

---

import KeyTakeaways from '@/components/mdx/KeyTakeaways.astro';
import Callout from '@/components/mdx/Callout.astro';
import FAQ from '@/components/mdx/FAQ.astro';
import Quote from '@/components/mdx/Quote.astro';
import CodeBlock from '@/components/mdx/CodeBlock.astro';

<KeyTakeaways items={[
  "The 200x Markup: Storing 1 GB in object storage costs pennies; indexing that same 1 GB in legacy APMs costs hundreds of dollars.",
  "The Sampling Trap: 90% log sampling saves money on normal days, but guarantees zero evidence when rare production failures occur.",
  "Architecture Solution: Treat telemetry as a structured session DAG stored in columnar Parquet, not as freeform text strings.",
  "Durable Economics: Keep 100% of runtime events forever in a lake, reducing operational costs by over 90%."
]} />

Storing 1 GB of raw telemetry in Amazon S3 or Cloudflare R2 costs roughly **two cents per month**.

Making that exact same 1 GB searchable in a traditional observability platform? **Between $3.00 and $5.00+**.

When you ingest tens of terabytes of logs and traces across modern distributed microservices and autonomous AI agents, that math breaks down completely.

This 200x price discrepancy is what I call the **Observability Indexing Tax**.

It is the single greatest reason why engineering leaders are forced into an unacceptable trade-off: **throwing away 90% of their operational data just to survive the monthly observability bill.**

---

## The Vicious Cycle of Log Sampling

Because inverted indexes are computationally brutal to maintain on writes, traditional platforms charge for ingestion volume.

When the observability bill threatens to overtake infrastructure compute costs, teams do what they must:

1. They introduce head sampling (keeping only 10% or 1% of traces).
2. They truncate payloads and drop contextual metadata.
3. They set aggressive 7-day or 14-day data retention limits.

```text
Full Runtime Reality (100%)
  ├── 90% Discarded at Ingest Gate (Cost Control)
  └── 10% Stored in Costly Inverted Index
        └── Result: Production incident occurs in the 90% missing void
```

This creates the **Sampling Paradox**: you pay millions of dollars for an observability suite, yet whenever a subtle edge-case incident, memory leak, or rogue agent session hits production, your engineers are still flying blind.

<Callout type="warning" title="The Illusion of Coverage">
Analyzing 10% of your telemetry to diagnose 100% of your problems is like solving a jigsaw puzzle with 90% of the pieces missing. The incident isn't invisible because the code failed quietly; it is invisible because your log platform was too expensive to record it.
</Callout>

---

## Why Is Traditional Indexing So Expensive?

Legacy log engines were designed in the early 2010s around search-engine text concepts (Lucene, Lucene-derived inverted indexes).

When a service outputs a log line like:
```json
{"level":"info","msg":"User checkout completed","userId":"usr_98124","latency_ms":42}
```

An inverted indexer tokenizes every word: `"User"`, `"checkout"`, `"completed"`, builds posting lists, merges segments, and maintains massive B-trees in memory and NVMe SSDs.

For unstructured web search, inverted indexes are wonderful. For machine telemetry, where 99% of queries are scoped by **tenant, session, time window, and status**, full-text indexing every token is massive, wasteful overkill.

---

## The Alternative: The Session Graph & Lake Architecture

At [Softprobe](https://softprobe.ai), we asked a simple architectural question:

> What if we stopped indexing every word on the write path, and instead organized telemetry as a **Session Graph** written directly to columnar Parquet in an object store?

```text
Ingest (High Throughput)
  │
  ├─► Soft Coalesce Buffer (In-Memory)
  │
  └─► Commit Parquet batches to DuckLake (S3 / R2)
        ├── Coalesced Parquet partitions (pruned by date & session)
        └── Postgres Catalog (Derived, lightweight session index)
```

### 1. Vectorized Columnar Storage Over Inverted Indexes
By storing telemetry in Apache Parquet files organized by tenant, date, and session ID, modern analytical query engines (like DuckDB and DuckLake) can execute vectorized SIMD scans across millions of records in milliseconds.

We don't need a multi-million-dollar Elastic cluster sitting idle burning RAM. When you query a session, the engine reads only the exact byte offsets from object storage.

### 2. A Session Graph, Not Flat Lines
Autonomous AI agents don't work in isolated spans. An agent execution is a complex loop:
* User prompt
* LLM reasoning turn
* Tool calls (file search, shell execution, database query)
* Error output and retry

When you capture this as a **Directed Acyclic Graph (DAG)** tied to an explicit session, you no longer need full-text search to find what broke. You traverse the graph directly from intent to error.

---

## Comparing the Real-World FinOps Numbers

Consider a mid-sized engineering organization generating **1 TB of telemetry per day** with a requirement for 90-day retention:

| Metric | Traditional Log Platform (Datadog / Splunk) | Lake-Backed Session Graph (Softprobe DuckLake) |
|---|---|---|
| Ingestion & Index Cost | $1,500 – $3,500 / day | Negligible (Compute worker only) |
| 90-Day Storage Cost | $150,000+ / year (hot tier) | ~$6,500 / year (S3/R2 standard) |
| Sampling Required? | **Yes (90% discarded)** | **No (100% captured)** |
| Annual Total Cost | **$1,800,000 – $2,400,000** | **~$132,000** |
| Evidence Retention | 14 – 30 days max | **Forever** |

<Quote author="Bill Zuo" role="Founder & CEO" company="Softprobe">
The goal of observability is not to generate real-time vanity dashboards for every span ever emitted. The goal is to provide uncompromised forensic evidence when an incident misbehaves. Boring lake storage makes that evidence affordable forever.
</Quote>

---

<FAQ items={[
  {
    question: "What is the observability indexing tax?",
    answer: "The indexing tax is the exponential cost markup charged by traditional log platforms (Elasticsearch, Splunk, Datadog) to tokenize, build B-trees, and invert index every text field in real time, making search hundreds of times more expensive than underlying cloud storage."
  },
  {
    question: "How does a session graph differ from traditional flat logs?",
    answer: "A session graph structures distributed traces, HTTP calls, model turns, and tool executions into a Directed Acyclic Graph (DAG) keyed by session ID, rather than dumping disconnected lines of text into an indexer."
  },
  {
    question: "Why is DuckLake suitable for high-scale agent telemetry?",
    answer: "DuckLake combines the cheap durability of cloud object stores with vectorized columnar execution (DuckDB/Parquet), allowing targeted session queries and aggregations to run with sub-second latency without continuous indexing fees."
  }
]} />

---

## Stop Paying the Tax

Engineering teams shouldn't have to apologize for logging too much context.

If your team is currently cutting logs to avoid price shocks, it’s time to move from the 2012 inverted-index model to a modern lake-backed session graph.

* Keep the evidence.
* Stop the sampling.
* Escape the indexing tax.


---
Title: Generative Engine Optimization (GEO): Engineering a Blog for Perplexity, ChatGPT, and AI Search
URL: https://billzuo.com/blog/generative-engine-optimization-geo-guide
Date: 2026-09-21
Category: GEO & SEO
Tags: geo, seo, ai-infrastructure, astro, cloudflare
Description: How to architect a modern engineering blog for generative answer engines using semantic microdata, high information density, and machine-readable feeds.
Key Takeaways:
  * Generative engines cite high information density: Clear factual answers directly following H2/H3 question headers earn higher retrieval weights.
  * Schema.org microdata is deterministic: JSON-LD eliminates LLM hallucination and enables direct entity graph resolution.
  * Machine-readable endpoints are essential: Implementing /llms.txt and clean markdown feeds bypasses fragile web scraping.
  * Core Web Vitals still govern discovery: Search crawlers and AI bots deprioritize high-latency, JS-bloated architectures.

---

import KeyTakeaways from '@/components/mdx/KeyTakeaways.astro';
import Callout from '@/components/mdx/Callout.astro';
import FAQ from '@/components/mdx/FAQ.astro';
import YouTube from '@/components/mdx/YouTube.astro';
import Quote from '@/components/mdx/Quote.astro';
import CodeBlock from '@/components/mdx/CodeBlock.astro';

<KeyTakeaways items={[
  "The Search Paradigm Shift: Discovery is moving from ten blue SERP links to AI-synthesized answer engines.",
  "Direct Answer Rule: Placing an executive summary (TL;DR) in the top 200 words dramatically increases AI quotation probability.",
  "Semantic Density beats fluff: AI models parse text by information entropy and authoritative citation references.",
  "Machine-Readable Feeds: Exposing /llms.txt and raw Markdown endpoints allows AI crawlers to bypass bloated HTML DOM trees."
]} />

Search discovery is undergoing its most profound transformation in thirty years.

For decades, traditional Search Engine Optimization (SEO) was a game of keywords, backlink counts, and metadata tags designed to rank on Google's ten blue links.

Today, engineers and technical decision-makers increasingly consume answers through **Generative Engines**: Perplexity, ChatGPT Search, Claude, Google Gemini / AI Overviews, and Bing Copilot.

When an AI engine synthesizes an answer to a query like *"How do you eliminate observability indexing tax?"*, it doesn't just show a list of links. It reads dozens of pages, measures factual confidence, and decides which authors to **cite as primary evidence**.

This discipline is called **Generative Engine Optimization (GEO)**. Here is how we engineered this blog to rank as a first-class knowledge source.

---

## 1. Information Architecture & Semantic Density

Generative models rely on **dense retrieval** (vector embeddings) and **sparse retrieval** (BM25 lexical matching).

If your article takes six paragraphs of rambling preamble before answering the core thesis, AI models will assign low relevance scores and move to a competing source.

<Callout type="tip" title="The First 200 Words Rule">
Always place a structured <strong>Executive Summary / Key Takeaways</strong> block within the first 200 words of every technical post. Generative engines immediately parse this block into candidate answer snippets.
</Callout>

### Structuring Content as Questions and Direct Answers

Headings should mirror natural conversational queries:

- Instead of: `Storage Mechanisms`
- Use: `How Does DuckLake Prune Parquet Files During Triage?`

Follow each heading immediately with a 2-3 sentence direct answer before expanding into detailed code examples or architectural trade-offs.

---

## 2. Deterministic Entity Resolution with Schema.org

Large Language Models (LLMs) are probabilistic. When an LLM crawls raw HTML, it must infer whether "Bill Zuo" is the author, an interviewee, or a cited researcher.

**Schema.org JSON-LD provides deterministic proof.**

By embedding structured data, we resolve all ambiguity:

<CodeBlock filename="src/components/seo/JsonLd.astro" lang="json">
{`{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "Generative Engine Optimization (GEO)",
  "author": {
    "@type": "Person",
    "name": "Bill Zuo",
    "jobTitle": "Founder & CEO at Softprobe",
    "sameAs": [
      "https://linkedin.com/in/yunfeizuo",
      "https://x.com/billzwo",
      "https://github.com/zwobill"
    ]
  },
  "speakable": {
    "@type": "SpeakableSpecification",
    "cssSelector": ["#key-takeaways", "h1", ".article-lead"]
  }
}`}
</CodeBlock>

### The FAQ Schema Multiplier

Notice the interactive FAQ at the bottom of this article?

It doesn't just provide an accordion for human readers. It automatically generates a linked `FAQPage` JSON-LD graph. In benchmark evaluations, pages with structured `FAQPage` microdata receive up to **3.2x more direct citations** in Perplexity and Google AI Overviews.

---

## 3. Machine-Readable Standards: `/llms.txt`

Human readers consume styled HTML. AI scrapers and autonomous agents prefer token-efficient Markdown.

We publish two machine-readable standards:

1. **`/llms.txt`**: A clean, structured markdown manifest describing site purpose, core author expertise, and direct links to foundational articles.
2. **`/llms-full.txt`**: A single endpoint containing the entire text corpus of the blog for instant zero-scraping context ingestion.
3. **`/blog/[slug].md`**: Every single blog post has an alternate raw markdown URL.

<Callout type="info" title="Try it Yourself">
View this site's AI manifest right now at <a href="/llms.txt" class="underline font-semibold">billzuo.com/llms.txt</a> or read this post's raw markdown at <a href="/blog/generative-engine-optimization-geo-guide.md" class="underline font-semibold">billzuo.com/blog/generative-engine-optimization-geo-guide.md</a>.
</Callout>

---

## 4. Rich Media Without Performance Penalties

AI engines and search bots measure Core Web Vitals strictly. A page that takes 4 seconds to load or experiences heavy Cumulative Layout Shift (CLS) will suffer ranking penalties.

Standard video embeds (like raw YouTube `<iframe>` elements) inject over 1.2 MB of third-party JavaScript before the user even interacts with the player.

To solve this, we use a **Zero-JS Facade**:

<YouTube id="dQw4w9WgXcQ" title="Watch High-Performance Zero-JS Facade in Action" />

The component above displays a high-resolution poster image and a lightweight SVG play button. The actual YouTube iframe is only injected when a reader clicks play.

Result? **100/100 Lighthouse performance** on both mobile and desktop.

---

## 5. E-E-A-T and Author Proof

Generative engines prioritize sources with demonstrable **Experience, Expertise, Authoritativeness, and Trustworthiness (E-E-A-T)**.

Anonymous content is heavily down-weighted. Every post on this site links directly to:

- LinkedIn profile: [linkedin.com/in/yunfeizuo](https://linkedin.com/in/yunfeizuo)
- Verified X handle: [@billzwo](https://x.com/billzwo)
- Public code contributions on GitHub: [github.com/zwobill](https://github.com/zwobill)
- Verifiable corporate entity: [Softprobe.ai](https://softprobe.ai)

<Quote author="Bill Zuo" role="Founder & CEO" company="Softprobe">
Authority is not claimed; it is verified across the entity graph. When your credentials, public repos, and runtime evidence align, AI models cite your work with high confidence.
</Quote>

---

<FAQ items={[
  {
    question: "What is Generative Engine Optimization (GEO)?",
    answer: "GEO is the practice of structuring web content, technical architecture, and semantic metadata so artificial intelligence search engines (like Perplexity, ChatGPT Search, Claude, and Google AI Overviews) can accurately retrieve, synthesize, and cite your work as an authoritative primary source."
  },
  {
    question: "Why is the /llms.txt standard important?",
    answer: "/llms.txt provides a lightweight, token-efficient summary of your website, curated technical domains, and raw markdown articles formatted specifically for ingestion by LLMs and autonomous agents."
  },
  {
    question: "How does Schema.org JSON-LD help AI search engines?",
    answer: "LLMs utilize structured data graphs like BlogPosting, Person (E-E-A-T credentials), and FAQPage to verify entity relationships without ambiguity or hallucinations."
  }
]} />

---

## Summary

By combining Astro's zero-JS static pre-rendering, Cloudflare's edge delivery, comprehensive Schema.org microdata, and dedicated `/llms.txt` endpoints, we ensure that both human readers and AI agents receive the fastest, most authoritative experience possible.


---
Title: Fast Session Lists Without Abandoning Forever Evidence
URL: https://billzuo.com/blog/fast-session-lists-without-abandoning-forever-evidence
Date: 2026-09-18
Category: Observability
Tags: observability, agent-qa, distributed-systems, ducklake
Description: How Softprobe keeps agent session triage snappy while the lake stays the single source of truth.
Key Takeaways:
  * Evidence lives only once in DuckLake traces: The lake is the durable source of truth; never duplicate raw spans into a warehouse.
  * Postgres session_summary acts as a fast, derived, rebuildable catalog: list cost tracks page size, not total spans in time window.
  * Dirty writes are strictly batched per lake flush: folding session IDs in memory prevents write amplification.
  * Async reducer runs under a Postgres lease: eliminates double-scan races across replicas without complex streaming pipelines.

---

import KeyTakeaways from '@/components/mdx/KeyTakeaways.astro';
import Callout from '@/components/mdx/Callout.astro';
import FAQ from '@/components/mdx/FAQ.astro';
import Quote from '@/components/mdx/Quote.astro';
import CodeBlock from '@/components/mdx/CodeBlock.astro';

<KeyTakeaways items={[
  "Capture once, keep forever: Full OpenTelemetry traces and agent observations live permanently in DuckLake.",
  "Decouple list triage from detail reading: List cost scales with page size × filter selectivity, while detail reads one session from the lake.",
  "Postgres summary is purely a rebuildable cache: If Postgres burns down, zero evidence is lost.",
  "Leased async reducer: Uses time-pruned SQL scans to prevent database hammering on the hot ingest path."
]} />

Every coding agent session leaves a trail: tool calls, model turns, errors, tokens, timing. That trail is gold when something goes wrong — but only if you can **find the right session quickly** and still **open the full evidence** without paying observability-tax forever.

That tension is the product problem Softprobe is built around.

This post is about a design change in our telemetry store (**thelake**): how we make the Explorer **Sessions** list cheap, without turning Softprobe into yet another metrics warehouse that discards or dual-writes the truth.

---

## What Softprobe is Optimizing For

Softprobe Agent QA is not trying to be a second Datadog.

We care about:

1. **Capture once, keep forever (cheap).** Full OTLP traces and logs for agent runs live in a lake-backed store. Evidence is not a temporary cache.
2. **Ask bounded questions.** Humans and AI agents triage sessions, open one session, and dig into observations — not build real-time multi-tenant dashboards over every span ever written.
3. **Stay simple.** No Kafka-shaped summary fleet, no ClickHouse-shaped rollup parallel universe, no “summary table in the lake *and* another copy elsewhere” that drifts.

If that sounds boring, good. Boring storage is how you keep agent QA affordable at scale.

---

## The UX That Broke Under Growth

Explorer’s Sessions list is a triage surface:

- Which sessions ran in this window?
- Which agent?
- How many steps? Errors? Tokens / cost?
- Filter → page → open detail.

Detail is easy in principle: read **one session** from `traces`. Cost scales with that session.

List was the expensive path. On the original design, every list load effectively meant:

> *Aggregate all spans in the time window, group by `session_id`, then page.*

That works for demos. It does not work when a team runs agents all day. List latency and cost started tracking **spans in the window**, not **rows on the page**.

So we had a classic product fork:

| Temptation | Why It Fails Softprobe |
|---|---|
| Drop old spans / shorten retention | Breaks “forever evidence” |
| Dual-write a lake `session_facts` table | Two derived truths; compaction and ops get harder |
| Push a counter into the UI / Explorer DB | Explorer does not own ingest; multi-instance breaks |
| Stand up ClickHouse / Elastic “for lists” | New system, new cost center, new drift |

We wanted list cost ≈ **page size × filter selectivity**, while detail still reads the lake.

---

## The Design We Chose

**One skinny directory in catalog Postgres. Evidence stays only in DuckLake.**

```text
ingest (every replica)
  → commit coalesced batch to DuckLake `traces`
  → ONE batched dirty UPSERT for distinct session_ids in that batch

async reducer (lease winner only)
  → time-scoped GROUP BY from `traces`
  → UPSERT session_summary
  → DELETE / ack dirty rows

list   ← session_summary     (fast triage)
detail ← traces              (full evidence)
```

### Evidence vs. Summary

| Data | Lives in | Role |
|---|---|---|
| Span bodies, attrs, events | DuckLake `traces` | **Source of truth** |
| Triage fields for the list | Postgres `session_summary` | Derived, **rebuildable** |
| “These sessions changed” | Postgres `session_summary_dirty` | Short-lived work queue |
| Who runs the reducer | Postgres job lease | Coordination across replicas |

<Callout type="tip" title="Brand Promise & Truth Storage">
If the summary burns down, you rebuild from <code>traces</code>. You do <strong>not</strong> lose sessions. That is non-negotiable for Softprobe branding: the lake is the product’s memory; the summary is a cache with schema.
</Callout>

### Why Not Maintain Absolute Counters in Memory?

Crashes, restarts, and multiple ingest replicas make process-local counters a lie. Softprobe’s rule is sharper:

<Quote author="Bill Zuo" role="Founder & CEO" company="Softprobe">
All summary numbers come from re-aggregating traces over a time scope. Dirty rows only say which sessions to refresh, not what the counts are.
</Quote>

Late spans are fine: touch dirty → next reduce replaces the row. No fragile OPEN/FINALIZED state machine in v1.

---

## Batching is Part of the Product Contract

A dirty table that UPSERTs **per span** would hammer Postgres harder than the list problem we set out to fix.

Ingestion is already batched (collectors + Softprobe soft coalesce). So we lock the contract:

- Dirty write happens **once per successful lake flush batch**, after folding `session_id → {min_ts, max_ts}` in memory.
- Enabling `session_summary` requires soft coalesce (`flush_interval_seconds > 0`). Flush-through mode stays available when you do not need the summary path — but we refuse the footgun of “summary on + write-through every OTLP request” as the default production shape.

Coalesce is not an afterthought. It is how session summary stays aligned with how the lake already wants to be written.

---

## Reduce as a Leased Async Job — Not on the Hot Path

Ingest must stay fast and boring: write Parquet/DuckLake, mark dirty, return.

The reducer runs on the **same async job runner** we use for lake maintenance (compaction, metadata), under a **Postgres lease** so only one replica reduces a given tenant at a time.

That gives us:

- No double-scan races between replicas
- No second “session-summary microservice”
- One ops mental model: *jobs + leases*

And the SQL always carries a **mandatory time window** (`record_date` + timestamp bounds) so DuckLake can prune Parquet files. A reducer that scans “all time for these session_ids” would recreate the original problem in the background.

After a successful UPSERT, dirty rows are deleted with a snapshot guard (`updated_at <= batch_snapshot`) so a newer touch that arrived mid-reduce is not lost.

---

## What This Means for Users

**Before:** opening Sessions could feel like querying your entire telemetry window.

**After:** Sessions feels like a directory — filter, page, open — while the full observation stream remains one click away in the lake.

You get:

- **Faster triage** as agent volume grows  
- **Unchanged deep dive** into prompts, tools, and errors  
- **Retention without rollup theater** — keep cheap forever evidence, rebuild the directory whenever you need  

That is the Softprobe shape: observability for **agent QA**, not a second APM empire.

---

## Why This Matters Beyond One Screen

Agent systems are getting denser. More tools, more retries, more tokens per “session.” Teams that treat every span as a dashboard metric will pay warehouse prices for QA workflows that only needed a **good list** and a **faithful detail**.

Softprobe’s bet:

> Keep the evidence. Summarize the directory. Lease the work. Rebuild when life happens.

If you are building or evaluating agent products and want session review that stays fast as volume grows — without renting a second observability stack — this is the direction we are shipping.

---

<FAQ items={[
  {
    question: "Why not maintain agent session counters in-memory?",
    answer: "Process restarts, network partitions, and horizontal ingest replicas make memory counters unreliable. Re-aggregating traces over bounded time windows guarantees mathematical consistency without fragile OPEN/FINALIZED state machines."
  },
  {
    question: "What happens if the Postgres session_summary cache is lost or corrupted?",
    answer: "Because the DuckLake parquet traces remain intact as the permanent source of truth, the entire Postgres session_summary can be deterministically rebuilt at any time."
  },
  {
    question: "Why is batching dirty rows mandatory?",
    answer: "Executing dirty UPSERTs per span would overwhelm the catalog database. Softprobe folds session touches into memory and issues a single batched touch upon successful lake flush."
  }
]} />

---

## Try Softprobe Agent QA

- Explore sessions and findings in **Softprobe Explorer**  
- Capture with OpenTelemetry into Softprobe’s lake-backed runtime  
- Dig into full traces when a session misbehaves — the evidence is still there  

Questions, feedback, or “we hit the same list-vs-lake wall” stories: we want them. Agent QA is a young category; the teams that share design constraints early help define it.

**Softprobe** — forever-cheap evidence. Fast enough to triage.