Published 2026-03-03.
Time to read: 2 minutes.

This page is part of the golang collection.

In a scenario where LLMs are generating the code, one quickly realizes how limited they are in terms of debugging capability.

I was having trouble getting LLMs to develop stable software for a distributed application. I concieved of an active test harness framework for quick, comprehensive, and accurate diagnosis. The intention was to make it would easy for an LLM to insert additional probes anywhere in a complex, multi-step process of a live end-to-end integration test.

My minions and I developed a Go framework for self-diagnostics that can handle error and as well as actively probing for system status and efficiently finding root causes. The ability to direct an LLM to add active probes between each debug session makes debugging a wonderful thing to behold.

The LLM user just says "Fix the problem", and the LLM observes the output of the probing test harness as it progressively diagnoses issue after issue, usually without any input from the LLM user.

This programming technique works for human-powered workflow, but it works especially well with autonomous debugging agents because of how it can scale. Nine LLMs can sometimes produce a digital baby in one month!

This type of work is best performed by using a combination of Claude Sonnet 4.6 and the Gemini Coding Assistant.

Claude

Key Learnings from the Diagnostic Journey

1. Act-Verify-Diagnose Creates Self-Documenting Failures

   The output tells a story:
   ✓ Pointer verified in index for test.txt: oid=7a67f… size=27
   ✓ Blob cached for test.txt at /tmp/…/7a/67/7a67f… (27 bytes)
   ✓ Original file preserved in working tree for test.txt (27 bytes)
   [git push happens]
   ✗ S3 object missing for test.txt (key=lfs/objects/7a/67/…) 404 NotFound

   Immediately we know:

   • Clean filter worked perfectly (all 3 verifications passed)
   • Push completed (we got to the S3 check)
   • Upload never happened (404, not a permission/auth error)

   Compare this to a traditional test that just says “test failed” - we’d have to add debug logging, re-run, guess at the problem.

2. Iterative Probe Refinement is Powerful

   We didn’t know the answer upfront. We asked questions:

   1. “Did git add work?” → Add pointer/cache/file verifications → ✓ Yes
   2. “Did git push work?” → Capture output → ✓ Yes, but no LFS activity
   3. “Is LFS configured?” → Dump git config → ✓ Yes, looks correct
   4. “Is pre-push hook installed?” → Check .git/hooks → ✗ Missing!

   Each probe answered one question and led to the next. This is much faster than trying to solve the whole problem at once.

3. Visibility = Debuggability

   The detailed logging (with ✓/✗ symbols, actual values, file paths) meant:

   • User can see progress in real-time
   • Failure point is immediately obvious
   • No need to re-run with verbose flags
   • Test output IS the diagnostic report

4. State Accumulator Enables Progressive Verification

    TestFile tracking across steps:
      type TestFile struct {
      OriginalSize int64 // Known after git add
      OriginalHash string // Known after git add
      PointerHash string // Known after git add
      S3Key string // Known after git push
      CachedPath string // Known after git add
    }
   

   
   Each step verifies it correctly built on previous state. If Step 3 fails, we know Steps 1-2 succeeded.

5. Fail-Fast with Context > Fail-Late

   We caught the problem immediately after git push, not later when:

   • Trying to clone
   • Running git lfs pull
   • Checking out files

   We knew exactly which operation failed and had full context (git config, git output, file state).

6. Custom Probes On-Demand

   When standard verification wasn’t enough, we added:

    // Diagnostic: Capture git push output to see if transfer agent is invoked
    c.T.Logf("Executing git push (should invoke transfer agent)...")
    pushResult := c.Runner.MustRun(c.Ctx, c.RepoDir, c.Env, "git", "push", "-u", "origin", "main")
    c.T.Logf("Git push stdout:\n%s", pushResult.Stdout)
   

   
   This answered: “What did git push actually do?” without changing the test structure.

What to Document

In diagnostics/integration/doc.go:

// Act-Verify-Diagnose: The Core Philosophy
//
// Integration tests should follow a pattern of incremental verification:
// 1. ACT: Perform an operation (git add, git push, etc.)
// 2. VERIFY: Check the operation produced expected state changes
// 3. DIAGNOSE: If verification fails, provide detailed context about what went wrong
//
// Example from S3 integration testing:
//
// // ACT: Add file (triggers clean filter)
// c.Runner.MustRun(c.Ctx, c.RepoDir, c.Env, "git", "add", filename)
//
// // VERIFY: Check clean filter results
// c.verifyPointerInIndex(filename) // Did pointer get created?
// c.verifyBlobCached(filename) // Did blob get cached?
// c.verifyOriginalFilePreserved(filename) // Is working tree unchanged?
//
// // DIAGNOSE: If any verification fails, the failure message includes:
// // - Which specific verification failed
// // - Expected vs actual state
// // - Relevant file paths, hashes, sizes
// // - Context from previous successful verifications
//
// Benefits:
// - Self-documenting: Test output shows exactly what worked and what failed
// - Fast debugging: No need to re-run with verbose flags
// - Guided investigation: Each failure suggests what to check next
// - Iterative probes: Add diagnostic logging when standard verification isn't
enough

In core/test/integration/s3/doc.go:

// State Accumulator Pattern for S3 Integration Testing
//
// The S3TestContext implements a State Accumulator pattern where each test step
// builds on previous state and verifies correctness incrementally:
//
// Step 1: Setup repo
// Verifies: Repo directory exists, git initialized
//
// Step 2: Start LFS server
// Verifies: Server listening, health check passes
//
// Step 3: Install filters
// Verifies: Git config has filter.lfs.clean, filter.lfs.smudge
//
// Step 6: Push workflow (git add + git push)
// ACT: git add test.txt
// VERIFY: - Pointer in index (verifyPointerInIndex)
// - Blob cached in .git/lfs/objects (verifyBlobCached)
// - Working tree unchanged (verifyOriginalFilePreserved)
// ACT: git commit + git push
// VERIFY: - S3 objects uploaded (verifyS3ObjectsUploaded)
// - Object sizes match (verifyS3ObjectSizes)
// - Transfer agent succeeded (verifyTransferAgentSuccess)
//
// The TestFile structure accumulates state across verifications:
//
// type TestFile struct {
// OriginalSize int64 // Set by verifyPointerInIndex
// OriginalHash string // Set by verifyPointerInIndex
// PointerHash string // Set by verifyPointerInIndex
// S3Key string // Set by verifyS3ObjectsUploaded
// CachedPath string // Set by verifyBlobCached
// }
//
// When a verification fails, the output shows:
// ✓ Which previous verifications passed (narrowing the failure point)
// ✗ Which verification failed (with detailed context)
// ✓ All accumulated state (hashes, paths, sizes) for diagnosis
//
// Adding Custom Diagnostic Probes:
//
// When standard verification isn't enough to diagnose a failure, add targeted
// diagnostic probes to answer specific questions:
//
// // Question: "Is Git LFS configured correctly?"
// lfsConfig := c.Runner.MustRun(c.Ctx, c.RepoDir, c.Env, "git", "config", "--list")
// c.T.Logf("Git LFS config:\n%s", lfsConfig.Stdout)
//
// // Question: "Did git push invoke the transfer agent?"
// pushResult := c.Runner.MustRun(c.Ctx, c.RepoDir, c.Env, "git", "push", ...)
// c.T.Logf("Git push output:\n%s", pushResult.Stdout)
//
// This iterative probe refinement quickly identifies root causes without
// modifying the core test structure.

Share to X, Hacker News, LinkedIn.