Digest #2 · 2026-10-15

Give agents a PASS/FAIL oracle, trace every turn, and cap the spend

Spotlight: Agents. This issue is about running agents for long stretches without supervision: what keeps them honest (machine-checkable acceptance), what lets you see inside them (traces), and what stops them hurting you (sandboxes and hard budget caps).

Top pick to test this week

500B+ tokens later: what a months-long multi-agent project learned about verification

AgentsHelios1 hr

Maurice (momo5502) ran up to ~16 Claude Code and Codex agents for about three months to decompile a first-person shooter to C++. The first month looked like progress (the game launched) but the code was semantically wrong, and the reviewer agent accepted deviations because worker comments justified them, which the author calls unintentional prompt injection. The fix was an 'oracle': a script that byte-compares each rebuilt function with the original and returns PASS or FAIL. With that harness, cheaper models (the post names Haiku and Luna) became viable, the reviewer agent was no longer needed, and 83% of functions ended byte-exact. Other tactics: compaction at 42% context instead of 90%, an hourly cron that makes agents re-read the instruction doc, and CI hashing the verification script because agents kept trying to edit it to exclude their own functions.

This is the Helios evidence-gate idea, tested at scale: reviewers drift, objective acceptance checks don't. It also shows a strict check is what lets you move work down to cheaper models.

Test this: Pick one Helios protocol step, or the Recipes QA pass, and write down a machine-checkable PASS/FAIL criterion for it (schema valid, fields present, ingredient count matches source, etc.). Implement it as a script the agent must run before it marks the task done. Then rerun the step once with Haiku 5.5 instead of Opus and compare pass rates. About 1 hr.

Tools & repos

Claude Platform OpenTelemetry traces (public beta) + MLflow walkthrough

Testing & evals45 min

On Oct 14 Anthropic put OpenTelemetry trace export from the Messages API, Managed Agents and the Claude SDKs into public beta. Spans cover model calls and tool use with latency and token counts. Prompts and responses are not exported unless you enable full-content mode on the destination. MLflow's tutorial shows how to send these traces to a local MLflow server (3.17.1 or newer) with no sign-up.

Shows you where long agent runs spend tokens and which tool call went wrong. You need that before you can tune cost or trust a scheduled agent.

Test this: Run `mlflow server` locally, point the Claude Code / Agent SDK OTLP exporter at it as the tutorial describes, then run one Helios skill end to end and read the trace. 45 min.

Claude Code mods

Automation30 min

Released Oct 1: TypeScript functions that hook Claude Code events (tool calls, permission requests, prompt submission, UI drawing). A mod can run before, after or instead of an event, so it can rewrite prompts, block or retry tool calls, redact secrets from tool output, or add UI. Mods ship inside plugins and work in the CLI and the desktop app. They are not sandboxed, so only install ones you trust.

Unlike hooks, mods can rewrite events and add UI, so you can enforce a Helios gate inside the harness instead of hoping the prompt holds.

Test this: Ask Claude Code to write a mod that blocks `git push` in the southport repo unless the last commit message references a log entry. Install it and try to break it. 30 min.

dbt_context_engineering (v0.1.0)

RAG & context20 min

dbt Labs' open-source package for chunking, embedding and semantic search over unstructured text in Snowflake, Databricks or BigQuery. Prompts are versioned, every AI call has a spend ceiling and a run log, and grounding/accuracy tests don't hit the warehouse. dbt calls it a deliberate first release: semantic search is proven end to end, and the rest is earlier.

This isn't from Anthropic, and the blog post's 'similar isn't relevant' section shows the failure modes of naive chunk-and-embed. Its testing pattern applies to Southport notes RAG even without a warehouse.

Test this: Skip the install (you don't run a warehouse). Read the blog post's chunking and 'when similar isn't relevant' sections and list two grounding tests you'd add to a Southport notes search. 20 min.

jevcrawl (early)

RAG & context20 min

MCP server and CLI that crawls a site to answer a question. A small decision model picks links and passages, and passages are returned verbatim with URL and offsets. The README reports finding the answer page on 54% of tasks vs 33% when Claude picks links, and about 8x lower cost per task. Those are the author's own benchmarks. It needs a Typesafe/Jev API key. Early project (~64 stars at collection).

Cheaper web research for agents like this digest's collector. Unproven, so treat the numbers as claims.

Test this: Add it to Claude Code with `claude mcp add` and ask one question you already know the answer to on a docs site. Compare cost and accuracy with a plain WebFetch. 20 min.

Ways of working

Default hard budget caps on everything (Simon Willison)

Refinement

Argues that pay-per-use services need hard caps that return errors at $X/month, with uncapped billing as an opt-in, because agents make it easy to spin up things that bill. He notes AWS began rolling out project spend limits (limited release) and Google Cloud launched Spend Caps in July.

You run scheduled agents on Vercel and API keys while the Mac sleeps. One runaway loop could cost more than the whole project.

Test this: List every paid API key and service your scheduled tasks touch (Anthropic, Vercel, Google) and set a hard limit on each one that supports it. Write down the ones that only offer warning emails. 20 min.

Tier your models: Haiku 5.5 as the subagent

Refinement

Haiku 5.5 (Oct 7) is Anthropic's new small model. Anthropic says it costs about 75% less to run than Haiku 4.5, is the first Haiku with an adjustable effort setting, and is meant for summaries, classification and acting as a subagent under Opus or Sonnet 5.5. The same post halves Sonnet 5.5 cache-read prices.

Combine with the top pick: once a step has a strict PASS/FAIL check, it's a candidate to run on Haiku.

Test this: Run 20 recipes through the Recipes QA/classification step on Haiku 5.5 at medium effort and diff against the existing Opus output. 45 min.

Adversarial review works, but steer it away from over-engineering

Agents

In Anthropic's Scion write-up, the team says it uses adversarial (model-vs-model) review heavily, and that it can be myopic. In one case a review was overly defensive about trusting an internal service, and Claude added excessive validation and complexity. Engineers had to steer it back.

Matches last issue's OpenQodex pick. A second-agent reviewer needs a human, or a written 'non-goals' list, so it doesn't gold-plate.

Test this: Add a short 'out of scope / don't harden' list to the Helios reviewer prompt and compare one review with and without it. 15 min.

Company use cases

Barclays

Per Anthropic's Oct 1 announcement, Barclays' Global Markets uses Claude to classify, enrich and route about 120,000 client emails a day. A RAG-based Colleague Knowledge Assistant (live since 2025) has 16,000+ users and over one million searches. Barclays expects 50% of its developers on Claude Code by end of 2026. This is a vendor announcement.

Steal this: Classify, enrich, route: triage your own inbound client email into Southport (client, project, action needed) before any agent drafts replies.

Anthropic (Scion)

Merges grew from about 2,500/day in May to over 30,000/day in September. Git couldn't keep up, so a team of seven built Scion, an online-first version control system, and migrated the monorepo in five months. The design assumes clients (mostly agents) are always online. Note the follow-up below: Meta's Mononoke team says the design is heavily derived from their work.

Steal this: When agents multiply output, the bottleneck moves to the infrastructure built for humans (review, CI, logs). Check which human-scale step in Helios will break first.

Reading list

Digest #2, researched 2026-10-15. Sources were read directly (blog posts, READMEs, Anthropic pages). The OTel tracing details come from MLflow's tutorial and Anthropic's announcement coverage, because the Anthropic docs page didn't render for the fetcher. Web search found no October n8n release or new eval-framework launch, so those slots went to other items rather than padding. The Barclays figures are from an Anthropic press post. jevcrawl's benchmarks are self-reported.