you're losing money on agents
AI is shifting from clever text assistants to full-time digital employees working in shifts. The real engineering problem isn't keeping them running — it's knowing what they actually did while you were asleep.
[01] the shift to shifts
We used to talk to AI.
You typed a prompt, watched tokens stream by, and hoped for a clean copy-paste of code or text. It was a simple, single-turn transaction.
Now, we’re hiring them.
AI is shifting from clever text assistants to full-time digital employees that can run for hours, days, or weeks across multiple sandboxes and context windows. They work in shifts.
Addy Osmani frames this transition cleanly:
"Imagine a software project staffed by engineers working in shifts, where each new engineer arrives with no memory of what happened on the previous shift."
Without an explicit infrastructure to preserve memory across those shifts, every handoff becomes a disaster.
But keeping them running is no longer the hardest part.
The real engineering problem is knowing what they actually did while you were asleep.
[02] the harness is the house
The model is just the engine. The harness is the chassis.
If you just drop a frontier LLM into a naked while true loop, it fails.
In early runs of Anthropic’s long-horizon coding agents, the model would hit "context anxiety." As the context window filled, the agent would panic, wrap up its work prematurely, and declare "task complete" on a broken, half-implemented codebase.
The fix wasn’t a bigger model. It was structural telemetry.
Anthropic solved this by decoupling the "brain" (the model) from the "hands" (the sandboxes and execution tools) and the "session" (the append-only event log).
The model reasoning engine, kept stateless and focused.
Ephemeral sandbox containers treated as cattle, not pets.
Append-only durable event log that survives sandbox reboots.
They made sandboxes cattle, not pets. If a sandbox container crashes, the stateless harness catches it as a tool error, pulls the event log from durable storage, and spins up a fresh sandbox without losing the session’s state.
When you decouple them, performance jumps: Anthropic’s p50 time-to-first-token dropped by 60%, and p95 plummeted by over 90%. The brain starts reasoning before the physical container even finishes booting.
The model itself might be amnesiac, but your filesystem isn’t.
[03] the silent failure of the final line
Traditional software is deterministic: same inputs, same outputs.
If a unit test passes, the code works.
But an agent can produce the correct final output through a completely broken, hallucinated, or inefficient process. Google’s Hugo Selbie calls this the "silent failure".
An agent tasked with running an inventory report pulls the exact right numbers, but references last year’s cached spreadsheet by mistake. The final output looks perfect. The CFO is happy. But the execution is a ticking time bomb.
This is why final outputs don’t tell you anything anymore. You have to evaluate the trajectory — the exact sequence of tool calls, reasoning steps, retries, and context states that led to the result.
When agents fail, they fail in weird, highly specific ways. Microsoft’s AgentRx team annotated 115 failed agent runs and mapped a concrete nine-category failure taxonomy, finding agents failing from:
- Plan Adherence Failure: Ignoring their own formulated steps midway through execution.
- Invention of New Information: Flat-out hallucinations injected into real tool parameters.
- Invalid Tool Invocations: Repeatedly firing schema-violating or unvalidated calls.
If you aren’t tracing every step, you’re flying blind.
[04] the coordination bottleneck
If one agent is good, are sixteen agents better?
Usually, no. They just step on each other’s toes.
When Nicholas Carlini at Anthropic tasked 16 parallel Claudes with writing a Rust-based C compiler from scratch, they hit the parallel wall.
When the agents tried to compile the giant monolithic Linux kernel, they got stuck.
Every agent would find the exact same bug, fix it in their own container, pull from upstream, and overwrite each other’s changes. Having 16 agents running didn’t help because each was stuck solving the same task.
They had to build a custom compiler oracle using GCC just to let the agents compare and isolate separate files to work on in parallel.
Cursor hit the same wall. Their first attempts at running multiple agents used simple file locks. The result? The agents became risk-averse, endlessly churning rather than committing.
Emit isolated, scoped tasks
Execute with zero big-picture cognitive overhead
Decide when an iteration is truly done
Left to themselves, agents are terrible coordinators. More agents usually just means higher coordination overhead.
[05] quality-adjusted cost and role matching
Let’s talk about the bill.
Running agents overnight is not cheap.
Carlini’s compiler experiment consumed 2 billion input tokens and 140 million output tokens, running up a bill of $20,000.
Addy Osmani warns: "Without budgets, circuit breakers, and a hard cap on tool spend, an agent can quietly burn through a week’s API budget in an afternoon."
This is why you can’t just throw your largest, most expensive model at every problem. You need role matching.
Cursor discovered in production that different models excel in different roles. A GPT model actually outperformed Opus for extended autonomous work because Opus had a tendency to take lazy shortcuts or stop early.
If you’re paying top-tier pricing for a high-reasoning model to do a simple regex find-and-replace, you are burning cash.
You need to measure quality-adjusted cost: matching the cheapest model that can reliably hit the success criteria for that specific sub-task.
[06] the observe-evaluate-optimize loop
Every major cloud player and open-source project is rushing to build this telemetry stack, but they’re looking at it from different angles.
GA’d evaluations that integrate their Agent SDK directly with DeepMind’s "adaptive rubrics." It uses a user simulator to play out multi-turn conversations and online monitors that grade live production traffic.
Attacking the loop with their Agent Optimizer and "intelligent trace sampling." It reads live production traces, converts them into offline test datasets, and automatically suggests prompt adjustments with side-by-side diffs.
Focuses heavily on structural developer tooling: git-backed "test ratchets" inside prompts to prevent coding agents from deleting failing tests to make them pass, and context resets to clear memory limits.
Langfuse, LangSmith, Phoenix (Arize), and OpenTelemetry standardizing the wires. OpenTelemetry has become the common trace language underneath, spanning custom code and frameworks in one unified view.
They all agree on the golden rule of agent engineering: never let an agent grade its own work. Models are inherently lenient on themselves.
Where they disagree is where the state belongs: Google wants cloud-managed Memory Banks and Sessions; Anthropic wants to decouple the session log entirely; open-source developers want self-hosted, lightweight trace containers.
[07] we are building forge
This is the exact wall I hit while building Orbit.
I knew Orbit was running, but I couldn’t tell why it decided to call a specific partner API, why it spent 50,000 tokens looping on a simple string formatting error, or whether a prompt tweak actually improved things across 100 historical runs.
So, I started building Forge.
An open-source telemetry and evaluation engine designed specifically to help developers understand what their AI coding agents did, what it cost, what worked, and what to improve.
It’s not another agent framework. It doesn’t write code for you.
It sits at the protocol layer. It captures the trajectory, normalizes your agent traces, and isolates the exact "critical failure step" where a long-running execution became unrecoverable. It calculates your quality-adjusted token cost per run and lets you run deterministic unit tests alongside LLM-as-a-judge rubrics.
Because if you can’t measure it, you can’t optimize it.
[08] the next unit of software
We are entering a world where agents, not classes or APIs, are the new unit of software development.
You won’t write code; you will write specs that survive contact with autonomous executors.
But when your codebase is written, reviewed, and debugged by independent, parallel brains, your standard DevOps stack of log files and exception catchers breaks down.
The next multi-billion-dollar tooling layer isn’t going to be a "smarter" agent.
It’s going to be the infrastructure that lets us understand, evaluate, and guide the agents we already have.
[09] the dashboard, not the gas pedal
We don’t need a heavier foot on the gas.
We need a better dashboard.
The dashboard, not the gas pedal.
Hidden Layer is a reader-supported publication. To receive new posts and support my work, follow along and check back often.