Most agent frameworks solve the happy path: define agents, wire tools, ship. The hard problem isn't getting agents to run — it's knowing when they're failing in production. Your MMLU score is great. Your SWE-Bench rank is impressive. And your users are still getting hallucinated outputs at 2am while nobody notices.

This is the gap I've been living in — at Atomicwork, at SuperAGI, and now with Synapse. The thesis is simple: the runtime layer is the right place to solve agent reliability, because the runtime is the only layer that sees everything. Evaluation and observability aren't features you bolt on. They're primitives you build into the execution engine itself.

Offline Evals Are Necessary but Not Sufficient

The standard playbook for agent quality goes something like: curate a golden dataset, run your agent against it, score the outputs, write a report. Repeat weekly. The problem, as Hamel Husain puts it bluntly, is that the most important difference between CI evals and production evals is the data. Your golden dataset is a snapshot. Your production traffic is a living thing that shifts daily — new edge cases, new user intents, new failure modes your test set never anticipated.

Eugene Yan makes a related argument: teams cargo-cult LLM-as-judge thinking another metric will fix quality, but automated evaluators can't compensate for neglect. If you're not actively reviewing real outputs and real user feedback, no amount of automated scoring will save your product. Evals are a practice — a scientific method applied continuously — not an artifact you generate and file away.

This doesn't mean offline evals are useless. You should absolutely run comprehensive benchmarks before deploying. But offline evals have a fundamental limitation: they test the system as it was, not as it is. The prompts changed on Tuesday, the tool schemas changed on Thursday, and Friday's eval report is testing a system that no longer exists.

The Case for Online Evaluation

Online evaluation means running classifiers and quality checks against production traffic continuously — not in a batch job, not on a cron, but as an integrated part of the agent execution lifecycle.

The idea isn't new. Traditional software has had production monitoring forever. But agent systems have unique properties that make standard APM inadequate:

  • Non-determinism is the default. The same input produces different outputs across runs. You can't diff against a known-good response — you need semantic classifiers that understand intent.
  • Failure is subtle. A hallucinated tool parameter doesn't throw an exception. A lazy response that skips half the user's question returns 200 OK. The system "works" in every way that traditional monitoring can detect.
  • Quality is multi-dimensional. A single run can be factually correct but frustrating, or helpful but off-brand, or thorough but dangerously slow. You need multiple signals evaluated simultaneously.

This is where the runtime layer becomes critical. The runtime sits between the agent logic and the outside world — it manages execution, tool dispatch, and state. It sees every input, every output, every tool call, every reasoning step, and every failure. No other layer has this visibility. Not the orchestrator above it. Not the model provider below it.

What Synapse Does Differently

Synapse is a Python SDK (synapse-agents) that unifies agent execution, online evaluation, and experimentation into a single runtime. The key design choice: these aren't separate systems stitched together — they share the same data model, the same persistence layer, and the same execution context.

Execution with Durability Guarantees

At the core is a ReAct execution loop with proper engineering around it. Agents define tools via a @tool decorator, the runtime handles parallel tool batching, and every run gets an idempotency fingerprint (SHA-256 of agent name + input) to prevent duplicate execution. Runs can be paused, resumed with new human input, or retried with prior conversation context — all with atomic state transitions that prevent race conditions.

Multi-agent delegation uses a parent-child model with tighten-only depth limits — a child agent can never escalate beyond what its parent allows. This is a small thing that prevents an entire class of runaway-delegation bugs.

The persistence layer makes a deliberate distinction between fail-closed operations (traces, tool calls, run events — these retry and propagate errors) and best-effort operations (usage stats, progress heartbeats — these swallow exceptions). Not every write is equally critical, and treating them the same is how you get cascading failures from a flaky database.

Signal-Based Online Evaluation

This is the part I'm most invested in. Every completed run flows through a MonitorPipeline that runs signal classifiers against the full execution trace — the user input, agent output, every reasoning step, every tool call and its result.

Synapse ships with eight built-in classifiers:

  • Hallucination — detecting fabricated content not grounded in tool results or context
  • Jailbreaking — catching prompt injection and boundary-testing attempts
  • User frustration — recognizing when a user is struggling or getting a runaround
  • Task failure — identifying runs where the agent didn't actually complete what was asked
  • Laziness — flagging minimal-effort responses that technically answer but don't help
  • Forgetting — detecting context loss across a conversation
  • NSFW — content safety classification
  • Wins — the positive signal, because you need to know what's working too

Each classifier is exception-isolated — one failing classifier doesn't take down the pipeline. And the system supports custom classifiers, because every domain has failure modes that no generic tool can anticipate. Husain's advice about starting from manual review of real traces, then codifying what you find into automated checks — that's exactly the workflow this enables.

Signals feed into a rule-based alerting engine that can notify via Slack or webhooks. The loop closes: agent runs, signals fire, alerts deliver, humans review, classifiers get refined.

Built-in Experimentation

The other piece that I think is undervalued in the agent ecosystem is proper A/B testing. Most teams evaluate prompt changes by vibes — "this version feels better." Synapse has a built-in experimentation engine with sticky variant assignment (deterministic hashing so users always see the same variant) and per-variant metrics: success rates, token consumption, cost, and signal frequency.

This matters because, as teams at Statsig and Braintrust have documented, offline eval scores and online A/B results often diverge. A prompt that scores higher on your test set might perform worse in production because the test set doesn't represent the tail of real traffic. The only way to know is to measure both.

Why the Runtime Is the Right Abstraction

You could build all of this as middleware on top of an existing framework. Plenty of teams do — a logging wrapper here, a webhook there, an eval script that runs nightly. But there's a reason this approach consistently fails to scale:

Middleware can only see what the framework exposes. If your orchestrator doesn't emit structured tool call results, your eval pipeline can't classify tool use quality. If your agent loop doesn't surface reasoning steps, your hallucination detector is working blind. You end up monkey-patching internals and parsing log strings — fragile, incomplete, and always one framework upgrade away from breaking.

The runtime doesn't have this problem because it is the framework. It owns execution, so it captures traces natively. It owns tool dispatch, so it records every call with full context. It owns state, so it can guarantee idempotent reruns and safe pause/resume. The OpenTelemetry community recognized this when they introduced semantic conventions specifically for GenAI spans — the insight being that agent execution has distinct observability needs that traditional APM can't model.

Think of it like this: you wouldn't bolt error handling onto a programming language as a library. You build it into the runtime — try/catch, Result types, panic handlers. Agent reliability deserves the same treatment. The question isn't "should we monitor our agents?" It's "at what layer should monitoring live?"

What I Got Wrong, and What's Next

Building Synapse has changed how I think about a few things. Early on, I assumed you'd want to evaluate every run. In practice, that's prohibitively expensive. The current approach uses queue-based processing with configurable concurrency — but smarter sampling (over-indexing on novel patterns and error states) is where the real leverage is.

I also underestimated how domain-specific failure modes are. The built-in classifiers cover common patterns, but the teams I've talked to always end up needing custom signals specific to their product. The classifier interface needs to be simple enough that writing a new one takes minutes, not hours.

What's next: better trace replay (rerun a production trace with modified configuration — time-travel debugging for agents), per-agent cost attribution tied back to traces, and tighter integration between the experimentation engine and the signal pipeline so you can automatically detect when a variant causes a spike in a specific failure mode.

Closing Thoughts

The agent ecosystem has gotten remarkably good at making agents work. What we haven't solved is knowing when they don't — and fixing it fast. That's an infrastructure problem, not a model problem.

Synapse is my bet that the right abstraction is a runtime that treats evaluation and observability as first-class concerns. Not because existing frameworks are bad, but because they were designed for a world where the hard part was getting agents to run at all. Now that agents run, the hard part is making them reliable.

If you're building agent systems and these problems resonate, I'd love to hear what you're seeing. Find me on X or email me.

Synapse is open-source under Apache 2.0. Code and docs on GitHub.