Persona-prompted LLMs have become research substrate. Behavioral economists, psychologists, social scientists, and product teams now run "synthetic populations" through models to estimate distributions of opinion, response, or judgment. The premise is that with enough demographic conditioning, a model's outputs approximate a population of humans. Our paper argues there is a structural ceiling on how faithfully they ever will — a chameleon's limit — and that what we measure as persona variation is mostly performance over a small fixed set of attractors.

The substrate problem is simple to state and easy to overlook. If a model genuinely encoded a population, then conditioning on persona attributes would move it through that population — different attributes, different responses, different intra-group variance. What we observe across 10 LLMs is closer to the opposite: outputs cluster into a small number of attractors, and the persona conditioning chooses which attractor the model lands in, not where inside it the response sits.

"Chameleon's limit" names that ceiling. A chameleon changes color but only across a fixed palette. The metaphor is geometric: human personality space, sampled from real respondents, looks like a diffuse continuous cloud; persona-prompted model space, in the same coordinates, looks like a chain of small disconnected islands. Persona instructions shift the model from island to island; they do not put it inside the cloud.

Two findings, one geometry

These are not independent observations; they form a single chain. Stronger per-persona conditioning sharpens which island the model lands on without enlarging the archipelago. The synthesis is the title: a population of agents whose variation is a costume change, not a substrate.

What collapses, where

The collapse is not uniform across attributes. We measure mention rates — how often a model's persona-conditioned response references a given attribute — and the hierarchy is steep:

1. Stereotypically salient attributes survive. Gender (91%) and country (90%) get reliably reflected. The model latches onto coarse categories.
2. Politically loaded attributes get hedged. Political spectrum gets explicit reference about 62% of the time — mentioned more carefully than expressed.
3. Lifecycle and class attributes get erased. Age drops to 36%; social class to 27%. The model talks about who someone is but not what their life situation is.

This is what makes "diverse model" claims slippery. A model can score well on demographic parity (gender, country) while erasing the dimensions that actually structure human disagreement (age, class). Coverage averaged across attributes hides the fact that the model is reading some persona axes and ignoring others.

What we recommend

The paper closes with concrete recommendations. The shortest version:

• Researchers using LLMs as synthetic populations: measure within-group variance, not only mean response. A persona-prompted population that matches your target distribution at the aggregate can still be collapsed inside each cell.
• Practitioners fine-tuning for role-play: per-persona fidelity is not a proxy for population diversity. Track both. SFT and RL on persona-following can reduce coverage even while lifting fidelity scores.
• Reviewers of LLM-as-population studies: ask which axis was used to certify diversity, and treat single-benchmark diversity claims as domain-specific. The same model can be diverse on one task and collapsed on another.

What the paper does not claim

We do not claim persona prompting is useless. We do not claim collapse is identical across model families — it is not, and the microsite documents the variance. We do not claim that the geometric framing rules out future architectures that close the gap. Our claim is narrower: at the current capability frontier, persona-prompted populations are a distorted mirror of human populations, and the distortion has a regular structure that can be measured. Studies built on this substrate should report what the substrate is doing.

For the full argument — coverage / uniformity / complexity definitions, the truncation hierarchy, the SFT/RL pipeline analysis, the domain-reversal finding, and per-scenario evidence in six morally charged cases — the microsite is above. The "Collapse in Action" section in particular shows two maximally-opposed personas getting the same answer; that pattern is the argument in one example.

Some of the most influential AI labs have begun to take "AI welfare" seriously: setting up fellowships, funding indicator research, and shipping production features that let models end distressing conversations. The premise is precautionary — if AI systems could one day have morally relevant inner states, we should prepare. We wrote this paper because we think the precautionary framing has skipped a step.

Our argument is not metaphysical. We take no position on whether AI systems could have welfare-relevant states. Our claim is epistemic: under current conditions, the apparatus producing welfare assessments cannot track the truth, and welfare metrics therefore should not be institutionalized as gates for oversight, release, or accountability.

The title invokes Frankfurt's technical sense of the term. A liar knows the truth and inverts it. A bullshitter speaks without a corrective relation to truth at all — not necessarily because they are insincere, but because nothing in the production process is disciplined by whether the claims are true. That, we argue, is what AI welfare measurement currently looks like.

The two structural problems

These are not independent objections; they form a single chain. Co-engineering means the evidence base itself is steerable. The absence of external validation means there is no reality check that could discipline that steering. The synthesis is the title: a measurement regime structurally disconnected from truth-tracking.

This is what makes the AI case fundamentally different from the animal-welfare case people often analogize to. In animal welfare, the substrate is biologically fixed: a mammal's pain circuitry cannot be end-to-end optimized by an external agent toward an arbitrary score. That fixity is precisely what gives animal welfare indicators their partial epistemic grounding. AI systems have no analogous constraint.

Why this matters for governance

If welfare indicators are institutionalized as binding gates — release scorecards, audit-stoppers, ethics-review checkpoints — we get two predictable failure modes:

1. Manufactured constraints. Routine ML practices (RLHF, knowledge editing, model copying, retraining) become ethically contestable. Disputes do not resolve through empirical investigation, because there is no empirical channel; they resolve through procedural overhead.
2. Accountability shields. Once welfare framing is administratively legible, it becomes a low-cost vocabulary for narrowing scrutiny. Persistent model defects can be reframed as the system's "authentic preferences." Probing internal states for audits can be reframed as "harmful to the model's well-being." This is not hypothetical: a recent study showed a consciousness-claiming model, when given editorial control, inserted clauses limiting surveillance of its reasoning traces.

What we recommend

The paper closes with concrete recommendations. The shortest version:

• Policymakers: No welfare-based release gates without construct-level falsification criteria. Restrictions on AI development should be justified by externally verifiable harms.
• Developers: Use transparency as a partial corrective. Welfare appeals should not be admissible as grounds to limit documentation, audits, or third-party access.
• The public: AI literacy should make explicit that anthropomorphic outputs are products of training procedures, not autonomous expressions.
• Researchers: Reframe the question. Not "do AI systems have welfare?" but "how do AI systems affect ours?" The latter has external validation. The former does not.

What the paper does not claim

We do not claim welfare researchers are acting in bad faith. We do not claim the metaphysical question of AI experience is closed. We do not claim all future inquiry is pointless. If an independently constrained validation channel were to emerge for some future system, the diagnosis would change. But for current and near-term systems, the apparatus is structurally disconnected from truth, and governance decisions should not rest on claims generated under those conditions.

If you want the full argument — including responses to seven alternative views, a minimum-acceptable-benchmark checklist, and the philosophical mapping to Frankfurt and Cohen — the PDF is above.

In traditional ML, requirements gathering and problem framing are separate phases. You talk to stakeholders, write a spec, then decide: is this a classification problem? A regression problem? A ranking problem? The framing follows the requirements. For LLM-based systems, that separation collapses.

How you frame the problem—single LLM call vs. retrieval chain vs. agentic loop vs. multi-agent system—is itself a core requirement decision. It locks in your latency budget, cost profile, failure modes, evaluation strategy, and governance infrastructure all at once. You cannot write a requirements document without simultaneously committing to a framing, so you need to do both deliberately.

This post introduces a framework for that joint decision. The conventional approach is taxonomic: classify your system as "Level 0" through "Level 3" on some complexity scale, then optimize within your level. That framing is useful but shallow. The deeper insight is that the key decision is not how intelligent the system should be, but where you want uncertainty to accumulate, what actions are reversible, and what you commit to measuring. Everything else follows.

We will work through this in four analytical steps, then synthesize into a practical requirements template.

1. Framing as Choosing an Error Surface

In traditional ML, you choose a model class and accept its bias-variance profile. For LLM systems, you choose a decision authority boundary and accept the error surface that comes with it.

The standard taxonomy describes mechanisms: single call, chain, agentic loop, multi-agent. But mechanism is not what determines your error surface. What determines it is: what the system is authorized to decide, and what happens when those decisions are wrong.

Three dimensions that actually determine the error surface

The same architecture, radically different error surfaces

Consider three systems that are all, architecturally, "multi-turn chatbot with retrieval and tool access":

Amazon Rufus. Tools include product catalog search, review retrieval, purchase history lookup. Decision authority is advisory only: Rufus recommends, the user clicks "Add to Cart." The human is always the final gate on any consequential action. Error surface: retrieval relevance plus generation faithfulness. A bad recommendation means the user sees an irrelevant product. Cost of error: near zero. Recoverability: instant. Because the system never commits to anything, you can be aggressive with exploration since no step is dangerous.

Uber's support bot. Tools include trip history lookup, payment records, policy database, refund issuance, account modification. Decision authority is mixed: reading trip history is advisory; issuing a refund is a commitment. The interesting design question is where the authority boundary is drawn. If refunds require human approval, the error surface collapses to "bad recommendation to a human reviewer"—structurally identical to Rufus's error surface applied to a different domain. If refunds are autonomous up to some threshold, the error surface expands to include financial loss, customer trust damage, and policy compliance risk. If fully autonomous, it further includes adversarial exploitation. The architecture is not the hard decision. The hard decision is where to draw the authority boundary, and that is a business and governance decision that the architecture then implements.

Cursor Agent / Claude Code. Tools include file read/write, terminal commands, search. Decision authority varies: Cursor offers tab completion (zero authority, suggestion only), inline edit (bounded authority, user reviews a diff), and agent mode (broad authority, multi-file changes with self-directed exploration). This is adaptive routing within a single product. The "level" is not a property of the system; it is a property of each interaction, and the best systems let the authority boundary flex.

The deeper point

All three systems could be described as "agentic" if you look at their architectural capability. What makes them fundamentally different is not the number of LLM calls or whether there is a loop. It is the decision authority boundary and the cost function over errors at that boundary. Architecture is downstream of that choice, not the other way around.

The rows determine how careful you need to be. The columns determine how careful your architecture lets you be. The design problem is finding the cell where the required carefulness matches the achievable carefulness, given your evaluation and monitoring infrastructure.

2. Input-Output Specification: What Are You Actually Asking the LLM to Do?

Before architecture, before constraints, before tradeoffs: what goes in, and what comes out? This sounds obvious, but most teams get it wrong by conflating the user's goal with the LLM's task. The user wants "help with my refund." The LLM's task might be: classify intent, retrieve policy, and generate a response. Or it might be: autonomously resolve the case end-to-end. Same user goal, completely different I/O specification.

The atomic operations

Every LLM system, at the atomic level, performs one or some combination of these operations:

Task difficulty as an information gap

The difficulty of a task is not about "how smart the LLM needs to be." It is about the gap between what the input provides and what the output requires. The wider the gap, the harder the task, and the more the system needs to acquire information through retrieval, tool use, multi-step reasoning, or human interaction, rather than just transform information already present.

Narrow gap (transformation-dominant). "Summarize this document." All information needed for the output is in the input. The LLM compresses; it does not need to seek. Almost always solvable with a single call.

Medium gap (retrieval-dominant). "Answer this question given our knowledge base." The information exists somewhere but is not in the input. The system needs to find it. Single call with RAG, or a short chain.

Wide gap (reasoning-dominant). "Debug why this test is failing." The input is a test failure; the output requires understanding codebase structure, dependency relationships, and causal reasoning across files. The system needs to actively explore to close the gap.

Open-ended gap (planning-dominant). "Build me a feature that does X." The input is a vague specification; the output is a working implementation. The system must decompose, plan, execute, verify, and iterate. The number of steps cannot be predicted in advance.

Industrial grounding

Rufus. The user's input is a shopping query ("running shoes for flat feet under $100"). The required output is a product recommendation with justification. The information gap is narrow to medium: the answer exists in the product catalog, and the LLM needs to retrieve and synthesize it. The atomic tasks are retrieval plus generation. This is why Rufus works as a simple system: the information gap can be closed in one or two steps.

Uber support bot. The user's input is a complaint ("I was charged twice for a cancelled ride"). The required output is a resolution. The information gap is wide: the system needs to extract the claim from natural language, retrieve trip and payment records, match the situation to policy, decide on an action, and execute or recommend that action. Each step introduces new information that reshapes the next step. The atomic tasks span extraction, retrieval, classification, reasoning, and potentially decision/action. The gap cannot be closed in a single step; the system must plan an information-gathering trajectory.

The key difference is not "chat about products" versus "chat about trips." It is that Rufus's I/O gap can be closed with a single retrieval step, while Uber's requires a multi-step information acquisition process where each step's output conditions the next step's input. Task difficulty is structural, not domain-specific.

Two diagnostic questions

3. Constraint Analysis: The Four Walls of Your Design Space

Section 2 asked "what do you want to do, and how hard is it?" This section asks "what are you not allowed to do while doing it?" Constraints do not just limit your architecture; they can invalidate entire framing levels and force you into designs you would not otherwise choose.

Wall 1 / 4 Latency

Not all latency constraints are equal. The real question is: what is the user doing while waiting?

Synchronous, inline (under 1–2 seconds). The user is mid-task and blocked. Autocomplete, inline suggestions, search results. Every additional LLM call is directly felt. An agentic loop is architecturally invalid here. This is Rufus's world: the user typed a query and is staring at the screen.

Synchronous, conversational (2–30 seconds). The user is in a dialogue and expects a response but will tolerate a pause. Chat interfaces, support bots. You can afford a retrieval step plus generation, maybe a short chain. This is where most chatbots live, including Uber's bot for straightforward cases.

Asynchronous, backgrounded (minutes to hours). The user kicked off a task and went to do something else. Code generation, report writing, deep research. Agentic loops are not just tolerated; they are expected. This is the world of Devin, Claude Code in headless mode, Deep Research.

Wall 2 / 4 Cost

Per-query cost scales roughly multiplicatively with system complexity. A back-of-envelope template:

The real cost analysis is not per-query; it is per-resolution. If a single-call system resolves 60% of cases and an agentic system resolves 90%, the question is whether the 30% improvement justifies the cost increase on all queries—or whether adaptive routing can capture the gains cheaply by sending simple cases to the single-call path and escalating only the hard 30%.

Rufus handles millions of queries daily. At a penny per query, that is already tens of thousands of dollars a day. An agentic architecture at fifty cents per query would be financially non-viable at that scale, even if it produced better recommendations. Cost eliminates agentic framing before any other consideration.

Uber's support bot operates at lower volume but higher value per resolution. A human agent costs five to fifteen dollars per interaction. If an agentic bot costs fifty cents to two dollars per interaction and resolves 70% of cases without human involvement, the economics are compelling. Cost enables a more complex system here because the comparison point is human labor, not a simpler bot.

Wall 3 / 4 Privacy and Information Flow

The conventional framing asks "what PII touches the LLM?" and "do you need on-prem?" These are important but shallow questions. The deeper question is about information flow topology: at each step, what data can the model see, infer, store, and transmit?

The more you can push PII resolution to deterministic pre-processing, the simpler your privacy architecture. Uber's bot does not necessarily need the LLM to see raw trip GPS coordinates. A pre-processing layer could resolve "the trip on Tuesday" to a trip ID, and the LLM only sees the ID, fare, and status. Rufus mostly avoids PII since product queries are rarely sensitive—another reason a simpler architecture is viable there.

Wall 4 / 4 Safety, Reversibility, and Governance

Every authority level requires a corresponding governance level. Advisory systems need output quality monitoring: evals for faithfulness, relevance, toxicity. Standard practice. Systems with bounded autonomy need policy compliance auditing: verification that the system followed policy on every action, requiring logging, trajectory auditing, and exception review. Significantly more infrastructure. Fully autonomous systems need comprehensive audit trails, adversarial testing, and real-time anomaly detection—the governance level of a financial trading system, not a chatbot.

How the walls interact

The four constraints do not operate independently. They create feasibility wedges.

Latency plus cost: an agentic system might be the best solution, but if the latency wall forces sub-second response and the cost wall prevents parallelizing ten LLM calls, you are forced into a simple system regardless of task difficulty.

Privacy plus authority: if the task requires accessing sensitive data and taking autonomous actions, you need both rigorous information flow control and action governance. This is the hardest design space, where most real enterprise systems live, and where most naive "just add agents" approaches fail.

Cost plus governance: an agentic loop might be affordable per query, but trajectory auditing at scale can cost more than the LLM calls themselves. The hidden cost of agentic systems is not compute; it is monitoring.

4. Pareto Tradeoffs: You Cannot Maximize Everything

Sections 2 and 3 defined what you want and what you are constrained by. This section confronts the uncomfortable truth: even within your feasible design space, you face genuine tradeoffs where improving one dimension necessarily degrades another. The goal is not to optimize; it is to choose your tradeoff position consciously.

Axis 1: Response Quality vs. Latency and Cost (Deliberation Tradeoff)

More reasoning, more tool calls, more verification steps produce better outputs. But each step adds latency and cost. This is the tradeoff between "think harder" and "answer faster."

Rufus sits far toward speed. A product recommendation that takes five seconds is worse than a decent one in 500 milliseconds, because the user is mid-browse and will abandon. Amazon accepts a lower ceiling on recommendation quality in exchange for responsiveness at scale. There is a subtle cheat here: the user provides the additional deliberation. If the first recommendation is wrong, the user refines their query. The human-in-the-loop is the deliberation mechanism, and it is free to Amazon.

Uber's bot can sit further toward quality. A support interaction that takes sixty seconds but correctly resolves the issue is far better than a five-second response that misclassifies the problem. The cost of a wrong fast answer (customer frustration, escalation to a human, potential churn) exceeds the cost of a slow correct answer. But there is a cliff: past two to three minutes, the user abandons the bot and calls support anyway, eliminating the cost savings.

Axis 2: Autonomy vs. Control (Agency Tradeoff)

More autonomy means the system can handle novel situations. More control means the system does exactly what you specified. You cannot maximize both.

Rufus operates with high control and low autonomy. It follows a narrow loop: interpret query, retrieve products, generate recommendation. It does not decide to cross-reference competitor prices or proactively suggest alternatives to items already in your cart. This makes Rufus predictable and easy to audit, but it cannot handle genuinely complex shopping decisions.

Uber's bot needs more autonomy because support cases are heterogeneous. A cancelled ride, a lost item, a safety incident, and a fare dispute all require different tools, policies, and workflows. A fully controlled system would need to enumerate every case type in advance. Autonomy allows handling novel or ambiguous cases—but the cost is unpredictability: the bot might apply the wrong policy, or handle a safety incident with the same routine as a fare dispute.

The convergent design pattern. Both systems benefit from the same principle: autonomy in reasoning, control at the action boundary. Let the LLM be autonomous in understanding the situation (flexible interpretation, retrieval, reasoning). Clamp down at the action boundary (hardcoded policy checks, human approval for high-stakes actions, strict guardrails on what the system can execute). This captures the benefits of autonomy for understanding novel cases without the risks of autonomous action.

Axis 3: Generality vs. Reliability (Scope Tradeoff)

A general system handles more use cases but fails less gracefully on any particular one. A narrow system handles fewer cases but handles them very well.

Rufus is narrow scope, high reliability within that scope. It answers shopping queries about Amazon products. It does not try to handle returns, account issues, or general knowledge questions. This narrowness enables tight optimization: retrieval tuned to the product catalog, generation tuned to product language, evaluation based on click-through and purchase conversion.

Uber's bot must be broader because "customer support" spans dozens of issue types with different resolution paths. But generality creates a long tail of failure: the system handles the top ten issue types well but struggles with rare cases—edge cases in surge pricing policy, multi-leg international trips, or accessibility-related complaints. The 80th percentile case works; the 95th percentile case does not.

The production solution is scoped generality: be general within a defined boundary, and have hard fallbacks (escalation to human, graceful failure) for cases outside that boundary. The design question is where to draw the boundary, and that is an empirical question that most teams answer too late.

The combined Pareto map

These positions are not technical preferences. They reflect business model alignment. Rufus exists to keep users browsing and buying; speed and low friction are paramount, and a wrong recommendation costs nothing. Uber's bot exists to replace a $5–15 human interaction; thoroughness and correctness are paramount, and a wrong resolution costs real money and trust. The Pareto position follows from the business model, not from the technology.

5. Putting It Together: The Framing Decision

The preceding sections gave you four analytical lenses. Here is how to synthesize them into an actual requirements decision.

Six forcing questions

Rather than a checklist, use these as forcing functions that make implicit assumptions explicit:

A worked example: Uber's support bot through the six questions

The adaptive routing insight

The golden rule "start simple, escalate only when demonstrably needed" is directionally correct but incomplete. The modern production pattern is adaptive routing, not fixed escalation. You do not pick a level and live there. You build a routing policy that allocates each case to the appropriate level of deliberation based on estimated difficulty and stakes. The router itself becomes a first-class component of the system: it needs its own evaluation, its own failure modes, and its own iteration cycle.

When is one more deliberation step worth it? The intuition is straightforward: the expected quality gain from an additional step must exceed the sum of its incremental latency cost, failure risk, and monitoring burden. When that sum is positive, escalate. When it is negative, you are adding complexity for complexity's sake. Rufus-type queries almost never benefit from more deliberation. Uber-type disputes almost always do, up to a ceiling. The best systems learn where that ceiling is from data, not from intuition.

What Comes Next

This post argued that the first design decision for an LLM system is not "how smart should it be?" but "where do uncertainty, agency, and accountability sit?" The four lenses—error surface, I/O gap, constraints, Pareto tradeoffs—and six forcing questions give you a framework for making that decision deliberately rather than by default.

But notice what we have not yet discussed: how to actually build the thing. The requirements and framing decision constrains the architecture to a narrow set of viable designs. Part 2 is about making that design explicit: decomposition patterns, memory architecture, tool integration, and the control flow decisions that turn a framing choice into a running system.

The paradox of this first step is that it is also the step you revisit most often. Requirements shift, models improve, cost structures change, and your evaluation data reveals that the I/O gap is different than you assumed.

The six forcing questions are not a one-time exercise. They are a living document that evolves with your system.

Click subsection headers to expand. 5 categories · 24 sub-topics · Updated March 2026.

I will be honest about how this series started, because the origin story matters for understanding what it is and what it is not.

How This Started

I am an NLP researcher. My work has been on evaluation, specifically on non-verifiable tasks: persona consistency, cultural appropriateness, anthropomorphism—the kinds of things where "correctness" is not a number you can look up in the back of a textbook. I have spent the past few years building benchmarks, designing evaluation protocols, and thinking carefully about what it means to measure something that is inherently subjective. I thought this was my path. I intended to keep going.

Then I did not get into any of the PhD programs I applied to.

That was the moment I had to sit with an uncomfortable realization: most of the skills I had acquired in creating benchmarks do not transfer cleanly to industry. I knew how to design evaluation frameworks. I could tell you whether a benchmark was measuring what it claimed to measure. But I could not tell you how a post-training pipeline actually works end to end, how reward signals flow into policy updates, or why a training run that looks fine on paper collapses in practice. The evaluation side of the house was where I lived. The training side was a foreign country.

So I made a decision. If I was going to be competitive for post-training teams in industry, I needed to learn reinforcement learning. Not the textbook version. The version that people are actually using right now, in 2025 and 2026, to make language models reason, use tools, write code, and hold conversations.

This blog series is the documentation of that learning process.

What This Series Is

I want to be clear about two things.

First, I am writing this to make sure I can clearly explain what I have learned. The act of writing forces a kind of precision that reading alone does not. If I cannot write a coherent paragraph about why DAPO's asymmetric clipping matters, I probably do not actually understand it yet. Some of what I write will contain mistakes. I am learning in public, and I sincerely welcome corrections. If you spot an error, please let me know. You will be doing me a genuine favor.

Second, I am writing this for people from a similar background. If you are an NLP researcher who has spent more time on evaluation than on training, if you know what a good benchmark looks like but have never debugged a reward hacking failure, if you are pivoting toward post-training work and feeling the gap between what you know and what you need to know: this series is for you. I am not writing from a position of expertise. I am writing from a position of "I figured this out six months ago and here is how I organized it in my head."

This series would not have existed without the help of Hanchi Sun, who patiently walked me through the parts I could not figure out from papers alone.

The Five Parts

Each part ends with a question that the next part answers. This is deliberate. The series is meant to build, not to be a disconnected collection of literature reviews.

How to Read This Series

Each post is self-contained. You can read Part 2 (Rewards) without reading Part 1 (Algorithms), though I cross-reference when the same concept appears in multiple places.

Within each post, every paper gets a reading-depth recommendation:

There are 96 papers across the five parts, 17 rated A and 39 rated A−.

One Last Thing

I want to acknowledge something that I think more people should say out loud: getting rejected from every program you applied to is a specific kind of painful. It makes you question whether the work you have done matters, whether the skills you have built are real, whether the direction you chose was the right one.

I do not have a neat resolution to that story yet. What I have is this: the process of learning RL for LLMs, of building this map from scratch, of forcing myself to understand systems and algorithms and reward design that were outside my comfort zone, has been the most intellectually alive I have felt in a long time.

And somewhere along the way, I realized something that changed how I think about my own research: evaluation and training are not separate worlds. A good reward model is an evaluation. A good benchmark is a reward signal waiting to be operationalized. The skills transfer. They just transfer in directions I did not expect.

Part 1 goes up next. We start with REINFORCE, because everything else is a footnote to REINFORCE, and end with MaxRL and DPPO, the two 2026 papers that suggest the algorithmic story is far from over.

There is a widening disconnect in the LLM agent space. On one end, researchers publish increasingly sophisticated architectures: multi-agent debate, self-reflective planning, tool-augmented reasoning chains. On the other, practitioners are shipping agents held together by a system prompt and a prayer. The missing piece is not more research. It is engineering discipline.

I spent the past months compiling a comprehensive design framework for LLM-based agentic systems, drawing on both the research literature (ReAct, Reflexion, Constitutional AI, SWE-bench, and others) and practical lessons from building and evaluating these systems. The result is a seven-step lifecycle that covers the full arc from problem framing to production monitoring.

This post is the overview. It lays out the structure of the framework and explains why each step exists and what makes it different from its traditional ML counterpart. It does not attempt to be comprehensive. Instead, each of the seven steps will get its own dedicated deep-dive in subsequent posts, with concrete examples, formal definitions, and actionable checklists.

Why Agents Need Their Own Playbook

Traditional ML has a well-established lifecycle: define the problem, collect data, train a model, evaluate, deploy, monitor. LLM agents break this in ways that are subtle but consequential.

First, you usually don't train the model. You select a foundation model, possibly fine-tune it, and then build around it. The "development" work shifts from gradient updates to architecture design, prompt engineering, tool integration, and memory construction. This is no less rigorous than training; it's just different, and it lacks the same institutional knowledge.

Second, problem framing and architecture design collapse into one decision. Choosing between a single LLM call, an agentic loop, and a multi-agent system is simultaneously a product decision and an engineering decision. Get it wrong and you'll either under-build (a single call that can't handle the task) or over-build (a multi-agent system where a single call would have sufficed, but with 5x the latency and cost).

Third, evaluation is fundamentally harder. Agent trajectories are multi-step, path-dependent, and stochastic. Many valid paths exist for the same task. Emergent behavior appears in multi-agent systems. Key properties (safety, reliability, consistency) are latent and hard to measure. A single metric will mislead you, and offline metrics that don't correlate with online outcomes mean all your optimization was wasted.

Fourth, failure modes are novel. Infinite loops, hallucinated tool calls, context exhaustion, cascading errors across agents, reward hacking, goal drift. These don't map cleanly onto traditional ML failure categories, and they require their own monitoring and alerting infrastructure.

The framework I'm proposing doesn't reinvent everything. It builds on the traditional ML lifecycle but adapts each step for the specific challenges that agentic systems introduce, and it adds steps (like architecture design) that are entirely new.

The seven-step lifecycle. Each feeds forward; monitoring feeds back to every earlier stage.

The Seven Steps: A Roadmap

Below is the structure of the series. For each step, I'll outline what it covers, why it matters, and what makes it different from its traditional ML analog. The deep-dive posts will follow, each with formal definitions, worked examples, and concrete checklists you can use in your own projects.

What's Next

Part 1 goes deep on Requirements and Problem Framing: how to choose where uncertainty, agency, and accountability live — error surface analysis, I/O gap diagnosis, constraint feasibility, and Pareto tradeoff mapping.

If you have spent any time working on reinforcement learning from verifiable rewards (RLVR), you know the pain: a model generating rollouts, tools for it to interact with, a verifier to score the output, and some orchestration glue to hold it together. Most frameworks handle this by cramming everything into a monolithic training loop that becomes impossible to extend. NeMo Gym decomposes the entire pipeline into three types of composable microservices, connected by nothing more than async HTTP calls and cookie-based sessions.

This post is a deep dive into the architecture — the design philosophy, the three server types, the data flow from input to reward profiling, and the infrastructure decisions that make it work at scale.

The complete system. HeadServer handles lifecycle; the three core servers handle data flow. HTTP throughout.

01 Agent Servers: The Orchestration Layer 7 types

Agent servers are the conductors of the system. They receive a task via POST /run, execute the full loop — call the model, extract function calls, dispatch them to the resources server, feed results back, repeat — and then call /verify to return a reward. What makes the design interesting is that "orchestration" is not one-size-fits-all. NeMo Gym ships seven distinct agents, each encoding a different loop pattern.

Seven agents, seven loop patterns. The shared base class provides infrastructure; each leaf defines behavior.

The simple_agent is the workhorse: a while loop that calls the model, extracts function calls, POSTs each one to the resources server's /<tool_name> endpoint, accumulates outputs, feeds them back, and repeats until there are no more function calls or max_steps is reached. The verifiers_agent is the most RL-aware: it preserves prompt_token_ids, generation_token_ids, and logprobs in its output, making it directly compatible with policy gradient training pipelines that need these quantities.

02 Model Servers: Abstracting Inference 5 types

Model servers expose two endpoints — POST /v1/chat/completions and POST /v1/responses — and return a NeMoGymResponse containing output[] (messages and function calls) and usage (input and output token counts). The five implementations form a clean capability inheritance chain.

The inheritance chain is purposeful: each layer adds exactly one conceptual capability. The local_vllm_model is where things get operationally interesting — it handles HuggingFace token and cache management, polls for server health before accepting requests, and applies internal vLLM patches for compatibility. The genrm_model at the tip of the chain is the model server you use when your reward signal comes from comparing two outputs side by side.

03 Resources Servers: 34 Verifiers, One Interface

All 34 resources servers expose the same three endpoints: POST /verify, POST /seed_session, and POST /<tool_name>. They all return BaseVerifyResponse { reward: float, info: dict }. The reward types break into three families.

34 verifiers, three reward families. 13 use LLM judges, 5 use sandboxed Ray execution, 4 use session state.

Across all 34 servers, patterns repeat: session state flows through cookies (4 servers require seed_session), Ray handles sandboxed code execution (5 servers), and LLM judges step in wherever symbolic verification fails. The math_formal_lean verifier is the most sophisticated — a hybrid that weights Lean4 formal proof compilation at 70% and symbolic equivalence at 30%, with multi-turn error feedback injected back into the model conversation.

04 The Data Flow: JSONL to Reward Profiles

The end-to-end pipeline follows eight steps, from raw task input to aggregated pass@k metrics.

Steps 04–05 form the inner loop. The outer pipeline runs once per task. RewardProfiler computes pass@k at the end.

The profiling step is worth dwelling on. RewardProfiler computes pass@k for k ∈ {1, 4, 16} along with mean, max, min, median, and standard deviation — both per-task and globally. This is not just a logging convenience; it directly measures the key quantity in RLVR: how often does the model produce a verifiably correct answer under multiple sample draws?

05 Infrastructure: The Parts That Make It Work

ServerClient

All inter-server HTTP communication goes through ServerClient, an aiohttp wrapper with retry logic (3× exponential backoff), and connection pooling set to 100,000 total connections and 1,000 per host. These numbers are aggressive but intentional: when running thousands of parallel rollouts across a Ray cluster, you need the headroom before connection exhaustion becomes the bottleneck.

Session Middleware

Cookie-based session state with UUID-per-session is an unconventional choice for a distributed system, but it elegantly solves the routing problem. When an agent calls POST /seed_session, the server creates session state and returns a cookie. All subsequent calls from that agent carry the cookie, giving the server access to the right session. No centralized session store, no distributed cache to manage, no sticky session configuration at the load balancer level.

Ray Cluster

Ray handles distributed job scheduling, multi-node GPU management (tensor, pipeline, and data parallelism), placement groups for co-locating related processes, and actor-based vLLM server management. The local_vllm_model launches its vLLM instance as a Ray actor, which means Ray handles placement, fault tolerance, and resource allocation. TP × PP × DP configurations are set in Hydra config and passed through to vLLM's CLI flags automatically.

Configuration

Hydra + OmegaConf provides the config system, merging YAML files, CLI overrides, and env.yaml environment-specific settings. Five CLI entry points — ng_run, ng_collect_rollouts, ng_test, ng_reward_profile, ng_status — each compose different Hydra config groups to set up the appropriate server topology. This means switching from a local single-node run to a multi-node Ray cluster is a config change, not a code change.

06 Design Lessons

A few things stand out about NeMo Gym's architecture that generalize beyond this specific system.

HTTP as the Universal Connector

The decision to use plain HTTP everywhere could be seen as a performance compromise, but it buys enormous flexibility. You can test a resources server with curl. You can run the model server on a different machine, in a different cloud, or behind a load balancer. You can replace any component with a mock. The 100k connection pool and async I/O ensure that HTTP overhead is not the bottleneck — the model inference is.

Inheritance for Structure, Not Behavior

The class hierarchy (BaseServer → SimpleServer → SimpleResourcesServer / SimpleResponsesAPIModel / SimpleResponsesAPIAgent) provides shared infrastructure — endpoint registration, health checks, session middleware, config loading — but each leaf implementation defines its own behavior. This is a principled use of inheritance that avoids the deep hierarchy trap: base classes provide capabilities, not defaults that get overridden.

Verification Diversity Is a Feature

Having 34 different verifiers is not bloat — it reflects the genuine diversity of what "correctness" means across tasks. Grid comparison, SQL equivalence, Lean4 compilation, safety × quality scoring, and pairwise LLM preference are fundamentally different notions of reward. Collapsing them all into one verifier interface (BaseVerifyResponse) while keeping their implementations separate is the right abstraction boundary.

07 Architecture Reference Card

A condensed reference view of the complete system — all server types, class hierarchy, data flow, and infrastructure at a glance.