Feedback-Guided Optimization of Agent Harnesses

Compound-system behavior and two timescales

A foundation model is one component in a deployed agent system. The effective policy also depends on the system prompt and standing instructions, task-specific context, memory retrieval, tool descriptions, orchestration logic, delegation rules, and the external work state. A model can therefore remain fixed while a change to persistent harness configuration changes which observations enter context, which tool is selected, or when control is delegated. The adaptation variable is the compound system configuration h; foundation-model parameters remain fixed during the intervention.

Two temporal scales must be kept separate. At inference time, an agent observes the current task state, selects an action, receives an observation, and updates its working trajectory. Revising a plan after a tool failure is execution-time adaptation within one mission. At the outer loop, an operator changes a persistent instruction or configuration used by later missions. That intervention changes the conditions under which a sequence of future inference-time trajectories is generated. The first loop targets progress on the current objective; the second asks whether a configuration generalizes across a distribution of future work.

A compact explanatory model is to write a trajectory as τ ~ P(τ | M, h, x), where M is a fixed foundation model, h is the harness configuration, and x is task context and environment state. A proposed evaluation objective could be J(h) = E_{x∼D, τ∼P(·|M,h,x)}[U(τ,x) − λC(τ,x)], where U scores accepted task outcomes and C captures relevant costs such as latency, rework, or human effort. This notation identifies the object a future optimizer would compare. The formal objective is a prospective evaluation specification; the implemented protocol supplies the intervention, observation, and adoption structure.

τ ~ P(τ | M, h, x)
J(h) = E[U(τ, x) − λC(τ, x)]   (proposed evaluation objective; not computed by the current protocol)

Related sources: [1], [2]

Prediction-led black-box intervention

The operating loop starts with a behavioral trace, not an abstract preference for more activity. A supervisor identifies a concrete failure mode, such as waiting for irrelevant correspondence, losing the task objective, choosing a poor delegation boundary, or repeating avoidable work. The diagnosis proposes a causal hypothesis about an instruction or configuration condition. The intervention changes a bounded part of the persistent harness, which may include standing text, orchestration settings, or a small executable rule. Because the system is treated as a black box at this layer, the change is evaluated through observed behavior rather than an assumed internal gradient or weight update.

Before applying a change, the supervisor states a prospective prediction: which observable behavior should change, under what conditions, and by what observation deadline. The previous configuration remains available for reversion. At the horizon, the operator compares the observed trajectory with the prediction and considers downstream outcome quality. If the prediction is absent or the intervention introduces regressions, the operator may revert or revise it. This defines a sequential hypothesis-driven intervention protocol with explicit rollback conditions.

A prospective prediction must identify an intervention mechanism and an observable transition. Aggregate activity alone confounds productive throughput with redundant execution. “When a task is blocked only on a recorded work result, inspect that result and continue without waiting for an unrelated acknowledgement” predicts a specific transition in the control flow. A test should then measure whether the agent recognized the dependency correctly, whether the task advanced, and whether the resulting work was accepted. The behavioral proxy is informative only when interpreted alongside rework, latency, and human effort.

A future configuration search could treat h as a structured program and propose candidate edits, but that raises problems of search-space definition, objective noise, safe exploration, and multiple comparisons. The present method constrains candidate selection through supervisor diagnosis and prospective predictions. Algorithmic candidate generation, acquisition functions, and population-based search are possible extensions of this intervention space.

• Observe a concrete failure in an agent trajectory and formulate a plausible harness-level cause.
• Specify one bounded intervention, its predicted observable effect, and an observation horizon.
• Preserve the prior configuration, apply the candidate change, and record the relevant work context.
• Compare subsequent observations with the prospective prediction and inspect regressions.
• Retain, revise, or revert the change using accepted outcomes and costs, not activity counts alone.

Within-agent two-period crossover evaluation

A historical analysis in the experimental tooling uses a two-period crossover structure. Two cohorts of supervisory agents experience opposite treatment sequences: one sequence receives the intervention in the first observation period and the comparison condition in the second; the other sequence reverses that order. Each agent contributes observations under both conditions when sampling is sufficient. The analysis uses fixed observation windows and a washout interval between periods. The periods are not assumed to have equal duration, and the available description does not establish randomized assignment. Normalizing a count to a rate adjusts for elapsed observation time; it does not remove period effects, secular trends, or sequence confounding.

The outcome is a per-agent rate of delegation events per unit time, derived from observed count growth over elapsed time. For each eligible agent i, the estimator forms a paired contrast dᵢ = rᵢ,treatment − rᵢ,comparison, then reports the mean of the within-agent contrasts. Agents without adequate observations in both periods do not contribute an estimate. A treatment-ingestion exclusion is declared before the calculation and removes a unit whose comparison period contains evidence that it received the intervention. The exclusion is intended to reduce contamination; it does not guarantee that all carryover has been removed.

The analysis enumerates sign assignments for the paired differences and compares the observed mean with the one-tailed reference distribution. Exact enumeration of all 2ⁿ sign patterns is computationally exact for that statistic and sample. Its inferential validity is a separate question. A sign-flip test requires a null under which the signs of unit-level effects are exchangeable or symmetric, along with an appropriate independence structure across units. It is not randomization inference unless treatment order was randomized under a known assignment mechanism. Small n also gives coarse p-value resolution, and a directional one-tailed test answers a narrower question than a two-sided test.

Persistent prompts and shared operating conditions create interference and carryover risks. An intervention given to one supervisor can affect tasks delegated to another, shared workload, or team coordination; one unit’s outcome may therefore depend on other units’ assignments. A washout interval cannot ensure erasure if instructions, learned habits, task queues, or context persist. The crossover estimator also remains sensitive to treatment-by-period interaction and changes in task mix. These conditions challenge the stable unit treatment value assumption (SUTVA), particularly its no-interference and well-defined-treatment components. The current paper describes the estimator and its limitations; it does not report a statistically significant effect or a causal performance claim.

dᵢ = rᵢ,treatment − rᵢ,comparison
Δ̂ = meanᵢ(dᵢ)
Sign-flip reference distribution: { meanᵢ(sᵢ dᵢ) : sᵢ ∈ {−1, +1} }
Inference requires a defensible symmetry/exchangeability null and suitable unit independence; it does not follow from enumeration alone.

Related sources: [4]

Trace: reduce coordination delay

Suppose a supervisor repeatedly waits for another agent’s acknowledgement even though the task dependency is a saved work result. The operator records the behavior, identifies the standing instruction that treats acknowledgement as a prerequisite, and proposes a narrow edit: inspect the recorded dependency and proceed when it is satisfied. The prediction specifies that the next eligible task will advance without that unrelated wait. The observation horizon and a condition for reverting the edit are written before the persistent configuration changes.

A later trace can confirm whether the supervisor consulted the dependency and advanced at the predicted point. That observation does not establish improved task performance by itself. The operator also checks whether the dependency was correctly interpreted, whether the work passed review, and whether the change increased rework or shifted cost to another agent. If progress becomes faster while acceptance falls, the intervention has not optimized the objective. The trace specifies a falsifiable intervention and the outcome checks required to adjudicate it.

Relation to agent learning and prompt optimization

Reflexion stores verbal feedback for subsequent attempts, while GEPA searches over prompt candidates using execution feedback and reflection. These methods show how language, traces, and external memory can participate in system improvement. They are related work, not descriptions of the current product protocol. The shared abstraction is that an agent system’s effective behavior can change through components outside foundation-model parameters. The distinction is in the optimization mechanism: the operating method described here relies on a supervisor to diagnose a failure, choose a bounded intervention, predict its effect, and inspect follow-up work.

The terms training and reinforcement learning require care. An outer loop may optimize a deployed system without updating neural weights, but feedback-driven changes alone do not identify an RL algorithm. Such a claim would require a defined policy representation, reward or objective, data-generating process, update rule, and evaluation protocol. The current work specifies none of these as an implemented learner. Supervised fine-tuning and reinforcement learning are possible future uses of high-quality execution evidence, subject to data governance and independent evaluation; they are not current capabilities.

Related sources: [2], [3]

Requirements for stronger evidence

A stronger evaluation would define a representative task distribution, a versioned configuration treatment, a task-level utility that includes accepted outcomes and costs, and an assignment procedure appropriate to the interference structure. If teams exchange work or share queues, cluster-level assignment or a design that models those dependencies may be needed. Period and sequence effects should be estimated or controlled, carryover should be assessed rather than assumed absent, and exclusions should be set before inspecting outcomes. An independent evaluation set would reduce the risk that repeated edits overfit the observed failures.

The product direction is to connect versioned missions, execution records, and evaluated outcomes so that future analyses can bind interventions to their context. A useful record would include task characteristics, harness version, available context, observed trajectory, evaluation, and outcome, subject to consent and privacy controls. Those data may support later supervised fine-tuning or reinforcement learning if the objective and sampling process justify them. More iterations do not prove generalization: an intervention that fixes one observed failure may degrade another behavior or fail on a new task distribution.