Durable Execution Across Agent Lifecycles

Crash-recovery model and persistence boundary

Consider an execution trajectory comprising stochastic inference steps, tool invocations, waits, and externally observable effects. Worker processes may crash, time out, or be replaced between any two steps. Volatile inference context can disappear independently of committed runtime state. External effects may remain in flight after local termination. The recovery model must therefore separate process failure, durable-state visibility, and effect completion instead of treating them as a single terminal event.

The central invariant is execution-identity continuity across process replacement: an authorized successor must recover the same run and slot obligation while retaining the predecessor attempt as historical state. Pinned configuration prevents replacement from implicitly resolving newer workflow or instruction defaults. Externalized results determine the recoverable prefix of semantic work. Recovery beyond that prefix requires recomputation or reconciliation with the service that owns the effect.

Execution identity and guarded reassignment

A package revision defines the requested outcome and its accepted content. A run resolves package and workflow configuration by pinned revision. It also uses an accepted workflow-agent binding set that identifies a binding revision, the selected agent revision for each workflow node, and resolved prompt, skill, or library attachments. The run therefore refers to a configuration snapshot selected at launch rather than silently following later defaults. Pins identify the configuration; they do not freeze external services or observations.

Let a run be R = (tenant, package revision, workflow revision, binding revision). Let slot q belong to R, and let attempts A(q) contain worker instances previously assigned to q. This run–slot–attempt hierarchy separates the execution contract from its instantiations. Reassignment preserves R and q; the subsequent claim introduces another attempt into A(q). The prior attempt remains addressable for provenance and recovery analysis.

The reassignment command is guarded by predicates over current runtime state. The slot must resolve within the command’s tenant and execution run and must be in retry-wait. It must refer to a prior worker whose worker status and lease are terminal. The command requires a nonempty reason. If the prior worker has an owner actor recorded, the command actor must match that owner. These checks reject a held or active slot, a slot from another tenant or run, and a reassignment by a different actor when ownership is recorded.

After validation, the runtime chooses a launch specification for the same tenant, run, and slot whose worker-claim compatibility matches the requested worker class. The command updates the existing slot with that specification, clears pending and active worker references, leaves the slot in retry-wait, and clears its retry time. It preserves slot identity and the prior-worker reference. Conceptually, the transition is (q=retry-wait, prior=a, terminal(a), authorized(c), compatible(s)) → (q=retry-wait, spec=s, prior=a, no active worker). This makes the slot available for the next claim without claiming that a new attempt already exists.

Reassignment and claim are separate state transitions. Reassignment selects the compatible launch contract and reopens the slot. The claim path later creates a worker and lease for that slot. Tests exercise this boundary by checking that reassignment updates the slot without creating an attempt; a subsequent claim is responsible for the new worker and lease. Separating these operations makes intermediate state observable and keeps scheduling or claim failure from being misreported as successful execution.

Lease state is an input to runtime ownership admissibility, not a semantic-progress detector or a distributed cancellation certificate. The terminal-worker and terminal-lease predicates govern reassignment locally. They do not establish that every downstream service has cancelled an outstanding request. Effect authorization and duplicate suppression must remain at the service boundary where those effects are committed.

Semantic continuation depends on externalized state: persisted results, decision records, source references, and any task-specific checkpoints emitted before termination. The run and slot preserve coordination state, while these artifacts preserve the recoverable computation. Unserialized inference context is outside the persistence contract. A successor reconstructs its working context from the pinned configuration and available artifacts rather than inheriting the predecessor process image.

Persistence, stochastic replay, and effect idempotency

The runtime persistence contract is orthogonal to parametric memory and cross-episode retrieval. Its state variables are package and workflow revisions, accepted binding revisions, slots, attempts, leases, and saved artifacts. These variables remain available independently of model-process residency. Persistence can therefore support recovery without modifying foundation-model parameters or maintaining an uninterrupted context window.

Durability also differs from deterministic replay. Two attempts can use the same package and binding revisions yet produce different outputs because model sampling, tool behavior, scheduling, or external observations differ. Bitwise reproducibility requires additional control of model versions, tool versions, random state, and external data. This architecture supports continuation under pinned configuration but does not claim those controls or equivalent conclusions.

Finally, retryability is not exactly-once execution. A service that sends a payment, changes a record, or deploys software must check current authority at the effect boundary and use an idempotency mechanism when applicable. Saved progress may guide the worker, but it cannot serve as authoritative proof of remote effect state. The effect owner must report that state and prevent duplicate or unauthorized effects under its own contract.

Related sources: [1], [2]

Recovery trace and ambiguous-effect boundary

A research worker is assigned slot q in run R. Its attempt reaches a terminal outcome, and q enters retry-wait while retaining the prior-worker reference. An authorized caller submits a reassignment command with a reason and requested worker class. The runtime resolves q within the same tenant and run, checks retry-wait state and terminal worker and lease status, and enforces the recorded owner actor if present.

The runtime then selects a launch specification whose claim compatibility matches that worker class and whose scope matches R and q. It records the selected specification and reassignment reason, clears active and pending worker references, preserves q and the prior-worker reference, and leaves the slot open for claim. No worker or lease has been created at this point. If no compatible specification exists, the reassignment cannot prepare the slot for that class.

A later claim performs the separate creation step. The new worker reads the pinned run configuration and any saved source notes or decisions. If the earlier worker did not externalize its source review, the new attempt may need to repeat it. If the previous attempt might have completed an external action before it terminated, the worker must consult the effect owner rather than infer success from the slot state. Recovery preserves the task boundary and exposes the prior attempt; it does not promise memory continuity or safe replay of every action.

Fault-injection evaluation and consistency scope

Evaluate the state machine at interruption points before work starts, after a tool result, during retry wait, and around an external effect. For reassignment, vary tenant, run, slot state, prior worker status, lease status, owner actor, reason, and worker class. Confirm that only an authorized retry-wait slot with a terminal prior attempt can be reopened and that the selected launch specification matches the requested class. Confirm separately that reassignment creates no worker or lease and that claim creates them later.

Recovery quality also depends on what the previous attempt saved. Check whether a new worker can locate required pinned inputs and explicit progress, and identify which steps it must repeat when those records are absent. Vary task types and effects: a text report has different recovery risks from an operation with a long-running remote side effect. When effect status is uncertain, the worker should ask the effect owner for the authoritative result rather than assume completion or failure.

This paper reports no recovery benchmark or exactly-once guarantee. The applicable correctness properties are preservation of run and slot identity, rejection of inadmissible reassignment, compatible launch selection, and separation of reopening from worker creation. External service availability, semantic validity of persisted results, and resolution of ambiguous remote effects remain outside those runtime invariants.