Workflow Orchestration: Carrying Probabilistic Models with Persistent State and Deterministic Control

The most direct impulse when organizing a group of model calls into a reliable process is to have the Agent itself remember which step it has reached. This approach does not work. Agent sessions can be truncated or have their context compressed at any time, or deviate from the original plan because the model replans. Session memory is not transactional state; it can neither guarantee that "what is the next step" is always traceable, nor guarantee that after a failure we can return exactly to the step that went wrong. Putting process state into model context is like betting reliability on a carrier that is neither reproducible nor auditable.

What Workflow Orchestration solves is exactly this misalignment: the model is responsible for producing results, but the model is not responsible for remembering the process. The orchestrator is a piece of deterministic control logic that persistently records each node's input, current state, attempt count, and output, and based on this handles dependencies, timeouts, and recovery. The "persistent" here is key—node state resides in stable storage, not in the memory of some session. In this way, after a process crashes and restarts, the orchestrator reads back the facts committed last time, not a conversation that may already be lost.

Probabilistic activities are compressed into clearly defined nodes. Large Language Model (LLM) nodes only perform local tasks such as classification, extraction, generation, or judgment, and when delivering results use a versioned schema to constrain the model's free-text output into structured, validatable results. The orchestrator keeps complete records of each node's input and output, so when the model makes an error, which node the error occurred in, what the input was, and what output was produced are all observable, replayable, and localizable. Probabilistic errors are no longer scattered throughout the entire process but are confined into bounded nodes one by one.

Inputs, outputs, and the overall causal chain can be understood as follows: the input is a set of describable steps, dependencies between steps, the schema for each step, and timeout and recovery rules; the orchestrator compiles these into a deterministic execution skeleton, invokes a probabilistic activity once at each node, and writes the result back into the history. The output is a restartable, auditable process instance—it records not only the final answer but also the facts of what happened at every step.

From this we get a usage boundary: fixed paths use workflows, and autonomous search is placed only in genuinely unknown local parts. When process steps are clear and dependencies are well-defined, use the deterministic skeleton for scheduling; only when a certain step faces a truly open search space that cannot be enumerated in advance do we embed a constrained Agent in that local part. The orchestrator preserves facts and decisions, and the LLM only handles local judgments; model context can carry a single inference, but it cannot carry the transactional state of the entire process. Separating these two layers is the starting point for orchestration reliability.

The orchestration skeleton is not a single graph; rather, it selects different representations based on the control structure of the process. A review process with a conditional loop—for example, 'if the document fails, send it back to supplement materials until the conditions are met before continuing'—if forcibly drawn as a directed acyclic graph (DAG), immediately becomes awkward: DAG edges can only go in one direction and cannot return to ancestor nodes, while 'return for re-review' is naturally a back edge, requiring nodes to be expanded into several copies to barely represent it. Choosing the wrong control model turns a clear process into a pile of copy-pasted pseudo-nodes.

The inputs for choosing the control model are whether the process itself has cycles, whether it has waits, whether it receives asynchronous events, and whether there are paths unknown in advance. The output is one or a combination of the four types of structures: DAG, state machine, event-driven, or Agent subprocess.

DAG excels at expressing acyclic dependencies, batch processing, and parallel fan-out: a batch of steps first runs independently to completion, then converges, making dependencies clear at a glance and naturally suited to parallel scheduling. Its limitation is that loops and waiting are cumbersome to express—any process that needs to 'go back to the previous step' or 'get stuck waiting for external conditions' will be forcibly flattened.

State machines treat 'which state you are currently in and what event triggers what transition' as first-class concepts, so conditional branches, loops, timeouts, and human pauses can all be modeled directly; review returns, approval timeout escalations, and orders awaiting payment are all state transitions. The cost is that when the number of states and transitions grows large, the state diagram itself becomes complex and hard to read.

Event-driven structures excel at cross-system asynchronous reactions: when an external system sends callbacks, messages, or webhooks, the corresponding handling is triggered. Their weakness is that order and duplication are harder to track—the order in which events arrive is not necessarily deterministic, and the same event may be delivered multiple times, requiring additional idempotency and deduplication mechanisms as a safety net.

Agent subprocesses are suitable for local tasks whose paths are unknown in advance, letting the model decide on its own what to do next within a constrained scope. Their cost and termination are both uncertain: the model may explore for a long time, or may never converge, so they can only be placed in a clearly bounded local area and must have timeout and termination conditions.

These four types of structures can be combined rather than mutually exclusive. In the same system, DAGs can be used to organize dependencies between large phases, state machines can be used to manage long-lived state flows such as orders, event-driven can be used to trigger external callbacks, and inside a certain node a constrained Agent can be called to handle local unknown judgments. The key to composition is to let each structure carry only the part of control it is best at, rather than squeezing all requirements into the same kind of graph. Forcing a conditional loop into a DAG loses exactly the clear expression of states—this is exactly the mistake to avoid first when selecting a model.

Putting the previously discussed principles into a real pipeline, the most intuitive example is invoice processing: from receiving a file to final posting, with probabilistic extraction, deterministic validation, potentially lengthy human approval, and a posting call that interacts with the external financial system in between. Each type of risk in this pipeline is assigned to the corresponding structure.

The receiving stage uses the file hash as the workflow ID. When the same file is uploaded again with the same hash, it matches the existing workflow instance instead of opening a new one. Deduplication thus becomes deterministic key matching, and duplicate submissions do not create duplicate workflows.

The extraction stage calls an LLM, asking it to output a versioned invoice schema—fields such as supplier, amount, tax amount, and date are constrained within a structured contract. When parsing fails, only the “extraction” activity is retried: the preceding receiving step has already settled, and the subsequent validation has not yet occurred, so the cost of retrying is limited to a single node. The free text output by the model is first converged here into a verifiable structure.

The validation stage is completely deterministic code that checks whether the total amount adds up, whether the supplier is valid, and whether it forms a duplicate invoice with existing records. It does not rely on any probabilistic judgment, so any validation result can be recomputed at any time and yield the same answer.

The approval stage routes high-amount invoices into a persistent human task. The approver may not respond for days, but this workflow instance does not occupy a worker process—it is saved in a waiting state and only wakes up when a human action or a timeout triggers it. The long wait is decoupled from the process lifecycle; it is a state machine, not an online call, that carries this part of the control.

The posting stage calls the financial API and carries an idempotency key, invoice_id. Multiple calls with the same invoice_id produce only one posting effect on the financial system side. Thus, even if a call times out or a response is lost, retrying will not cause duplicate posting.

The notification stage is sent only after the posting number is confirmed. If the notification itself fails, it can be retried independently, because the posting fact is already established and the notification is merely a replayable side effect.

The key property of the entire pipeline is reflected in recovery: after a workflow restart, the persistent history is read and confirmed activities are not repeated. If the posting result is unknown—for example, the request was sent but no response was received—the correct approach is not to retry blindly, but first to query the financial system to confirm whether this invoice was actually posted, and then decide whether to retry or escalate to human investigation. Recovery means continuing from confirmed nodes, not directly re-executing unknown side effects.

The inputs of this pipeline are the invoice file, the versioned schema, validation rules, and the approval and financial APIs; the outputs are a posting number or a paused state with a clear reason. Hash-based deduplication blocks duplicate submissions, extraction failure retries only that node, human approval waits persistently, and posting timeout first checks the status; each step's failure is confined within observable, recoverable boundaries.

Any “at-least-once execution” semantics implies a corollary: the same activity may be executed multiple times. To understand this, first see how the one-shot success rate of a long chain decreases with the number of steps.

If three independent steps each have a one-shot success rate of 0.99, the one-shot success rate of the entire chain without retries is 0.99³ ≈ 97.03%; if the chain has ten steps, it drops to 0.99¹⁰ ≈ 90.44%. Each additional independent step makes the chain's success probability without retries the product of all step success rates. The more steps there are, the higher the probability that the chain as a whole fails. This is not because some step is not good enough; it is the inevitable result of multiplying probabilities.

Retries are a direct way to improve the final success rate: if a step fails and you retry, the final probability of success is significantly higher than a single attempt. But the cost of retries is that the same activity may be executed multiple times—you may never know whether the first call actually took effect. Thus, while availability improves, the risk of duplicate execution is also created. To retry safely, side effects must be idempotent: executing the same business intent repeatedly produces only one business effect.

The idempotency key K consists of three parts: workflow identifier, activity type, and logical attempt identifier, that is, K = workflowId ∥ activityType ∥ logicalAttempt. workflowId identifies a specific workflow instance, activityType identifies the kind of activity (for example, "posting to ledger" rather than "sending a notification"), and logicalAttempt identifies the same business intent — it remains the same when retrying the same intent and changes only when a new intent is attempted. The symbol ∥ denotes unambiguous concatenation encoding, not numeric addition. The key obtained by concatenating the three is stable and unique, and the business system uses it for deduplication.

This clarifies the true meaning of "exactly once". It is usually a business effect, not a network transmission guarantee: the network layer may still deliver the same request multiple times, but as long as the receiver uses a unique key, a deduplication table, and state checks so that duplicate requests produce only one payment or one write, the business effect is exactly once.

The input to this logic is the workflow ID, activity type, and logical attempt identifier; the output is a stable idempotency key and the guarantee that "duplicate requests produce only one business effect". Its boundaries must be clear: idempotency applies only to defined business keys; it does not mean the network transmits only once, nor can it fix incorrect business parameters — if the request itself carries the wrong amount or supplier, no matter how many retries or how thorough the deduplication, it will only cement the error. Idempotency solves the problem of "repeated execution of the same intent", not the problem of "the intent itself being wrong".

After a process crashes, how does the orchestrator know which steps have completed, which are halfway, and which haven't started? The answer is that it does not rely on memory but on persistent history. What the history records is not what was said in the session, but every scheduling, start, completion, failure, timer, and approval event—these are facts and decisions, not intermediate artifacts of reasoning.

The core role of persistent history is to turn “restart” into “replaying decisions,” rather than “redoing side effects.” During recovery, the orchestrator replays the control logic: it reads from history the state that has already occurred at each node, determines which edge to take next, which wait to wake, and which failure to retry or compensate. Activities that are already confirmed complete will not execute again, because their results have been written into history as facts. Activities that produce external side effects rely on idempotency keys and status queries to avoid redoing—retrying the same idempotency key causes the external system to recognize the effect only once; when the result is unknown, query first rather than blindly sending again.

This mechanism can be understood as a causal chain. The input is the events accumulated in persistent history: when an activity was scheduled, started, completed, failed, and when a human approval arrived or timed out. During recovery, the orchestrator reads these recorded results, reconstructs control decisions, and outputs replayable control flow. Business activities themselves carry idempotency keys and initiate status queries for uncertain external results, thereby ensuring that “replaying decisions” does not degenerate into “redoing side effects.”

This structure has a boundary that must be clearly drawn: history proves that the system recorded an event, but it does not automatically prove that the facts in external systems are true. The history may say “the posting call has been sent,” but that does not mean the finance system actually posted successfully; if the result is unknown, you still need to query the external system or escalate to human handling. Persistent history provides a reliable local sequence of facts and decisions, building recovery determinism on “what was recorded,” while the truth of the outside world must still be confirmed separately through idempotency and querying.

Retries are a means to improve availability, but they are effective for only one class of failure: transient failures. When a Large Language Model (LLM) continuously outputs invalid fields, exponential backoff cannot solve the problem—that is not network jitter, but rather the output itself not matching the schema. Applying retries to the wrong failure type only causes the same bad result to be repeatedly produced and discarded.

The boundary between transient and permanent failures is drawn by error type. Network timeouts, rate limiting, and brief unavailability are transient errors that can be retried: the problem usually disappears on its own within a short time, and retries have a genuine chance of success. Schema incompatibility, permission denial, and business validation failures are permanent errors; retries will not make them better. What is needed is to fix data, perform a migration, or transfer to manual handling. The criterion is whether this failure might disappear automatically on the next attempt. Only failures that can disappear are worth retrying.

Even when the failure type is suitable for retries, boundaries must be set. The maximum number of attempts, exponential backoff, and jitter together form a bounded policy: after each failure, the wait time grows exponentially, and random jitter is added to prevent many clients from retrying at the same moment; once the attempt count reaches the limit, the task enters a dead-letter queue or a manual queue rather than retrying indefinitely. The purpose of this boundary is to prevent retry storms from amplifying downstream failures—when the downstream is already overloaded, unrestrained retries only push more requests into it, turning a local failure into a full-scale avalanche.

Each retry must also keep the same logical activity ID and a different physical attempt number. The logical activity ID represents the same business intent and is used for deduplication; the physical attempt number distinguishes which specific physical execution it is and is used for diagnostics. Only by separating the two can you ensure that duplicate requests take effect only once in the business sense while precisely tracking which physical execution had a problem.

The inputs to a retry policy are the error type, the number of attempts already made, the backoff parameters, and the maximum time limit; the output is one of four dispositions: retry, enter dead-letter, transfer to manual handling, or stop. Network timeouts and rate limiting can have bounded retries; schema, permission, and business rule errors must be fixed. Exponential backoff cannot turn permanent errors into transient errors, nor can it allow retry storms to run unchecked—it only gives transient failures a finite and polite waiting rhythm.

A process spanning multiple external systems cannot be like a single-database transaction: 'either all succeed or all roll back.' If a hotel has already been booked but the subsequent payment fails, the workflow cannot pretend nothing happened—there is already a real booking in the hotel system. Long-running processes cannot lock multiple external systems for a long time waiting for an atomic commit, so cross-system consistency must be restored through compensation rather than database-style rollback.

The mechanism of compensation is: for each successful side effect that has already occurred, predefine a corresponding inverse action. The inverse of booking is canceling the booking; the inverse of occupying inventory is returning the inventory; the inverse of creating a draft is revoking the draft. When a later step fails, the workflow executes these inverse actions in order, using new business actions to pull the system back to a consistent state. Compensation itself is also an ordinary external call, so it can also fail—therefore compensation also needs idempotency, retries, and human escalation as a fallback. If a request to cancel a booking times out, you cannot assume the cancellation definitely succeeded; you must be able to resend it safely, and if necessary transfer to manual processing.

Here there is an unavoidable boundary: not all actions are reversible. An email that has already been sent cannot be taken back; you can only append a correction notice. Therefore, during design, you should first minimize irreversible steps and, as much as possible, place them later in the process—the later an irreversible action is executed, the fewer things actually need to be reversed when an earlier failure occurs, and the shorter and more reliable the compensation chain becomes.

This leads to a point that beginners most easily confuse: a workflow 'final completion' is not equivalent to database-style all-or-nothing. A process may stop in a state of partial completion, unknown side-effect outcome, or compensation failure; these states must be explicitly defined and included in a recoverable model, rather than pretending they do not exist.

The inputs of the compensation mechanism are completed side effects, subsequent failures, and the inverse actions predefined for each side effect; the output is one of three states: partial recovery, compensation failure, or requiring human escalation. It uses new business actions to restore cross-system consistency, not database atomic rollback; compensation itself can fail, and some actions are irreversible, so irreversible steps should be minimized and placed later. Only by thinking through these constraints can you maintain truly usable consistency in real-world cross-system processes.

The most insidious damage from deploying new code occurs in those old instances that were started long ago and are still running. Suppose the new version deletes a state in the workflow definition, and a workflow instance started yesterday is still waiting in that state. When it resumes or replays, the new code cannot find that state, so replay fails or takes the wrong branch. The core problem that version migration must solve is to allow running old instances to continue safely under the new code.

The approach has two layers. First, version both the workflow definition and activity schemas. Old instances either continue using compatible branches or undergo explicit migration to convert old states into new states—migration is an explicit, auditable state transition, not an implicit assumption in the new code that “the old state does not exist.” Second, deterministic replay requires that the same history produce the same control decisions. This means current time, random numbers, and results of external queries cannot directly participate in orchestration logic: if replay re-fetches the current time, yesterday’s and today’s decisions will differ; if replay re-calls an external query, returning a different value will lead the workflow down another path. These non-deterministic values should be written into history as activity results, so that replay reads the values that were fixed at the time rather than values generated on the spot.

Combining these two points makes replay truly reproducible: given the same persistent history, the orchestration logic produces the same set of control decisions whenever it replays. Sources of nondeterminism are isolated into the history record, while the orchestration logic itself remains purely deterministic.

Before release, also replay the new code against snapshots of production history: run the new version over histories that actually ran, observe whether control decisions match the old version, and catch incompatibilities during the canary phase instead of discovering only when production instances resume that the migration does not hold.

The inputs to version migration are the history of running instances, the workflow code, and the activity schema versions; the outputs are compatible branches or states after explicit migration. Historical snapshot replay can significantly reduce the risk of incompatibility, but it has a clear boundary: it cannot prove that all future external responses are compatible. Snapshots can only cover histories that have already occurred and cannot enumerate every result external systems may return in the future. Deterministic replay guarantees “consistent decisions for recorded history,” while responses from the future external world must still be handled at runtime through idempotency, queries, and compensation.

Human approval may wait three days or even longer. This wait cannot be implemented by "leaving the process hanging"; otherwise a deployment, a restart, or a scale-out will drop the waiting tasks. It also cannot hang indefinitely, otherwise the workflow will be permanently stuck. Human tasks must be modeled as persistent state that can time out and be escalated.

An approval node stores not just a "to-do" marker, but a complete set of fields: owner, qualification, evidence, SLA, reminders, and escalation path. Owner and qualification determine who the task should be assigned to and who is authorized to process it; evidence is the material the approver needs to make a judgment; SLA and reminders define when the task should be chased; the escalation path specifies who it is handed to after timeout and what process to follow. All of this information is persisted together, so the worker process can release resources after issuing the human request—it does not need to retain any memory or thread for this wait.

When an external approval event arrives, it carries a task token. The orchestrator uses this token to wake the corresponding workflow instance from persistent state and continue executing subsequent nodes. The wait is thus completely decoupled from the process lifecycle: the process can crash, restart, or scale out, and as long as the persistent state and task token remain, the workflow will not be lost.

Timeout is a path that must be explicitly handled in this design. Timeout can be escalated to a higher-level owner, can be rejected, can be canceled, but must never silently auto-approve—auto-approval amounts to secretly bypassing the intent of the human gate. Review results and reasons must enter history, ensuring that afterwards it is possible to audit "who made what decision, when, and on what grounds." Sensitive feedback is handled according to retention policy; just because it entered history does not mean all of its content is retained forever.

The inputs to a human task are owner, qualification, evidence, SLA, reminders, and escalation path; the output is one of four events: approve, reject, escalate on timeout, or cancel. Its two boundaries must both be maintained: persisting the wait does not mean the approval can hang indefinitely; timeout cannot silently auto-approve; sensitive reasons remain subject to retention policy. Persistence solves "not losing" the task; timeout and escalation solve "not getting stuck"; both are indispensable.

A single ideal path that runs from start to finish does not mean the system can go live. An ideal path only verifies that “it works when there is no failure,” but the value of an orchestration system emerges precisely when failures occur: can it recover after a crash, will duplicate delivery produce duplicate side effects, and what to do when compensation fails. Evaluation must actively inject these anomalies to prove that reliability is not luck.

The metrics in an evaluation report should be end-to-end success, node failures and retries, duplicate side effects, compensation success, unknown states, queue backlog, wait and execution times, cost per successful task, and human SLA. Together these metrics answer not just “can the process complete,” but also “where are failures contained, where does recovery continue from, and whether resources are out of control.”

The fault injection checklist corresponds to the mechanisms discussed in each preceding chapter: response loss after an activity completes, verifying whether external calls such as posting validation correctly query rather than blindly redo when the result is unknown; orchestrator restart, verifying whether persistent history allows recovery to continue from the correct node; duplicate external callbacks, verifying whether idempotent deduplication actually blocks duplicate side effects; old schema, verifying the compatibility of version migration for in-flight instances; compensation failure, verifying whether the human escalation path can handle it; queue full, verifying whether backpressure and boundaries stop a retry storm.

The means of verification is to inspect the event history. By comparing event sequences, confirm that recovery indeed continues from the correct node, rather than redoing activities that have already completed. Event history is the most direct source of evidence in this evaluation; it records what the system actually did, not what the design intent claims it would do.

Evaluation can also be used to compare architectural choices: place the “deterministic backbone plus LLM node” approach and the all-Agent approach under the same set of faults and compare quality, cost, and diagnosability. The advantage of a deterministic backbone is that it is replayable and locatable, while an all-Agent approach, under failures, is often harder to explain where it stopped and why it did not continue.

The inputs to a recovery evaluation are an ideal task and a set of faults—crash, packet loss, duplicate callbacks, old schema, compensation failure, and queue full; the outputs are observed values such as end-to-end success, duplicate side effects, compensation result, backlog, SLA, and recovery node. Be clear about its boundaries: passing one ideal path does not mean it is ready for production, and fault injection can only cover scenarios that have already been designed. It proves the system's resilience to known fault categories; it cannot guarantee that the same holds for new faults never imagined.

When restoring a workflow, calling the same model again may lead to a different branch. The reason is that the model call itself is nondeterministic: model version, temperature parameter, server-side version, and sampling randomness can all cause the same input to produce different outputs. If, during replay, it is regenerated once, the new output may differ from the output recorded in history, so the workflow will proceed along a path different from the previous one. Deterministic replay therefore fails at a Large Language Model (LLM) node—not because the orchestration logic is nondeterministic, but because the probabilistic result it depends on has been resampled.

The solution is to have the results of probabilistic activities go into history, rather than regenerating them during replay. After the first call completes, the orchestrator writes the input version, model version, output, and validation result together into persistent history. When replaying the control logic, it reads these recorded results and does not call the model again. Thus, “which branch was taken last time” is determined by history, not by a new round of random sampling; the same history necessarily yields the same control branch.

Only explicitly initiated new attempts produce new activity events. By an explicit new attempt, we mean that the workflow explicitly decides to “try again with new parameters or new inputs,” rather than unintentionally regenerating during replay. A new attempt produces a new result, while the old result remains for audit comparison—history shows what each attempt took as input and what it obtained.

For cacheable read-only activities, content hashing can be used to avoid repeated computation. But the cache key must include four types of elements: prompt, model, tool, and data version. If any one is missing, the cache will disguise stale answers as determinism: when the prompt changes, the model is upgraded, tool behavior changes, or data has been updated, the old cached answer still hits, and the workflow proceeds on an old fact that no longer holds. The value of caching is to save repeated read-only computation, not to erase the impact of input changes.

The inputs for replaying a probabilistic activity are the recorded model input version, model version, output, and validation result; the output is the same control branch under the same history. During recovery, read the old result rather than generating it again; only explicit new attempts create new events. Content-hash caching must also include prompt, model, tool, and data version in the key. By settling probabilistic results into historical facts, the deterministic skeleton can still hold in workflows that include model calls.

String the previously scattered mechanisms into a causal chain, and you can see how this concept starts from an original problem and ends up as verifiable engineering practice.

The starting point is to model tasks as dependencies and state transitions. This step determines whether the workflow is a DAG, a state machine, event-driven, or local Agent subprocesses, and also determines which component carries loops, waiting, and unknown paths. With a clear control structure in place, next define versioned input and output schemas for each node, so that the model's free-form text is converted into verifiable, migratable structures.

After the structure is fixed, annotate every side effect and equip it with an idempotency key and compensating action. Posting has an idempotency key, reservations have a cancellation action, and irreversible steps like email are minimized and deferred. This way, repeated execution produces only one business effect, and cross-system failures can be brought back to consistency with new business actions rather than pretending to be atomic rollback.

During execution, the orchestrator durably records every decision—scheduling, start, completion, failure, timing, and approval events are all recorded in history. Workers are responsible for executing specific LLM or business activities: the results of probabilistic activities are written back to history, and during replay old results are read rather than regenerated. On failure, handle based on error type: transient errors get bounded retries, permanent errors are fixed or escalated, and those exceeding thresholds go to a dead-letter or manual queue. Human and external events are modeled as durable waits, resumed with task tokens after events arrive, not occupying worker processes nor hanging indefinitely.

Finally, there is the verification stage: fault recovery, history replay, and version migration. By injecting crashes, packet loss, duplicate callbacks, old schemas, compensation failures, and queue saturation, inspect the event history and confirm that recovery indeed continues from the correct node, and that new code replaying old history yields consistent decisions. The entire chain is thus closed—from the graphical representation of tasks, to schemas and idempotency, to durable history and replay, to repeatable failure evaluation, each link builds on the previous one, and together they answer the same question: how to reliably support probabilistic models with durable state and deterministic control.