From 14c806d38faa3bf3e065b9d38692c92a2d78d74d Mon Sep 17 00:00:00 2001 From: Hare Date: Wed, 2 Sep 2026 14:43:49 +0900 Subject: [PATCH] docs: define durable operation boundaries --- docs/README.md | 35 ++-- docs/design/durable-operations.md | 288 ++++++++++++++++++++++++++++++ 2 files changed, 306 insertions(+), 17 deletions(-) create mode 100644 docs/design/durable-operations.md diff --git a/docs/README.md b/docs/README.md index d02c1065..5ba0e732 100644 --- a/docs/README.md +++ b/docs/README.md @@ -7,23 +7,24 @@ It is not a dumping ground for external research, old plans, API inventories, or ## Reading order 1. [`design/overview.md`](design/overview.md) — the system map. -2. [`design/context-history.md`](design/context-history.md) — the highest-risk invariant: inputs that affect the model must be committed to history before they enter context. -3. [`design/worker-session-state.md`](design/worker-session-state.md) — Worker identity, replayable session logs, current metadata, and live process hints. -4. [`design/session-observation.md`](design/session-observation.md) — common session captures, `SessionEntryRef`, Memory evidence, and host-authorized Worker observation. -5. [`design/flow-state-graph.md`](design/flow-state-graph.md) — Workspace Flow sources, immutable revisions, transition attempts, and bounded internal verification. -6. [`design/profiles-manifests-prompts.md`](design/profiles-manifests-prompts.md) — reusable Profiles, resolved Manifests, and prompt resources. -7. [`design/tool-permissions-scope.md`](design/tool-permissions-scope.md) — tool policy and filesystem scope. -8. [`design/plugin-packages.md`](design/plugin-packages.md) — plugin package distribution, discovery, and enablement boundaries. -9. [`development/plugin-development.md`](development/plugin-development.md) — how to build, package, enable, and inspect Yoi Plugins. -10. [`design/memory-knowledge.md`](design/memory-knowledge.md) — generated memory and audit records. -11. [`design/workspace-kanban-orchestrator-runtime.md`](design/workspace-kanban-orchestrator-runtime.md) — how Kanban operations become durable orchestration events and backend-internal routing decisions. -12. [`design/workspace-runtime-docker.md`](design/workspace-runtime-docker.md) — the WebUI / Backend / Runtime split, Docker image layout, worker launch path, and workdir materialization boundary. -13. [`development/server-runtime-auth.md`](development/server-runtime-auth.md) — manual Workspace Server / Runtime public-key exchange and authenticated Runtime startup checks. -14. [`development/work-items.md`](development/work-items.md) — how project work is recorded and reviewed. -15. [`development/rust-testing-strategy.md`](development/rust-testing-strategy.md) — what Yoi Rust tests should prove, where they belong, and how to name them. -16. [`development/validation.md`](development/validation.md) — how to check changes. -17. [`development/workspace-schema-migrations.md`](development/workspace-schema-migrations.md) — how to preflight, apply, verify, and roll back control-plane SQLite schema changes. -18. [`design/standalone-agent-host.md`](design/standalone-agent-host.md) — in-process standalone Worker host の依存方向、authority、lifecycle、非目標。 +2. [`design/durable-operations.md`](design/durable-operations.md) — cross-domain operation identity, checkpoints, retries, child operations, and disposition. +3. [`design/context-history.md`](design/context-history.md) — the highest-risk invariant: inputs that affect the model must be committed to history before they enter context. +4. [`design/worker-session-state.md`](design/worker-session-state.md) — Worker identity, replayable session logs, current metadata, and live process hints. +5. [`design/session-observation.md`](design/session-observation.md) — common session captures, `SessionEntryRef`, Memory evidence, and host-authorized Worker observation. +6. [`design/flow-state-graph.md`](design/flow-state-graph.md) — Workspace Flow sources, immutable revisions, transition attempts, and bounded internal verification. +7. [`design/profiles-manifests-prompts.md`](design/profiles-manifests-prompts.md) — reusable Profiles, resolved Manifests, and prompt resources. +8. [`design/tool-permissions-scope.md`](design/tool-permissions-scope.md) — tool policy and filesystem scope. +9. [`design/plugin-packages.md`](design/plugin-packages.md) — plugin package distribution, discovery, and enablement boundaries. +10. [`development/plugin-development.md`](development/plugin-development.md) — how to build, package, enable, and inspect Yoi Plugins. +11. [`design/memory-knowledge.md`](design/memory-knowledge.md) — generated memory and audit records. +12. [`design/workspace-kanban-orchestrator-runtime.md`](design/workspace-kanban-orchestrator-runtime.md) — how Kanban operations become durable orchestration events and backend-internal routing decisions. +13. [`design/workspace-runtime-docker.md`](design/workspace-runtime-docker.md) — the WebUI / Backend / Runtime split, Docker image layout, worker launch path, and workdir materialization boundary. +14. [`development/server-runtime-auth.md`](development/server-runtime-auth.md) — manual Workspace Server / Runtime public-key exchange and authenticated Runtime startup checks. +15. [`development/work-items.md`](development/work-items.md) — how project work is recorded and reviewed. +16. [`development/rust-testing-strategy.md`](development/rust-testing-strategy.md) — what Yoi Rust tests should prove, where they belong, and how to name them. +17. [`development/validation.md`](development/validation.md) — how to check changes. +18. [`development/workspace-schema-migrations.md`](development/workspace-schema-migrations.md) — how to preflight, apply, verify, and roll back control-plane SQLite schema changes. +19. [`design/standalone-agent-host.md`](design/standalone-agent-host.md) — in-process standalone Worker host の依存方向、authority、lifecycle、非目標。 ## What belongs here diff --git a/docs/design/durable-operations.md b/docs/design/durable-operations.md new file mode 100644 index 00000000..4915f4b5 --- /dev/null +++ b/docs/design/durable-operations.md @@ -0,0 +1,288 @@ +# Durable operation design + +Durable operation records make retries converge on the same authorized intent +and preserve the facts needed to explain an externally visible result. They are +not execution traces and must not mirror every Rust function or implementation +step as persisted state. + +This document defines the cross-domain rules for operation identity, state, +checkpoints, child operations, failure evidence, and terminal disposition. +Domain code may use different records where its atomicity boundary differs, but +it should classify the operation before choosing a schema. + +## Core rule + +Persist authority and non-reconstructable facts, not control flow. + +A value belongs in durable operation state only when at least one of the +following is true: + +- it identifies the caller's stable intent and detects conflicting reuse; +- it freezes authority or configuration that an exact retry must continue to + use; +- it binds a preallocated or created resource to the operation; +- it records an externally visible side effect that cannot be safely derived or + repeated; +- it records the final domain result or disposition; +- it provides bounded evidence needed for retry, reconciliation, or audit. + +A local step does not become durable merely because it occurs before or after +another function call. If current authority can be reread or the step can be +repeated safely, derive or repeat it instead of adding a stage. + +## Resilience boundary + +These rules cover normal product retry and recovery boundaries: duplicate +requests, returned errors, timeouts, known partial-completion outcomes, process +restart from the last committed facts, and retries of provider operations with +an explicit idempotency or observation contract. + +They do not require the system to survive an unexpected stop at every point +while Rust code is executing. A panic, abort, process kill, machine loss, or +power failure may occur between an external side effect and its next durable +checkpoint. Yoi does not attempt to close every such instruction-level crash +window by writing a stage before and after every `await`, function, or provider +call. + +Consequently: + +- do not claim general exactly-once execution; +- do not introduce write-ahead stages solely to model arbitrary Rust + control-flow interruption; +- rely on SQLite transaction atomicity for work inside one database transaction; +- prefer provider idempotency, compare-and-swap, stable resource identity, and + authoritative observation for external effects; +- when an uncovered crash window cannot be reconciled automatically, retain the + last committed facts and surface an `unknown` or attention-required + disposition rather than guessing that the side effect did or did not happen. + +A domain may require a stronger crash-consistency contract for a specific +destructive or security-sensitive effect. That requirement must be explicit and +must define the provider protocol, checkpoint ordering, replay behavior, and +reconciliation evidence. It is not implied by calling a record a durable +operation. + +## Classify the record before adding state + +Not every record containing an `operation_id` is a state machine. Use one of the +following shapes. + +### Atomic idempotency ledger + +Use an idempotency ledger when all authoritative mutations and result recording +commit in one database transaction. + +The record normally contains: + +- Workspace and operation identity; +- a fingerprint of stable caller intent; +- the created resource or result identity; +- the committed revision and timestamp where relevant. + +It does not need `pending`, `executing`, or intermediate stages. An exact retry +returns the recorded result. Reusing the same operation identity with a +different fingerprint fails. + +Repository secret mutation results, Workspace resource creation results, and +transactionally appended domain events are examples of this shape. + +### Reservation + +Use a reservation when an identity or exclusive right must exist before a later +binding can complete. + +Persist factual transitions such as: + +- the reserved resource identity; +- the immutable request fingerprint and authority snapshot; +- the concrete resource or assignment bound to the reservation; +- reservation expiry or release evidence when the contract requires it. + +Do not model internal dispatch, validation, construction, or callback steps as +reservation states. A nullable result binding or a small `reserved | created` +state can be sufficient when those values correspond to real authority facts. + +### Durable side-effect operation + +Use a durable side-effect operation when work crosses a database/provider +boundary and a retry needs durable intent or result evidence. + +The default lifecycle is deliberately small: + +```text +pending -> completed +pending -> failed +failed -> pending # only when the domain explicitly permits retry +``` + +Existing code may use `succeeded` for the successful terminal value; new naming +should prefer `completed`. Do not rewrite applied migrations or historical audit +text only to normalize that word. + +The operation should contain: + +- stable operation identity and request fingerprint; +- immutable resolved authority needed by an exact retry; +- preallocated resource identity where it prevents duplicate creation; +- only the necessary irreversible checkpoints; +- bounded failure evidence; +- the final result and domain disposition. + +`pending` means that the intent remains open and current authority must be +reread before progress. It does not identify which Rust function should execute +next. `failed` records the latest terminal attempt outcome; retryability is an +explicit domain rule, not something inferred from the word. `completed` means +the operation's required result and evidence are durably committed. + +### Parent workflow + +A parent workflow coordinates domain operations but does not duplicate their +lifecycle. + +Persist: + +- the parent intent and fencing authority; +- stable child operation identities; +- the final workflow result or disposition; +- bounded attention or decision evidence. + +Read child state from the child authority. Do not copy child states, provider +stages, Worker status, attachment status, or Workdir status into a second parent +state machine. A parent cleanup workflow will often need only +`pending | completed`; child failure remains on the child operation and appears +in the parent as current attention metadata. + +Creating or binding a child must itself be idempotent. Prefer a deterministic +child operation identity or persist the child reference atomically with the +parent decision so a retry cannot create siblings for one intent. + +## Checkpoint rules + +A checkpoint records a fact that changes retry semantics. It is not a progress +notification. + +Add a checkpoint only when all of the following hold: + +1. A side effect may already have occurred outside the current transaction. +2. Current authority cannot derive the fact reliably enough for safe retry, or + repeating the effect is not safe under the provider contract. +3. The retry algorithm changes after the fact is committed. +4. Tests can exercise behavior before and after the checkpoint. + +Prefer factual fields over stage names: + +- `provider_deleted_at` is evidence that provider deletion succeeded; +- `child_operation_id` binds delegated work; +- `result_revision` identifies the committed result; +- `target_ref_after` records verified merge evidence. + +Avoid fields such as `validating`, `closing_session`, `detaching`, +`deleting_registry`, or `finalizing`. Those names describe code location, not +durable authority. If those steps are safe to rerun or their result can be read +from Worker, attachment, Workdir, repository, or provider authority, they are +not checkpoints. + +A checkpoint must never claim more than the authority that produced it. For +example, sending a provider request is not proof that provider deletion +completed, and receiving a Worker notification is not proof that a Ticket or +cleanup workflow completed. + +## State, failure, blockers, and disposition are separate + +Do not overload one enum with unrelated dimensions. + +- **Operation state** says whether the intent is open, completed, or has a + recorded failed attempt. +- **Failure evidence** records a bounded category, timestamp, and safe + diagnostic detail for the latest failure. +- **Blockers and eligibility** are normally derived by rereading current + authority. Persist them only as audit or attention evidence, not as a + substitute for live validation. +- **Disposition** records what the domain decided to retain, delete, release, + tombstone, abandon, or leave unknown. +- **Attention metadata** explains why automated progress currently cannot + continue and what authority must change. + +Values such as `blocked`, `executing`, `stale`, `dirty`, `retained`, and +`deleted` therefore do not all belong in one operation-state enum. Some are +derived conditions, some describe transient execution, and some are domain +results. + +Before every retry or side effect, reread live authority and revalidate its +fence. A previously recorded blocker does not prove that the operation remains +blocked, and a previously unblocked operation does not retain permission after +assignment, ownership, revision, or attachment authority changes. + +## Identity and fingerprinting + +Every externally retryable operation has a stable identity in its owning +Workspace or authority scope. The operation fingerprint represents stable caller +intent, not generated results or mutable observations. + +Include inputs whose change would mean a different requested operation. Exclude: + +- generated resource IDs when the Server allocates and persists them as the + result; +- timestamps assigned by the Server; +- retry counters and diagnostics; +- current provider observations that are expected to change; +- secret bytes and credential material. + +Resolved authority snapshots may be stored separately from the caller +fingerprint. An exact retry uses the persisted snapshot where replay convergence +requires it; a new operation resolves current authority. Unknown, foreign, or +conflicting operation identity fails closed. + +## Transactions and external providers + +Keep database work in one transaction whenever the owning authority and result +live in the same database. Do not create a durable operation merely to split a +transaction that can remain atomic. + +When an external provider is involved: + +1. reserve stable intent and identity if retry needs them; +2. invoke the provider with the strongest available idempotency, expected-old + revision, or stable resource key; +3. verify the provider result through authoritative response or observation; +4. commit only the checkpoint or result evidence that changes retry behavior; +5. on retry, reread both the operation and current domain/provider authority + before acting. + +Compensation is a domain operation, not an invisible `finally` block. If +compensation has its own external side effects or retry lifecycle, give it a +stable child operation identity rather than expanding the parent into a list of +cleanup stages. + +## Diagnostics and audit + +Persist bounded error categories and identifiers needed to investigate or retry. +Do not persist credentials, provider handles, raw command output, raw prompts, +full session transcripts, or host paths in ordinary operation diagnostics. + +Attempt counts, last-attempt timestamps, and safe provider categories may help +operations, but they are telemetry and evidence rather than lifecycle authority. +Logs may describe detailed execution stages; the durable record should remain +centered on intent, checkpoints, result, and disposition. + +## Applying this rule + +For a new or materially changed operation: + +1. identify the owning authority and transaction boundary; +2. classify it as an atomic ledger, reservation, durable side-effect operation, + or parent workflow; +3. define stable identity, fingerprint, and exact-retry behavior; +4. list external side effects and decide which are idempotent or authoritatively + observable; +5. add only checkpoints that change retry behavior; +6. keep child operation state in the child authority; +7. separate failure, blocker, attention, and disposition from lifecycle state; +8. state the unsupported crash windows honestly; +9. test fingerprint conflict, exact retry, authority revalidation, checkpoint + replay, and result/disposition projection as applicable. + +Existing operation schemas need not be rewritten solely for vocabulary +consistency. When an operation is changed for functional reasons, use this +classification to remove derived or control-flow stages rather than adding +another special-case lifecycle.