docs: define durable operation boundaries
This commit is contained in:
+18
-17
@@ -7,23 +7,24 @@ It is not a dumping ground for external research, old plans, API inventories, or
|
|||||||
## Reading order
|
## Reading order
|
||||||
|
|
||||||
1. [`design/overview.md`](design/overview.md) — the system map.
|
1. [`design/overview.md`](design/overview.md) — the system map.
|
||||||
2. [`design/context-history.md`](design/context-history.md) — the highest-risk invariant: inputs that affect the model must be committed to history before they enter context.
|
2. [`design/durable-operations.md`](design/durable-operations.md) — cross-domain operation identity, checkpoints, retries, child operations, and disposition.
|
||||||
3. [`design/worker-session-state.md`](design/worker-session-state.md) — Worker identity, replayable session logs, current metadata, and live process hints.
|
3. [`design/context-history.md`](design/context-history.md) — the highest-risk invariant: inputs that affect the model must be committed to history before they enter context.
|
||||||
4. [`design/session-observation.md`](design/session-observation.md) — common session captures, `SessionEntryRef`, Memory evidence, and host-authorized Worker observation.
|
4. [`design/worker-session-state.md`](design/worker-session-state.md) — Worker identity, replayable session logs, current metadata, and live process hints.
|
||||||
5. [`design/flow-state-graph.md`](design/flow-state-graph.md) — Workspace Flow sources, immutable revisions, transition attempts, and bounded internal verification.
|
5. [`design/session-observation.md`](design/session-observation.md) — common session captures, `SessionEntryRef`, Memory evidence, and host-authorized Worker observation.
|
||||||
6. [`design/profiles-manifests-prompts.md`](design/profiles-manifests-prompts.md) — reusable Profiles, resolved Manifests, and prompt resources.
|
6. [`design/flow-state-graph.md`](design/flow-state-graph.md) — Workspace Flow sources, immutable revisions, transition attempts, and bounded internal verification.
|
||||||
7. [`design/tool-permissions-scope.md`](design/tool-permissions-scope.md) — tool policy and filesystem scope.
|
7. [`design/profiles-manifests-prompts.md`](design/profiles-manifests-prompts.md) — reusable Profiles, resolved Manifests, and prompt resources.
|
||||||
8. [`design/plugin-packages.md`](design/plugin-packages.md) — plugin package distribution, discovery, and enablement boundaries.
|
8. [`design/tool-permissions-scope.md`](design/tool-permissions-scope.md) — tool policy and filesystem scope.
|
||||||
9. [`development/plugin-development.md`](development/plugin-development.md) — how to build, package, enable, and inspect Yoi Plugins.
|
9. [`design/plugin-packages.md`](design/plugin-packages.md) — plugin package distribution, discovery, and enablement boundaries.
|
||||||
10. [`design/memory-knowledge.md`](design/memory-knowledge.md) — generated memory and audit records.
|
10. [`development/plugin-development.md`](development/plugin-development.md) — how to build, package, enable, and inspect Yoi Plugins.
|
||||||
11. [`design/workspace-kanban-orchestrator-runtime.md`](design/workspace-kanban-orchestrator-runtime.md) — how Kanban operations become durable orchestration events and backend-internal routing decisions.
|
11. [`design/memory-knowledge.md`](design/memory-knowledge.md) — generated memory and audit records.
|
||||||
12. [`design/workspace-runtime-docker.md`](design/workspace-runtime-docker.md) — the WebUI / Backend / Runtime split, Docker image layout, worker launch path, and workdir materialization boundary.
|
12. [`design/workspace-kanban-orchestrator-runtime.md`](design/workspace-kanban-orchestrator-runtime.md) — how Kanban operations become durable orchestration events and backend-internal routing decisions.
|
||||||
13. [`development/server-runtime-auth.md`](development/server-runtime-auth.md) — manual Workspace Server / Runtime public-key exchange and authenticated Runtime startup checks.
|
13. [`design/workspace-runtime-docker.md`](design/workspace-runtime-docker.md) — the WebUI / Backend / Runtime split, Docker image layout, worker launch path, and workdir materialization boundary.
|
||||||
14. [`development/work-items.md`](development/work-items.md) — how project work is recorded and reviewed.
|
14. [`development/server-runtime-auth.md`](development/server-runtime-auth.md) — manual Workspace Server / Runtime public-key exchange and authenticated Runtime startup checks.
|
||||||
15. [`development/rust-testing-strategy.md`](development/rust-testing-strategy.md) — what Yoi Rust tests should prove, where they belong, and how to name them.
|
15. [`development/work-items.md`](development/work-items.md) — how project work is recorded and reviewed.
|
||||||
16. [`development/validation.md`](development/validation.md) — how to check changes.
|
16. [`development/rust-testing-strategy.md`](development/rust-testing-strategy.md) — what Yoi Rust tests should prove, where they belong, and how to name them.
|
||||||
17. [`development/workspace-schema-migrations.md`](development/workspace-schema-migrations.md) — how to preflight, apply, verify, and roll back control-plane SQLite schema changes.
|
17. [`development/validation.md`](development/validation.md) — how to check changes.
|
||||||
18. [`design/standalone-agent-host.md`](design/standalone-agent-host.md) — in-process standalone Worker host の依存方向、authority、lifecycle、非目標。
|
18. [`development/workspace-schema-migrations.md`](development/workspace-schema-migrations.md) — how to preflight, apply, verify, and roll back control-plane SQLite schema changes.
|
||||||
|
19. [`design/standalone-agent-host.md`](design/standalone-agent-host.md) — in-process standalone Worker host の依存方向、authority、lifecycle、非目標。
|
||||||
|
|
||||||
## What belongs here
|
## What belongs here
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,288 @@
|
|||||||
|
# Durable operation design
|
||||||
|
|
||||||
|
Durable operation records make retries converge on the same authorized intent
|
||||||
|
and preserve the facts needed to explain an externally visible result. They are
|
||||||
|
not execution traces and must not mirror every Rust function or implementation
|
||||||
|
step as persisted state.
|
||||||
|
|
||||||
|
This document defines the cross-domain rules for operation identity, state,
|
||||||
|
checkpoints, child operations, failure evidence, and terminal disposition.
|
||||||
|
Domain code may use different records where its atomicity boundary differs, but
|
||||||
|
it should classify the operation before choosing a schema.
|
||||||
|
|
||||||
|
## Core rule
|
||||||
|
|
||||||
|
Persist authority and non-reconstructable facts, not control flow.
|
||||||
|
|
||||||
|
A value belongs in durable operation state only when at least one of the
|
||||||
|
following is true:
|
||||||
|
|
||||||
|
- it identifies the caller's stable intent and detects conflicting reuse;
|
||||||
|
- it freezes authority or configuration that an exact retry must continue to
|
||||||
|
use;
|
||||||
|
- it binds a preallocated or created resource to the operation;
|
||||||
|
- it records an externally visible side effect that cannot be safely derived or
|
||||||
|
repeated;
|
||||||
|
- it records the final domain result or disposition;
|
||||||
|
- it provides bounded evidence needed for retry, reconciliation, or audit.
|
||||||
|
|
||||||
|
A local step does not become durable merely because it occurs before or after
|
||||||
|
another function call. If current authority can be reread or the step can be
|
||||||
|
repeated safely, derive or repeat it instead of adding a stage.
|
||||||
|
|
||||||
|
## Resilience boundary
|
||||||
|
|
||||||
|
These rules cover normal product retry and recovery boundaries: duplicate
|
||||||
|
requests, returned errors, timeouts, known partial-completion outcomes, process
|
||||||
|
restart from the last committed facts, and retries of provider operations with
|
||||||
|
an explicit idempotency or observation contract.
|
||||||
|
|
||||||
|
They do not require the system to survive an unexpected stop at every point
|
||||||
|
while Rust code is executing. A panic, abort, process kill, machine loss, or
|
||||||
|
power failure may occur between an external side effect and its next durable
|
||||||
|
checkpoint. Yoi does not attempt to close every such instruction-level crash
|
||||||
|
window by writing a stage before and after every `await`, function, or provider
|
||||||
|
call.
|
||||||
|
|
||||||
|
Consequently:
|
||||||
|
|
||||||
|
- do not claim general exactly-once execution;
|
||||||
|
- do not introduce write-ahead stages solely to model arbitrary Rust
|
||||||
|
control-flow interruption;
|
||||||
|
- rely on SQLite transaction atomicity for work inside one database transaction;
|
||||||
|
- prefer provider idempotency, compare-and-swap, stable resource identity, and
|
||||||
|
authoritative observation for external effects;
|
||||||
|
- when an uncovered crash window cannot be reconciled automatically, retain the
|
||||||
|
last committed facts and surface an `unknown` or attention-required
|
||||||
|
disposition rather than guessing that the side effect did or did not happen.
|
||||||
|
|
||||||
|
A domain may require a stronger crash-consistency contract for a specific
|
||||||
|
destructive or security-sensitive effect. That requirement must be explicit and
|
||||||
|
must define the provider protocol, checkpoint ordering, replay behavior, and
|
||||||
|
reconciliation evidence. It is not implied by calling a record a durable
|
||||||
|
operation.
|
||||||
|
|
||||||
|
## Classify the record before adding state
|
||||||
|
|
||||||
|
Not every record containing an `operation_id` is a state machine. Use one of the
|
||||||
|
following shapes.
|
||||||
|
|
||||||
|
### Atomic idempotency ledger
|
||||||
|
|
||||||
|
Use an idempotency ledger when all authoritative mutations and result recording
|
||||||
|
commit in one database transaction.
|
||||||
|
|
||||||
|
The record normally contains:
|
||||||
|
|
||||||
|
- Workspace and operation identity;
|
||||||
|
- a fingerprint of stable caller intent;
|
||||||
|
- the created resource or result identity;
|
||||||
|
- the committed revision and timestamp where relevant.
|
||||||
|
|
||||||
|
It does not need `pending`, `executing`, or intermediate stages. An exact retry
|
||||||
|
returns the recorded result. Reusing the same operation identity with a
|
||||||
|
different fingerprint fails.
|
||||||
|
|
||||||
|
Repository secret mutation results, Workspace resource creation results, and
|
||||||
|
transactionally appended domain events are examples of this shape.
|
||||||
|
|
||||||
|
### Reservation
|
||||||
|
|
||||||
|
Use a reservation when an identity or exclusive right must exist before a later
|
||||||
|
binding can complete.
|
||||||
|
|
||||||
|
Persist factual transitions such as:
|
||||||
|
|
||||||
|
- the reserved resource identity;
|
||||||
|
- the immutable request fingerprint and authority snapshot;
|
||||||
|
- the concrete resource or assignment bound to the reservation;
|
||||||
|
- reservation expiry or release evidence when the contract requires it.
|
||||||
|
|
||||||
|
Do not model internal dispatch, validation, construction, or callback steps as
|
||||||
|
reservation states. A nullable result binding or a small `reserved | created`
|
||||||
|
state can be sufficient when those values correspond to real authority facts.
|
||||||
|
|
||||||
|
### Durable side-effect operation
|
||||||
|
|
||||||
|
Use a durable side-effect operation when work crosses a database/provider
|
||||||
|
boundary and a retry needs durable intent or result evidence.
|
||||||
|
|
||||||
|
The default lifecycle is deliberately small:
|
||||||
|
|
||||||
|
```text
|
||||||
|
pending -> completed
|
||||||
|
pending -> failed
|
||||||
|
failed -> pending # only when the domain explicitly permits retry
|
||||||
|
```
|
||||||
|
|
||||||
|
Existing code may use `succeeded` for the successful terminal value; new naming
|
||||||
|
should prefer `completed`. Do not rewrite applied migrations or historical audit
|
||||||
|
text only to normalize that word.
|
||||||
|
|
||||||
|
The operation should contain:
|
||||||
|
|
||||||
|
- stable operation identity and request fingerprint;
|
||||||
|
- immutable resolved authority needed by an exact retry;
|
||||||
|
- preallocated resource identity where it prevents duplicate creation;
|
||||||
|
- only the necessary irreversible checkpoints;
|
||||||
|
- bounded failure evidence;
|
||||||
|
- the final result and domain disposition.
|
||||||
|
|
||||||
|
`pending` means that the intent remains open and current authority must be
|
||||||
|
reread before progress. It does not identify which Rust function should execute
|
||||||
|
next. `failed` records the latest terminal attempt outcome; retryability is an
|
||||||
|
explicit domain rule, not something inferred from the word. `completed` means
|
||||||
|
the operation's required result and evidence are durably committed.
|
||||||
|
|
||||||
|
### Parent workflow
|
||||||
|
|
||||||
|
A parent workflow coordinates domain operations but does not duplicate their
|
||||||
|
lifecycle.
|
||||||
|
|
||||||
|
Persist:
|
||||||
|
|
||||||
|
- the parent intent and fencing authority;
|
||||||
|
- stable child operation identities;
|
||||||
|
- the final workflow result or disposition;
|
||||||
|
- bounded attention or decision evidence.
|
||||||
|
|
||||||
|
Read child state from the child authority. Do not copy child states, provider
|
||||||
|
stages, Worker status, attachment status, or Workdir status into a second parent
|
||||||
|
state machine. A parent cleanup workflow will often need only
|
||||||
|
`pending | completed`; child failure remains on the child operation and appears
|
||||||
|
in the parent as current attention metadata.
|
||||||
|
|
||||||
|
Creating or binding a child must itself be idempotent. Prefer a deterministic
|
||||||
|
child operation identity or persist the child reference atomically with the
|
||||||
|
parent decision so a retry cannot create siblings for one intent.
|
||||||
|
|
||||||
|
## Checkpoint rules
|
||||||
|
|
||||||
|
A checkpoint records a fact that changes retry semantics. It is not a progress
|
||||||
|
notification.
|
||||||
|
|
||||||
|
Add a checkpoint only when all of the following hold:
|
||||||
|
|
||||||
|
1. A side effect may already have occurred outside the current transaction.
|
||||||
|
2. Current authority cannot derive the fact reliably enough for safe retry, or
|
||||||
|
repeating the effect is not safe under the provider contract.
|
||||||
|
3. The retry algorithm changes after the fact is committed.
|
||||||
|
4. Tests can exercise behavior before and after the checkpoint.
|
||||||
|
|
||||||
|
Prefer factual fields over stage names:
|
||||||
|
|
||||||
|
- `provider_deleted_at` is evidence that provider deletion succeeded;
|
||||||
|
- `child_operation_id` binds delegated work;
|
||||||
|
- `result_revision` identifies the committed result;
|
||||||
|
- `target_ref_after` records verified merge evidence.
|
||||||
|
|
||||||
|
Avoid fields such as `validating`, `closing_session`, `detaching`,
|
||||||
|
`deleting_registry`, or `finalizing`. Those names describe code location, not
|
||||||
|
durable authority. If those steps are safe to rerun or their result can be read
|
||||||
|
from Worker, attachment, Workdir, repository, or provider authority, they are
|
||||||
|
not checkpoints.
|
||||||
|
|
||||||
|
A checkpoint must never claim more than the authority that produced it. For
|
||||||
|
example, sending a provider request is not proof that provider deletion
|
||||||
|
completed, and receiving a Worker notification is not proof that a Ticket or
|
||||||
|
cleanup workflow completed.
|
||||||
|
|
||||||
|
## State, failure, blockers, and disposition are separate
|
||||||
|
|
||||||
|
Do not overload one enum with unrelated dimensions.
|
||||||
|
|
||||||
|
- **Operation state** says whether the intent is open, completed, or has a
|
||||||
|
recorded failed attempt.
|
||||||
|
- **Failure evidence** records a bounded category, timestamp, and safe
|
||||||
|
diagnostic detail for the latest failure.
|
||||||
|
- **Blockers and eligibility** are normally derived by rereading current
|
||||||
|
authority. Persist them only as audit or attention evidence, not as a
|
||||||
|
substitute for live validation.
|
||||||
|
- **Disposition** records what the domain decided to retain, delete, release,
|
||||||
|
tombstone, abandon, or leave unknown.
|
||||||
|
- **Attention metadata** explains why automated progress currently cannot
|
||||||
|
continue and what authority must change.
|
||||||
|
|
||||||
|
Values such as `blocked`, `executing`, `stale`, `dirty`, `retained`, and
|
||||||
|
`deleted` therefore do not all belong in one operation-state enum. Some are
|
||||||
|
derived conditions, some describe transient execution, and some are domain
|
||||||
|
results.
|
||||||
|
|
||||||
|
Before every retry or side effect, reread live authority and revalidate its
|
||||||
|
fence. A previously recorded blocker does not prove that the operation remains
|
||||||
|
blocked, and a previously unblocked operation does not retain permission after
|
||||||
|
assignment, ownership, revision, or attachment authority changes.
|
||||||
|
|
||||||
|
## Identity and fingerprinting
|
||||||
|
|
||||||
|
Every externally retryable operation has a stable identity in its owning
|
||||||
|
Workspace or authority scope. The operation fingerprint represents stable caller
|
||||||
|
intent, not generated results or mutable observations.
|
||||||
|
|
||||||
|
Include inputs whose change would mean a different requested operation. Exclude:
|
||||||
|
|
||||||
|
- generated resource IDs when the Server allocates and persists them as the
|
||||||
|
result;
|
||||||
|
- timestamps assigned by the Server;
|
||||||
|
- retry counters and diagnostics;
|
||||||
|
- current provider observations that are expected to change;
|
||||||
|
- secret bytes and credential material.
|
||||||
|
|
||||||
|
Resolved authority snapshots may be stored separately from the caller
|
||||||
|
fingerprint. An exact retry uses the persisted snapshot where replay convergence
|
||||||
|
requires it; a new operation resolves current authority. Unknown, foreign, or
|
||||||
|
conflicting operation identity fails closed.
|
||||||
|
|
||||||
|
## Transactions and external providers
|
||||||
|
|
||||||
|
Keep database work in one transaction whenever the owning authority and result
|
||||||
|
live in the same database. Do not create a durable operation merely to split a
|
||||||
|
transaction that can remain atomic.
|
||||||
|
|
||||||
|
When an external provider is involved:
|
||||||
|
|
||||||
|
1. reserve stable intent and identity if retry needs them;
|
||||||
|
2. invoke the provider with the strongest available idempotency, expected-old
|
||||||
|
revision, or stable resource key;
|
||||||
|
3. verify the provider result through authoritative response or observation;
|
||||||
|
4. commit only the checkpoint or result evidence that changes retry behavior;
|
||||||
|
5. on retry, reread both the operation and current domain/provider authority
|
||||||
|
before acting.
|
||||||
|
|
||||||
|
Compensation is a domain operation, not an invisible `finally` block. If
|
||||||
|
compensation has its own external side effects or retry lifecycle, give it a
|
||||||
|
stable child operation identity rather than expanding the parent into a list of
|
||||||
|
cleanup stages.
|
||||||
|
|
||||||
|
## Diagnostics and audit
|
||||||
|
|
||||||
|
Persist bounded error categories and identifiers needed to investigate or retry.
|
||||||
|
Do not persist credentials, provider handles, raw command output, raw prompts,
|
||||||
|
full session transcripts, or host paths in ordinary operation diagnostics.
|
||||||
|
|
||||||
|
Attempt counts, last-attempt timestamps, and safe provider categories may help
|
||||||
|
operations, but they are telemetry and evidence rather than lifecycle authority.
|
||||||
|
Logs may describe detailed execution stages; the durable record should remain
|
||||||
|
centered on intent, checkpoints, result, and disposition.
|
||||||
|
|
||||||
|
## Applying this rule
|
||||||
|
|
||||||
|
For a new or materially changed operation:
|
||||||
|
|
||||||
|
1. identify the owning authority and transaction boundary;
|
||||||
|
2. classify it as an atomic ledger, reservation, durable side-effect operation,
|
||||||
|
or parent workflow;
|
||||||
|
3. define stable identity, fingerprint, and exact-retry behavior;
|
||||||
|
4. list external side effects and decide which are idempotent or authoritatively
|
||||||
|
observable;
|
||||||
|
5. add only checkpoints that change retry behavior;
|
||||||
|
6. keep child operation state in the child authority;
|
||||||
|
7. separate failure, blocker, attention, and disposition from lifecycle state;
|
||||||
|
8. state the unsupported crash windows honestly;
|
||||||
|
9. test fingerprint conflict, exact retry, authority revalidation, checkpoint
|
||||||
|
replay, and result/disposition projection as applicable.
|
||||||
|
|
||||||
|
Existing operation schemas need not be rewritten solely for vocabulary
|
||||||
|
consistency. When an operation is changed for functional reasons, use this
|
||||||
|
classification to remove derived or control-flow stages rather than adding
|
||||||
|
another special-case lifecycle.
|
||||||
Reference in New Issue
Block a user