fix tracked job dispatch and delivery recovery

This commit is contained in:
2026-10-01 11:58:49 +09:00
parent 213ed76061
commit 027a346e9d
4 changed files with 423 additions and 118 deletions
+29 -19
View File
@@ -316,39 +316,49 @@ The durable split is deliberate:
- `backend_jobs` stores immutable bounded input, its fingerprint, the current
attempt number, and the final structured result or terminal failure;
- `backend_job_attempts` binds one attempt and input revision to one dedicated
`(Workspace, Runtime, Worker)` identity, optional Runtime run evidence, a
deadline, and its bounded outcome;
`(Workspace, Runtime, Worker)` identity, a precommitted tracked Runtime
submission request identity, a deadline, and its bounded outcome;
- `backend_job_deliveries` records best-effort notification separately from the
result, so a missing or stopped recipient never discards a result or reruns
model work.
result. Its durable delivery ID is also the Runtime's tracked notification
request ID, so restart recovery can replay an ambiguous `sending` claim
without duplicating Worker context. A missing or stopped recipient never
discards a result or reruns model work.
Exact dispatch retry reuses the deterministic attempt and Worker allocation key.
Only explicit re-evaluation increments the bounded attempt number. The runner
Only explicit re-evaluation of a definitively failed attempt increments the
bounded attempt number; unknown outcomes are not replayable. The runner
uses the existing embedded Runtime spawn, Worker registry, Session input,
observation, usage, and lifecycle paths with no Workdir attachment. It uses the
dedicated `builtin:backend-job` profile, whose tools cannot mutate Workspace
state, recursively create Workers or Jobs, or inherit caller Workdir authority.
Success is accepted only by the structured result boundary. The submission is
fenced by Workspace, Job ID, attempt ID, input revision, and the bound Runtime
Worker identity, and its JSON bytes are bounded before persistence. Worker final
prose, Idle, Stopped, or a successful model call are not success evidence.
Identical structured submission is idempotent; conflicting replay, stale or late
attempts, wrong revisions, and other Workers fail closed.
Success is accepted only by the structured result boundary. Before Runtime
input can execute, the attempt is fenced by Workspace, Job ID, attempt ID, input
revision, bound Runtime Worker identity, and the deterministic tracked submission
request identity. Ordinary Worker input and Console mutation are rejected for
Job-owned Workers, so no later run in the same Session can exercise the result
capability. Result JSON bytes are bounded before persistence. Worker final prose,
Idle, Stopped, or a successful model call are not success evidence. Identical
structured submission is idempotent; conflicting replay, stale or late attempts,
wrong revisions, and other Workers fail closed.
On restart, an unbound reservation is redispatched with its stable allocation.
A dispatched attempt continues only when its bound Worker remains observable;
otherwise it becomes `unknown`. Durable deadlines fence acceptance before the
runner requests Worker stop. The design does not claim general exactly-once
model execution: a crash between Runtime creation and durable binding converges
through stable allocation where available, and unrecoverable external evidence
is surfaced as `unknown` rather than guessed. Historical attempts preserve the
reason for retry without duplicating the Runtime Worker lifecycle state machine.
A dispatched attempt continues only when its bound Worker and precommitted
tracked submission identity remain observable; otherwise it becomes `unknown`.
Ambiguous dispatch outcomes are also `unknown` and cannot be retried as a new
attempt. Durable deadlines fence acceptance before the runner requests Worker
stop. Notification recovery is independent: `sending` or `unknown` delivery
claims return to `pending` and replay the same tracked notification request ID.
The design does not claim general exactly-once model execution: a crash between
Runtime creation and durable binding converges through stable allocation where
available, and unrecoverable external evidence is surfaced as `unknown` rather
than guessed. Historical attempts preserve the reason for retry without
duplicating the Runtime Worker lifecycle state machine.
Workspace Worker subscriptions expose an explicit Job binding (`job_id`,
`attempt_id`, and `purpose`) in snapshot and live projections. Sidebar hiding is
a Web View policy over that canonical metadata; the Worker remains registered,
observable, and eligible for its canonical Console route.
observable, and eligible for its canonical read-only Console route.
## Diagnostics and audit