fix tracked job dispatch and delivery recovery
This commit is contained in:
@@ -316,39 +316,49 @@ The durable split is deliberate:
|
||||
- `backend_jobs` stores immutable bounded input, its fingerprint, the current
|
||||
attempt number, and the final structured result or terminal failure;
|
||||
- `backend_job_attempts` binds one attempt and input revision to one dedicated
|
||||
`(Workspace, Runtime, Worker)` identity, optional Runtime run evidence, a
|
||||
deadline, and its bounded outcome;
|
||||
`(Workspace, Runtime, Worker)` identity, a precommitted tracked Runtime
|
||||
submission request identity, a deadline, and its bounded outcome;
|
||||
- `backend_job_deliveries` records best-effort notification separately from the
|
||||
result, so a missing or stopped recipient never discards a result or reruns
|
||||
model work.
|
||||
result. Its durable delivery ID is also the Runtime's tracked notification
|
||||
request ID, so restart recovery can replay an ambiguous `sending` claim
|
||||
without duplicating Worker context. A missing or stopped recipient never
|
||||
discards a result or reruns model work.
|
||||
|
||||
Exact dispatch retry reuses the deterministic attempt and Worker allocation key.
|
||||
Only explicit re-evaluation increments the bounded attempt number. The runner
|
||||
Only explicit re-evaluation of a definitively failed attempt increments the
|
||||
bounded attempt number; unknown outcomes are not replayable. The runner
|
||||
uses the existing embedded Runtime spawn, Worker registry, Session input,
|
||||
observation, usage, and lifecycle paths with no Workdir attachment. It uses the
|
||||
dedicated `builtin:backend-job` profile, whose tools cannot mutate Workspace
|
||||
state, recursively create Workers or Jobs, or inherit caller Workdir authority.
|
||||
|
||||
Success is accepted only by the structured result boundary. The submission is
|
||||
fenced by Workspace, Job ID, attempt ID, input revision, and the bound Runtime
|
||||
Worker identity, and its JSON bytes are bounded before persistence. Worker final
|
||||
prose, Idle, Stopped, or a successful model call are not success evidence.
|
||||
Identical structured submission is idempotent; conflicting replay, stale or late
|
||||
attempts, wrong revisions, and other Workers fail closed.
|
||||
Success is accepted only by the structured result boundary. Before Runtime
|
||||
input can execute, the attempt is fenced by Workspace, Job ID, attempt ID, input
|
||||
revision, bound Runtime Worker identity, and the deterministic tracked submission
|
||||
request identity. Ordinary Worker input and Console mutation are rejected for
|
||||
Job-owned Workers, so no later run in the same Session can exercise the result
|
||||
capability. Result JSON bytes are bounded before persistence. Worker final prose,
|
||||
Idle, Stopped, or a successful model call are not success evidence. Identical
|
||||
structured submission is idempotent; conflicting replay, stale or late attempts,
|
||||
wrong revisions, and other Workers fail closed.
|
||||
|
||||
On restart, an unbound reservation is redispatched with its stable allocation.
|
||||
A dispatched attempt continues only when its bound Worker remains observable;
|
||||
otherwise it becomes `unknown`. Durable deadlines fence acceptance before the
|
||||
runner requests Worker stop. The design does not claim general exactly-once
|
||||
model execution: a crash between Runtime creation and durable binding converges
|
||||
through stable allocation where available, and unrecoverable external evidence
|
||||
is surfaced as `unknown` rather than guessed. Historical attempts preserve the
|
||||
reason for retry without duplicating the Runtime Worker lifecycle state machine.
|
||||
A dispatched attempt continues only when its bound Worker and precommitted
|
||||
tracked submission identity remain observable; otherwise it becomes `unknown`.
|
||||
Ambiguous dispatch outcomes are also `unknown` and cannot be retried as a new
|
||||
attempt. Durable deadlines fence acceptance before the runner requests Worker
|
||||
stop. Notification recovery is independent: `sending` or `unknown` delivery
|
||||
claims return to `pending` and replay the same tracked notification request ID.
|
||||
The design does not claim general exactly-once model execution: a crash between
|
||||
Runtime creation and durable binding converges through stable allocation where
|
||||
available, and unrecoverable external evidence is surfaced as `unknown` rather
|
||||
than guessed. Historical attempts preserve the reason for retry without
|
||||
duplicating the Runtime Worker lifecycle state machine.
|
||||
|
||||
Workspace Worker subscriptions expose an explicit Job binding (`job_id`,
|
||||
`attempt_id`, and `purpose`) in snapshot and live projections. Sidebar hiding is
|
||||
a Web View policy over that canonical metadata; the Worker remains registered,
|
||||
observable, and eligible for its canonical Console route.
|
||||
observable, and eligible for its canonical read-only Console route.
|
||||
|
||||
## Diagnostics and audit
|
||||
|
||||
|
||||
Reference in New Issue
Block a user