feat: execute durable standalone Jobs through Host-owned Internal Workers (T-708)
This commit is contained in:
@@ -26,7 +26,8 @@ worker
|
||||
- Controller transport は `InProcess` に固定する。通常起動で Worker subprocess、HTTP/WS server、Unix socket を作らない。
|
||||
- model provider は通常の resolved Manifest から構築する。埋め込み host と deterministic test は `start_with_model_client` で同じ bootstrap に process-owned client を注入できる。
|
||||
- feature plan/install は既存 `WorkerController` が行う。Task や optional direct SubWorker を standalone 側で再実装しない。
|
||||
- `shutdown()` は既存 `Method::Shutdown` を送り、controller が active run、SubWorker registry、Workdir session、MachineScope allocation を順に片付けた後の confirmation を待つ。
|
||||
- `shutdown()` は Host-owned Job service の scheduling を閉じ、Internal Worker の取消・cleanup とローカル SQLite connection の close を確認してから、既存 `Method::Shutdown` を送る。controller が active run、SubWorker registry、Workdir session、MachineScope allocation を順に片付けた後の confirmation を待つ。cleanup を確認できない drop/error では lease を保持する。
|
||||
- `jobs()` は Feature へ明示的に渡す汎用 Job 実行 capability。対話 Session・モデル作成 SubWorker とは別で、Profile/result authority と永続化の契約は [Standalone Jobs](standalone-jobs.md) に記録する。
|
||||
- startup error は category のみを公開し、credential、prompt 本文、session metadata、内部 path を error text に含めない。
|
||||
|
||||
## CLI / TUI routing
|
||||
|
||||
@@ -0,0 +1,152 @@
|
||||
# Host-owned standalone Jobs (T-708)
|
||||
|
||||
## Implementation design recorded before code changes
|
||||
|
||||
Base: `3c424fd17d1a5e8bb5542de2d0ad88f55338e0ea` (current origin/develop; T-704 integrated). T-709's subjektiv consumer is not part of this change.
|
||||
|
||||
The reusable `job` crate owns the transport-neutral immutable request (registry Profile selector, input snapshot/reference/revision, caller Job ID, instruction, serialization key and bounded limits), result envelope/digest and validation. Backend retains its existing SQLite state machine, Runtime binding, domain grants, authorization, unknown-outcome and cleanup implementation; its adapter uses the shared request validation and result contract, not a second runner.
|
||||
|
||||
Standalone owns a separate adapter, not a WorkspaceClient impersonation. `StandaloneHost` owns a Job service and its connection lifecycle. The concrete local placement is `<standalone state directory>/<WorkerId>/jobs.sqlite3` (using the existing Worker directory layout; alongside, not inside, interactive sessions). The `standalone` crate owns the versioned local schema: durable intents, immutable attempts, execution state, accepted result/digest, cancellation and consumer acknowledgement. A single Host connection is protected by the existing Worker lease. SQLite transactions serialize reserve, dispatch, result acceptance and terminal transitions. No subjektiv tables or fake Workspace/Runtime registry are created.
|
||||
|
||||
Local adapter states follow the existing intent/attempt/result distinctions. Exact immutable request replay returns the same Job, never starts another attempt. Explicit retry creates a bounded new attempt only for known failure; unknown outcomes require caller reconciliation and never auto-replay. Unstarted pending attempts remain pending on shutdown/reopen. Interrupted dispatch/running attempts without an accepted result become unknown; accepted results survive restart and remain available until consumer acknowledgement. Cancellation is durable and never auto-resumed. Consumer acknowledgement/delivery failure is separate from model success and never reruns the model.
|
||||
|
||||
Execution uses the existing private Internal Worker substrate with a new Host-facing bounded Job entry point. It gets a distinct ephemeral session, explicitly resolved selected Profile/system prompt/model, no ambient parent tools, filesystem, Workspace or Feature connections, and only a Job/attempt/revision-bound structured result submission capability. Profiles requiring unavailable tools/features are explicitly rejected rather than silently stripped. Result authority is independent of Profile name; final prose, Idle and Worker termination are never success evidence. This first generic adapter grants no domain operations; future consumers must provide explicit domain capabilities rather than gaining them through a profile tag.
|
||||
|
||||
The Host limits concurrent active resources independently of caller limits, enforces timeout/turn/output limits, routes cancellation into the Internal Worker lifecycle and waits for resource cleanup. Accepted results are committed before resource release; required domain postprocessing must finish before submission (or be independently durable in its consumer). No daemon, pool, subprocess, HTTP/WS listener or Server dependency is introduced.
|
||||
|
||||
Validation will distinguish scripted in-process Internal Worker executions and SQLite reopen/recovery fixtures from any real-provider/process tests. It will cover alternative Profiles, immutable retry, invalid/stale results, resource limits, failure/cancellation/timeout, shutdown and restart. Backend focused regression tests must still prove its existing fences and cleanup.
|
||||
|
||||
## API and ownership
|
||||
|
||||
`job::{JobRequest, JobLimits, JobResultSubmission, JobOutcome}` are the common
|
||||
Feature-facing data contract. `JobOutcome` distinguishes intent state from attempt
|
||||
state and omits Workspace/Runtime identities. These enums are read vocabulary,
|
||||
not another transition engine. Backend converts neutral requests with
|
||||
`BackendJobRequest::from_job_request` and projects reservation/acceptance with
|
||||
`.outcome()`; Backend domain grants, provenance, delivery and cleanup remain on
|
||||
its own adapter and retain their existing durable/wire format and fingerprints.
|
||||
|
||||
Standalone consumers receive `StandaloneHost::jobs()` explicitly, not a fake
|
||||
Workspace client and not a tool on the interactive Worker. They can persist
|
||||
`request`, read `get`/`pending`, explicitly `start` or `retry`, `cancel`, and
|
||||
`acknowledge`. `.wait()` observes execution cleanup; `.outcome()` on the returned
|
||||
snapshot is the common read view. `Completed` with its accepted structured result
|
||||
is success authority; merely returning from `wait` is not. The injected-client
|
||||
entry point resolves/validates the same selected Profile and model configuration;
|
||||
it replaces only the model transport for embedders/tests.
|
||||
|
||||
A typical generic consumer supplies its own stable Job ID/revision/reference,
|
||||
immutable JSON input, instruction and registry Profile:
|
||||
|
||||
```rust,ignore
|
||||
let jobs = host.jobs();
|
||||
let snapshot = jobs.request(request)?;
|
||||
if snapshot.state == standalone::jobs::JobState::Pending {
|
||||
jobs.start(&snapshot.request.job_id).await?;
|
||||
}
|
||||
let outcome = jobs.wait(&snapshot.request.job_id).await?.outcome();
|
||||
// Validate/apply the consumer's domain result here. A failed notification or
|
||||
// delivery leaves the successful result unacknowledged; do not retry the model.
|
||||
if outcome.state == job::JobState::Completed {
|
||||
deliver_idempotently(&outcome)?;
|
||||
jobs.acknowledge(&outcome.job_id)?;
|
||||
}
|
||||
```
|
||||
|
||||
The Host retains tasks/cancellation even while a consumer waits; dropping a wait
|
||||
future does not detach cancellation ownership or release a concurrency slot.
|
||||
Active resources are bounded to eight by the Host and by the minimum of all live
|
||||
requests' ceilings, including after result acceptance and before cleanup.
|
||||
Serialization keys likewise fence live resources, not just pending result state.
|
||||
Busy `start` returns a typed capacity error and leaves the reserved attempt
|
||||
unchanged; callers decide when to start/resume pending work. There is no pool,
|
||||
background daemon, implicit retry or independent Feature-owned Internal Worker.
|
||||
|
||||
## Profile and capability policy
|
||||
|
||||
The caller must select an existing built-in or user registry Profile. Path/raw
|
||||
selectors and `inherit` are not accepted. Registry existence, selected instruction
|
||||
and model configuration are checked before dispatch. The ordinary user settings
|
||||
registry and Prompt catalog are used; repository-local recipes are not discovered
|
||||
by the standalone Host, and default never silently maps to a different Profile.
|
||||
|
||||
`builtin:job` is an **optional**, compatible result-only policy. It is not a
|
||||
runner-selected default or authority token. User registry Profiles with different
|
||||
instructions/models/bounds execute the same way, as demonstrated by the scripted
|
||||
Worker tests. This adapter deliberately rejects active unsupported Feature,
|
||||
filesystem/delegation scope, MCP, skills, compaction and durable event-trace
|
||||
requirements. Existing `builtin:backend-job` has such requirements and is rejected
|
||||
by this standalone adapter, while its Backend behavior is unchanged. A future
|
||||
consumer requiring domain operations must arrange explicit Host capabilities;
|
||||
choosing `builtin:subjektiv-memory-consolidation` alone grants nothing.
|
||||
|
||||
Every run uses the existing private Internal Worker substrate, a distinct identity
|
||||
and ephemeral Session. Only `SubmitJobResult` is installed by a FeatureModule.
|
||||
Its model input is `{result: ...}`; Job/attempt/input-revision bindings come from
|
||||
the Host capability, not model arguments. The sink checks those bindings, the
|
||||
current persisted attempt, deadline, byte limit and result digest. Exact result
|
||||
replay is idempotent; changed/foreign/stale results are rejected. Submission is
|
||||
sealed before acceptance, so uncertain responses permit only identical retries.
|
||||
An ambiguous sink failure is reconciled against SQLite; absent accepted evidence
|
||||
it is `Unknown`, not a known retryable model failure. No domain logic is embedded
|
||||
in this sink. Required consumer/domain postprocessing must complete before
|
||||
submission or have its own durable ownership outside the ephemeral Worker.
|
||||
|
||||
Selected instructions are retained. The Host execution contract is rendered from
|
||||
`resources/prompts/internal/job_system.md` before the run's initial history is
|
||||
created, not injected into a previous/transient turn. Turn/output policy is
|
||||
attenuated to at most 32 turns and 8192 output tokens per model request, preserving
|
||||
tighter selected Profile bounds. Structured results retain the shared 16 KiB
|
||||
default / 64 KiB absolute byte limits, not a larger standalone exception.
|
||||
|
||||
## Persistence and termination
|
||||
|
||||
Local schema version **1** is owned only by `standalone::job_store`: `job_intents`
|
||||
contains immutable serialized intent/fingerprint, serialization key, current
|
||||
attempt and acknowledgement; `job_attempts` contains immutable identity/revision,
|
||||
execution state, failure and accepted result/digest. Foreign keys, immutable
|
||||
identity/result triggers, transactions, `WAL` and `synchronous=FULL` protect the
|
||||
boundary. No Server schema or subjektiv/Memory schema is copied or opened.
|
||||
The Host opens and recovers the store only under its existing top-level Worker
|
||||
lease; clones cannot open or recover it themselves.
|
||||
|
||||
- **Exact request replay:** same immutable intent returns its current snapshot.
|
||||
Changing any intent field under the same ID is a conflict, not a new attempt.
|
||||
- **Known failure:** only explicit `retry` creates a new immutable attempt, up to
|
||||
the intent's bounded maximum. Prior attempts/results remain in SQLite.
|
||||
- **Timeout/shutdown before acceptance:** fence `Unknown` before requesting Engine
|
||||
cancellation; late result submissions cannot create success. Await normal
|
||||
cleanup; retain the Host lease and report an error if confirmation is lost.
|
||||
- **Normal Host shutdown:** seal scheduling/mutations, cancel active runs, await
|
||||
resource cleanup, close SQLite, then stop the interactive controller and release
|
||||
its lease. Reserved/unstarted requests remain resumable. Old cloned handles
|
||||
return `Closed`, rather than mutating a store reopened by another Host.
|
||||
- **Unexpected Host drop:** request cancellation and retain the lease because
|
||||
synchronous drop cannot attest cleanup. No daemon continues after process exit.
|
||||
- **Process loss/reopen:** reserved attempts stay pending; unaccepted dispatched
|
||||
attempts become unknown; accepted results/acknowledgements remain intact.
|
||||
SQLite result evidence, never final prose or an ephemeral Session, decides success.
|
||||
- **Cancellation:** durable and never automatically resumed. Cancelling already
|
||||
unknown intent does not erase its attempt's uncertain execution evidence or
|
||||
release its serialization fence. `retry` still refuses it. There is no generic
|
||||
operation that declares uncertain domain side effects unexecuted; callers must
|
||||
reconcile domain evidence rather than bypassing the fence with an implicit replay.
|
||||
- **After acceptance:** model error, cancellation or consumer delivery failure
|
||||
cannot erase success or rerun the model. Acknowledgement is independently durable
|
||||
and idempotent. Execution resources are reclaimed independently of result lifetime.
|
||||
|
||||
Emergency cancellation has a bounded grace (one second for cancellation setup,
|
||||
two seconds for normal cleanup). Expiry is **not** cleanup confirmation: it returns
|
||||
an error, keeps unknown evidence/accepted results, and the Host retains its lease.
|
||||
The result-only Feature owns no async/background tasks or domain connections;
|
||||
the dropped engine stream/tool pumps request abort. Synchronous SQLite submission
|
||||
is bounded by its five-second busy timeout and cannot be preempted mid-transaction.
|
||||
No resource or side effect is claimed rolled back merely because a future dropped.
|
||||
|
||||
## Validation evidence
|
||||
|
||||
See [T-708 validation ledger](../report/2026-10-06-t708-validation.md). It separates
|
||||
scripted in-process Worker/Host runs, direct SQLite/reopen/rollback fixtures and
|
||||
an actual test-owned child-process loss/Host restore test from live-provider or
|
||||
live dogfood validation. T-709's subjektiv consumer is intentionally not migrated
|
||||
here; Backend T-704 Profile selection and T-690 cleanup remain regression-tested.
|
||||
@@ -0,0 +1,134 @@
|
||||
# T-708 implementation and validation ledger
|
||||
|
||||
Base: clean detached assigned checkout `3c424fd17d1a5e8bb5542de2d0ad88f55338e0ea`,
|
||||
also the inspected `origin/develop`. T-704 was authoritative `done` with that
|
||||
integrated source. Created `work/T-708-standalone-jobs` without overwriting an
|
||||
existing branch or discarding changes. T-709's subjektiv consumer is separate.
|
||||
The concrete SQLite placement/schema ownership was recorded in
|
||||
[the design](../design/standalone-jobs.md) before implementation; it uses the
|
||||
existing `<state_dir>/<WorkerId>` directory layout, not a new `workers` subdirectory.
|
||||
|
||||
Three scoped implementation children supplied the common contract/Backend
|
||||
projection, Worker execution adapter, and local SQLite store. Parent inspected
|
||||
committed sessions, exact source changes and concrete test results, then integrated
|
||||
Host/service ownership, live-resource ceilings, deadline fencing, independent
|
||||
acknowledgement, shared outcome projection, process-loss tests and documentation.
|
||||
Independent published-MR review is separate from these implementation children.
|
||||
|
||||
## Implemented boundaries
|
||||
|
||||
- `job`: neutral immutable request/limits/result envelope, fingerprint, strict
|
||||
registry selectors, bounded digests and read-only intent/attempt/outcome DTOs.
|
||||
No HTTP, Workspace, Runtime identity, domain grants, runner or workflow DSL.
|
||||
- Backend reuses validation/result helpers and neutral projections while retaining
|
||||
its exact request wire/fingerprint/envelope, domain grants, persistent states,
|
||||
Runtime authorization, unknown fences, resource serialization and cleanup.
|
||||
- `worker::job`: selected registry Profile/instruction/model, explicit requirement
|
||||
rejection, bounded policy, distinct ephemeral Internal Worker, result-only
|
||||
FeatureModule and Host-fenced structured acceptance. No parent filesystem,
|
||||
Workspace, Feature or domain connections; no Profile-name authorization.
|
||||
Ambiguous sink acceptance is Unknown unless the Host's durable result proves success.
|
||||
- Standalone Host owns schema-1 SQLite intent/attempt/result/ack state and a bounded
|
||||
execution service. It opens and recovers under its Worker lease; shutdown cancels
|
||||
and awaits executions, closes SQLite, then releases the lease after controller
|
||||
confirmation. Old cloned handles are closed. Drop cannot attest cleanup and
|
||||
retains the lease. Busy/unknown/cancelled/terminal work is not blindly replayed.
|
||||
- Exact immutable request resend differs from explicit bounded new-attempt retry.
|
||||
Results commit before resource release, survive model/notification failure, and
|
||||
are independently acknowledged by the consumer. Waiting does not relinquish
|
||||
Host ownership of cancellation/concurrency. Cancelling unknown intent retains
|
||||
its uncertain attempt evidence and serialization fence.
|
||||
- Optional `builtin:job` is a caller-selected compatible policy, not a fixed runner
|
||||
default. Different user registry Profiles are demonstrated in Worker fixtures.
|
||||
Existing `builtin:backend-job` retains Backend behavior; standalone rejects its
|
||||
unavailable requirements rather than silently stripping or replacing them.
|
||||
- No production Server/Runtime process, HTTP/WS listener, Worker registry,
|
||||
subprocess Worker, daemon/pool or subjektiv-specific runner is added.
|
||||
|
||||
## Validation results
|
||||
|
||||
Development began with narrow contract/module targets before full changed-crate
|
||||
validation. Final root check covers the normal default-members TUI/Server closure.
|
||||
|
||||
| Command | Concrete result |
|
||||
| --- | --- |
|
||||
| `cargo test -p job` | 9 contract tests passed; no unit/doc tests |
|
||||
| `cargo test -p standalone --lib job_store` | 19 fixtures passed: transactional replay/conflict, all limits, result fences, retry/old-attempt preservation, cancellation, concurrency races, SQL rollback, schema rejection and reopen recovery |
|
||||
| `cargo test -p standalone --lib jobs::tests` | 2 passed: Host capability binding/deadline/replay and success/cancellation/closed-handle lifetime |
|
||||
| `cargo test -p standalone --test jobs process_loss_recovers_durable_jobs_without_replaying_unknown` | Actual test-owned process exit and Host stale-lease/SQLite recovery passed |
|
||||
| Final `cargo test -p standalone` | 25 library + 10 existing Host integration + 8 Job integration passed; 1 child-process fixture entry point marked ignored in parent harness, explicitly invoked with `--ignored` by the passing process-loss test |
|
||||
| `cargo test -p manifest` | All 185 tests passed, including new optional builtin Profile catalogue resolution; no resolver/runtime/schema implementation changed |
|
||||
| `cargo test -p worker` | All 826 library + 110 integration tests passed, including 11 new Job tests and 8 existing Internal Worker substrate tests |
|
||||
| `cargo test -p yoi-workspace-server --lib backend_job` | All 38 passed: neutral adapter/wire/fingerprint, T-704 Profiles/no fallback, structured fences, unknown/recovery, resource serialization and T-690 cleanup |
|
||||
| `cargo test -p yoi-workspace-server --bins` | All 11 binary tests passed |
|
||||
| `cargo test -p worker-runtime --lib backend_job` | All 3 Job binding/dispatch regressions passed |
|
||||
| `cargo test -p server-api --features typescript` | All 77 passed, including checked-in generated/OpenAPI/TypeScript freshness; existing ts-rs serde-attribute warnings |
|
||||
| Root `cargo check` | Passed after integration and again after final local adapter/tests changes |
|
||||
| `cargo fmt --all -- --check` / `git diff --check HEAD` | Passed |
|
||||
| `cargo test -p yoi-workspace-server` | **Failed: 643 passed, 1 failed** in library; see retained anomaly below. Binary tests were therefore run separately. This is not a green full Server suite. |
|
||||
| `cargo test -p yoi-workspace-server -- --test-threads=4` | **Failed: 643 passed, 1 failed**, same unchanged checker assertion; not claimed passed |
|
||||
|
||||
Profile fixtures use two different existing user registry entries with distinct
|
||||
instructions and budgets, plus optional `builtin:job`. They prove only
|
||||
`SubmitJobResult` is exposed, selected instructions/policy are retained, scopes and
|
||||
Backend/domain tools are absent, malformed/oversized/forged results are rejected,
|
||||
uncertain sink submission permits only identical replay, final prose is not
|
||||
success, provider failure/cancellation/turn limits do not manufacture success,
|
||||
and accepted results survive later provider failure.
|
||||
|
||||
Host Job integration additionally proves explicit known-model-failure retry,
|
||||
request replay, live concurrency while a consumer waits, cancellation, timeout,
|
||||
shutdown, pending resume, consumer acknowledgement and unsupported Profile
|
||||
rejection without model calls. Direct-store fixtures prove persisted facts and
|
||||
rollback/reopen behavior, not live model or process behavior.
|
||||
|
||||
### Actual process verification versus fixtures
|
||||
|
||||
`process_loss_recovers_durable_jobs_without_replaying_unknown` starts **only a
|
||||
test-owned copy of the integration test executable**, using scripted model clients.
|
||||
The child owns a real StandaloneHost/SQLite connection/Worker lease, persists a
|
||||
reserved request, a cancelled request and an accepted result, then exits with code
|
||||
23 while a separate Internal Worker is executing. It deliberately skips destructors
|
||||
and graceful shutdown. The parent verifies that exit and restores the Host through
|
||||
the real stale-process lease path: accepted result retained/unacknowledged,
|
||||
cancelled work never resumed, reserved work explicitly resumable and interrupted
|
||||
work Unknown/no automatic retry. The helper's ignored annotation is an entry-point
|
||||
selector, not skipped semantic coverage.
|
||||
|
||||
These are real Rust test/compiler processes plus scripted in-process model
|
||||
executions and SQLite fixtures. They are **not** live-provider LLM calls, a live
|
||||
standalone CLI/TUI session, a Backend/Runtime deployment or dogfood environment
|
||||
update. No operating Yoi Worker or live dogfood process was stopped or updated.
|
||||
Production Jobs do not create the subprocess used solely by that crash test.
|
||||
|
||||
## Retained failures and evidence limits
|
||||
|
||||
Initial integrated compilation exposed `get` snapshot/Option disagreement between
|
||||
concurrently authored store/service; signatures were aligned and the subsequent
|
||||
store/Host/full standalone runs passed. A private tool-type import during child
|
||||
integration was corrected before focused/full Worker validation. No passing
|
||||
result is attributed to those initial failing builds.
|
||||
|
||||
Both full Server runs and one isolated run of
|
||||
`server::tests::ticket_checker_suppresses_stale_and_clean_results_and_diagnoses_invalid_output`
|
||||
failed the unchanged post-stale-result `assert!(execution.take_inputs().is_empty())`
|
||||
at `server.rs:44748`. All 38 Job-filter tests pass, including unchanged Backend
|
||||
envelope/fingerprint compatibility and late-result/cleanup fences. The same
|
||||
checker assertion/race was independently reproduced on the earlier original base
|
||||
in the [T-702 ledger](2026-10-06-t702-stale-checker-validation.md), and explicitly
|
||||
retained as unfixed in the [T-704 ledger](2026-10-06-t704-validation.md).
|
||||
No new isolated build of this Ticket's exact base was run, and its root cause is
|
||||
not claimed proven or fixed here. The checker implementation/test was neither
|
||||
changed nor skipped to manufacture a green suite. This validation anomaly remains
|
||||
visible for Reviewer/Orchestrator attention.
|
||||
|
||||
No whole-workspace tests, E2E, Nix/Docker builds or live-provider validation were
|
||||
claimed. Backend generated contracts were regression checked despite no wire
|
||||
change. Result-only Jobs reject unsupported Feature/authority requirements;
|
||||
subjektiv consumer capabilities/postprocessing are T-709 work, not inferred grants.
|
||||
|
||||
Publication must use only the clean committed Ticket work branch by normal
|
||||
non-force push, verified with exact provider source ref. No target/tags/unrelated
|
||||
refs, replacement MR, integration branch, merge or Ticket completion is performed
|
||||
by this Coder. Final immutable source/MR/review identities belong in the MR thread
|
||||
and approved Ticket handoff rather than being guessed in this ledger.
|
||||
Reference in New Issue
Block a user