fix: compose bounded execution with Ticket public callers
This commit is contained in:
@@ -0,0 +1,159 @@
|
||||
# Backend bounded synchronous execution
|
||||
|
||||
## Ownership and admission
|
||||
|
||||
`ControlPlaneStore::db_executor()` returns a clone of the bounded execution owner.
|
||||
Use `executor.run(move || synchronous_operation(...)).await` with owned input and
|
||||
services. The closure owns the **whole** synchronous operation, including connection
|
||||
acquisition, mutex wait, the existing SQL transaction, commit, and ordered local
|
||||
publication. Do not split source authorization, receipts, CAS, audit, or saved
|
||||
results around an await. Do not change connection ownership or introduce a read pool.
|
||||
|
||||
Production stores share one DB worker and a queue of 32 jobs. Admission uses a
|
||||
nonblocking queue send and a separate actual-job budget (worker count plus queue
|
||||
capacity); rejection means the operation was **not accepted**. HTTP maps capacity
|
||||
and executor unavailability to 503, not authentication failure or deletion conflict.
|
||||
Tests can inject a dedicated executor with explicit limits, without process-global
|
||||
settings or environment variables. Ordinary SQLite fixtures use `open_for_test`,
|
||||
`in_memory_for_test`, or `from_connection_for_test`: their clones share their own
|
||||
1-worker/32-queued owner, not the production singleton or another fixture's drain.
|
||||
Server/Workspace fixture constructors likewise inject a lazy `HostExecutors` owner;
|
||||
all child Workspaces and weak retention adapters use that same owner. Host and
|
||||
cleanup still have the production 2-worker/16-queued limits. Dropping the final
|
||||
sender closes its queue after accepted work finishes; idle fixture threads are
|
||||
not kept in a process-global fixture cache.
|
||||
|
||||
The restore intent journal and Workspace deletion use
|
||||
`with_immediate_transaction`: read authority/journal state only after acquiring
|
||||
SQLite write ownership. A DEFERRED read-to-write upgrade can return BUSY without
|
||||
invoking SQLite's busy handler when another connection is writing. Moving these
|
||||
whole helpers to an offloaded caller must not create an automatic mutation retry.
|
||||
|
||||
|
||||
Accepted work is owned by the worker, not the Tokio waiter. Cancelling a waiter:
|
||||
|
||||
- does not interrupt, resubmit, or retry SQL;
|
||||
- does not return its queue/job slot before execution and discarded-result cleanup;
|
||||
- does not release the request lifecycle lease before actual completion.
|
||||
|
||||
Synchronous post-commit continuations inherit the admitted lifecycle source when
|
||||
submitting another bounded job. This is not implicit inheritance by detached Tokio
|
||||
tasks. An operation panic has an unknown outcome even if SQL already committed;
|
||||
return a structured unknown-outcome error and diagnostic, never retry the mutation.
|
||||
A cancelled fire-and-forget job still emits its diagnostic. `drain()` observes all
|
||||
jobs accepted before its barrier is first polled and waits for actual cleanup; it
|
||||
is not a queue close and does not include later admissions. Lifecycle callers must
|
||||
stop their producers/admission before destroying resources.
|
||||
|
||||
## Runtime/network boundaries
|
||||
|
||||
A synchronous Runtime workflow must not monopolize the sole DB worker while
|
||||
waiting for callbacks that themselves need DB authority. The Server's separate
|
||||
finite Host lane has two workers and 16 queued jobs. It uses the same accepted-job
|
||||
ownership mechanism, not an unbounded `spawn_blocking` queue.
|
||||
|
||||
Pure DB stages use the DB lane. Existing mixed resource-lifecycle workflows can
|
||||
use the Host lane, with no Runtime I/O or await inside a SQL transaction. Where an
|
||||
owned workflow needs async resource locks, `run_host_workflow` polls it on a Host
|
||||
worker using the original Tokio handle; do not build a replacement runtime for
|
||||
cached services and do not submit to and await the Host lane from inside itself.
|
||||
Control/session guards needed by an accepted mutation belong to that actual job.
|
||||
Subjektiv session Read/Search, subject response projection, general memory staging
|
||||
and legacy consolidation-scope helpers also use Host ownership: they can
|
||||
synchronously call Runtime/network services outside their SQL transactions.
|
||||
Pure DB-resident Subjektiv scope and memory projection helpers stay on the DB lane.
|
||||
Host admission must not be acquired again from an already-owned Host workflow,
|
||||
including indirectly through a Runtime's mandatory Backend callback.
|
||||
|
||||
Mandatory Session attribution has Subject-only authority and whole attribution
|
||||
SQL on the independent DB lane. Store-only scope discovery distinguishes Subject,
|
||||
current Job consolidation, and legacy consolidation authority; attribution rejects
|
||||
both consolidation roles **before** any Runtime profile/tag lookup. Legacy
|
||||
consolidation checks for mixed callers remain on Host. Subject-only memory
|
||||
operations (resident context/summary, proposal validation, receipt lookup, explicit
|
||||
staging) use the same store-only authority boundary and DB ownership. Runtime
|
||||
bootstrap/restore can request resident context while its Host parent is waiting.
|
||||
This is not permission to bypass Workspace deletion admission: only the already
|
||||
allowed exact callback endpoints may reenter deletion drain.
|
||||
|
||||
Backend Job cleanup has a third finite lane (two workers, 16 queued jobs), so a
|
||||
long Run teardown wait cannot starve ordinary Host control/stop requests. It is
|
||||
admitted before its durable cleanup claim. Rejection leaves Pending/Failed
|
||||
ownership recoverable instead of abandoning an Executing claim. Ticket
|
||||
checker dispatch and timeout processing use bounded admission; overload leaves the
|
||||
saved reservation/deadline diagnosable. They are not automatically retried after an
|
||||
unknown DB outcome. Runtime teardown is still its own lifecycle fence, not proof
|
||||
of model processing.
|
||||
|
||||
Remote subscription and Workdir authorization are async callbacks: their complete
|
||||
live binding check and fresh request signing are offloaded together. Cold protocol
|
||||
source lookup in `connect_workspace_worker_protocol` is also awaited as an owned
|
||||
DB stage, rather than invoking its synchronous signing callback on Tokio.
|
||||
`WorkspaceWorkerRuntime::observation_source` is fallible: live binding/signing
|
||||
failure and bounded admission errors propagate explicitly, not as a missing
|
||||
source or a fallback to a configured bearer token. Synchronous CLI/migration
|
||||
callers and existing synchronous Runtime adapters retain their explicit blocking
|
||||
boundary.
|
||||
|
||||
## Caller recipe (including assignment APIs)
|
||||
|
||||
```rust
|
||||
let store = api.store.clone();
|
||||
let workspace_id = api.workspace_id().to_owned();
|
||||
let worker = worker.clone();
|
||||
let record = api.store.db_executor().run(move || {
|
||||
store.get_worker_registry(&workspace_id, &worker)
|
||||
}).await?;
|
||||
```
|
||||
|
||||
This is an illustration, not a new authority API. Assignment delivery scan,
|
||||
reserve, prepare, acknowledgement and start authorization must keep their existing
|
||||
whole transaction helpers. Additional assignment diagnostics, answer mutations,
|
||||
and Ticket queries use the same owned caller boundary; do not add a second ledger
|
||||
or reconstruct processing evidence from Worker liveness.
|
||||
|
||||
Worker discovery uses the same whole `workers_response` DB stage as Worker list
|
||||
and resource detail: projection snapshot, Workdir registry/links, and authority
|
||||
projection never acquire the shared connection on Tokio. Source-proof requirement
|
||||
and pure query/cursor validation still precede admission; overload remains 503.
|
||||
Runtime resource list/detail use an accepted Host workflow for synchronous Runtime
|
||||
summary I/O and await independent DB binding/audit reads. Launch options are also
|
||||
mixed Runtime/config helpers and use Host, not the sole DB worker. Runtime
|
||||
registration validates its signing identity in a whole DB stage, scans existing
|
||||
Workers on Host, then owns SQL key/CAS mutation, local registration publication
|
||||
and response observation in one accepted workflow. Metadata updates likewise keep
|
||||
commit and publication together after waiter cancellation. Trust-key revoke also
|
||||
owns whole CAS/audit SQL, expectation removal, broker unregister/Shutdown and
|
||||
response observation in one accepted Host workflow. Future request proof denial
|
||||
alone does not terminate existing subscriptions. Host admission must precede the
|
||||
mutation, so overload cannot commit a revoke then fail on response admission.
|
||||
Public resource/detail helpers admit Host ownership; already-owned workflows call their `_on_host`
|
||||
variants directly, never await another admission to the same occupied lane.
|
||||
Connection tests ping on finite Host ownership and then await the whole current
|
||||
binding fence on DB. A replaced/revoked binding cannot attest the earlier ping.
|
||||
Do not confuse a Runtime summary callback with a pure stored Worker projection
|
||||
or run network waits on the sole DB worker.
|
||||
|
||||
## Regression evidence
|
||||
|
||||
- `db_executor::tests`: real SQLite IMMEDIATE write contention while a current-thread
|
||||
timer/control task progresses; accepted/cancelled queue bounds; rollback; panic
|
||||
before/after commit; discarded reply cleanup and retained authority; child jobs.
|
||||
- `server::db_execution_tests`: HTTP Ticket writes and cold/warm Subjektiv scope
|
||||
opening/mutation under a separate SQLite connection's write lock; overload;
|
||||
cancelled HTTP jobs and Workspace deletion drain; same-budget fixture isolation,
|
||||
saturated/cancelled Host jobs versus cleanup and paused Tokio time; restore intent
|
||||
contention before the first authority read, with one persisted tuple and no retry.
|
||||
The terminal checker fixture keeps its original Tokio driver alive through accepted cleanup,
|
||||
then shuts down projection producers and drains actual DB jobs.
|
||||
- `worker_projection::tests`: real SQL contention, commit/payload publication,
|
||||
ordered snapshot registration, and replacement draining a cancelled consumer's
|
||||
accepted SQL before applying a new Runtime snapshot.
|
||||
- Auth/proof and Runtime subscription tests protect availability classification,
|
||||
replay rejection, and no bearer fallback.
|
||||
- `workdir`'s `http-client` tests protect awaited fresh authorization and exact
|
||||
request claims before any HTTP send.
|
||||
|
||||
These fixtures use temporary authority data and observable synchronization, not
|
||||
live databases or a fixed sleep to guess completion. This change does not claim a
|
||||
latency benchmark or a measured production incident.
|
||||
@@ -0,0 +1,287 @@
|
||||
# T-748: async caller ownership inventory and scoped repair evidence
|
||||
|
||||
## Current status and evidence boundary
|
||||
|
||||
MR `01a12d3f-e90c-7b02-9ed9-cf3bd2e822a1` seq13 requested changes for published
|
||||
`ebce7ab7b1230654b8eb5340afca8de835f24f12`. The independent review confirmed the
|
||||
revoke cancellation/publication fix, but found three further direct async callers.
|
||||
Earlier full suites exercised successful outcomes without proving every composed
|
||||
caller boundary. Repeatedly treating a helper's apparent purpose as its lane
|
||||
classification missed Runtime lookup in Ticket projection and post-await reads.
|
||||
A successful suite is not a substitute for the entry-to-publication inventory.
|
||||
|
||||
The seq13 and adjacent fixes were committed as `5873f250`; embedded protocol
|
||||
and Feature shutdown fixes as `4d981194`. The remaining confirmed caller-placement
|
||||
and Worker stop repairs are included in the follow-up below. The read-only audit
|
||||
children were released after result-observation gaps; that release was not treated
|
||||
as completion. No audit-to-implementation approval gate is imposed.
|
||||
No target integration, Ticket conclusion, live update or attachment cleanup is
|
||||
performed. T743 domain APIs are not copied or implemented here; their owner still
|
||||
needs normal composition and exact combined-source review.
|
||||
|
||||
The tables and line numbers below preserve the **pre-repair audit snapshot**.
|
||||
“Unfixed”, “pending”, and “remaining” in those historical sections describe that
|
||||
snapshot, not the current implementation. The final section supersedes their
|
||||
placement/stop status and explicitly distinguishes tested boundaries from limits.
|
||||
|
||||
Audited snapshot `server.rs` SHA-256:
|
||||
`dae05a2649f5d4f884e8118ba4ce6baa902de6e61727c20e97c0afa3661ef3a6`.
|
||||
Audited snapshot `worker_operations.rs` SHA-256:
|
||||
`e3ad2327370440fdc40fb2c76f9604b984667e3472570d76145f4d0987d343fd`.
|
||||
These file hashes identify the inspected implementation, not a provider review ref.
|
||||
|
||||
## Current confirmed ownership changes
|
||||
|
||||
| Entry / composed family | Authority and synchronous dependencies | Current owner / completion boundary | Evidence and limits |
|
||||
|---|---|---|---|
|
||||
| `ticket_implementation_cancel` → `scoped_cancel_ticket_implementation` | Ticket authority/assignment reads; Runtime cancel; current-session close; intact assignment cancellation transaction; detail | One accepted finite Host workflow owns cancellation through transaction and response. No Runtime wait on DB, no nested Host admission | Pre-authenticated generated POST route; real separate SQLite writer holds actual cancellation transaction; timer/control progress; cancelled waiter retains actual lease, then one operation/cleared assignment after drain. Saturated Host returns 503 without mutation. Existing live Coder cancellation policy passes |
|
||||
| `merge_request_selector_repair` → `scoped_repair_merge_request_selector` | Browser auth/explicit confirmation; MR store; current assignment; provider ref; intact repair/audit; public projection | Auth remains an owned DB read; confirmed post-auth mixed stage is one finite Host job | Browser auth is drained before real SQL/shared-mutex contention; post-auth stage stays Pending without blocking Tokio and retains cancelled Host lease. Imported NULL-selector/no-assignment fixture rejects without repair/thread effects. Host overload preserves confirmation priority. Successful provider repair is not claimed by this narrow fixture |
|
||||
| restore routes → `restore_runtime_worker_with_context` → `project_worker_restore_response_owned` | Existing authorized Host restore returns historical result; then whole stored Worker/links/Workdir/capability projection | Whole independent DB stage. A result with no Worker remains a pure response and needs no DB admission | Real SQL writer/shared-mutex hold; timer and independent Host control progress; bounded overload; cancellation retains actual DB lease; historical state preserved and no registry observation write. This tests post-restore policy, not a complete restore endpoint/E2E |
|
||||
| Ticket detail/show/query and alias `get_ticket`; edit/event detail and assignment preparation | `TicketAuthority` projection includes open-MR `RuntimeTicketSourceCommitResolver` → synchronous provider HTTP | Mixed stages use finite Host, not sole DB. Subsequent pure assignment mutation stages keep their existing owner | Detail/query held loopback `/v1/repository-refs/observe`; unrelated DB/timer/control progress; cancel retains actual Host budget; overload stays 503. Provider-placement fixture is explicitly unsigned, not proof of signed request authority |
|
||||
| Ticket tracking/case decision authority | Retained assignment/CAS/body/event checks plus `find_workspace_orchestrator` → live `RuntimeRegistry::worker` | Whole existing decision helper on finite Host; transaction and notification order unchanged | Held loopback live Orchestrator lookup for both tracking/case; independent SQL/timer/control; cancelled actual Host lease retained; failed live authority leaves Ticket unchanged. Existing non-assignee/browser bypass denial passes |
|
||||
| Browser Ready | Existing mutation/notifications, followed by detail (including source ref observation) | Mutation and detail use one accepted Host job and explicit `execute_ticket_rest_operation_on_host`, never a second Host admission | Fake promptly denies notification GET, holds the subsequent source-ref POST; exactly one Ready event/readiness is durable before held detail; SQL/timer/control progress and cancelled Host ownership persist. Rejected fixture setup/misidentified notification GET are excluded |
|
||||
| Worker control input/cancel/stop/restore source preflight | Synchronous live source Runtime lookup precedes `resolve_async` and operation admission | `authenticate_worker_mutation_source_owned` admits finite Host; already-owned synchronous callers retain the original helper | Actual control handler's held source GET proves DB/timer/control progress and cancelled finite Host budget; saturation returns availability, not source denial. The fake's lookup fails before an actual Worker cancellation; no product process is used |
|
||||
| Worker observation capture | Source lookup, initial SQL grant, control lock, live grant re-fence and target Runtime lookup, awaited cold protocol connection | Source on Host; initial grant on DB; actual re-fence/live lookup on Host owns the acquired control guard; cold protocol signing retains existing DB stage | Source/permission/cold-signing contracts remain covered separately. Full capture success/held-protocol composition is not claimed by the new source lookup fixture |
|
||||
|
||||
## Regressions and causal checks
|
||||
|
||||
New `server_residual_execution_tests.rs` contains eight focused tests, including
|
||||
three read/Ready and two decision-authority table cases. Exact old methods or the
|
||||
old direct stored-projection placement were restored diagnostically in turn for
|
||||
cancel, repair, restore, detail, tracking, and control. All six fail at their expected
|
||||
first-poll/shared-mutex/independent-SQL boundary. Five-second fixture watchdog
|
||||
release and its induced SQLite BUSY/panic are diagnostic bailout, not progress.
|
||||
The product source is restored in `finally` after each case.
|
||||
|
||||
Artifacts (local, not tracked product code):
|
||||
- `target/t748-residual-old-{cancel,repair,restore,detail,tracking,control}-red.log`
|
||||
- `target/t748-residual-final-restored-green.log`: **26 DB-boundary tests pass**
|
||||
- `target/t748-residual-decision-policy.log`: non-assignee/browser bypass policy passes
|
||||
- `target/t748-residual-root-check.log`: root compile closure passes
|
||||
- `target/t748-residual-provisional-full-suite.log`: original default-parallel Server
|
||||
**979 passed / 0 failed / 1 existing ignored**, binary11; **178.38 seconds**.
|
||||
Only the three deliberate executor unknown-outcome panic diagnostics appeared.
|
||||
No retained test/repro processes were observed after completion.
|
||||
- `target/t748-residual-provisional-workdir-{default,http}.log`: default153 and
|
||||
http-client156 tests pass. This does not complete the pending transitive audit.
|
||||
- fmt/diff and state-contract names **558 / 0 unapproved / 0 stale** pass.
|
||||
|
||||
All Cargo validation uses `target/t748-candidate-build`; no parallelism reduction,
|
||||
production budget increase, skipped tests or mutation retry was introduced.
|
||||
Initial test-only private enum/state/DTO mistakes failed compilation; missing
|
||||
repository seeding and Ready notification GET confusion were diagnosed and fixed.
|
||||
Those attempts are not semantic proof. The full-suite result above is provisional:
|
||||
it does **not** conclude the caller inventory or approval.
|
||||
|
||||
## Remaining inventory (not complete)
|
||||
|
||||
`target/t748-async-entry-candidates.tsv` records 661 lexical async-entry candidates
|
||||
from Server, Worker operations, auth, projection and subscription sources. Visible
|
||||
lane markers/await counts are discovery hints, not an AST/type-checked call graph
|
||||
or proof that all transitive operations are safe. Forwarders, actual handlers,
|
||||
service adapters and callback/background callers must be matched by name and
|
||||
traced through their real dependencies before marking a family inspected.
|
||||
|
||||
| Inventory work | Current status / next evidence |
|
||||
|---|---|
|
||||
| Ticket/MR/Objective/Memory/Subjektiv/Backend Job: canonical/alias/browser/Runtime entries, assignment/routing/decision/projection/publication and callback edges | First committed child result inspected. Major handler owner table returned, but follow-up final result was not observable. Parent released the child and is directly checking remaining adapters/forwarders/launch edges. Macro trait dispatch and representative Objective/Memory/Ticket forwarding plus legacy Memory/cleanup owners inspected; complete per-operation mapping remains open |
|
||||
| Auth/Runtime/Worker/control/observation/resources/Workdir/repository/Drive/settings/config/catalog/uploads/sessions/subscriptions and lifecycle background edges | First committed child result inspected; parent verified the residual structures below against actual source. The follow-up result was not observable after two Idle notifications; parent released the read-only child and directly traced the remaining four edge groups below. This is not delegated completion evidence |
|
||||
| Existing pure DB, mixed Host, mandatory Subject callback and accepted publication boundaries | Earlier documented evidence remains relevant; unchanged code does not exempt newly found composed callers from tracing |
|
||||
| `WorkspaceWorker::stop` first Host result → waiter-owned session close → second Host stage | Parent verified the split in actual source. Structural ownership gap, not yet a cancellation reproducer; tracked below rather than silently marked safe |
|
||||
|
||||
## Additional residuals found before republication (unfixed)
|
||||
|
||||
The first two read-only audit results were inspected through their committed
|
||||
sessions. The parent then read the concrete handler/helper source listed below.
|
||||
These are **static execution-placement/continuation findings**, not new successful
|
||||
endpoint, cancellation or overload tests. None has been repaired or published in
|
||||
this snapshot. The provisional suite above therefore cannot establish safety for
|
||||
these families. Lower recovery/transport edges are still being traced by the
|
||||
parent after releasing the read-only children; this is not independent review.
|
||||
|
||||
`S` below means `crates/workspace-server/src/server.rs` at the frozen hash;
|
||||
`O` means `crates/workspace-server/src/server/worker_operations.rs`.
|
||||
|
||||
| Entry / family | Entry → authorization → synchronous work → completion | Verified residual and required boundary |
|
||||
|---|---|---|
|
||||
| `repository_access_projection`, `repository_ssh_credential_{list,create,generate,get,delete,public_key,rotate}`, `repository_ssh_host_trust_{list,put,get,delete}` | ServerApi entries `S:7879–8107` directly await handlers `S:12934–13129`; owner authentication is awaited first, then active config projection / secret / trust / CAS / audit helpers execute synchronously | Auth ownership does not carry past await. Parent read all handlers and actual `repository_access.rs:391–436` connection reads. Whole existing synchronous operation needs an owner; pure DB and any external provider work must be classified by dependency, not by the SSH label |
|
||||
| `repository_create` → `scoped_create_repository` | Owner authentication await → validation → `config_store.insert_repository` → intent/replay classification → response, `S:23157–23214` | Insert is outside an accepted job. Preserve the existing intact insert/intent/commit helper and successful replay vs conflicting intent semantics |
|
||||
| `repository_ssh_connection_probe`, `repository_ssh_host_trust_confirm` | Owner and active binding awaits → catalog / `probe_repository_ssh_connection` → Runtime observation → trust read/CAS/write → response, `S:23247–23358` | Post-auth mixed work has no finite Host owner. Confirm must keep observed host-key selection and fingerprint comparison before the existing trust mutation |
|
||||
| `worker_session`, `worker_session_history`, `scoped_worker_session_attachment` | Workspace checks → synchronous Runtime Session API → response/body, `S:8696–8769,12011–12069`; registry facade `hosts.rs:2129–2195` | No finite Host admission at these request boundaries. Binding/signing and remote transport dependencies must stay off Tokio and out of the sole DB lane; preserve Session diagnostics and response limits |
|
||||
| `runtime_worker_attachment_upload` → `upload_runtime_worker_file_contract` | Grant claim → registry resolution → Runtime upload → grant completion or cancellation artifact deletion, `S:28081–28152` | Whole body-upload operation is still direct synchronous mixed work, distinct from the already-owned grant/cancel/delete routes. Admission must precede grant claim; accepted upload must own completion and cancellation cleanup |
|
||||
| Drive Worker authorization / `drive_grant_create` | `server_drive_grants.rs:54–74,126–166`: source/header checks → `current_worker_identity` or live Runtime worker → later DB grant precheck/mutation | Pre-DB source/live Runtime lookup has no Host owner. `S:18244–18292` confirms `current_worker_identity` directly reaches synchronous `runtime.worker`. Retain no-actor-fallback policy; owner unavailability must not be misreported as authority denial |
|
||||
| `current_worker_workspace_config_{get,attach,observe,read,commit}` | `S:11625–11699` directly invokes `workspace_config::identity`; `server_workspace_config.rs:65–74` reaches the same synchronous Runtime source helper before later owned DB stages | All five logical-config entries share the unowned identity preflight. Later accepted persistence/publication does not cover this earlier stage |
|
||||
| `workspace_workdir_list` → `scoped_list_working_directories` | DB job `S:25943–25959` → summaries `32974–32999` → all Runtime workdir observations `33515–33524` → list/detail Runtime calls and SQL reconciliation `33421–33467` | Mixed Runtime/SQL helper is incorrectly on the sole DB worker. Preserve live reconciliation and diagnostics, but move the whole mixed stage to finite Host rather than pretending the list is stored-only |
|
||||
| `workspace_config_tree_commit` → `scoped_commit_workspace_config_tree` | DB job `S:13444–13460` → prepare → repository-access projection → config commit → Runtime prompt publication, `13498–13517`; remote publication `hosts.rs:5090–5099` | SQL commit is followed by synchronous network publication inside sole DB job. Keep commit-before-best-effort-publication and accepted ownership; do not leave the continuation on the waiter. Logical config's separate owned Host dispatch is not this UI path |
|
||||
| `repository_{list,detail,log}` and aliases | DB jobs `S:29108–29183` → live repository reader → summary/log provider; `repositories.rs:276–375` invokes Git subprocess for materialized local repositories | Provider work is not pure DB. Mixed catalog/provider work needs finite Host, without changing existing diagnostics for unmaterialized sources |
|
||||
| `worker_control_stop`, `runtime_worker_stop`, `runtime_worker_stop_alias` → `WorkspaceWorker::stop` | `O:379–425`: grant/lock/re-fence → Host #1 Runtime stop returns result/control guard → waiter session lock/close → Host #2 linked Workdir observation/SQL | Structural ownership gap confirmed by parent: accepted Runtime stop has no owner for session close and subsequent Workdir synchronization if its waiter vanishes. `S:18609–18619` awaits real session close; the second Host admission also occurs after side effects. Whole accepted stop must own this continuation without nested Host acquisition. Cancellation reproduction remains pending |
|
||||
|
||||
### Parent takeover of the four remaining Runtime edge groups
|
||||
|
||||
The Runtime child's first result remains useful, but its follow-up final result
|
||||
was not available through authorized observation. The parent did not infer
|
||||
completion from Idle and did not loop Submit or inspect raw session storage.
|
||||
See `docs/report/2026-10-12-worker-observation-followup-gap.md` for the bounded
|
||||
observations and handling. The parent read the following actual source directly:
|
||||
|
||||
| Edge | Source and actual stages | Classification / unresolved contract |
|
||||
|---|---|---|
|
||||
| Embedded protocol connect and method pump | `S:31845–31902` directly calls synchronous `attach_worker_protocol_scoped` before spawning its async pump; the pump directly calls `send_connected_protocol_method_scoped`. `worker-runtime/src/runtime.rs:3486–3598` takes synchronous operation/state locks; Shutdown performs `begin_stop_operation` and `commit_stopped_worker` (`4359–4445`), with durable record/snapshot writes in fs-store builds (`5256–5269`) | **Additional confirmed placement residual**: Backend async connect/pump is not an accepted finite Host owner. The existing endpoint identity/settled-operation checks and commit/publication helpers must remain intact; no Runtime-store refactor is needed |
|
||||
| Workdir creation reservation → Runtime result → finalization | `S:26316–26609`: DB preparation/reservation, Host runtime selection/observation/create, then waiter-admitted failure or final registry/receipt DB jobs. Same-id reentry observes the reserved Runtime Workdir before attempting create (`26473–26517`) | **Confirmed continuation split**, with durable same-operation recovery intent, not automatic mutation retry. Existing recovery does not make the original create job own finalization after its waiter disappears. Final classification/regression must cover accepted Runtime outcome, final receipt, cancellation and overload without creating a second provider operation |
|
||||
| External grant fence → attachment/provider cleanup | `S:25595–25709`: DB authority/read or revoke/fence/audit, then waiter-owned cleanup/provider stop; terminal GET/revoke explicitly retries cleanup. Cleanup itself commits pending, awaits release, then admits completion/audit DB (`25719–25799`). Expiry task retains its async owner and retries (`24790–24854`) | **Confirmed continuation split plus explicit recovery**, not a claim that revocation failed or is unsafe after its DB fence. Grants remain denied by their committed fence. Need causal cancellation/lease evidence for cleanup and preserved retry/expiry policy before choosing the smallest ownership fix |
|
||||
| Command provider acceptance → registration/audit | `S:19507–19631`: Host identity, DB attachment fence, async session/provider, then synchronous command-handle registration. External `operate` admits audit-requested DB, sends provider request with explicit cancel guard, receives result, then admits audit-completed DB (`888–1024`); External command-handle registration follows another await (`1372–1428`). Local command start retains Runtime resources in its actual task and owns terminal-close admission (`workdir/src/local.rs:1831–1897`) | **Partially protected / continuation question**. SQL jobs are owned and local actual command resources are retained; explicit external cancel is not SQL retry. Provider acknowledgement-to-registration can cross a cancellable audit await. Do not assert an orphan process or whole command success without a focused protocol/registration reproducer; do not rewrite external cancellation semantics speculatively |
|
||||
| Catalog and single-workspace shutdown | `S:12222–12279`: catalog waits for HTTP serve, drains Worker projection and owns Feature shutdown DB (`2651–2664`); single-workspace `serve` aborts its hook but calls synchronous Feature shutdown directly. `feature-storage/src/lib.rs:220–241,521–523` fences and waits on connection close. Executor `drain` (`db_executor.rs:290–315`) covers only jobs accepted before its first poll, not later submissions | **Additional confirmed direct shutdown placement residual** in single-workspace serve. Separately, catalog does not demonstrate hook/expiry/Host/cleanup producer quiescence before storage shutdown. Whole producer/drain ordering remains a lifecycle question; no live shutdown was invoked to test it |
|
||||
|
||||
Domain generated adapters, remaining Tool/WIP forwarding, Backend Job / legacy
|
||||
Memory launch lower edges remain under follow-up inspection. Existing domain SQL
|
||||
helpers are not being reimplemented or globally refactored; ownership tracing is
|
||||
separate from re-auditing every SQL semantic branch. None of the additional rows
|
||||
above has new executable regression evidence yet.
|
||||
|
||||
|
||||
T743's additional assignment page/diagnostics/answer APIs are absent from this
|
||||
source. Ticket seq8–9 identifies their separate owner's normal composition with
|
||||
the published T748 facade. This inventory is **not** an attestation of that future
|
||||
combined source; no duplicate API or combined approval is introduced here.
|
||||
|
||||
## Republication conditions
|
||||
|
||||
Review-ready requires a consolidated caller-name list with each entry's auth,
|
||||
SQL/provider/Runtime stage, post-await projection, post-commit publication and
|
||||
owner identified; confirmed residuals repaired with focused causal regressions;
|
||||
remaining uncertainty explicit; and final root/changed-crate/affected-feature,
|
||||
format/diff/name checks on that final source. Only then normal same-selector push,
|
||||
provider exact-HEAD verification and fresh trusted independent review of this MR.
|
||||
No new Reviewer Runtime or separate Coder is created. Read-only audit children do
|
||||
not grant review verdict or integration authority.
|
||||
|
||||
## Scoped caller repair and validation history
|
||||
|
||||
The same Ticket branch is retained for advancing the existing MR. This section
|
||||
records validation history, not approval or integration. The final-validation
|
||||
section below supersedes the intervening failed runs without deleting their evidence.
|
||||
The existing domain helpers,
|
||||
authorization fences, receipt/replay rules, and transaction/publication order
|
||||
remain intact. No automatic mutation retry, production limit increase, nested
|
||||
Host admission, or test parallelism reduction was introduced.
|
||||
|
||||
| Repaired family | Current owner / executable evidence |
|
||||
|---|---|
|
||||
| Repository access/SSH credential and host-trust CRUD, Repository registration | Whole post-owner-auth synchronous operation on DB. Registration regression observes the **config store's actual connection** waiting behind a separate SQLite writer, then cancels the waiter; after drain exactly one Repository exists and the same registration intent returns HTTP200/replayed without another row |
|
||||
| SSH probe/host-trust confirm | Whole post-owner/binding-auth mixed probe/observed-key selection/fingerprint comparison/trust mutation on Host. Existing narrow SSH/trust policy tests pass in the full Server suite; no successful real SSH connection or host-key exchange is claimed |
|
||||
| Repository list/detail/log and aliases | Live Git/catalog helpers on Host, not DB. With DB admission saturated, empty catalog and absent-key detail/log retain their ordinary success/404 responses. This regression proves lane selection, not held Git subprocess behavior |
|
||||
| Worker Session/history/attachment | Typed Session service and attachment handler admit Host before synchronous Runtime observation. Loopback held Runtime lookup proves independent SQL/timer/control progress, cancelled actual lease retention, bounded overload and release after actual drain |
|
||||
| Body upload | Host admission precedes the upload grant claim and owns resolution/upload/completion/cancellation cleanup. The same loopback table holds the Runtime lookup after claim; cancelled waiter leaves actual owner, and provider refusal completes the claim back to Pending. Successful artifact upload/delete and established remote sessions are not exercised by this refusal fixture |
|
||||
| Drive Worker authorization / grant create; all five logical-config identity entries | Live source/target preflight uses Host; later DB authorization and mutations retain their previous transaction owner. Three held-lookup table cases prove independent SQL/timer/control, cancelled lease retention, and typed HTTP503 availability (not false authority denial) when Host admission is full. Existing no-actor-fallback and source-fence policies remain in the suite |
|
||||
| Workdir list | Whole live Runtime reconciliation/SQL helper on Host. Held Runtime list lookup in the Session/Workdir/upload table proves sole DB remains free and cancellation retains the actual Host budget |
|
||||
| UI config commit | Whole existing evaluate/project/commit/best-effort prompt publication helper on Host. Loopback held prompt POST confirms the changed tree is already durable while independent DB/timer/control proceeds. Cancelling the waiter retains publication ownership; refused publication neither repeats nor rolls back the commit |
|
||||
| Worker stop canonical/alias/control entries | One accepted Host workflow owns Runtime stop, session fence/close, stored read and linked Workdir synchronization. An in-process Runtime emits its stop receipt while a real session fence is held; after waiter abort and fence release the registered session is removed. Restoring the exact old stop method makes this test fail at the missing cleanup assertion; restoring the fix passes. No real Worker/product process is spawned or stopped. Linked Workdir provider success is not claimed by this session-only fixture |
|
||||
| Embedded protocol connect/method; single-workspace Feature shutdown | `4d981194` owns these operations on the existing finite Host/DB lanes. Focused protocol and actual Feature connection shutdown regressions remain green in the final suite |
|
||||
|
||||
Final logs in the Workdir (build artifacts, not committed product content):
|
||||
- `target/t748-stop-old-placement-red.log`: old stop fails **missing cleanup**, not timeout.
|
||||
- `target/t748-stop-restored-green.log`: restored accepted-stop ownership passes.
|
||||
- `target/t748-caller-host-green.log`: Session/history/attachment/Workdir/upload table passes.
|
||||
- `target/t748-repository-create-green.log`: actual SQLite commit/cancel/replay passes.
|
||||
- `target/t748-drive-config-preflight-green.log`: all three live authority cases pass.
|
||||
- `target/t748-ui-prompt-owner-green.log`: commit-before-publication passes.
|
||||
- `target/t748-repository-provider-green.log`: all three Repository read lanes pass.
|
||||
- `target/t748-caller-final-root-check.log`: workspace-root `cargo check` passes.
|
||||
- `target/t748-caller-final-server-suite.log`: original default-parallel Server
|
||||
**988 passed, 0 failed, 1 existing ignored**, plus **11 binary tests**, 185.38s.
|
||||
- `target/t748-caller-final-workdir-default.log`: **153** tests pass across targets.
|
||||
- `target/t748-caller-final-workdir-http.log`: **156** tests pass across targets.
|
||||
- fmt/diff checks pass; `target/t748-caller-final-names.log`: **558 / 0 unapproved / 0 stale**.
|
||||
|
||||
Test setup errors are excluded from semantic evidence: the first stop fixture
|
||||
omitted the Runtime's live Worker summary; the Repository busy observer initially
|
||||
watched the general store rather than the config store's distinct connection.
|
||||
Both were corrected before the successful runs. Missing package/Python command
|
||||
and compile-time DTO/Duration/Drive-error mistakes also prove no product behavior.
|
||||
|
||||
### Explicit limits, not an expanding audit gate
|
||||
|
||||
The four lower-edge rows above separate confirmed direct placement repairs from
|
||||
Workdir create / external grant cleanup / provider acknowledgement continuation
|
||||
questions and catalog producer quiescence. The latter are not redesigned here,
|
||||
per the user's instruction not to expand speculative Workdir/grant or unrelated
|
||||
Feature work indefinitely. Their existing durable recovery, explicit cleanup
|
||||
retry and provider cancellation policy are preserved, not reclassified as
|
||||
end-to-end cancellation proof. This report is not a mechanically complete
|
||||
transitive call graph or an assertion that every future/composed caller is safe.
|
||||
T743's separate new API composition still requires its owner's exact combined
|
||||
source validation/review. None of these limits constitutes Ticket conclusion,
|
||||
MR approval, target integration or permission to update the running environment.
|
||||
|
||||
### Post-suite focused validation blocker
|
||||
|
||||
After the successful full suite above, the current candidate was checked with
|
||||
`cargo test -p yoi-workspace-server --lib db_execution_tests` using the same
|
||||
candidate target directory and ordinary parallel execution. Result: **31 passed /
|
||||
1 failed**, in 8.06s. Failure:
|
||||
`cancelled_runtime_key_revoke_commits_audit_and_removes_broker_registration`
|
||||
reported the shared `observed` helper's two-second `Elapsed(())` timeout, without
|
||||
identifying which of its several lifecycle waits timed out. Source inspection
|
||||
confirms projection shutdown is already present in this fixture; it is not
|
||||
legitimate to attribute the failure to omitted projection shutdown, treat it as
|
||||
load-only, increase limits, reduce parallelism, or rerun until green.
|
||||
|
||||
Evidence is preserved in `target/t748-caller-final-boundaries.log`. The prior
|
||||
full-suite success is real but insufficient to declare final validation complete.
|
||||
The new caller and stop cases all passed in this focused run. Next diagnosis must
|
||||
identify the exact wait and inspect actual HTTP/job/broker ownership events before
|
||||
choosing a fixture or product repair. No normal push, fresh review, approval,
|
||||
integration or Ticket conclusion was performed for this candidate.
|
||||
|
||||
## Continued diagnosis and final validation
|
||||
|
||||
Returning the failed test as a stop condition was a workflow mistake, not an
|
||||
external outage, lost authority, or additional audit approval gate. Work continued
|
||||
on the same branch. The failed run above is retained as evidence.
|
||||
|
||||
The `observed` helper now reports its caller location. A diagnostic 32-case run
|
||||
passed; that result alone was **not** treated as a diagnosis or used to erase the
|
||||
failure. Source tracing identified an uncontrolled cold broker signing producer
|
||||
in the revoke fixture. The test now receives the actual signed WebSocket
|
||||
handshake and drains its completed DB work **before** holding the revoke writer.
|
||||
The handshake remains pending, so broker shutdown is still tested. An early HTTP
|
||||
response is selected against actual SQLite entry, reporting status/body rather
|
||||
than waiting for an event that can no longer occur. The original failed run did
|
||||
not identify its exact wait; attributing it conclusively to signing would exceed
|
||||
the captured evidence.
|
||||
|
||||
Subsequent full runs exposed the Ready table's distinct failure **before** source
|
||||
POST arrival (caller location, not timer/control failure). Artifacts:
|
||||
`target/t748-post-fixture-server-suite.log` and
|
||||
`target/t748-final-framing-server-suite.log`. The HTTP fake previously assumed
|
||||
one TCP read contained the request line and answered without consuming a body.
|
||||
It now parses the complete header/body; a deterministic two-byte-fragment reader
|
||||
proves complete frame consumption and truncated-body rejection. This is a real
|
||||
fixture flaw but does not by itself establish the historical timeout cause.
|
||||
|
||||
Ready setup includes mutation and best-effort notification before the source
|
||||
POST. The held-provider arrival now uses the existing five-second fixture
|
||||
watchdog as its setup bound, selected against handler completion. The subsequent
|
||||
**timer and independent SQL/control observations retain their two-second budget**;
|
||||
actual held-request cancellation/lease/overload/cleanup assertions are unchanged.
|
||||
Production admission limits, normal test parallelism, and provider timeout are
|
||||
unchanged. This setup-bound change is explicit, not a hidden limit increase or
|
||||
claim that Ready completes within two seconds under a whole-suite workload.
|
||||
Restoring the exact published old Ready method still fails this revised test
|
||||
(`target/t748-ready-old-boundary-red.log`, Ready returns only after synchronous
|
||||
provider wait); restored code passes, so the offload distinction is not weakened.
|
||||
|
||||
Final code validation after these fixture changes:
|
||||
- `target/t748-boundary-setup-fixed-green.log`: **33 DB-boundary cases pass**.
|
||||
- `target/t748-final-setup-root-check.log`: root compile closure passes.
|
||||
- `target/t748-final-setup-server-suite.log`: original default-parallel Server
|
||||
**989 passed / 0 failed / 1 existing ignored**, **11 binary tests**, 189.66s.
|
||||
- `target/t748-final-setup-workdir-default.log`: **153 pass** across targets.
|
||||
- `target/t748-final-setup-workdir-http.log`: **156 pass** across targets.
|
||||
- `cargo fmt --all -- --check`, `git diff --check HEAD`, and state-contract names
|
||||
**558 / 0 unapproved / 0 stale** pass.
|
||||
|
||||
No failed semantic run was merely repeated until green: the intervening source
|
||||
changes and the limits of each diagnosis are recorded above. These results support
|
||||
publication for fresh exact-source independent review, not reviewer approval,
|
||||
MR integration, Ticket conclusion, or updates to the live dogfooding environment.
|
||||
@@ -0,0 +1,444 @@
|
||||
# T-748: default-parallel suite failure diagnosis
|
||||
|
||||
## Sources and preserved observations
|
||||
|
||||
- Original local candidate: `61533fadb6b39465af60cb55aed2b4d0650319cd`.
|
||||
- Comparison source: `56795fb92cb9def9bcd701bd51d92e99526f8eff` (develop/T742).
|
||||
- Same implementation branch: `work/T-748-bounded-db-execution`.
|
||||
- The original `target/t748-server-tests.log` remains untouched. Its 97 FAILED
|
||||
lines and 600-second timeout were symptoms, not proof of one cause.
|
||||
- Initial diagnosis used `cargo test -p yoi-workspace-server -- --nocapture`,
|
||||
default libtest parallelism, without changing queue limits, features, or startup
|
||||
settings. `target/t748-diagnostic-environment.txt` records the environment;
|
||||
this host reports 24 available CPUs. Diagnostic captures were bounded to
|
||||
90/150 seconds to obtain errors before the original timeout, not counted as
|
||||
successful full-suite runs.
|
||||
|
||||
Logs under `target/` are local diagnostic artifacts, not tracked review inputs.
|
||||
This report records their meaning and the permanent regressions that reproduce it.
|
||||
|
||||
## 1. Different fixtures consumed one global Host admission budget
|
||||
|
||||
The first non-intentional errors in `target/t748-server-nocapture.log` were
|
||||
`DbCapacityExceeded` in manual Worker setup (`server.rs` at the then-current
|
||||
line 50614), followed by assignment fixture construction (52990/53045) and
|
||||
HTTP 503 responses. The deliberate before/after-commit executor panic tests
|
||||
are different: those passed and verified unknown-outcome diagnostics.
|
||||
|
||||
Temporary submit/start/cleanup instrumentation recorded submitting test names,
|
||||
closure types, actual job IDs and the execution-owner identity. It showed a
|
||||
single Host lane with `max=18`, with accepted jobs from unrelated fixtures.
|
||||
A rejection snapshot contained `{32,45,46,...,61}`. The complete diagnostic
|
||||
capture later contained cleanup for every submitted Host job on that owner;
|
||||
it did not show permanently abandoned accepted Host jobs. Instrumentation was
|
||||
removed before the source was committed/published.
|
||||
|
||||
`target/t748-live-process-18.txt` sampled the test process and threads at the
|
||||
actual rejection. That sample had 126 threads; the Host workers were runnable,
|
||||
and the process was using approximately 17 CPU cores. There was no captured
|
||||
thread-creation EAGAIN/unavailable error. This evidence does not establish an
|
||||
OS thread/CPU hard limit as the cause. No test processes remained after the
|
||||
bounded diagnostic run or the completed suite.
|
||||
|
||||
A representative failure passed alone (0.56 seconds); the eight logging cases
|
||||
passed with four test threads (2.46 seconds). The isolated comparison source's
|
||||
full default-parallel suite passed: 931 library tests, one pre-existing ignored
|
||||
migration fixture, 11 binary tests; library duration 168.74 seconds. This did
|
||||
not prove that all new failures were capacity-related, but did distinguish the
|
||||
new shared-admission rejection from an observed baseline failure.
|
||||
|
||||
### Ownership fix (`f3d4aba5`)
|
||||
|
||||
Production budgets remain shared and finite: DB 1 worker/32 queued, Host 2/16,
|
||||
cleanup 2/16. Ordinary fixtures now explicitly own a finite DB executor and lazy
|
||||
Host/cleanup owners at those same limits. API clones, child Workspace APIs, and
|
||||
weak retention adapters use their injected owner. No global fixture lane cache,
|
||||
parallelism reduction, unlimited queue, or capacity-wait retry was added.
|
||||
|
||||
The permanent saturated-Host regression fills both running slots and all 16
|
||||
queued slots, cancels waiters, advances paused Tokio time, and still observes
|
||||
rejection until actual completion. Another fixture and the separate cleanup lane
|
||||
can progress; draining restores the original budget and completes exactly 18
|
||||
accepted jobs. Existing real-SQLite cancellation/lease/drain tests remain active.
|
||||
|
||||
## 2. SQL offload invalidated fixture assumptions about first poll and publication
|
||||
|
||||
After Host isolation the default-parallel suite finished in 171.57 seconds:
|
||||
954 passed, three failed, one existing ignored. Actual errors were:
|
||||
|
||||
- removal subscriber saw a legitimate queued Runtime stop Upsert before Removed;
|
||||
- external session-open expiry exceeded its one-second bound while its DB lane
|
||||
was still shared across other fixtures;
|
||||
- grant replacement returned success because one initial poll had only submitted
|
||||
the offloaded authority read, not reached the held original-grant lock.
|
||||
|
||||
The ordinary DB fixtures were isolated too. The one-second expiry assertion was
|
||||
not increased. Grant testing now drains its actual authority read then polls the
|
||||
saved response to reach the held lock before replacing the grant. Removal testing
|
||||
observes the committed stopped projection before taking the removal baseline.
|
||||
The Removed payload assertion is unchanged. These are observable synchronization,
|
||||
not sleeps or retries. Focused expiry, grant, and removal tests all passed.
|
||||
|
||||
A captured cleanup job also attempted to poll a Tokio timer after its fixture
|
||||
runtime had shut down. The terminal checker fixture now retains that original
|
||||
runtime through accepted Host and cleanup jobs, then stops projection producers
|
||||
and drains actual DB work before ending. Terminal SQL state is not treated as
|
||||
actual cleanup completion. The subsequent full run contained only the three
|
||||
intentional executor panic diagnostics, not the shutdown-context panic.
|
||||
|
||||
The synchronization/teardown changes are in `df9d056d`.
|
||||
|
||||
## 3. Restore intent's DEFERRED read-to-write upgrade returned SQLite BUSY
|
||||
|
||||
The next default-parallel run finished in 171.48 seconds with 956 passed, two
|
||||
failed, one existing ignored. Both failures were SQLite `DatabaseBusy`, extended
|
||||
code 5, not capacity rejection:
|
||||
|
||||
- production-profile zero-input Orchestrator restore;
|
||||
- pending manual Worker claim restore reconciliation (500 instead of 409).
|
||||
|
||||
Both failed alone too. `target/t748-busy-origin.log` captured the error and a
|
||||
backtrace through `pin_internal_worker_restore_intent` -> the transaction helper
|
||||
-> internal restore -> manual claim. The journal read authority first in a
|
||||
DEFERRED transaction while the offloaded projection's other SQLite connection
|
||||
could write. Such an upgrade can return BUSY without invoking SQLite's busy
|
||||
handler; treating this as a retryable whole mutation would be incorrect.
|
||||
|
||||
A new real-SQLite regression holds an IMMEDIATE writer on another connection,
|
||||
registers an owned per-connection busy observer, and attempts one journal pin.
|
||||
The old helper fails deterministically with BUSY/5 before waiting for ownership
|
||||
(`target/t748-deferred-upgrade-red.log`, 0.23 seconds). The fixed helper waits
|
||||
before its first authority read, while a current-thread timer and control task
|
||||
progress. After releasing the writer it persists exactly one requested tuple;
|
||||
there is no application-level mutation retry.
|
||||
|
||||
`f597f1b5` makes this existing mutation helper explicitly
|
||||
`with_immediate_transaction`. Restore journal and deletion authority reads,
|
||||
mutation and commit remain in one transaction. Focused DB/HTTP tests (8), the
|
||||
original restore and claim cases, and deletion tests (13) passed.
|
||||
|
||||
## 4. Why a pre-dispatch failure could become the 600-second suite timeout
|
||||
|
||||
The original long-running logging case joined an HTTP request with a fake provider
|
||||
waiting for command dispatch. On pre-dispatch rejection the HTTP future can finish
|
||||
with 503 while the receiver remains Pending: its channel is still owned by the
|
||||
fixture, but no command will arrive. An unbounded `join!` waits forever in that
|
||||
state and hides the actual error behind the outer suite timeout.
|
||||
|
||||
The permanent `saturated_host_before_command_start_returns_availability_without_a_provider_operation`
|
||||
regression uses the real signed HTTP route with a controlled full Host budget. It
|
||||
observes the final 503/`db_capacity_exceeded` response and the still-live Pending
|
||||
provider receiver, with no process or command dispatch. The original logging
|
||||
fixture now asserts an early HTTP response before waiting for both branches, so
|
||||
an unexpected rejection reports its body immediately rather than hanging. This
|
||||
reproduces the causal timeout mechanism; the old log itself has no stack trace
|
||||
from the hung provider future.
|
||||
|
||||
The nine logging cases pass together at default parallelism (`49bfbfb2`).
|
||||
|
||||
## Artifact/source isolation during comparison
|
||||
|
||||
Initially the archived baseline used the existing Cargo target directory. That
|
||||
replaced a same-package artifact/fingerprint; one later post-fix representative
|
||||
command ran the baseline's 932-test binary (and `db_executor --list` had no tests).
|
||||
That result was rejected as candidate evidence. All later candidate checks/tests
|
||||
use `--target-dir target/t748-candidate-build`, separate from the baseline cache.
|
||||
This changes artifact location, not test parallelism, features or admission policy.
|
||||
The fresh candidate build listed the expected executor tests and total test count.
|
||||
|
||||
## Validation and remaining composition boundary
|
||||
|
||||
Pre-review code source `49bfbfb2` passed the full default-parallel Server suite,
|
||||
including the command-overload case: 960 passed, zero failures, one existing
|
||||
ignored migration fixture; 11 binary tests, library duration 175.29 seconds.
|
||||
The only captured panics were the three deliberate unknown-outcome tests.
|
||||
Root `cargo check` compile closure passed (including TUI/Server), as did Workdir
|
||||
default (153 tests) and `http-client` (156 tests) full suites, format/diff checks
|
||||
and state-contract names (558 matches, zero unapproved, zero stale exceptions).
|
||||
These commands used the isolated candidate artifact directory; libtest default
|
||||
parallelism and the production budgets were unchanged. The MR records the exact
|
||||
published source, which also includes this report and the boundary document.
|
||||
|
||||
## Post-publication independent review and causal fixes
|
||||
|
||||
MR `01a12d3f-e90c-7b02-9ed9-cf3bd2e822a1` seq3 independently reviewed published
|
||||
source `66bb9d0b` and requested changes despite the successful original suite.
|
||||
Its two major findings exposed boundaries the earlier tests did not prove:
|
||||
|
||||
- `T748-DB-NETWORK-LANE`: mixed Subjektiv helpers could wait for synchronous
|
||||
Runtime HTTP on the sole DB worker. Fix `d3fcfb29` places session read/search,
|
||||
subject projection, memory staging, and legacy consolidation-scope helpers on
|
||||
the existing finite Host owner; pure DB-resident operations keep DB placement.
|
||||
Whole transaction helpers remain intact, and no Runtime I/O is introduced
|
||||
inside a SQL transaction.
|
||||
- `T748-ASYNC-PROTOCOL-SIGNING`: cold protocol source lookup called live binding
|
||||
checks and signing directly on Tokio. Fix `c0223a0d` awaits the owned DB stage.
|
||||
The source API now propagates failure explicitly: an invalid live binding or
|
||||
signing/admission failure cannot be hidden as `None` or a legacy bearer fallback.
|
||||
|
||||
Permanent real loopback HTTP regressions hold upstream Runtime replies during
|
||||
session read/search, subject and legacy-scope requests, while independently
|
||||
observing SQL, timer and control progress. Cancelling waiters retains the actual
|
||||
Host lifecycle lease until completion. Cold protocol source tests control another
|
||||
SQLite writer and shared connection mutex wait, and verify bounded overload,
|
||||
fresh signing and revocation without fallback. Existing queue/cancellation and
|
||||
deletion evidence now observes the actual mutation owner; the mixed fixtures
|
||||
explicitly inject a small finite Host budget. Production budgets are unchanged.
|
||||
|
||||
Focused boundary tests passed (10). Restoring the old placement produced a
|
||||
failing regression, and fixed placement passed; an earlier syntax-rejected
|
||||
instrumentation attempt is excluded from evidence. Diagnostic artifacts under
|
||||
`target/` are not production source or replacement domain implementations.
|
||||
|
||||
Exact code source `c0223a0d` then passed root `cargo check` and the original
|
||||
**default-parallel** Server suite: **962 passed, zero failed, one pre-existing
|
||||
ignored**, 11 binary tests; library duration **173.36 seconds**. Only the three
|
||||
intentional executor panic diagnostics appeared, with no residual test processes.
|
||||
Workdir default (153) and `http-client` (156) full suites, format/diff checks and
|
||||
state-contract naming (558 matching lines, zero unapproved, zero stale) passed.
|
||||
All used the isolated `target/t748-candidate-build` artifact directory. Logs:
|
||||
`target/t748-review-fix-{root-check,full-suite,workdir-default,workdir-http,format,names}.log`.
|
||||
The documentation-only publication `b10b7315` did not change that tested code.
|
||||
MR seq5 independently confirmed the two old fixes and repeated the checks above,
|
||||
but requested changes for a new mandatory-callback cycle; see the next section.
|
||||
|
||||
## Mandatory callback must be independent of its waiting Host parent
|
||||
|
||||
MR seq5 finding `T748-HOST-CALLBACK-CYCLE` traced Host create/restore -> Runtime
|
||||
bootstrap -> mandatory Session attribution -> the same finite Host lane. Two
|
||||
waiting parents could consume both workers while successfully admitted callbacks
|
||||
could not run. Cancelling a waiter correctly retains the actual parent, so it is
|
||||
not a solution. The Reviewer's compiled executor artifact proved this placement
|
||||
cycle, not a complete create endpoint/E2E reproduction.
|
||||
|
||||
Fix `a2487b57` separates store-only scope discovery into typed Subject, current
|
||||
Job consolidation and legacy consolidation authority. Session attribution accepts
|
||||
only Subject authority and executes its whole existing SQL operation on the
|
||||
independent DB lane. It rejects consolidation roles without calling Runtime.
|
||||
Mixed read/search, subject projections and allowed legacy consolidation operations
|
||||
keep their Runtime checks on Host; the previous DB/network fix is not undone.
|
||||
|
||||
The same dependency audit identified Subject resident context reads used by
|
||||
Runtime bootstrap/restore. All Subject-only memory operations use store-only
|
||||
Subject authority and DB ownership too (resident summary/context, proposal
|
||||
validation, receipt lookup, explicit staging). Other memory operations retain
|
||||
Host ownership because legacy authority can require synchronous Runtime lookup.
|
||||
No transaction is split across an await and no production budget is increased.
|
||||
|
||||
The strengthened signed route regression holds both actual production-sized Host
|
||||
workers (2/16) in `run_host_workflow` parents and retains their create admission
|
||||
leases until callbacks finish. Resident and attribution routes authenticate real
|
||||
Runtime source proofs, check current singleton ownership, respond successfully,
|
||||
and persist both distinct Subject session attributions **before** either parent
|
||||
releases capacity. It also preserves Session callback reentry during deletion
|
||||
drain. Runtime's wait is modeled at this policy boundary; no product process or
|
||||
complete create endpoint is spawned. Separate denial coverage proves legacy
|
||||
consolidation callbacks return 403 without Runtime lookup, leave no attribution,
|
||||
and do not bypass authentication (401).
|
||||
|
||||
Only Session attribution is an allowed deletion-drain callback; that endpoint
|
||||
allowlist was not expanded. An initial fixture sent resident reads during deletion
|
||||
and correctly received 409. The fixture now tests those reads before deletion,
|
||||
then Session attribution while deletion waits. This is alignment with the existing
|
||||
fence, not a relaxed product gate or success-picking retry. A diagnostic Python
|
||||
invocation was unavailable before making any source change; only the subsequent
|
||||
Deno lane-restoration runs are causal red evidence.
|
||||
|
||||
Restoring just Session's old Host placement fails at its expected callback
|
||||
watchdog; restoring just resident memory's old Host placement fails at its own
|
||||
watchdog. Fixed placement succeeds without parent bailout. Evidence:
|
||||
`target/t748-callback-old-{session,resident}-placement-red.log`,
|
||||
`target/t748-callback-restored-green.log`. Final focused signed callback tests (5),
|
||||
DB boundaries (10), and Subjektiv regressions (25) passed.
|
||||
|
||||
Exact code `a2487b575df1f348251214c53da61595860fcb40` passed root `cargo check` and
|
||||
original default-parallel Server **963 passed / 0 failed / 1 existing ignored**,
|
||||
11 binary tests; library duration **173.01 seconds**. Only the three intentional
|
||||
executor panics appeared, with no residual test processes. Workdir default153 /
|
||||
http-client156, fmt/diff and naming558/0/0 passed. The isolated candidate cache
|
||||
remains in use. Logs: `target/t748-callback-{root-check,full-suite,workdir-default,workdir-http}.log`.
|
||||
The docs-only publication `27b6c688` changed no tested code. MR seq7 independently
|
||||
confirmed the callback fix and earlier fixes, but identified another direct async
|
||||
caller; its independent verdict was `request_changes`, not approval.
|
||||
|
||||
## Worker discovery and adjacent metadata caller audit
|
||||
|
||||
MR seq7 major `T748-ASYNC-WORKER-DISCOVERY` traced async discovery directly into
|
||||
synchronous `workers_response`. That helper acquires the shared SQL connection
|
||||
for the Worker projection snapshot, Workdir registry/links and authority projection.
|
||||
The Reviewer's independently compiled public snapshot helper reproduced a
|
||||
300.246ms shared-mutex stall while its Tokio timer could not progress. This was
|
||||
helper-boundary evidence, not an authenticated endpoint/E2E run.
|
||||
|
||||
Fix `dbcea6c3` awaits the **whole existing** `workers_response` on the independent
|
||||
bounded DB owner. Source-proof requirement, Workspace/query/cursor validation,
|
||||
visibility filtering, pagination and control-grant behavior are unchanged. The
|
||||
other production callers were traced: Worker list/resource detail already use DB;
|
||||
mixed Runtime cleanup plans use Host. No network work is added to the DB owner.
|
||||
|
||||
The adjacent metadata audit found async `workspace_runtime_resources_response`
|
||||
also directly calling synchronous Runtime summaries (including remote signed HTTP
|
||||
and its awaited DB authorization). This call now uses finite Host ownership,
|
||||
then the existing independent async binding read. Runtime list/detail share this
|
||||
boundary. Whole SQL helpers, admission budgets and authentication fallback rules
|
||||
are not changed.
|
||||
|
||||
The permanent post-auth discovery handler regression uses a separate SQLite
|
||||
IMMEDIATE writer and busy observer to hold the actual shared connection mutex.
|
||||
While the accepted discovery read is queued, current-thread timer and independent
|
||||
Host control progress. A second read receives 503/`db_capacity_exceeded`; a source
|
||||
without proof receives 403 before admission. Cancellation retains the actual
|
||||
queued read's lease/budget until result cleanup; ordinary completion preserves the
|
||||
empty filtered catalog response. Signed route proof verification remains covered
|
||||
by the existing dedicated-permission test, separately from this post-auth policy
|
||||
boundary so middleware's own DB wait cannot hide the handler bug.
|
||||
|
||||
Restoring the exact old discovery call fails its first-poll assertion after the
|
||||
fixture watchdog releases the synchronous mutex wait. Restoring only the old
|
||||
Runtime-summary placement fails because Tokio cannot observe the real loopback
|
||||
request until the upstream watchdog releases it. Fixed summary placement lets
|
||||
SQL/timer/control progress during the held signed upstream request and keeps the
|
||||
actual Host lease after cancellation. Explicit upstream-release observation
|
||||
prevents watchdog failure cleanup from masquerading as progress. Logs:
|
||||
`target/t748-discovery-old-placement-red.log`,
|
||||
`target/t748-runtime-summary-old-placement-red.log`,
|
||||
`target/t748-discovery-summary-fixed-boundaries.log` (12 passed),
|
||||
`target/t748-discovery-summary-policy.log` (4), and callbacks (5).
|
||||
|
||||
Exact code `dbcea6c34ba55453ce588d68272df238cde19f47` passed root `cargo check` and
|
||||
original default-parallel Server **965 passed / 0 failed / 1 existing ignored**,
|
||||
11 binary tests; library duration **175.00 seconds**. Only three deliberate
|
||||
executor panics appeared, with no residual test processes. Workdir default153 /
|
||||
http-client156 and fmt/diff/naming558/0/0 passed. Logs use the same isolated
|
||||
candidate cache: `target/t748-discovery-summary-{root-check,full-suite,workdir-default,workdir-http}.log`.
|
||||
Docs-only publication `b63bdb44` changed no tested code. MR seq9 independently
|
||||
confirmed these fixes, then requested changes for Runtime management's residual
|
||||
synchronous authority calls; it was not an approval of the complete source.
|
||||
|
||||
## Runtime management: authority, network, and accepted publication ownership
|
||||
|
||||
Seq9 major `T748-ASYNC-RUNTIME-MANAGEMENT` identified registration's direct
|
||||
`get_validated` signing identity read and post-ping direct live-binding fence.
|
||||
The Reviewer reproduced both exact compiled helpers waiting approximately 300ms
|
||||
on the shared SQL mutex with no Tokio timer progress (helper-boundary, not endpoint
|
||||
or E2E proof). Registration's synchronous Runtime Worker scan also requires a
|
||||
network/authority boundary; ping previously used an unbounded blocking admission.
|
||||
|
||||
Fix `925bce2a` introduces whole owned DB stages for validated active signing
|
||||
identity and the post-ping current binding fence. Runtime Worker scans and ping
|
||||
use existing finite Host admission. A changed/revoked binding still produces
|
||||
conflict rather than attesting the earlier connection; owner/key/CAS validation
|
||||
and existing whole store transactions are retained.
|
||||
|
||||
The adjacent management audit also closes cancellation separation between SQL
|
||||
commit and local Runtime publication. Registration and metadata updates now own
|
||||
mutation, expectations/registry/subscription publication, and response observation
|
||||
in one accepted Host workflow, with their SQL helper awaited on the independent
|
||||
DB owner. Public Runtime resource/detail helpers admit ownership; the already-owned
|
||||
workflows call explicit `_on_host` variants directly instead of reacquiring the
|
||||
same lane. There is no Runtime/network await inside a SQL transaction or nested
|
||||
Host admission. A cancelled waiter cannot strand a committed metadata change
|
||||
without publishing the matching binding.
|
||||
|
||||
Launch options were another mixed Runtime summary/config helper placed on sole
|
||||
DB. Its signed Runtime authorization could enqueue DB work behind its own waiting
|
||||
parent. It now uses Host, preserving SQL/control progress and finite ownership.
|
||||
The audit traced remaining signing validation/binding-fence calls: UI signing
|
||||
operations already use DB, Runtime gate calls occur on Host, and removal already
|
||||
has an owned Host workflow. DNS validation stays async; pure local registration
|
||||
construction/notification stays in the accepted publication owner.
|
||||
|
||||
Permanent regressions cover both authority stages under a real separate SQLite
|
||||
IMMEDIATE writer and shared-mutex busy observer: current-thread timer/control
|
||||
progress, overload503, cancelled actual DB lease retention/cleanup, positive
|
||||
validation and revoked binding409. An actual signed loopback ping is held while
|
||||
SQL contention starts; after its reply the real handler's post-ping fence queues
|
||||
without blocking Tokio or Host control. Another signed loopback scan proves SQL/
|
||||
timer progress and retained cancelled finite Host admission. The metadata mutation
|
||||
case cancels while its own SQL is waiting, then observes one durable update and
|
||||
an equal published binding after actual Host/DB drain. Producers are stopped and
|
||||
drained while the original fixture runtime remains alive.
|
||||
|
||||
Old identity, binding, scan and launch placements each fail their expected
|
||||
contention/watchdog boundary; fixed code passes. Logs:
|
||||
`target/t748-management-old-{identity,binding,scan,launch}-placement-red.log`,
|
||||
`target/t748-management-final-restored-green.log` (16 boundaries), registration
|
||||
policy11 and connection policy13. A zero-match launch-options exploration is not
|
||||
counted as proof; its real held-summary case is in the 16 boundaries. The first
|
||||
fixture teardown named an unavailable method and failed compilation; it was
|
||||
corrected to the existing `shutdown_async`, not counted as a semantic test run.
|
||||
|
||||
Exact code `925bce2a405b6d3255887fba88867ab8d23882dc` passed root `cargo check` and
|
||||
original default-parallel Server **969 passed / 0 failed / 1 existing ignored**,
|
||||
11 binary tests; library duration **172.25 seconds**. Only three deliberate
|
||||
executor panics appeared, with no residual test processes. Workdir default153 /
|
||||
http-client156, fmt/diff/naming558/0/0 passed. Same isolated cache; evidence logs:
|
||||
`target/t748-management-{root-check,full-suite,workdir-default,workdir-http}.log`.
|
||||
Docs-only publication `6d8c49b7` changed no tested code. MR seq11 independently
|
||||
confirmed those fixes and repeated the full validation, but requested changes for
|
||||
cancelled trust-key revocation; it did not approve the complete snapshot.
|
||||
|
||||
## Trust-key revocation must own post-commit subscription termination
|
||||
|
||||
Seq11 major `T748-RUNTIME-REVOKE-CANCEL-PUBLICATION` reproduced the authenticated
|
||||
public DELETE trust-key route waiting for an actual SQLite writer. Cancelling its
|
||||
HTTP waiter did not cancel accepted SQL: revoke, binding-ID replacement and audit
|
||||
committed, but waiter-owned expectation removal and broker unregister were lost.
|
||||
The independent artifact used the normal public Server library's inner router,
|
||||
not outer Workspace dispatch or live-stream/E2E. New signed requests still fail
|
||||
the live binding fence; that does not shut down an already connected subscription.
|
||||
|
||||
Fix `1a065db0` gives the whole key-revoke outcome one accepted finite Host owner:
|
||||
intact CAS/audit SQL on the independent DB lane, local expectation removal, broker
|
||||
unregister/Shutdown, and response observation. Owner/scope/input preflight, typed
|
||||
trust conflicts and current-binding error responses remain unchanged. Response
|
||||
reads use the already-owned `_on_host` detail helper, with no nested Host admission.
|
||||
Admission/unknown-outcome errors retain the normal typed availability mapping;
|
||||
there is no transaction split, new queue budget or automatic mutation retry.
|
||||
The adjacent key-mutation caller audit confirms registration and metadata already
|
||||
own publication and this is the sole production key-revoke caller.
|
||||
|
||||
The permanent cancellation regression exercises the **real outer Workspace
|
||||
dispatch**, owner browser cookie, generated DELETE route, whole store mutation and
|
||||
broker. Projection producers stop before the write lock is acquired, so the
|
||||
fixture-local busy observer identifies this actual revoke transaction, not startup
|
||||
SQL. Timer/control progress before the watchdog releases SQLite. After waiter
|
||||
cancellation, the real admitted RequestLease keeps deletion Pending. Releasing the
|
||||
writer and draining actual Host/DB owners persists exactly one revoke audit,
|
||||
changes binding ID, removes expectations and broker registration, and delivers
|
||||
`Closed/ServerShutdown` to the downstream. The loopback endpoint holds a WebSocket
|
||||
handshake; this proves broker termination, not an established upstream socket or
|
||||
complete product E2E. Existing broker protocol tests cover socket replacement.
|
||||
|
||||
A separate full-Host route regression observes 503 before mutation and unchanged
|
||||
binding, audit and publication. Restoring only the exact old revoke method fails
|
||||
both permanent tests: the cancelled job leaves broker registration behind, and
|
||||
under overload the old handler commits revoke before its response helper returns
|
||||
503. Fixed code passes both without retry or reduced parallelism. Evidence:
|
||||
`target/t748-revoke-final-old-publication-red.log`,
|
||||
`target/t748-revoke-fixed-boundaries.log` (18), `target/t748-revoke-policy.log` (3),
|
||||
`target/t748-revoke-binding-policy.log` (16).
|
||||
An initial counter probe outside dispatch was invalid because dispatch replaces it
|
||||
with the real RequestLease; it was corrected to observe actual deletion drain.
|
||||
A test-only overload channel type-inference compile rejection was corrected with
|
||||
an explicit unit channel. Neither setup error is counted as semantic evidence.
|
||||
|
||||
Exact code `1a065db0fc5d07e2dcc1b9dac2fcfc0607b9a23c` passed root `cargo check` and
|
||||
the original default-parallel Server **971 passed / 0 failed / 1 existing ignored**,
|
||||
11 binary tests; library duration **173.05 seconds**. Only three deliberate executor
|
||||
panic diagnostics appeared, and no retained test/repro processes were observed.
|
||||
Workdir default153/http-client156, fmt/diff and naming558/0/0 passed. All Cargo
|
||||
checks/tests use the isolated candidate cache. Logs:
|
||||
`target/t748-revoke-{root-check,full-suite,workdir-default,workdir-http,fmt,names}.log`.
|
||||
Docs-only source `ebce7ab7` changed no tested code. MR seq13 independently confirmed
|
||||
the revoke ownership fix and repeated the full checks, but requested changes for
|
||||
Ticket cancellation, selector repair and post-restore projection's async callers.
|
||||
The current uncommitted follow-up and the requested comprehensive caller audit
|
||||
are tracked in [the caller inventory](2026-10-12-t748-caller-inventory.md).
|
||||
Successful suites are not evidence that uninspected caller boundaries are safe.
|
||||
|
||||
No latency benchmark, production incident measurement, target integration,
|
||||
Ticket conclusion, or running-environment update is claimed. T743 has now
|
||||
published its caller inventory/source; its owner will compose the T748 facade and
|
||||
new callers, then obtain review of that exact combined source. T748 does not copy
|
||||
those domain APIs or wait on the former unpublished branch.
|
||||
@@ -0,0 +1,34 @@
|
||||
# T748 / T743 orchestrated composition
|
||||
|
||||
## Exact inputs and ownership
|
||||
|
||||
- Integrated standalone T743 target: `05ba6a43adafc2a74fac29295b906bcb30f83f11`.
|
||||
- Independently approved T748 source: `e436b2ae58e940db416a5d6a9c2edb1368ec7600`.
|
||||
- Shared original develop base: `56795fb92cb9def9bcd701bd51d92e99526f8eff`.
|
||||
|
||||
T743's corrective revert deliberately removed the prematurely imported T748 tree while preserving published history. Ordinary merge therefore reports historical modify/delete conflicts; explicit composition against the original develop base restores the approved T748 platform while retaining standalone T743's public APIs/UI/Tools/Prompt. The resulting commit has both real input parents: no history rewrite, substituted source candidate, or change to T743's concluded requirements. The Orchestrator owns this composition and target integration; T743 is not reopened or gated on another lane.
|
||||
|
||||
## New caller execution boundaries
|
||||
|
||||
- `ticket_assignments_page` and `ticket_assignment_diagnostics`: owned clones and reference/scope lookup plus the whole existing synchronous query helper run on the shared finite DB executor.
|
||||
- `ticket_assignment_answer`: mixed Runtime source discovery, intact source/CAS/replay/audit/receipt/commit helper, and post-commit orchestrator discovery/delivery run under one accepted finite Host owner. Runtime/network waits never occupy the sole DB worker. The transaction is not divided by awaits, nested Host admission is not introduced, and request cancellation does not cancel/replay accepted work or lose commit-before-publication responsibility.
|
||||
- Browser forwarders remain delegates to those canonical methods; no second domain authority or DTO is introduced.
|
||||
|
||||
## Orchestrator validation
|
||||
|
||||
Commands used the repository's original default parallelism and locked dependencies:
|
||||
|
||||
- Root `cargo check --locked`: pass.
|
||||
- Assignment tracking library suite: 28 pass, including public answer source/replay/CAS/reassignment and page/cursor authorization semantics.
|
||||
- DB execution regressions: 36 pass (the 33 approved upstream tests plus 3 new public caller tests).
|
||||
- Complete `yoi-workspace-server --all-targets`: library 1007 pass / 0 fail / 1 pre-existing ignored; binary 11 pass. Library completed in 189.81 seconds.
|
||||
- `workdir --all-targets`: 153 pass; with `http-client`: 156 pass.
|
||||
- Root format check and staged/unstaged diff checks: pass.
|
||||
|
||||
New public-caller tests use the real generated Browser handlers on a current-thread Tokio runtime, explicit isolated finite executor budgets, and a deterministic actual owner/connection hold. After job admission, independent control and timer progress, cancellation leaves the queued slot occupied, overload is typed HTTP 503, drain waits for actual completion, and subsequent normal lookup has reusable capacity. They do not claim successful answer commit or Runtime E2E delivery; the existing domain and real SQLite regressions cover those distinct boundaries.
|
||||
|
||||
Causal red/green: replacing only the three new adapters with their standalone T743 synchronous bodies made all 3 new tests fail after the bounded 5-second watchdog released the held connection; restoring the finite adapters made all 3 pass. Both logs remain in the integration Workdir's `target/t748-integration-public-boundary{,-red}.log`. Complete suite, DB, and Workdir logs use the same `target/t748-integration-*` prefix. An initial Workdir command used the nonexistent package name `yoi-workdir` and failed selection before running tests; the correct Cargo package is `workdir`, as subsequently validated. No failed test was retried to pick a green result.
|
||||
|
||||
## Review and integration status at publication
|
||||
|
||||
This is an Orchestrator-authored candidate for exact-source independent review, not a claim that prior T748 approval covers this new composition. The stable `work/T-748-orchestrator-composition` ref publishes the merge candidate without updating develop. The existing assigned T748 Coder receives it for a normal merge into the same immutable MR source selector and owns the trusted Reviewer/fix loop. Only the Orchestrator will update develop after fresh exact-source readiness. No replacement MR, new Reviewer Runtime Worker, target integration, live deployment, Ticket conclusion, or resource cleanup is claimed here.
|
||||
@@ -0,0 +1,49 @@
|
||||
# Internal SubWorker follow-up observation gap
|
||||
|
||||
## Observed barrier
|
||||
|
||||
During T748's read-only caller inventory, the parent could read the first Runtime
|
||||
inventory assistant result by stable entry ref
|
||||
`E01a12e76-3893-70a0-bb73-28ee0c52f87b`. It then sent a follow-up to the same
|
||||
Idle direct child, `t748-runtime-caller-inventory`, to close four remaining
|
||||
execution/ownership boundaries.
|
||||
|
||||
After a turn-ended/Idle notification, `WorkerList` confirmed Idle, but bounded
|
||||
`SearchSessionEntries` for assistant entries after follow-up user ref
|
||||
`E01a12e76-d696-73d3-80d7-b8cde9935b37` returned no entries. Tool observations
|
||||
were visible. The overview showed only the initial and first follow-up user
|
||||
anchors, not a final result. A single additional Submit asking for the existing
|
||||
findings (not re-investigation) returned success. After its turn-ended/Idle
|
||||
notification, the same bounded assistant query still returned no entries, and the
|
||||
overview still ended at the first follow-up user anchor.
|
||||
|
||||
This is an observation-surface symptom, **not proof** that an assistant result
|
||||
was generated, lost, or that the underlying investigation completed. No raw
|
||||
session storage or hidden worker state was read to bypass the projection.
|
||||
|
||||
The same symptom also occurred for `t748-domain-caller-inventory`: its first
|
||||
result `E01a12e73-7ae4-7f02-9a7e-203140971635` was readable, but after the
|
||||
follow-up turn-ended/Idle notification the overview ended at the follow-up's
|
||||
starting commentary `E01a12e75-9c19-7bd0-9284-299116a05761` and intervening tool
|
||||
records. No final follow-up result was visible. The parent released that
|
||||
read-only delegation without another Submit and took over the remaining generated
|
||||
adapter/forwarder and launch lower-edge checks. Neither `WorkerStop` tool outcome
|
||||
was treated as evidence that the investigation completed.
|
||||
|
||||
## Handling
|
||||
|
||||
The parent did not mark the delegated work or Ticket complete, publish source,
|
||||
or replace independent review with the notification. It stopped only the two
|
||||
parent-owned read-only children to release delegation and took over the open
|
||||
Runtime and domain boundaries. No live product process, parent Worker, target
|
||||
branch, or attachment was stopped or cleaned.
|
||||
|
||||
## Improvement proposal
|
||||
|
||||
Expose enough committed turn metadata through the authorized observation surface
|
||||
to distinguish a finalized assistant result, a turn ending without a result, and
|
||||
an observation capture not yet including an accepted Submit. An explicit
|
||||
completion marker tied to the visible Submit entry would prevent an Idle
|
||||
notification from being confused with inspectable evidence. Preserve append-only
|
||||
session entries and the current authorization projection; do not expose raw
|
||||
prompts/reasoning or require polling/retry loops as a workaround.
|
||||
Reference in New Issue
Block a user