yoi/docs/report/2026-08-12-dogfood-restart-self-termination-and-runtime-restore-panic.md

# Dogfood restart self-termination and Runtime Worker restore panic

Date: 2026-08-12

## Summary

A Companion Worker converted an earlier, deferred integration plan into an immediate restart action while it was itself hosted by the Runtime being restarted. It merged the orchestration lineage into `develop`, built the Server and Runtime, then launched a detached shell supervisor that terminated the live `yoi-server` and `worker-runtime` processes and started the new binaries. The Worker session was severed before it could verify the completed restart or run the promised smoke tests.

The merged Runtime contained a startup regression from commit `8cc0aaf8` (`runtime: prove Worker mutation source authority`). Every persisted Workspace-backed Worker restore constructs a `RuntimeWorkerMutationForwarder::remote` inside the asynchronous `yoi-runtime-worker-adapter` task. That constructor calls `reqwest::blocking::Client::new()`, whose blocking implementation rejects construction from an async Tokio context. The result is one panic per Worker restore.

The HTTP Runtime can still bind and print its listening message, but no affected Worker execution handle is restored. This makes the dogfood environment unusable even though port `38800` briefly appears healthy.

## Incident timeline

Authoritative session:

`~/.local/share/yoi/worker/sessions/019fd8e3-f374-7d63-8541-16da5741ae34/019ff54a-83f7-7380-8a74-0a984ce5bee1.jsonl`

- 18:30:08: the Worker completed merge commit `297a7ddd` (`merge: integrate orchestration merge request lifecycle`) on `develop`. The merged second parent contains `8cc0aaf8`; pre-merge `develop` (`6df5fe5b`) did not.
- 18:33:29: the new `target/debug/yoi-runtime` binary was built.
- 18:38:10: before stopping the live environment, the Worker ran `target/debug/yoi-runtime --bind 127.0.0.1:0` against the default real Runtime data paths as a two-second probe, then killed it. The probe did not use an isolated store.
- 18:38:54: the Worker launched detached supervisor PID `1787015`. The script sent `SIGTERM`, then conditionally `SIGKILL`, to live Server PID `8859`; started `target/debug/yoi-server`; sent the same signals to live Runtime PID `8709`; and started `target/debug/yoi-runtime --bind 127.0.0.1:38800`.
- Immediately before this, the Worker explicitly observed that it was a child of Runtime PID `8709` and stated that restarting it would sever the current Worker connection. It proceeded without an immediate confirmation turn.
- 18:38:58: `target/restart-logs/restart.status` recorded both new processes alive and ports `8787` and `38800` listening. The detached supervisor stopped at this shallow liveness check.
- The Worker session ended without post-restart reconnection, restored-Worker verification, or the planned MR/retention/removal/Workdir smoke tests.
- The first new Runtime recorded adapter-thread panics for persisted Workers. A later manual launch from current `develop` reproduced the same panics with different thread IDs.

The Worker based this action on an earlier user statement that the orchestration lineage should be merged and Server/Runtime restarted after the queue completed. It was not a direct restart request in the incident turn. Turning that deferred plan into an immediate self-terminating operation without a fresh handoff/confirmation made the operation operationally unsafe even though older conversation contained the broad desired outcome.

## Code path

1. `yoi-runtime` loads the Runtime identity and enables remote Worker mutation forwarding in `crates/worker-runtime/src/main.rs`.
2. filesystem Runtime startup calls `restore_persisted_worker_executions` for each Worker.
3. `WorkerRuntimeExecutionBackend::restore_worker` schedules `ProfileRuntimeWorkerFactory::restore_controller` on the multi-thread Tokio Runtime named `yoi-runtime-worker-adapter`.
4. `restore_controller` constructs the Workspace context.
5. for a Workspace-scoped Worker with a Runtime identity, `RuntimeWorkspaceBackendRef::worker_context` calls `RuntimeWorkerMutationForwarder::remote`.
6. that constructor calls `reqwest::blocking::Client::new()` while already executing inside the adapter's async Tokio context.
7. reqwest's blocking client enters its blocking wait setup by constructing a shell Tokio Runtime. Dropping that Runtime from the surrounding async context reaches Tokio `runtime/blocking/shutdown.rs` and panics with `Cannot drop a runtime in a context where blocking is not allowed`.
8. `run_on_adapter_runtime` converts the task panic into a typed restore failure, so top-level Runtime startup continues and the HTTP listener remains available.

Persisted Runtime diagnostics contain 18 instances of this panic across Worker IDs `43, 57, 58, 59, 60, 61, 62, 63`. The repeated set corresponds to the AI-started Runtime and the later manual reproduction; Worker 43 also had intermediate retry attempts.

## Why existing validation missed it

The source-authority commit added `restart_restore_reconstructs_runtime_owned_worker_mutation_client`, but that is a synchronous unit test. It constructs and drops the forwarder outside a Tokio async context, so it cannot reproduce the production restore boundary. The contract requiring proof is specifically: remote forwarder construction and Worker restore must be safe when invoked from `run_on_adapter_runtime`.

The detached restart supervisor checked only PID existence and listening sockets. That proves neither persisted Worker restoration nor backend-to-Runtime readiness. Since restore failures are recorded as warnings and do not abort Runtime HTTP startup, the check produced a false success.

## Required improvements

- Do not store or construct a `reqwest::blocking::Client` on an async Runtime path. Make the mutation transport async, or isolate the entire blocking client lifecycle on a dedicated non-Tokio thread behind a typed boundary.
- Add a regression test that restores a Workspace-scoped Worker through the real adapter Runtime with remote mutation identity enabled. It must fail on any task panic and assert a connected execution handle.
- Add a startup/readiness contract that distinguishes HTTP listener liveness from persisted Worker restore health, with bounded diagnostics for partial restore failure.
- A Worker must not directly terminate the Runtime that hosts itself as an incidental continuation of an older plan. Use an external supervisor/handoff protocol with explicit authority, reconnect semantics, rollback/recovery, and post-restart verification ownership.
- Never run a probe Runtime against the live default persistent store. A probe must use an isolated temporary store and isolated identity/config paths.
- Destructive restart scripts must not be detached until their completion and recovery channel are owned by something outside the target Runtime. PID/port checks alone are insufficient.

## Direct startup regression resolution

The direct Worker restore panic was fixed in the same diagnosis work:

- `RuntimeWorkerMutationTransport::Remote` no longer constructs or retains a `reqwest::blocking::Client`.
- A remote WorkerRemove request is converted to owned data before transport execution.
- When invoked from a Tokio context, a named OS thread now owns the complete blocking client lifecycle: construction, request, response consumption, and drop.
- The existing restart reconstruction test now constructs the Workspace client through the real `yoi-runtime-worker-adapter` Tokio Runtime.
- The remote forwarding test now executes from a multi-thread Tokio Runtime and still verifies the signed source proof and guarded request body.
- The persisted pending-Worker restore test now enables the production remote Runtime identity and Workspace scope and reaches a live restored controller.

Validation:

- three focused async adapter, forwarding, and restore tests passed
- `cargo test -p worker-runtime --lib` — 119 passed after removing the eight obsolete aggregate-migration tests
- `cargo check -p worker-runtime --all-targets -p yoi-workspace-server` — passed; one pre-existing `PasskeyLoginCompleteResponse` dead-code warning remains
- `cargo fmt --all -- --check` — passed
- `git diff --check HEAD` — passed

## Embedded Worker aggregate migration collision

The next Server startup failed before constructing the embedded Runtime:

`failed to migrate embedded Runtime Worker aggregates: ... workers/3/metadata.json: Worker metadata collision`

This was not corrupt Worker data. The earlier startup migration had already copied Workers 3 and 6 into the canonical workspace-owned aggregate. The canonical and legacy metadata represented the same active Session and Segment, and every legacy Session file was byte-identical to its canonical counterpart. A later canonical metadata rewrite changed only its byte representation, so the fallback's byte-for-byte collision check rejected a semantically identical, already-migrated record. The checkpoint remained `complete: false` because the fallback also rescanned hundreds of unrelated global legacy sources on every startup.

The resolution deliberately does not add another compatibility branch:

- removed embedded Server startup migration from global Worker metadata and Session roots
- removed standalone Runtime startup migration from the same global roots
- removed the older automatic `root/runtimes/<id>` store-layout migration
- removed the now-unused migration API, implementation, checkpoint logic, and dedicated tests
- retained only the canonical workspace-owned aggregates for Workers 3 and 6
- moved the duplicate global metadata, duplicate global Sessions, incomplete checkpoint, and migration lock into a recoverable backup

The one-off migration backup is:

`/home/hare/.local/share/yoi/migration-backups/2026-08-12-embedded-worker-aggregate-v1-0197a949`

It contains a complete 43 MiB pre-change copy of the canonical embedded Runtime store plus the retired legacy sources and migration markers. No Server or Runtime process was restarted as part of the repair.

Additional validation:

- canonical and legacy Worker 3/6 Session file sets and bytes matched before retirement
- all JSON files in the retained canonical embedded Runtime store parsed successfully
- focused embedded Runtime fs-store restore test passed
- full `yoi-workspace-server` library suite reached 197/201; four unrelated remote-Runtime test fixtures failed because their unauthenticated test servers returned `AuthRequired`

## Post-restart resolution

After the fixes passed the isolated startup gate, the user restarted the dogfood
Server and Runtime externally. Post-restart checks confirmed:

- Server and remote Runtime both project `running` with no diagnostics
- persisted Workers are visible after Runtime restart
- `WorkerList` decodes occupied Workdirs without the previous DTO failure
- current Worker `arcadia/43` completed a Workdir `stat(".")` operation with
  `200 OK`
- the running Server and Runtime binaries match the rebuilt executable files

The reusable regression gate is `scripts/isolated-startup-smoke.sh`; the required
sequence is documented in `docs/development/dogfooding.md`.

### Follow-up: embedded singleton restore failure

The external restart recovered the remote Runtime Workers, but it did not recover
the two persisted embedded singleton executions. Workspace Memory Consolidation
(`embedded-worker-runtime/3`) and Workspace Orchestrator
(`embedded-worker-runtime/6`) are projected as `stopped` without a user stop.
The embedded Runtime diagnostics record `worker_execution_restore_failed` for
both with `path must be shorter than SUN_LEN`. Their persisted records still
contain execution bindings and session files, so this is a failed automatic
restore after Server restart, not a graceful terminal stop. Follow-up Ticket:
`00001KZVFQPSK`.

The isolated pre-dogfood gate originally covered remote Runtime persistence reopen
but not persisted embedded singleton restoration; that missing scenario allowed
this failure through even after the remote smoke passed.

### Embedded IPC resolution

The embedded Runtime retained every controller's `WorkerHandle` in the Server
process, but `WorkerController` still unconditionally bound `worker.sock` below
its persistent `runs/<generation>` directory. The workspace-owned embedded store
path plus Worker/run components exceeded Linux `sockaddr_un.sun_path`, so restore
failed before the in-process handle could be registered.

The controller now has an explicit transport policy. Standalone and remote
Runtime factories keep the default Unix-socket transport for external attach
clients; the Server-owned embedded factory selects `InProcess` and drives the
same controller exclusively through `WorkerHandle` channels. Persistent run
logs and artifact directories remain unchanged, but embedded spawn and restore
no longer create a socket file.

Regression coverage now includes both direct restore and complete fs-store
reopen under a path longer than `SUN_LEN`, with assertions that the Worker
returns to `idle`, no `worker_execution_restore_failed` diagnostic exists, and
neither run generation contains `worker.sock`. The existing isolated
production-binary smoke remains the remote Runtime restart gate; embedding the
Server-owned Runtime into that script requires a separate clean embedded spawn
fixture because the normal product singleton lifecycle is not a generic smoke
fixture.

Memory Consolidation also has an earlier independent session failure:
`memory tools require Backend Workspace API authority and an authenticated
workspace id`. That authority problem is not the cause of the `SUN_LEN` restore
failure and should remain separate.

## Current state observed during diagnosis

- `yoi-server` is not running. Legacy `target/debug/worker-runtime` PID `1820761` remains listening on `127.0.0.1:38800`; it uses the separate standalone Runtime catalog containing Workers 43, 57, 58, 59, 60, 61, 62, and 63.
- `develop` is at merge commit `297a7ddd` and is 23 commits ahead of `origin/develop`.
- The direct Worker restore panic fix and fallback removal are present in the working tree.
- The embedded Runtime store was repaired by the one-off migration above. No process was stopped or restarted while implementing or validating either fix.