yoi/docs/report/2026-09-23-remote-worker-repository-access-restore.md

Repository secret delivery expiry deletes a live Workdir helper and blocks restore

Editorial note (2026-10-10): obsolete state terminology is summarized by purpose below; cited IDs, commits and validation results still describe the original investigation, not new executions.

Symptom

The workspace:wip T-3 Coder completed a clean local commit but could not publish it:

branch=work/T-3-wip-protocol
head=742442f17417bfe0828291addd0249bb8a754ae9
origin=git@gitea.hareworks.net:Hare/wip-rs.git

The non-force push failed before contacting the Git remote:

fatal: cannot exec '/home/hare/.local/share/yoi/workdirs/.repository-access/f536517181e31ae9/ssh-command': No such file or directory
fatal: unable to fork

Stopping and restoring the Coder to reconstruct Repository access then failed twice with:

worker_workdir_attachments_replace_failed: Remote Runtime 'arcadia' rejected request
(HTTP 401 Unauthorized): invalid Workspace capability token:
Workspace capability does not authorize this operation

No branch was pushed and no Merge Request was opened.

Preserved work

The implementation was not lost:

• Runtime Worker: arcadia:01a0cfe9-2763-7070-8098-2088696ec9d5
• implementation Workdir: 001a0cfe90e4e000003
• read-only reference Workdir: 001a0cfe90c9a000002
• branch: work/T-3-wip-protocol
• commit: 742442f17417bfe0828291addd0249bb8a754ae9
• observed checkout: clean

The Worker is stopped, but its persisted restore request, logical attachments, execution metadata, and checkout remain. Cleanup, recreation, manual credential injection, and direct repository mutation are unnecessary.

Cause 1: a delivery deadline was reused as the bound helper lifetime

The Workspace Server makes Repository SSH secret delivery available for 300 seconds. It stores the serialized private-key candidates and pinned host entry behind a Runtime-bound FetchOnce resource handle. Runtime consumes that resource once before binding the Workdir; successful fetch removes the Server-side resource.

This deadline has a meaningful role only while the one-shot secret has not been consumed. It limits how long an unused delivery handle remains valid. It is not a lifetime intrinsic to the SSH key, Repository permission, or Git operation.

The Runtime nevertheless copied the same absolute deadline into an active expiry scheduler after Workdir bind. At expiry the scheduler stopped and removed the already-bound Repository command access while the Workdir session remained live.

The observed timing matches that behavior:

Event	Timestamp
T-3 Workdir linked to Coder	2026-09-23 20:15:59 UTC
push failed on missing helper	2026-09-23 20:29:42 UTC
elapsed	13 minutes 43 seconds

The push happened well after the five-minute delivery deadline.

What the missing helper was

GIT_SSH_COMMAND did not point to a private key. It pointed to a Runtime-generated executable script:

.repository-access/f536517181e31ae9/ssh-command

A bound access consists of:

.repository-access/<access-id>/
├── ssh-command   # invokes yoi-runtime __repository-ssh <b.sock> "$@"
├── known_hosts   # pinned host key
└── b.sock        # policy-enforcing Runtime broker socket

.repository-agents/<agent-id>/
└── agent.sock    # Runtime-internal ssh-agent socket

Runtime loads the fetched key into its internal ssh-agent through ssh-add -. The Worker receives neither the raw key nor the agent socket: its ordinary SSH_AUTH_SOCK is /dev/null. A Git SSH invocation reaches only ssh-command, whose broker validates the destination, port, Repository path, Git service, and read/write ceiling before starting ssh with the internal agent.

Thus the lending boundary is the brokered Git operation, not a five-minute key lifetime.

At the erroneous deadline, RepositoryCommandAccess::stop:

1. stopped the broker and removed b.sock;
2. killed the internal ssh-agent and removed its directory;
3. removed the complete .repository-access/<access-id> directory, including ssh-command and known_hosts.

The live LocalWorkdirSession retained the original GIT_SSH_COMMAND string, leaving a stale path. Git therefore failed at exec; it never reached Gitea. This is not evidence of a rejected key or missing remote write permission.

Cause 2: Server and Runtime disagreed on attachment-replacement permission

Restore reacquires Repository SSH authority before rebinding persisted Workdirs. In WorkspaceApi::restore_workspace_worker, the relevant order is:

1. send fresh Repository access for each SSH-backed Workdir;
2. replace the Worker's Workspace API binding;
3. replace the Worker's logical Workdir attachments;
4. restore the Runtime Worker and bind the Workdirs.

The observed diagnostic is from step 3, at:

POST /v1/workers/{worker_id}/workdir-attachments

The Server capability classifier recognized paths containing /attachments, but /workdir-attachments does not contain that literal substring. Server therefore fell through to runtime:read. Runtime explicitly classified /workdir-attachments as workers:input.

Server-issued operation: runtime:read
Runtime expectation:      workers:input

The exact 401 text is WorkspaceCapabilityVerificationError::WrongOperation. Verification had already accepted token shape, Workspace trust/signature, Workspace, Runtime, the historical binding counter, and Worker identity. Method, path, body-digest, and expiry checks occur later and were not reached.

Retrying the same restore without correcting the Server classifier deterministically returns the same 401.

Failure chain

1. Runtime consumed the one-shot Repository secret and bound a policy broker to the Workdir session.
2. Five minutes later, the delivery deadline incorrectly stopped that live broker and deleted its helper files.
3. The Worker retained a stale GIT_SSH_COMMAND.
4. A later push failed before remote contact.
5. The Orchestrator stopped the Worker to reconstruct the binding.
6. Restore reacquired Repository access but signed attachment replacement as runtime:read.
7. Runtime required workers:input and rejected restore before Workdir bind.

The two defects are distinct. The first breaks long-running Workdir sessions; the second blocks the normal recovery path.

Implemented correction

The correction preserves the existing security boundary without inventing a credential lifetime:

• the 300-second expiry remains on unconsumed one-shot secret delivery and pending pre-bind authority;
• binding validates and consumes that authority before its deadline;
• after bind, RepositoryCommandAccess is retained only as a Workdir session resource;
• every Worker Git SSH invocation still passes through the Runtime broker's host, Repository, service, and read/write policy checks;
• closing the Workdir session drops the resource, stops the broker and agent, and removes all helper files;
• expiry scheduler state and tests for active bound access are removed;
• a regression test waits beyond the consumed delivery deadline and proves the live helper remains until binding drop.

The restore correction adds /workdir-attachments to the Server's workers:input classifier and locks both Server issuance and Runtime verification with focused POST-route assertions.

Safe recovery boundary

After deployment through the normal operator-controlled process:

1. restore the same stopped Worker;
2. confirm that the existing implementation Workdir remains clean at 742442f17417bfe0828291addd0249bb8a754ae9;
3. let the assigned Coder perform the normal non-force push, remote-ref verification, Merge Request, and review flow.

No Runtime trust reprovisioning, credential rotation, manual helper construction, force push, Workdir copy, or Worker deletion is supported by the evidence.