diff --git a/AGENTS.md b/AGENTS.md index 1a9ae9cfc..56a8f46a1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,9 +1,10 @@ すでにシステムのドッグフーディングに成功しており、ブラウザ/TUI Client/backend/runtimeの分離とチームスペースとしてのworkspaceの実装を進めている。 -## このシステムに置ける設計要旨 +## 文書と設計 -- プロンプトはすべて`resources/prompts`に集約している。管理効率の向上のためであると同時に、ユーザーがオーバーライドする形式でもある。 -- 変更量を最小にするために設計を歪めたり、設計問題に対して不必要な後方互換性を作らない。長期的なメンテナンスと型安全性を追求すること。 +文書の追加・変更・削除は[文書管理規則](docs/README.md)に従う。設計理由は所有するRust module・itemのrustdocへ置き、crate READMEは短い概要と参照にする。利用ガイドや設定説明は追加しない。 + +変更量を最小にするために設計を歪めたり、不必要な後方互換性を作ったりしない。長期的な保守性と型安全性を優先する。 ### 状態契約の命名・競合制御 @@ -15,13 +16,7 @@ ### LLM コンテキストの加工原則 -LLM に投げる context はappend-onlyが基本であり、またその永続化形式からAPIコールの形式を純粋に再現可能である必要が有る。 - -一時的にメッセージを差し込む等の、揮発性の有るコンテキストの改変や、過去のメッセージを改ざんすることは基本的に禁止されている。 -これを行うと、 LLM はそのコンテキストに基づいて生成を行う一方、次以降のターンでhistoryに残らないため、「自分がなぜその発言/tool call をしたか」の根拠が消えるうえ、prompt cache のヒット率も低下させることになる。 - -過去のコンテキストの圧縮は、キャッシュ破壊とトークン消費のトレードオフであり、必要であれば行っている。 -しかし、キャッシュを破壊するタイミングと頻度は正確にコントロールされる必要があり、実際のセッションデータの解析に基づいて慎重に設計されるべきである。 +[正本・権限・履歴の原則](docs/principle/authority-and-history.md)に従う。LLM入力へ影響する情報は履歴へ先に確定し、永続化形式から再構成できるようにする。圧縮の保証と限界は所有moduleに記述し、実測に基づいて変更する。 --- diff --git a/README.md b/README.md index 36e716c4c..4276ff36f 100644 --- a/README.md +++ b/README.md @@ -1,85 +1,16 @@ -# 夜居 / Yoi agent +# 夜居 / Yoi -Yoi is an agent runtime for building, running, and orchestrating LLM Workers while preserving explicit history, scoped capabilities, and developer-controlled workflows. +Yoiは、LLM Workerの実行履歴と明示的な権限を管理するエージェント実行基盤である。Workspaceの記録と操作権限をBackendが管理し、RuntimeがWorkerと作業ディレクトリの実行資源を管理する。単一プロセスで実行するStandaloneは、Workspaceの権限やRuntime Workerの登録とは分離している。 -## 1. Yoi agent +## 実装の入口 -Yoi focuses on long-running agent operation rather than one-off prompt execution. A named Worker can keep durable session history, run with explicit tool and filesystem authority, delegate bounded work to child Workers, and be inspected or restored through CLI/TUI surfaces. +- [agen](crates/agen/README.md)：LLMターン、履歴、ツール呼出し。 +- [worker](crates/worker/README.md)：セッション、機能、ツールと実行権限の接続。 +- [workspace-server](crates/workspace-server/README.md)：Workspaceの正本とBackend操作。 +- [worker-runtime](crates/worker-runtime/README.md)：Workerの実行とRuntime資源。 +- [standalone](crates/standalone/README.md)：プロセス内の独立した実行ホスト。 +- [yoi](crates/yoi/README.md)・[tui](crates/tui/README.md)・[Web](web/workspace/README.md)：利用者との接点。 -Main highlights: +各crateのREADMEをcrateドキュメントとして使い、具体的な設計理由はmodule・itemのrustdocに置く。文書の追加・維持・削除は[文書管理規則](docs/README.md)、変更の検証は[検証規則](docs/development/validation.md)に従う。 -- Named long-running **Workers** with durable session and metadata records. -- Explicit tool permissions and filesystem scopes. -- Multi-agent orchestration with scoped coder/reviewer Workers. -- Profile, Manifest, and prompt-based runtime configuration. -- Local Tickets and workflow files for auditable project coordination. -- TUI and CLI entry points, including the `yoi panel` workspace Dashboard and single-Worker Console. - -Yoi is actively dogfooded in this repository. Public APIs, configuration formats, and workflows may still change. - -## 2. Quick Install - -From source: - -```sh -cargo build --release -p yoi -./target/release/yoi --help -``` - -With Nix: - -```sh -nix build .#yoi -./result/bin/yoi --help -``` - -## 3. Getting Started - -```sh -yoi --help -yoi -yoi panel -yoi --worker -yoi worker --help -``` - -Typical flow: - -1. Configure providers, models, profiles, prompts, and scopes. -2. Start or attach to a named Worker in the Console, or inspect workspace activity in the Dashboard. -3. Use explicit tools and scoped delegation for multi-agent work. -4. Record project work through Tickets, workflow files, and git history. - -Client and execution surfaces use the `yoi` binaries, XDG client/server configuration and data directories, Runtime-owned Workdirs/session artifacts, Server data directories, `YOI_*`, and the preserved legacy home secret-store boundary under `~/.yoi`. Repository-local `.yoi` trees are ignored legacy input, not runtime authority. - -## 4. Documentation - -Start with [`docs/README.md`](docs/README.md). - -Key docs: - -- [`docs/design/overview.md`](docs/design/overview.md) — architecture and crate ownership map. -- [`docs/design/context-history.md`](docs/design/context-history.md) — history/context invariants. -- [`docs/design/worker-session-state.md`](docs/design/worker-session-state.md) — Worker identity, metadata, and session logs. -- [`docs/design/profiles-manifests-prompts.md`](docs/design/profiles-manifests-prompts.md) — Profiles, Manifests, and prompt resources. -- [`docs/design/tool-permissions-scope.md`](docs/design/tool-permissions-scope.md) — tool policy and filesystem scope. -- [`docs/development/work-items.md`](docs/development/work-items.md) — Ticket workflow and project records. -- [`docs/development/validation.md`](docs/development/validation.md) — validation expectations. - -## 5. Development - -This repository dogfoods Yoi to develop Yoi. Work is tracked through Workspace Server-backed Tickets and `yoi ticket ...`; Git history plus append-only Ticket and Merge Request evidence are the authoritative project record. Repository-local `.yoi/tickets` trees are ignored legacy input. - -Common checks: - -```sh -yoi ticket doctor -git diff --check -cargo fmt --check -cargo check --workspace --all-targets -cargo test --workspace -``` - -Real-process E2E testing is opt-in. Do not design or implement E2E coverage unless the work explicitly requires E2E; otherwise protect the narrower parser, API, protocol, authority, or runtime boundary. - -License: MIT. See [`LICENSE`](LICENSE). +ライセンスは[MIT](LICENSE)。 diff --git a/crates/agen-macros/README.md b/crates/agen-macros/README.md index 679cc1ab2..73825cb59 100644 --- a/crates/agen-macros/README.md +++ b/crates/agen-macros/README.md @@ -1,32 +1,9 @@ # agen-macros -Procedural macros used by [`agen`](https://crates.io/crates/agen) to declare LLM tools from Rust methods. +`agen-macros` generates `agen` tool definitions and implementations from Rust +method declarations. It isolates procedural macro expansion from `agen`, which +owns tool registration and execution. -Applications should normally depend only on `agen` and import its re-exports: +## Design entry points -```rust -use agen::tool_registry; - -#[derive(Clone)] -struct Tools; - -#[tool_registry] -impl Tools { - /// Returns the supplied text. - #[tool] - async fn echo( - &self, - #[description = "Text to return"] text: String, - ) -> Result { - Ok(text) - } -} -``` - -`#[tool_registry]` generates the argument schema, a `Tool` implementation, and an `_definition` constructor. It rejects arguments of its own, duplicate `#[tool]` markers, malformed or duplicate `#[description = "..."]` attributes, and non-identifier argument patterns. - -Generated code targets the canonical `::agen` path and uses implementation dependencies re-exported by `agen`; consumers do not need direct `serde`, `schemars`, `serde_json`, or `async-trait` dependencies. Renaming the `agen` dependency in `Cargo.toml` is not currently supported. - -This companion package is published before the matching `agen` release. Its public contract is the generated API consumed by `agen`, and its minor version compatibility follows the `agen` 0.2 series. - -Licensed under the [MIT License](https://gitea.hareworks.net/Hare/yoi/src/branch/develop/LICENSE). +- [`tool_registry`]: declaration validation, argument schemas, and execution wrappers diff --git a/crates/agen-macros/src/lib.rs b/crates/agen-macros/src/lib.rs index e1ef6f96f..2999057cc 100644 --- a/crates/agen-macros/src/lib.rs +++ b/crates/agen-macros/src/lib.rs @@ -1,9 +1,4 @@ -//! Procedural macros for declaring [`agen`](https://docs.rs/agen) tools. -//! -//! [`tool_registry`] expands methods marked with `#[tool]` into `agen::tool::Tool` -//! implementations and tool definitions. Applications normally use the re-exports from -//! `agen`; this companion crate exists so those macros can be published and versioned -//! independently. +#![doc = include_str!("../README.md")] use proc_macro::TokenStream; use quote::{format_ident, quote}; @@ -12,30 +7,16 @@ use syn::{ spanned::Spanned, }; -/// Generates tools for methods marked with `#[tool]` in an `impl` block. +/// Validates tool declarations and generates JSON Schema and `agen` execution wrappers. /// -/// Method doc comments become the tool description. An argument can use -/// `#[description = "..."]` to supply its JSON Schema description. +/// Method doc comments and argument `description` attributes provide schema descriptions. +/// Invalid attributes, duplicate markers, and non-identifier argument patterns produce +/// `compile_error!` diagnostics rather than panicking during macro expansion. /// -/// ```ignore -/// #[derive(Clone)] -/// struct MyApp; -/// -/// #[agen::tool_registry] -/// impl MyApp { -/// /// Retrieves a user by ID. -/// #[tool] -/// async fn get_user( -/// &self, -/// #[description = "The user ID"] user_id: String, -/// ) -> Result { -/// todo!() -/// } -/// } -/// ``` -/// -/// This generates a `ToolGetUser` wrapper, a `GetUserArgs` schema type, and -/// `MyApp::get_user_definition()`. +/// Generated code references helper dependencies through `::agen::__private` so +/// downstream crates do not need to depend on implementation details directly. +/// The expansion uses the canonical `::agen` path and does not resolve renamed dependencies. +/// `ToolExecutionContext` is supplied by the executor and excluded from model input schemas. #[proc_macro_attribute] pub fn tool_registry(attr: TokenStream, item: TokenStream) -> TokenStream { let attr = proc_macro2::TokenStream::from(attr); diff --git a/crates/agen/Cargo.toml b/crates/agen/Cargo.toml index e01b2ca40..8da0da2d3 100644 --- a/crates/agen/Cargo.toml +++ b/crates/agen/Cargo.toml @@ -11,7 +11,7 @@ homepage = "https://gitea.hareworks.net/Hare/yoi" documentation = "https://docs.rs/agen" keywords = ["llm", "agent", "tools", "streaming", "orchestration"] categories = ["api-bindings", "asynchronous"] -include = ["src/**", "tests/**", "examples/*.rs", "docs/**", "README.md", "LICENSE"] +include = ["src/**", "tests/**", "examples/*.rs", "README.md", "LICENSE"] autoexamples = false [features] diff --git a/crates/agen/README.md b/crates/agen/README.md index 13478848e..11d8dbe1b 100644 --- a/crates/agen/README.md +++ b/crates/agen/README.md @@ -1,91 +1,14 @@ # agen -`agen` is a provider-neutral Rust engine for streaming LLM applications that use tools. It owns the turn loop, typed conversation history, provider wire-format adapters, tool execution, interceptors, usage accounting, and cache-aware state transitions. +`agen` integrates streaming LLM responses and tool execution in an agent engine. +It manages turns, history appends, pause/resume, and retries, while the LLM client +layer handles wire formats and authentication. The caller owns history storage, +worker lifecycle control, and compaction. -> `agen` is pre-1.0. Public APIs may change between minor releases. +## Design entry points -## Installation - -```toml -[dependencies] -agen = "0.2.1" -``` - -The default feature set is intentionally empty. Enable the experimental Codex/ChatGPT authentication adapter when needed: - -```toml -agen = { version = "0.2.1", features = ["codex"] } -``` - -`agen` requires Rust 1.86 or newer. The companion `agen-macros` package requires Rust 1.85 or newer. - -## Quick start - -Supply an implementation of [`LlmClient`](https://docs.rs/agen/latest/agen/llm_client/trait.LlmClient.html), keep conversation history in your application, then run a turn. The first call consumes the mutable engine and returns a cache-locked engine for later turns. - -```no_run -use agen::{Engine, EngineError, History}; -use agen::llm_client::LlmClient; - -async fn conversation(client: C) -> Result<(), EngineError> { - let mut history = History::new(); - let output = Engine::new(client) - .system_prompt("You are a concise assistant.") - .run(&mut history, "Explain typed state in one sentence.") - .await; - - let mut engine = output.engine; - let _result = engine.run(&mut history, "Give a Rust example.").await; - Ok(()) -} -``` - -## Declaring tools - -The tool macros are re-exported by `agen`; applications do not need direct dependencies on `serde`, `schemars`, `serde_json`, or `async-trait` for generated code. - -```rust -use agen::tool_registry; - -#[derive(Clone)] -struct Tools; - -#[tool_registry] -impl Tools { - /// Returns the supplied text. - #[tool] - async fn echo( - &self, - #[description = "Text to return"] text: String, - ) -> Result { - Ok(text) - } -} - -let definition = Tools.echo_definition(); -assert_eq!(definition().0.name, "echo"); -``` - -The generated API uses the canonical crate name `agen`. Renaming the `agen` dependency in `Cargo.toml` is not currently supported by these macros. - -## Features - -| Feature | Default | Adds | -|---|---:|---| -| `codex` | No | Experimental Codex/ChatGPT auth-file loading and token refresh support | - -The base crate includes provider-neutral transport and Anthropic, OpenAI-compatible, Gemini, and Ollama wire-format schemes. See [`llm_client`](https://docs.rs/agen/latest/agen/llm_client/) for the client boundary. - -## Architecture and API scope - -The current public modules cover the engine, typed history, client transport/schemes, timeline events, tools, interceptors, pruning, token estimation, and usage records. Their relationships are described in [Architecture](https://gitea.hareworks.net/Hare/yoi/src/branch/develop/crates/agen/docs/architecture.md); behavioral requirements are summarized in [Requirements](https://gitea.hareworks.net/Hare/yoi/src/branch/develop/crates/agen/docs/requirements.md). - -Low-level modules remain public in the 0.2 series because downstream Yoi components implement custom clients, event handlers, pruning policies, and tool registries against them. This surface is versioned as pre-1.0 API rather than declared stable. - -## Packaging and security - -The published package contains source, public documentation, curated examples, and deterministic tests/fixtures. Credentialed fixture-recording utilities are intentionally excluded. Examples that contact a provider read credentials from environment variables and never embed production credentials. - -## License - -Licensed under the [MIT License](https://gitea.hareworks.net/Hare/yoi/src/branch/develop/LICENSE). +- [`Engine`]: turn execution, interruption, and resumption +- [`History`]: model input paired with caller-owned annotations +- [`state`] and [`prune`]: prefix protection and request-only history projections +- [`llm_client`] and [`timeline`]: wire-format conversion and event processing +- [`interceptor`]: host-owned control decisions diff --git a/crates/agen/docs/architecture.md b/crates/agen/docs/architecture.md deleted file mode 100644 index 1209bd48b..000000000 --- a/crates/agen/docs/architecture.md +++ /dev/null @@ -1,62 +0,0 @@ -# agen architecture - -`agen` separates orchestration, event projection, and provider transport so applications can replace an LLM client without changing the turn loop or tool model. - -```text -┌────────────────────────────────────────────┐ -│ Engine │ -│ turn loop · interceptors · tool execution │ -│ typed state: Mutable → Locked → Mutable │ -└─────────────────────┬──────────────────────┘ - │ -┌─────────────────────▼──────────────────────┐ -│ Timeline │ -│ event dispatch · block collectors │ -└─────────────────────┬──────────────────────┘ - │ -┌─────────────────────▼──────────────────────┐ -│ LlmClient │ -│ transport · provider wire-format schemes │ -└────────────────────────────────────────────┘ -``` - -## Main modules - -| Module | Responsibility | -|---|---| -| `engine` | Turn execution, pause/resume, retries, tool integration, and callbacks | -| `state` | Sealed `Mutable` and `Locked` type-state markers | -| `interceptor` | Application-owned control decisions at orchestration boundaries | -| `tool` / `tool_server` | Tool metadata, registration, execution, and bounded output | -| `timeline` | Streaming event dispatch, handlers, and block assembly | -| `llm_client` | Provider-neutral request, response, auth, transport, and scheme contracts | -| `providers` | Optional higher-level provider adapters such as the `codex` feature | -| `prune` / `token_counter` | Cache-aware history reduction and token estimation | -| `usage_record` | Request and token usage accounting | - -## Request flow - -```text -Engine history - → provider-neutral Request - → Scheme::build_request - → Provider transport -``` - -## Response flow - -```text -streaming response bytes - → Scheme event parsing - → unified Event values - → Timeline handlers and collectors - → Engine history/tool decisions -``` - -## Type state and cache protection - -`Engine` permits configuration and history editing. `Engine::run` or `Engine::lock` commits the current prefix and produces `Engine`. The locked engine may append turns without mutating the committed prefix. `Engine::unlock` explicitly returns to mutable state when an application accepts losing that cache guarantee. - -## Public surface - -The 0.2 series exposes the low-level client, timeline, tool, pruning, and usage modules because custom clients and orchestration hosts build directly on them. These APIs are intentionally provider-neutral but remain pre-1.0 and may change in later minor releases. diff --git a/crates/agen/docs/requirements.md b/crates/agen/docs/requirements.md deleted file mode 100644 index d564b9502..000000000 --- a/crates/agen/docs/requirements.md +++ /dev/null @@ -1,39 +0,0 @@ -# agen requirements - -## R1: Turn execution and continuation - -- `Engine::run` starts a turn and loops through provider output and tool calls. -- An `Interceptor` may continue, cancel, or pause work at defined orchestration boundaries. -- `Engine::resume` continues paused generation without fabricating another user message. -- Cancellation and provider errors are represented as typed `EngineError` values. - -## R2: Explicit cache-preserving state - -- `Engine` permits configuration and history edits. -- `Engine::run` or `Engine::lock` transitions to `Engine` and records the committed prefix. -- A locked engine appends turns but cannot mutate that prefix through mutable-only APIs. -- `Engine::unlock` explicitly abandons the lock before configuration or history changes. - -## R3: Tool declarations and execution - -- `#[tool_registry]` generates a schema and `Tool` implementation for methods marked `#[tool]`. -- `#[description = "..."]` supplies argument descriptions in generated JSON Schema. -- Generated code resolves its runtime and helper dependencies through `::agen`. -- Invalid and duplicate marker attributes produce compile errors rather than panics. -- Tools execute through `ToolServer` with typed context, errors, and output limits. - -## R4: Provider-neutral orchestration - -- `LlmClient` is the boundary between the engine and provider-specific transport. -- Request/response schemes translate provider wire formats into shared request and event types. -- Interceptors, tool execution, timeline collection, and pruning stay above the provider transport. -- Provider-specific capabilities are optional features when they require additional policy or dependencies. - -## R5: Publication quality - -- crates.io metadata includes license, repository, documentation, README, categories, keywords, and MSRV. -- The default feature set and each optional feature compile and test independently. -- Macro expansion compiles in a downstream-style integration test without direct helper dependencies. -- rustdoc builds without dependency documentation. -- Package contents are explicitly bounded and exclude credentialed fixture-recording utilities. -- `cargo package` and `cargo publish --dry-run` are run for `agen-macros` before `agen` because the main package depends on its companion package. diff --git a/crates/agen/examples/engine_cli.rs b/crates/agen/examples/engine_cli.rs index ee90ca7ec..58d3b9e1f 100644 --- a/crates/agen/examples/engine_cli.rs +++ b/crates/agen/examples/engine_cli.rs @@ -368,7 +368,7 @@ fn create_client(args: &Args) -> Result, String> { ResolvedAuth::ApiKey(api_key), )), Provider::Ollama => { - // Ollama = Anthropic scheme + base_url 差し替え + 認証なし + // Ollama uses the Anthropic scheme with a custom base_url and no authentication. let scheme = AnthropicScheme::new(); let cap = default_capability(); Ok(Box::new(HttpTransport::new( diff --git a/crates/agen/examples/record_test_fixtures/main.rs b/crates/agen/examples/record_test_fixtures/main.rs index f1978efab..07e4a7fe5 100644 --- a/crates/agen/examples/record_test_fixtures/main.rs +++ b/crates/agen/examples/record_test_fixtures/main.rs @@ -108,7 +108,7 @@ async fn run_scenario_with_ollama( subdir: &str, model: Option, ) -> Result<(), Box> { - // Ollama = Anthropic scheme + base_url 差し替え + 認証なし + // Ollama uses the Anthropic scheme with a custom base_url and no authentication. let model = model.as_deref().unwrap_or("llama3"); let client = HttpTransport::new( AnthropicScheme::new(), diff --git a/crates/agen/src/engine.rs b/crates/agen/src/engine.rs index 16f76d790..dd1e57e85 100644 --- a/crates/agen/src/engine.rs +++ b/crates/agen/src/engine.rs @@ -1,3 +1,20 @@ +//! Provider-independent orchestration of turns and tool execution. +//! +//! The Engine uses common Request and Event types to keep wire formats and +//! authentication out of the turn loop. It delegates caller control decisions to +//! Interceptor and does not own notification intake, storage, or segment switches +//! during compaction. +//! +//! New history items enter in-memory history only after their annotation callback +//! succeeds. Hosts that require persistence record items in that callback. +//! Request-time reordering and pruning operate on a clone, preserving stored order +//! and caller-owned annotations. +//! +//! Retries before a stream opens differ from continuation requests after partial +//! output has been recorded. A stream failure after early tool dispatch disables +//! automatic continuation: regenerating the same side effect under a new call ID +//! is unsafe, so the Engine finalizes the executions it owns and returns an error. + use std::collections::{HashMap, HashSet, VecDeque}; use std::{future::Future, marker::PhantomData, pin::Pin, sync::Arc, time::Instant}; @@ -464,7 +481,7 @@ const MAX_STREAM_CONTINUATIONS: u32 = 3; /// /// Calling `run()` on a `Mutable` Engine consumes it and returns a /// `Locked` Engine together with the result. The engine borrows the caller's -/// [`History`](crate::History) only while running, so host annotations stay with +/// [`History`] only while running, so host annotations stay with /// the host-owned history and are never projected to providers. /// /// ```ignore @@ -488,7 +505,7 @@ const MAX_STREAM_CONTINUATIONS: u32 = 3; /// ``` #[derive(Debug, Clone, PartialEq, Eq)] pub struct LlmRetryNotice { - /// 直近で失敗した attempt 番号。1 origin。 + /// Number of the most recently failed attempt, starting at 1. pub failed_attempt: u32, pub max_attempts: u32, pub wait: std::time::Duration, @@ -539,6 +556,14 @@ impl Drop for ResponseStreamPump { } } +/// Coordinates LLM responses, tool results, and control callbacks in one turn loop. +/// +/// History belongs to the caller. `run` appends new input and starts a logical run; +/// `resume` continues a paused or yielded run without creating a new user message. +/// The turn budget survives pauses and yields, so resuming does not reset the limit. +/// +/// Type states restrict Engine configuration changes; they do not guarantee that +/// external History remains unchanged or that the provider reports a cache hit. pub struct Engine { /// LLM client client: C, @@ -550,8 +575,8 @@ pub struct Engine { text_block_collector: TextBlockCollector, /// Tool call collector (Timeline handler) tool_call_collector: ToolCallCollector, - /// Thinking block collector (Timeline handler)。metadata 付きで完了した - /// Thinking block を 1 ターン分バッファし、history に append する。 + /// Thinking block collector (Timeline handler). Buffers completed blocks and + /// their metadata for one turn before appending them to history. thinking_block_collector: ThinkingBlockCollector, /// Tool server handle tool_server: ToolServerHandle, @@ -561,12 +586,9 @@ pub struct Engine { system_prompt: Option, /// History length at lock time (only meaningful in Locked state) locked_prefix_len: usize, - /// AgentTurn count across the lifetime of this Engine. + /// Cumulative AgentTurns started during this Engine's lifetime. /// - /// Once retry (`agen-stream-continuation`) is implemented, an - /// AgentTurn collapses N retried `LlmCall`s with identical input; - /// today retry is not implemented so AgentTurn and LlmCall fire 1:1 - /// and the increment site (the LLM-call loop) is shared. + /// The logical run's turn budget is tracked separately in `active_run_turn_count`. turn_count: usize, /// AgentTurns consumed by the currently active logical run. /// @@ -1299,10 +1321,9 @@ impl Engine { /// Build assistant response items from reasoning items, text blocks, and tool calls. /// - /// Reasoning items come first (Anthropic / OpenAI Responses 双方ともに - /// アシスタント応答内で reasoning は先頭に並ぶ仕様)。これは Anthropic - /// が新世代モデルで thinking ブロックを assistant メッセージの先頭に - /// 置くことを要求するためでもある。 + /// Reasoning items precede other assistant output for both Anthropic and + /// OpenAI Responses. This also satisfies newer Anthropic models' requirement + /// that thinking blocks appear at the start of the assistant message. fn build_assistant_items( &self, reasoning_items: &[crate::llm_client::event::ReasoningBlockData], @@ -3293,7 +3314,7 @@ impl Engine { let event = match result { Ok(event) => event, Err(err) => { - // 部分情報でも発火しておく（料金会計用） + // Emit even partial usage for billing accounting. self.timeline.flush_usage(); return Ok(StreamResponseOutput { completion: StreamCompletion::Interrupted { @@ -3585,7 +3606,7 @@ impl Engine { } } } - // ストリーム完了時に集約済み Usage を 1 度だけ発火 + // Emit aggregated usage once when the stream completes. self.timeline.flush_usage(); debug!(event_count = event_count, "Stream completed"); Ok(StreamResponseOutput { @@ -3900,19 +3921,16 @@ impl Engine { } } - /// Lock and transition to Locked state + /// Finalizes configuration and transitions to Locked. /// - /// Flushes pending tool factories, then fixes the current system prompt - /// and history as a "committed prefix". After this, only `run()` / `resume()` - /// may append to history, ensuring cache hits. - /// - /// Most callers should use [`run()`](Self::run) instead, which calls - /// this internally. Use `lock()` directly only when you need the - /// `Locked` engine back on error (e.g. in a persistence layer). + /// Flushes pending tool factories and records the current history length as the + /// prefix boundary. External History mutations remain possible, so the caller + /// is also responsible for preserving the prefix. This transition changes the + /// type state; it does not persist history. /// /// # Panics /// - /// Panics if a pending tool factory produces a duplicate name. + /// Panics if pending tool factories produce duplicate names. pub fn lock(self, history: &History) -> Engine { self.tool_server.flush_pending(); let locked_prefix_len = history.len(); diff --git a/crates/agen/src/history.rs b/crates/agen/src/history.rs index 4bb198cbc..b4b21d5f7 100644 --- a/crates/agen/src/history.rs +++ b/crates/agen/src/history.rs @@ -1,9 +1,8 @@ -//! Typed conversation history containers. +//! History that pairs model input with host-owned annotations. //! -//! Agen keeps provider-visible [`Item`](crate::Item) values separate from any -//! host-domain provenance. The host chooses the annotation type `A`, while Agen -//! preserves each item and annotation as one entry for clone/truncate/restore -//! style history operations. +//! Only [`Item`] is projected to providers; annotation `A` holds storage identifiers +//! and input provenance. Cloning, truncation, and restoration keep each pair together. +//! The caller's append callback owns persistence; this container does not. use serde::{Deserialize, Serialize}; @@ -40,7 +39,13 @@ impl HistoryEntry<()> { } } -/// Conversation history with one annotation per item. +/// Ordered history of model input and host-owned annotations. +/// +/// [`History::append_with`] runs the callback before adding an item. A host that +/// persists in that callback keeps failed writes out of subsequent model requests. +/// Batch appends commit one item at a time; later failures do not roll back earlier +/// items. Direct append and replacement APIs do not persist, so the host must +/// distinguish restoration from normal appends. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Default)] pub struct History { entries: Vec>, diff --git a/crates/agen/src/interceptor.rs b/crates/agen/src/interceptor.rs index 67d34166d..5d662809b 100644 --- a/crates/agen/src/interceptor.rs +++ b/crates/agen/src/interceptor.rs @@ -1,8 +1,9 @@ -//! Interceptor - control flow delegation for the Engine execution loop +//! Host-owned control decisions at Engine execution boundaries. //! -//! Defines the [`Interceptor`] trait that callers implement to inject -//! orchestration decisions (approval, skip, pause, abort) into the Engine's -//! turn loop without the Engine knowing about host-application concepts. +//! [`Interceptor`] delegates approval, skip, pause, and abort decisions to the host, +//! keeping host-specific policy out of the Engine. Additional model input must go +//! through history appends: inserting new facts only into an in-flight request +//! would make its input impossible to reconstruct from stored history. use std::sync::Arc; diff --git a/crates/agen/src/lib.rs b/crates/agen/src/lib.rs index fca7069f1..321ab81c2 100644 --- a/crates/agen/src/lib.rs +++ b/crates/agen/src/lib.rs @@ -42,10 +42,9 @@ pub use tool::{ }; pub use usage_record::UsageRecord; -/// Implementation dependencies used by code generated from `agen` macros. +/// Implementation dependencies referenced by macro-generated code. /// -/// This module is not a stable user-facing API. It is public only because macro expansion -/// happens in the downstream crate. +/// Public for access from downstream macro expansions, not a stable user-facing API. #[doc(hidden)] pub mod __private { pub use async_trait; diff --git a/crates/agen/src/llm_client/auth.rs b/crates/agen/src/llm_client/auth.rs index 08cc61a59..d3dabecb3 100644 --- a/crates/agen/src/llm_client/auth.rs +++ b/crates/agen/src/llm_client/auth.rs @@ -1,43 +1,43 @@ -//! `Scheme` 実装と通信層が要求する認証要件、および動的認証プロバイダ。 +//! Authentication requirements for `Scheme` implementations and the transport layer, +//! plus dynamic authentication providers. //! -//! `AuthRequirement` は scheme が宣言する「この scheme はどんな認証を -//! 期待するか」のランタイム記述で、設定ファイルや環境変数などから -//! [`super::transport::ResolvedAuth`] を組み立てる責務は呼び出し側にある。 +//! `AuthRequirement` describes the authentication a scheme expects at runtime. +//! The caller is responsible for building [`super::transport::ResolvedAuth`] +//! from configuration files, environment variables, or other sources. //! -//! リクエスト毎にトークンが変わり得る認証は [`AuthProvider`] trait を -//! 実装し、[`super::transport::ResolvedAuth::Custom`] 経由で transport に渡す。 +//! Authentication whose tokens may change per request implements [`AuthProvider`] +//! and is passed to the transport via [`super::transport::ResolvedAuth::Custom`]. use async_trait::async_trait; use reqwest::header::{HeaderName, HeaderValue}; use super::error::ClientError; -/// `Scheme::required_auth()` が返す認証要件。 +/// Authentication requirements returned by `Scheme::required_auth()`. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum AuthRequirement { - /// 認証を行わない（Ollama など） + /// No authentication (e.g. Ollama). None, - /// `Authorization: Bearer ` ヘッダ（token は API key 相当） + /// `Authorization: Bearer ` header (the token acts as an API key). Bearer, - /// `x-api-key: ` ヘッダ（Anthropic 形式） + /// `x-api-key: ` header (Anthropic format). XApiKey, - /// クエリパラメータ `?=`（Gemini 形式） + /// Query parameter `?=` (Gemini format). QueryParam { name: &'static str }, - /// 複合ヘッダ（呼び出し側が [`AuthProvider`] で解決） + /// Combined headers (resolved by the caller through [`AuthProvider`]). Custom, } -/// リクエスト毎に認証ヘッダを動的に組み立てるプロバイダ。 +/// Provider that builds authentication headers dynamically for each request. /// -/// access token が refresh で更新されたり、複数ヘッダを同時に注入する -/// 必要があるケースで使う。実体は呼び出し側に置き、agen は -/// trait を知るだけ。 +/// Use this when access tokens are refreshed or multiple headers must be injected +/// together. The implementation belongs to the caller; agen only knows the trait. /// -/// 返したヘッダはそのまま `HeaderMap` に挿入される。`Authorization` -/// 含む scheme 既定の認証ヘッダは送出されないので、必要なら -/// 実装側でセットすること。 +/// Returned headers are inserted directly into `HeaderMap`. The scheme's default +/// authentication headers, including `Authorization`, are not sent; the +/// implementation must set them if needed. #[async_trait] pub trait AuthProvider: Send + Sync + std::fmt::Debug { - /// 1 リクエスト分の認証ヘッダを返す。refresh が必要なら内部で行う。 + /// Return authentication headers for one request, refreshing internally if needed. async fn headers(&self) -> Result, ClientError>; } diff --git a/crates/agen/src/llm_client/capability.rs b/crates/agen/src/llm_client/capability.rs index a2fde57a9..b80d279ac 100644 --- a/crates/agen/src/llm_client/capability.rs +++ b/crates/agen/src/llm_client/capability.rs @@ -1,16 +1,16 @@ -//! モデル能力メタデータ +//! Model capability metadata. //! -//! `ModelCapability` はモデルが持つ機能差を表現する。scheme は同じでも -//! モデルごとに reasoning 可否や prompt caching 方式が違うため、scheme -//! から分離して保持する。 +//! `ModelCapability` represents differences in model features. Reasoning support +//! and prompt caching strategies may vary even within one scheme, so this +//! metadata is kept separate from the scheme. //! -//! 値の供給経路は 2 通り: -//! 1. scheme 実装側の `model_id → ModelCapability` 静的テーブル（既知モデル） -//! 2. `ModelConfig::capability` での明示 override（未知モデル、または上書き） +//! The client construction layer supplies model-specific capabilities and overrides. +//! Schemes provide only wire-level defaults through `Scheme::default_capability`; +//! this module does not own a model registry or application configuration format. use serde::{Deserialize, Deserializer, Serialize, Serializer}; -/// モデル能力メタデータ +/// Model capability metadata. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] pub struct ModelCapability { pub tool_calling: ToolCallingSupport, @@ -23,7 +23,7 @@ pub struct ModelCapability { } impl ModelCapability { - /// 何もサポートしない安全側デフォルト。未知モデルのフォールバック用。 + /// Conservative default with no supported features, used for unknown models. pub const fn minimal() -> Self { Self { tool_calling: ToolCallingSupport::None, @@ -35,55 +35,55 @@ impl ModelCapability { } } -/// ツール呼び出しサポート +/// Tool calling support. #[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] pub enum ToolCallingSupport { - /// 非サポート + /// Unsupported. None, - /// 1 回のレスポンスで 1 ツールのみ + /// Only one tool per response. Sequential, - /// 1 回のレスポンスで複数ツール並行 + /// Multiple parallel tools per response. Parallel, } -/// Structured output サポート +/// Structured output support. #[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] pub enum StructuredOutput { None, - /// `json_object` モード（スキーマなし JSON 強制） + /// `json_object` mode (enforces JSON without a schema). JsonObject, - /// JSON Schema 指定で構造化出力 + /// Structured output with a JSON Schema. JsonSchema, } -/// Reasoning（extended thinking）サポート +/// Reasoning (extended thinking) support. #[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] pub enum ReasoningSupport { - /// OpenAI 形式: `reasoning.effort` (low/medium/high) + /// OpenAI format: `reasoning.effort` (low/medium/high). Effort, - /// Anthropic 形式: `thinking.budget_tokens` + /// Anthropic format: `thinking.budget_tokens`. BudgetTokens, - /// 両対応（内部では共通 `ReasoningControl` として扱い、各 scheme で投影） + /// Both (handled internally as shared `ReasoningControl`, projected by each scheme). Both, } -/// Prompt caching 戦略 +/// Prompt caching strategy. #[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] #[serde(tag = "kind", rename_all = "snake_case")] pub enum CacheStrategy { - /// Anthropic: `cache_control` マーカーを明示挿入 + /// Anthropic: insert explicit `cache_control` markers. Explicit { max_breakpoints: u8 }, - /// それ以外: サーバ側自動 prefix、または未サポート + /// Other providers: automatic server-side prefix caching, or unsupported. Auto, } -/// Reasoning 制御（共通型、scheme 側で各社形式に投影）。 +/// Shared reasoning control, projected into each provider's format by the scheme. /// -/// 文字列は provider-native な effort label、数値は provider-native な -/// thinking budget token として扱う。どちらか一方だけを型で表現する。 +/// Strings are provider-native effort labels; numbers are provider-native +/// thinking token budgets. The type represents only one at a time. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[serde(untagged)] pub enum ReasoningControl { diff --git a/crates/agen/src/llm_client/client.rs b/crates/agen/src/llm_client/client.rs index b112d4291..f2bda6262 100644 --- a/crates/agen/src/llm_client/client.rs +++ b/crates/agen/src/llm_client/client.rs @@ -1,4 +1,4 @@ -//! LLMクライアント共通trait定義 +//! Shared LLM client trait definitions. use std::pin::Pin; @@ -6,19 +6,19 @@ use crate::llm_client::{ClientError, Request, RequestConfig, event::Event}; use async_trait::async_trait; use futures::Stream; -/// 設定に関する警告 +/// Configuration warning. /// -/// プロバイダがサポートしていない設定を使用した場合に返される。 +/// Returned when using a configuration option unsupported by the provider. #[derive(Debug, Clone)] pub struct ConfigWarning { - /// 設定オプション名 + /// Configuration option name. pub option_name: &'static str, - /// 警告メッセージ + /// Warning message. pub message: String, } impl ConfigWarning { - /// 新しい警告を作成 + /// Create a new warning. pub fn unsupported(option_name: &'static str, provider_name: &str) -> Self { Self { option_name, @@ -48,19 +48,19 @@ pub enum ToolCallCompletionSupport { ResponseComplete, } -/// LLMクライアントのtrait +/// LLM client trait. /// -/// 各プロバイダはこのtraitを実装し、統一されたインターフェースを提供する。 +/// Each provider implements this trait to expose a unified interface. #[async_trait] pub trait LlmClient: Send + Sync { - /// ストリーミングリクエストを送信し、Eventストリームを返す + /// Send a streaming request and return an Event stream. /// /// # Arguments - /// * `request` - リクエスト情報 + /// * `request` - Request information. /// /// # Returns - /// * `Ok(Stream)` - イベントストリーム - /// * `Err(ClientError)` - エラー + /// * `Ok(Stream)` - Event stream. + /// * `Err(ClientError)` - Error. async fn stream(&self, request: Request) -> Result; /// Clone this client into a new `Box`. @@ -74,15 +74,15 @@ pub trait LlmClient: Send + Sync { ToolCallCompletionSupport::ResponseComplete } - /// 設定をバリデーションし、未サポートの設定があれば警告を返す + /// Validate configuration and return warnings for unsupported options. /// /// # Arguments - /// * `config` - バリデーション対象の設定 + /// * `config` - Configuration to validate. /// /// # Returns - /// サポートされていない設定に対する警告のリスト + /// List of warnings for unsupported options. fn validate_config(&self, config: &RequestConfig) -> Vec { - // デフォルト実装: 全ての設定をサポート + // Default implementation: all options are supported. let _ = config; Vec::new() } @@ -94,9 +94,9 @@ impl Clone for Box { } } -/// `Box` に対する `LlmClient` の実装 +/// `LlmClient` implementation for `Box`. /// -/// これにより、動的ディスパッチを使用するクライアントも `Engine` で利用可能になる。 +/// Allows clients using dynamic dispatch to be used with `Engine`. #[async_trait] impl LlmClient for Box { async fn stream(&self, request: Request) -> Result { diff --git a/crates/agen/src/llm_client/error.rs b/crates/agen/src/llm_client/error.rs index 2566e6c26..36c183de3 100644 --- a/crates/agen/src/llm_client/error.rs +++ b/crates/agen/src/llm_client/error.rs @@ -1,17 +1,22 @@ -//! LLMクライアントエラー型 +//! Report provider communication failures to turn control. +//! +//! Context limit rejections are classified from structured provider codes; +//! fuzzy body matches must not imply success or retryability. The Engine maps +//! client errors to typed execution results, which the host records as the Run's +//! terminal outcome. Live diagnostic display and persistence are separate concerns. use std::{fmt, time::Duration}; -/// LLMクライアントのエラー +/// LLM client error. #[derive(Debug)] pub enum ClientError { - /// HTTPリクエストエラー + /// HTTP request error. Http(reqwest::Error), - /// JSONパースエラー + /// JSON parse error. Json(serde_json::Error), - /// SSEパースエラー + /// SSE parse error. Sse(String), - /// APIエラー (プロバイダからのエラーレスポンス) + /// API error (an error response from the provider). Api { status: Option, code: Option, @@ -26,7 +31,7 @@ pub enum ClientError { phase: &'static str, timeout: Duration, }, - /// 設定エラー + /// Configuration error. Config(String), } @@ -137,16 +142,16 @@ pub(crate) fn is_image_size_diagnostic(code: Option<&str>, message: &str) -> boo || (message.contains("image dimensions") && message.contains("exceed"))) } -/// transient な失敗としてリトライ対象になるかを判定する。 +/// Determine whether a failure is transient and eligible for retry. /// -/// 対象: -/// - `Api { status }` のうち 408 / 425 / 429 / 500 / 502 / 503 / 504 / 529 -/// - `Http(reqwest::Error)` のうち `is_connect()` または `is_timeout()` -/// - `Timeout { .. }` の lifecycle hard timeout +/// Eligible failures: +/// - `Api { status }` with 408 / 425 / 429 / 500 / 502 / 503 / 504 / 529. +/// - `Http(reqwest::Error)` with `is_connect()` or `is_timeout()`. +/// - Lifecycle hard timeouts from `Timeout { .. }`. /// -/// それ以外（Json、Sse、Config、上記以外の Api ステータス）は false。 -/// SSE 読み出し開始後の失敗は呼び出し側で `Sse` として上に流すため、 -/// ここで対象外にしておけば自動的に弾かれる。 +/// Other failures (Json, Sse, Config, and other Api statuses) return false. +/// Callers propagate failures after SSE reading begins as `Sse`, so excluding +/// them here automatically prevents retries. pub fn is_retryable(error: &ClientError) -> bool { match error { ClientError::Api { diff --git a/crates/agen/src/llm_client/event.rs b/crates/agen/src/llm_client/event.rs index 9f1e65bfe..d1e947f1f 100644 --- a/crates/agen/src/llm_client/event.rs +++ b/crates/agen/src/llm_client/event.rs @@ -1,6 +1,6 @@ -//! LLMクライアント層のイベント型 +//! Event types for the LLM client layer. //! -//! 各LLMプロバイダからのストリーミングレスポンスを表現するイベント型。 +//! Events representing streaming responses from LLM providers. use serde::{Deserialize, Serialize}; @@ -8,43 +8,42 @@ use serde::{Deserialize, Serialize}; // Core Event Types (from llm_client layer) // ============================================================================= -/// LLMからのストリーミングイベント +/// Streaming event from an LLM. /// -/// 各LLMプロバイダからのレスポンスは、この`Event`のストリームとして -/// 統一的に処理されます。 +/// Responses from all LLM providers are handled uniformly as streams of `Event`. /// -/// # イベントの種類 +/// # Event types /// -/// - **メタイベント**: `Ping`, `Usage`, `Status`, `Error`, `UnhandledSse` -/// - **ブロックイベント**: `BlockStart`, `BlockDelta`, `BlockStop`, `BlockAbort` +/// - **Meta events**: `Ping`, `Usage`, `Status`, `Error`, `UnhandledSse` +/// - **Block events**: `BlockStart`, `BlockDelta`, `BlockStop`, `BlockAbort` /// -/// # ブロックのライフサイクル +/// # Block lifecycle /// -/// テキスト、thinking、ツール呼び出しは、`BlockStart` → `BlockDelta`(複数) → `BlockStop` -/// の順序でイベントが発生します。thinking の round-trip metadata は -/// `BlockStop.reasoning` に載ります。 +/// Text, thinking, and tool calls emit `BlockStart` → `BlockDelta` (multiple) → +/// `BlockStop` in order. Thinking round-trip metadata is carried in +/// `BlockStop.reasoning`. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub enum Event { - /// ハートビート + /// Heartbeat. Ping(PingEvent), - /// トークン使用量 + /// Token usage. Usage(UsageEvent), - /// ストリームのステータス変化 + /// Stream status change. Status(StatusEvent), - /// エラー発生 + /// Error occurred. Error(ErrorEvent), - /// Scheme が生成内容として解釈しない未対応 SSE イベント。 + /// Unsupported SSE event that the Scheme does not interpret as generated content. /// - /// stream trace 用の観測イベントであり、timeline / history には反映しない。 + /// An observation event for stream tracing, not reflected in the timeline or history. UnhandledSse(UnhandledSseEvent), - /// ブロック開始（テキスト、ツール使用等） + /// Block start (text, tool use, etc.). BlockStart(BlockStart), - /// ブロックの差分データ + /// Block delta data. BlockDelta(BlockDelta), - /// ブロック正常終了 + /// Normal block completion. BlockStop(BlockStop), - /// ブロック中断 + /// Block abort. BlockAbort(BlockAbort), } @@ -52,57 +51,57 @@ pub enum Event { // Meta Events // ============================================================================= -/// Pingイベント（ハートビート） +/// Ping event (heartbeat). #[derive(Debug, Clone, PartialEq, Default, Serialize, Deserialize)] pub struct PingEvent { pub timestamp: Option, } -/// 使用量イベント +/// Usage event. /// -/// プロバイダから受信した 1 LLM リクエスト分のトークン会計。 -/// 各 scheme で正規化され、フィールドの意味は全プロバイダ共通: +/// Token accounting received from the provider for one LLM request. +/// Each scheme normalizes it; field meanings are shared across providers: /// -/// - `input_tokens` は **送信した prompt prefix 全体の占有量**（プロンプト全長）。 -/// キャッシュヒット分も含まれる。Anthropic は raw API では非キャッシュ分のみを -/// `input_tokens` として返すため、`AnthropicScheme::convert_usage` で -/// `cache_read + cache_creation` を加算してこの規約に揃えている。 -/// - `cache_read_input_tokens` / `cache_creation_input_tokens` は上記の内訳で、 -/// 料金会計用。占有量からは差し引かない。 +/// - `input_tokens` is the **occupancy of the entire submitted prompt prefix** +/// (full prompt length), including cache hits. Anthropic's raw API returns only +/// uncached tokens as `input_tokens`, so `AnthropicScheme::convert_usage` adds +/// `cache_read + cache_creation` to follow this convention. +/// - `cache_read_input_tokens` / `cache_creation_input_tokens` are breakdowns +/// for billing. They are not subtracted from occupancy. #[derive(Debug, Clone, PartialEq, Default, Serialize, Deserialize)] pub struct UsageEvent { - /// 送信した prompt prefix の総トークン数（占有量、キャッシュ込み） + /// Total tokens in the submitted prompt prefix (occupancy, including cached tokens). pub input_tokens: Option, - /// このリクエストで生成された出力トークン数 + /// Output tokens generated by this request. pub output_tokens: Option, /// `input_tokens + output_tokens` pub total_tokens: Option, - /// `input_tokens` のうちキャッシュから読まれた分（割引料金） + /// Portion of `input_tokens` read from cache (discounted rate). pub cache_read_input_tokens: Option, - /// `input_tokens` のうちこのリクエストでキャッシュに書かれた分（割増料金、Anthropic） + /// Portion of `input_tokens` written to cache by this request (premium rate, Anthropic). pub cache_creation_input_tokens: Option, } -/// ステータスイベント +/// Status event. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub struct StatusEvent { pub status: ResponseStatus, } -/// レスポンスステータス +/// Response status. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub enum ResponseStatus { - /// ストリーム開始 + /// Stream started. Started, - /// 正常完了 + /// Completed normally. Completed, - /// キャンセルされた + /// Cancelled. Cancelled, - /// エラー発生 + /// Error occurred. Failed, } -/// エラーイベント +/// Error event. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub struct ErrorEvent { pub code: Option, @@ -118,10 +117,10 @@ impl ErrorEvent { } } -/// 未対応 SSE イベントの観測用メタイベント。 +/// Observation-only meta event for unsupported SSE events. /// -/// `data_preview` は provider から受け取った raw SSE data の bounded preview、 -/// `data_len` は preview 前の raw data byte length。 +/// `data_preview` is a bounded preview of raw SSE data received from the provider; +/// `data_len` is the raw data's byte length before preview truncation. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub struct UnhandledSseEvent { pub provider: String, @@ -134,27 +133,27 @@ pub struct UnhandledSseEvent { // Block Types // ============================================================================= -/// ブロックの種別 +/// Block type. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] pub enum BlockType { - /// テキスト生成 + /// Text generation. Text, - /// 思考 (Claude Extended Thinking等) + /// Thinking (e.g. Claude Extended Thinking). Thinking, - /// ツール呼び出し + /// Tool call. ToolUse, - /// ツール結果 + /// Tool result. ToolResult, } -/// ブロック開始イベント +/// Block start event. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub struct BlockStart { - /// ブロックのインデックス + /// Block index. pub index: usize, - /// ブロックの種別 + /// Block type. pub block_type: BlockType, - /// ブロック固有のメタデータ + /// Block-specific metadata. pub metadata: BlockMetadata, } @@ -164,7 +163,7 @@ impl BlockStart { } } -/// ブロックのメタデータ +/// Block metadata. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub enum BlockMetadata { Text, @@ -173,28 +172,28 @@ pub enum BlockMetadata { ToolResult { tool_use_id: String }, } -/// ブロックデルタイベント +/// Block delta event. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub struct BlockDelta { - /// ブロックのインデックス + /// Block index. pub index: usize, - /// デルタの内容 + /// Delta content. pub delta: DeltaContent, } -/// デルタの内容 +/// Delta content. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub enum DeltaContent { - /// テキストデルタ + /// Text delta. Text(String), - /// 思考デルタ + /// Thinking delta. Thinking(String), - /// ツール引数のJSON部分文字列 + /// Partial JSON string for tool arguments. InputJson(String), } impl DeltaContent { - /// デルタのブロック種別を取得 + /// Get the delta's block type. pub fn block_type(&self) -> BlockType { match self { DeltaContent::Text(_) => BlockType::Text, @@ -204,20 +203,20 @@ impl DeltaContent { } } -/// ブロック停止イベント +/// Block stop event. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub struct BlockStop { - /// ブロックのインデックス + /// Block index. pub index: usize, - /// ブロックの種別 + /// Block type. pub block_type: BlockType, - /// 停止理由 + /// Stop reason. pub stop_reason: Option, - /// Thinking block の停止時に確定した reasoning round-trip metadata。 + /// Reasoning round-trip metadata finalized when a Thinking block stops. /// - /// `None` の Thinking block は live streaming / trace 用で、history に - /// `Item::Reasoning` として永続化しない。`Some` の場合は block lifecycle - /// が永続化の authoritative source になる。 + /// Thinking blocks with `None` are for live streaming / tracing and are not + /// persisted in history as `Item::Reasoning`. With `Some`, the block lifecycle + /// is the authoritative source for persistence. pub reasoning: Option, } @@ -227,14 +226,14 @@ impl BlockStop { } } -/// ブロック中断イベント +/// Block abort event. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub struct BlockAbort { - /// ブロックのインデックス + /// Block index. pub index: usize, - /// ブロックの種別 + /// Block type. pub block_type: BlockType, - /// 中断理由 + /// Abort reason. pub reason: String, } @@ -244,36 +243,36 @@ impl BlockAbort { } } -/// Thinking block stop で確定した reasoning material。 +/// Reasoning material finalized at Thinking block stop. /// -/// `Item::Reasoning` の round-trip に必要な provider material を保持する。 -/// `text` は deltas から収集した本文を上書きするために使う（metadata-only -/// reasoning block や provider completion event で全文が届くケース）。 +/// Holds provider material needed to round-trip `Item::Reasoning`. +/// `text` overrides the body collected from deltas (for metadata-only reasoning +/// blocks or provider completion events that deliver the full text). #[derive(Debug, Clone, PartialEq, Default, Serialize, Deserialize)] pub struct ReasoningBlockData { - /// scheme 側で観測した item id（OpenAI Responses の `id`）。 + /// Item ID observed by the scheme (OpenAI Responses `id`). pub id: Option, - /// reasoning 本体テキスト。`None` の場合は block delta 収集結果を使う。 + /// Reasoning body text. `None` uses the collected block deltas. pub text: Option, - /// summary (OpenAI Responses の `summary_text[]`)。他 scheme は空。 + /// Summary (OpenAI Responses `summary_text[]`); empty for other schemes. pub summary: Vec, - /// 暗号化された opaque blob（Anthropic `redacted_thinking.data` / - /// OpenAI Responses `encrypted_content`）。 + /// Encrypted opaque blob (Anthropic `redacted_thinking.data` / + /// OpenAI Responses `encrypted_content`). pub encrypted_content: Option, - /// Anthropic extended thinking signature。round-trip 必須。 + /// Anthropic extended thinking signature, required for round-tripping. pub signature: Option, } -/// 停止理由 +/// Stop reason. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub enum StopReason { - /// 自然終了 + /// Natural end of turn. EndTurn, - /// 最大トークン数到達 + /// Maximum token count reached. MaxTokens, - /// ストップシーケンス到達 + /// Stop sequence reached. StopSequence, - /// ツール使用 + /// Tool use. ToolUse, } @@ -282,7 +281,7 @@ pub enum StopReason { // ============================================================================= impl Event { - /// テキストブロック開始イベントを作成 + /// Create a text block start event. pub fn text_block_start(index: usize) -> Self { Event::BlockStart(BlockStart { index, @@ -291,7 +290,7 @@ impl Event { }) } - /// テキストデルタイベントを作成 + /// Create a text delta event. pub fn text_delta(index: usize, text: impl Into) -> Self { Event::BlockDelta(BlockDelta { index, @@ -299,7 +298,7 @@ impl Event { }) } - /// テキストブロック停止イベントを作成 + /// Create a text block stop event. pub fn text_block_stop(index: usize, stop_reason: Option) -> Self { Event::BlockStop(BlockStop { index, @@ -309,7 +308,7 @@ impl Event { }) } - /// ツール使用ブロック開始イベントを作成 + /// Create a tool use block start event. pub fn tool_use_start(index: usize, id: impl Into, name: impl Into) -> Self { Event::BlockStart(BlockStart { index, @@ -321,7 +320,7 @@ impl Event { }) } - /// ツール引数デルタイベントを作成 + /// Create a tool argument delta event. pub fn tool_input_delta(index: usize, json: impl Into) -> Self { Event::BlockDelta(BlockDelta { index, @@ -329,7 +328,7 @@ impl Event { }) } - /// ツール使用ブロック停止イベントを作成 + /// Create a tool use block stop event. pub fn tool_use_stop(index: usize) -> Self { Event::BlockStop(BlockStop { index, @@ -339,7 +338,7 @@ impl Event { }) } - /// 使用量イベントを作成 + /// Create a usage event. pub fn usage(input_tokens: u64, output_tokens: u64) -> Self { Event::Usage(UsageEvent { input_tokens: Some(input_tokens), @@ -350,7 +349,7 @@ impl Event { }) } - /// Pingイベントを作成 + /// Create a Ping event. pub fn ping() -> Self { Event::Ping(PingEvent { timestamp: None }) } diff --git a/crates/agen/src/llm_client/mod.rs b/crates/agen/src/llm_client/mod.rs index 3037820a0..9957f7f81 100644 --- a/crates/agen/src/llm_client/mod.rs +++ b/crates/agen/src/llm_client/mod.rs @@ -1,20 +1,13 @@ -//! LLMクライアント層 +//! Boundary between the Engine and provider communication. //! -//! 各LLMプロバイダと通信し、統一された[`Event`] -//! ストリームを出力します。 +//! [`LlmClient`] accepts a shared [`Request`] and returns normalized [`Event`]s. +//! Provider-specific bodies, authentication requirements, and stream parsing are +//! separated into [`scheme`] and [`transport`] to limit how API changes affect +//! turn progression and tool execution semantics. //! -//! # サポートするプロバイダ -//! -//! - Anthropic (Claude) -//! - OpenAI (GPT-4, etc.) -//! - Google (Gemini) -//! - Ollama (ローカルLLM) -//! -//! # アーキテクチャ -//! -//! - [`LlmClient`] - プロバイダ共通のtrait -//! - `providers`: プロバイダ固有のクライアント実装 -//! - `scheme`: APIスキーマ（リクエスト/レスポンス変換） +//! Scheme capabilities describe what the wire format can safely handle. +//! The client construction layer owns per-model capability tables and alias +//! resolution; the Engine does not depend on that selection process. pub mod auth; pub mod capability; diff --git a/crates/agen/src/llm_client/retry.rs b/crates/agen/src/llm_client/retry.rs index 123084c5a..9a464bf14 100644 --- a/crates/agen/src/llm_client/retry.rs +++ b/crates/agen/src/llm_client/retry.rs @@ -1,25 +1,25 @@ -//! LLM response stream を開く前の transient error 向けリトライポリシー。 +//! Retry policy for transient errors before opening an LLM response stream. //! -//! `LlmClient::stream` の open error に対して `is_retryable` を見て -//! retry / backoff / cancellation をまとめて管理する。 -//! SSE 読み出し開始後の失敗は対象外。 +//! Uses `is_retryable` on `LlmClient::stream` open errors to manage retries, +//! backoff, and cancellation together. Failures after SSE reading begins +//! are excluded. use std::time::Duration; -/// 指数バックオフ + ジッター + 累積タイムアウトを表すポリシー。 +/// Policy combining exponential backoff, jitter, and a cumulative timeout. /// -/// `Default` は agen 全体の固定値を返す。呼び出し側からの上書きが -/// 必要になったら拡張する。 +/// `Default` returns fixed values shared across agen. Extend this if caller +/// overrides become necessary. #[derive(Debug, Clone)] pub struct RetryPolicy { - /// 指数の基準値。`base * 2^attempt` を `cap` で頭打ちにした上限から - /// フルジッターで実際の wait を抽選する。 + /// Exponential base. Full jitter samples the actual wait from an upper bound + /// of `base * 2^attempt`, capped at `cap`. pub base: Duration, - /// 1 回あたりの wait の上限。 + /// Maximum wait per attempt. pub cap: Duration, - /// 試行の合計回数（初回 + リトライ）。`1` ならリトライしない。 + /// Total attempts (initial attempt + retries). `1` disables retries. pub max_attempts: u32, - /// 初回送信開始からの累積タイムアウト。これを超える wait は打ち切る。 + /// Cumulative timeout from the first send. Waits exceeding it are cut short. pub total_timeout: Duration, } @@ -35,8 +35,8 @@ impl Default for RetryPolicy { } impl RetryPolicy { - /// `attempt` 回目の失敗（0-indexed）後に待つ時間を返す。 - /// `Retry-After` で上書きしたい場合は呼び出さず、その値をそのまま使う。 + /// Return the wait after failure number `attempt` (zero-indexed). + /// For a `Retry-After` override, use its value directly instead of calling this. pub fn backoff(&self, attempt: u32) -> Duration { let shift = attempt.min(20); let base_nanos = self.base.as_nanos() as u64; @@ -47,9 +47,9 @@ impl RetryPolicy { } } -/// `[0, max_nanos]` から擬似乱数的に 1 つ取り出す。`SystemTime` の -/// 下位ビットを splitmix64 で攪拌するだけの軽量実装で、暗号的乱数性は -/// 持たないがフルジッターのぶつかり回避には十分。 +/// Sample a pseudorandom value from `[0, max_nanos]`. This lightweight +/// implementation mixes the low bits of `SystemTime` with splitmix64. +/// It is not cryptographically random, but suffices to avoid full-jitter collisions. fn jitter_nanos(max_nanos: u64) -> u64 { if max_nanos == 0 { return 0; diff --git a/crates/agen/src/llm_client/scheme/anthropic/capability.rs b/crates/agen/src/llm_client/scheme/anthropic/capability.rs index 3a81974dc..f3bd1a5ed 100644 --- a/crates/agen/src/llm_client/scheme/anthropic/capability.rs +++ b/crates/agen/src/llm_client/scheme/anthropic/capability.rs @@ -1,17 +1,17 @@ -//! Anthropic scheme の wire-level 既定 capability。 +//! Wire-level default capability for the Anthropic scheme. //! -//! モデル ID 固有のテーブル(`claude-*` など)は client construction layer -//! の責務。ここでは未知モデルでも「この wire で -//! 安全に送れる最小共通項」を返すだけに留める。 +//! Model ID tables (e.g. `claude-*`) belong to the client construction layer. +//! This module returns only the minimum common features safe to send over +//! this wire, even for unknown models. use crate::llm_client::capability::{ CacheStrategy, ModelCapability, StructuredOutput, ToolCallingSupport, }; -/// Scheme 既定の capability。 +/// Scheme default capability. /// -/// Ollama の `/v1/messages` 流用を想定して `cache_control` を送らない -/// `CacheStrategy::Auto` にする。 +/// Uses `CacheStrategy::Auto` to avoid sending `cache_control`, allowing reuse +/// with Ollama's `/v1/messages` endpoint. pub(crate) fn default_capability() -> ModelCapability { ModelCapability { tool_calling: ToolCallingSupport::Parallel, diff --git a/crates/agen/src/llm_client/scheme/anthropic/events.rs b/crates/agen/src/llm_client/scheme/anthropic/events.rs index bc5482fd8..3d3706893 100644 --- a/crates/agen/src/llm_client/scheme/anthropic/events.rs +++ b/crates/agen/src/llm_client/scheme/anthropic/events.rs @@ -1,6 +1,6 @@ -//! Anthropic SSEイベントパース +//! Anthropic SSE event parsing. //! -//! Anthropic Messages APIのSSEイベントをパースし、統一Event型に変換 +//! Parse Anthropic Messages API SSE events and convert them to the unified Event type. use crate::llm_client::{ ClientError, @@ -14,7 +14,7 @@ use serde::Deserialize; use super::AnthropicScheme; use super::scheme_impl::{AnthropicState, PendingThinking}; -/// Anthropic SSEイベントタイプ +/// Anthropic SSE event type. #[derive(Debug, Clone, PartialEq, Eq)] pub(crate) enum AnthropicEventType { MessageStart, @@ -28,7 +28,7 @@ pub(crate) enum AnthropicEventType { } impl AnthropicEventType { - /// イベントタイプ文字列からパース + /// Parse an event type string. pub(crate) fn parse(s: &str) -> Option { match s { "message_start" => Some(Self::MessageStart), @@ -45,10 +45,10 @@ impl AnthropicEventType { } // ============================================================================ -// SSEイベントのJSON構造 +// SSE event JSON structures // ============================================================================ -/// message_start イベント +/// message_start event. #[derive(Debug, Deserialize)] pub(crate) struct MessageStartEvent { pub message: MessageStartMessage, @@ -62,7 +62,7 @@ pub(crate) struct MessageStartMessage { pub usage: Option, } -/// content_block_start イベント +/// content_block_start event. #[derive(Debug, Deserialize)] pub(crate) struct ContentBlockStartEvent { pub index: usize, @@ -79,15 +79,15 @@ pub(crate) enum ContentBlock { Thinking { #[serde(default)] thinking: String, - /// 非ストリーミングレスポンス由来の初期 signature（通常はストリームでは - /// 空 → `signature_delta` で埋まる）。 + /// Initial signature from a non-streaming response (usually empty in + /// streams, then filled by `signature_delta`). #[serde(default)] signature: Option, }, #[serde(rename = "redacted_thinking")] RedactedThinking { - /// 暗号化された opaque blob。signature ではなく、まるごと - /// `redacted_thinking.data` として送り返す必要がある。 + /// Encrypted opaque blob, not a signature. Must be sent back in full + /// as `redacted_thinking.data`. #[serde(default)] data: String, }, @@ -99,7 +99,7 @@ pub(crate) enum ContentBlock { }, } -/// content_block_delta イベント +/// content_block_delta event. #[derive(Debug, Deserialize)] pub(crate) struct ContentBlockDeltaEvent { pub index: usize, @@ -120,13 +120,13 @@ pub(crate) enum DeltaBlock { SignatureDelta { signature: String }, } -/// content_block_stop イベント +/// content_block_stop event. #[derive(Debug, Deserialize)] pub(crate) struct ContentBlockStopEvent { pub index: usize, } -/// message_delta イベント +/// message_delta event. #[allow(dead_code)] #[derive(Debug, Deserialize)] pub(crate) struct MessageDeltaEvent { @@ -141,7 +141,7 @@ pub(crate) struct MessageDeltaData { pub stop_sequence: Option, } -/// 使用量データ +/// Usage data. #[derive(Debug, Deserialize)] pub(crate) struct UsageData { pub input_tokens: Option, @@ -150,7 +150,7 @@ pub(crate) struct UsageData { pub cache_creation_input_tokens: Option, } -/// エラーイベント +/// Error event. #[derive(Debug, Deserialize)] pub(crate) struct ErrorEventData { pub error: ErrorDetail, @@ -164,20 +164,20 @@ pub(crate) struct ErrorDetail { } // ============================================================================ -// イベント変換 +// Event conversion // ============================================================================ impl AnthropicScheme { - /// SSEイベントをEvent型に変換 + /// Convert an SSE event to Event. /// /// # Arguments - /// * `event_type` - SSEイベントタイプ - /// * `data` - イベントデータJSON文字列 + /// * `event_type` - SSE event type. + /// * `data` - Event data as a JSON string. /// /// # Returns - /// * `Ok(Some(Event))` - 変換成功 - /// * `Ok(None)` - イベントを無視（unknown event等） - /// * `Err(ClientError)` - パースエラー + /// * `Ok(Some(Event))` - Conversion succeeded. + /// * `Ok(None)` - Event ignored (e.g. an unknown event). + /// * `Err(ClientError)` - Parse error. pub(crate) fn parse_event( &self, event_type: &str, @@ -191,11 +191,11 @@ impl AnthropicScheme { match event_type { AnthropicEventType::MessageStart => { let event: MessageStartEvent = serde_json::from_str(data)?; - // message_start時にUsageイベントがあれば出力 + // Emit a Usage event at message_start if available. if let Some(usage) = event.message.usage { return Ok(Some(Event::Usage(self.convert_usage(&usage)))); } - // Statusイベントとして開始を通知 + // Notify start via a Status event. Ok(Some(Event::Status(StatusEvent { status: ResponseStatus::Started, }))) @@ -210,18 +210,18 @@ impl AnthropicScheme { } AnthropicEventType::ContentBlockStop => { let event: ContentBlockStopEvent = serde_json::from_str(data)?; - // Note: BlockStopにはblock_typeが必要だが、AnthropicはStopイベントに含めない - // Timeline層がBlockStartを追跡して正しいblock_typeを知る + // Note: BlockStop requires block_type, but Anthropic omits it from Stop events. + // The stateful parser tracks BlockStart to restore the correct block_type. Ok(Some(Event::BlockStop(BlockStop { index: event.index, - block_type: BlockType::Text, // Timeline層で上書きされる + block_type: BlockType::Text, // Placeholder for the stateful parser. stop_reason: None, reasoning: None, }))) } AnthropicEventType::MessageDelta => { let event: MessageDeltaEvent = serde_json::from_str(data)?; - // Usage情報があれば出力 + // Emit Usage information if available. if let Some(usage) = event.usage { return Ok(Some(Event::Usage(self.convert_usage(&usage)))); } @@ -271,7 +271,7 @@ impl AnthropicScheme { DeltaContent::InputJson(partial_json.clone()) } DeltaBlock::SignatureDelta { .. } => { - // signature_delta は無視 + // Ignore signature_delta. return None; } }; @@ -282,15 +282,14 @@ impl AnthropicScheme { })) } - /// state を持ち回す上位パース。 + /// Higher-level parser carrying state across events. /// - /// `parse_event` の単発 Event に加えて、以下を行う: - /// - `content_block_stop` の `block_type` を直前の Start 値で書き戻す - /// - `thinking` / `redacted_thinking` ブロックの本体・signature・data を - /// `state.pending_thinking` に蓄積し、`content_block_stop` の Thinking - /// BlockStop metadata に載せる - /// - `signature_delta` を蓄積（Stream channel には流さず、reasoning metadata - /// にだけ反映する） + /// In addition to the single Event from `parse_event`, it: + /// - Writes the preceding Start's `block_type` into `content_block_stop`. + /// - Accumulates the body, signature, and data of `thinking` / + /// `redacted_thinking` blocks in `state.pending_thinking` and attaches them + /// to Thinking BlockStop metadata at `content_block_stop`. + /// - Accumulates `signature_delta` only in reasoning metadata, not the Stream channel. pub(crate) fn parse_with_state( &self, event_type: &str, @@ -301,8 +300,8 @@ impl AnthropicScheme { return Ok(Vec::new()); }; - // signature_delta はストリーム表示には流さず、state にだけ蓄積。 - // それ以外は parse_event で標準 Event 化する。 + // Accumulate signature_delta in state only, not in the stream display. + // Convert everything else to standard Events via parse_event. let mut emitted: Vec = Vec::new(); match parsed_event_type { @@ -351,7 +350,7 @@ impl AnthropicScheme { } DeltaBlock::SignatureDelta { signature } => { if let Some(pending) = state.pending_thinking.as_mut() { - // 通常 1 回しか来ないが、複数 fragment 来ても連結しておく + // Usually received once, but concatenate multiple fragments if present. match &mut pending.signature { Some(acc) => acc.push_str(signature), None => pending.signature = Some(signature.clone()), @@ -391,7 +390,7 @@ impl AnthropicScheme { reasoning, })); } - // 残りは state を必要としない。既存 parse_event に委譲。 + // The rest needs no state; delegate to the existing parse_event. _ => { if let Some(event) = self.parse_event(event_type, data)? { emitted.push(event); @@ -403,13 +402,13 @@ impl AnthropicScheme { } fn convert_usage(&self, usage: &UsageData) -> UsageEvent { - // Anthropic の `input_tokens` は **キャッシュ外** の入力トークンのみで、 - // プロンプト全長は input_tokens + cache_read + cache_creation。 - // UsageEvent の `input_tokens` には「占有量（プロンプト全長）」を載せる - // 規約に合わせて、ここでキャッシュ分を足し込む。 - // cache_read_input_tokens / cache_creation_input_tokens は内訳として - // 別フィールドに残るので、料金計算側で `input - cache_read - cache_creation` - // により非キャッシュ入力分は逆算可能。 + // Anthropic's `input_tokens` counts only **uncached** input tokens. + // Full prompt length is input_tokens + cache_read + cache_creation. + // Add cached tokens here to follow UsageEvent's convention of reporting + // occupancy (full prompt length) in `input_tokens`. + // cache_read_input_tokens / cache_creation_input_tokens remain separate + // breakdown fields, so billing can recover uncached input with + // `input - cache_read - cache_creation`. let raw_input = usage.input_tokens.unwrap_or(0); let cache_read = usage.cache_read_input_tokens.unwrap_or(0); let cache_creation = usage.cache_creation_input_tokens.unwrap_or(0); @@ -438,7 +437,7 @@ mod tests { let event = scheme.parse_event("message_start", data).unwrap().unwrap(); match event { Event::Usage(u) => { - // キャッシュなしなので input_total = raw_input = 10 + // No cache, so input_total = raw_input = 10. assert_eq!(u.input_tokens, Some(10)); } _ => panic!("Expected Usage event"), @@ -447,9 +446,9 @@ mod tests { #[test] fn test_convert_usage_includes_cache_in_input_total() { - // Anthropic の input_tokens はキャッシュ外のみで、占有量は - // input + cache_read + cache_creation。 - // UsageEvent.input_tokens は占有量に正規化される。 + // Anthropic's input_tokens counts only uncached tokens; occupancy is + // input + cache_read + cache_creation. + // UsageEvent.input_tokens is normalized to occupancy. let scheme = AnthropicScheme::new(); let usage = UsageData { input_tokens: Some(100), @@ -531,7 +530,7 @@ mod tests { #[test] fn thinking_block_stop_carries_reasoning_with_signature() { - // thinking ブロックが完了したら reasoning metadata に text+signature が乗ること + // Completed thinking blocks carry text+signature in reasoning metadata. let scheme = AnthropicScheme::new(); let mut state = AnthropicState::default(); diff --git a/crates/agen/src/llm_client/scheme/anthropic/mod.rs b/crates/agen/src/llm_client/scheme/anthropic/mod.rs index 7b6a58054..f51c50e92 100644 --- a/crates/agen/src/llm_client/scheme/anthropic/mod.rs +++ b/crates/agen/src/llm_client/scheme/anthropic/mod.rs @@ -1,7 +1,7 @@ -//! Anthropic Messages API スキーマ +//! Anthropic Messages API scheme. //! -//! - リクエストJSON生成 -//! - SSEイベントパース → Event変換 +//! - Request JSON generation. +//! - SSE event parsing → Event conversion. mod capability; mod events; @@ -10,14 +10,14 @@ mod scheme_impl; pub use scheme_impl::AnthropicState; -/// Anthropicスキーマ +/// Anthropic scheme. /// -/// Anthropic Messages APIのリクエスト/レスポンス変換を担当 +/// Handles request/response conversion for the Anthropic Messages API. #[derive(Debug, Clone)] pub struct AnthropicScheme { - /// APIバージョン + /// API version. pub api_version: String, - /// 細粒度ツールストリーミングを有効にするか + /// Whether to enable fine-grained tool streaming. pub fine_grained_tool_streaming: bool, } @@ -31,12 +31,12 @@ impl Default for AnthropicScheme { } impl AnthropicScheme { - /// 新しいスキーマを作成 + /// Create a new scheme. pub fn new() -> Self { Self::default() } - /// 細粒度ツールストリーミングを有効/無効にする + /// Enable or disable fine-grained tool streaming. pub fn with_fine_grained_tool_streaming(mut self, enabled: bool) -> Self { self.fine_grained_tool_streaming = enabled; self diff --git a/crates/agen/src/llm_client/scheme/anthropic/request.rs b/crates/agen/src/llm_client/scheme/anthropic/request.rs index b546f4c94..610844c91 100644 --- a/crates/agen/src/llm_client/scheme/anthropic/request.rs +++ b/crates/agen/src/llm_client/scheme/anthropic/request.rs @@ -41,7 +41,7 @@ pub(crate) struct AnthropicRequest { pub thinking: Option, } -/// Anthropic extended thinking 指示。 +/// Anthropic extended thinking instructions. #[derive(Debug, Serialize)] #[serde(tag = "type", rename_all = "snake_case")] pub(crate) enum AnthropicThinking { @@ -90,8 +90,7 @@ pub(crate) enum AnthropicContentPart { }, #[serde(rename = "redacted_thinking")] RedactedThinking { - /// 暗号化済み reasoning blob。`Item::Reasoning::encrypted_content` - /// から渡る。 + /// Encrypted reasoning blob from `Item::Reasoning::encrypted_content`. data: String, #[serde(skip_serializing_if = "Option::is_none")] cache_control: Option, @@ -181,9 +180,9 @@ pub(crate) struct AnthropicTool { impl AnthropicScheme { /// Build Anthropic request from Request. /// - /// `capability.prompt_caching` が [`CacheStrategy::Auto`] のときは - /// `cache_control` マーカーを一切挿入しない（Ollama の `/v1/messages` - /// 流用時など、サーバ側が `cache_control` を受け付けないケース）。 + /// No `cache_control` markers are inserted when `capability.prompt_caching` + /// is [`CacheStrategy::Auto`] (e.g. servers that reject `cache_control`, + /// such as Ollama when reusing `/v1/messages`). pub(crate) fn build_request( &self, model: &str, @@ -198,9 +197,8 @@ impl AnthropicScheme { let messages = self.convert_items_to_messages(&request.items, &breakpoints); let tools = request.tools.iter().map(|t| self.convert_tool(t)).collect(); - // Reasoning の投影: capability が BudgetTokens / Both をサポート - // していて、request 側で budget_tokens が指定されているときだけ - // thinking フィールドを付ける。 + // Project reasoning: add the thinking field only when the capability + // supports BudgetTokens / Both and the request specifies budget_tokens. let supports_budget_tokens = matches!( capability.reasoning, Some(ReasoningSupport::BudgetTokens | ReasoningSupport::Both), @@ -361,23 +359,23 @@ impl AnthropicScheme { .. } => { flush_pending(&mut messages, &mut pending_user, "user", &mut locations); - // Anthropic はアシスタントターン中の `thinking` / - // `redacted_thinking` ブロックを必ず assistant role の - // content_part として送り返す必要がある。 + // Anthropic requires `thinking` / `redacted_thinking` blocks + // from an assistant turn to be sent back as assistant-role + // content_parts. // - // - signature あり: `thinking` content_part を投影 - // - signature 無し + encrypted_content あり: - // `redacted_thinking` content_part を投影 - // - どちらも無い: 他 scheme（OpenAI 等）から流入した - // 素の reasoning text。Anthropic に投げる意味も - // round-trip の根拠も無いので drop。 + // - With signature: project a `thinking` content_part. + // - No signature, but encrypted_content present: + // project a `redacted_thinking` content_part. + // - Neither: plain reasoning text from another scheme + // (e.g. OpenAI). Drop it: there is no reason to send it to + // Anthropic and no basis for round-tripping it. if let Some(sig) = signature.clone() { pending_assistant .push((i, AnthropicContentPart::thinking(text.clone(), sig))); } else if let Some(data) = encrypted_content.clone() { pending_assistant.push((i, AnthropicContentPart::redacted_thinking(data))); } - // どちらも None なら何も pend せず、本 item は無視。 + // If both are None, queue nothing and ignore this item. } } } @@ -486,7 +484,7 @@ mod tests { CacheStrategy, ReasoningEffort, StructuredOutput, ToolCallingSupport, }; - /// cache_control が有効になる既定の capability。 + /// Default capability with cache_control enabled. fn cap_explicit() -> ModelCapability { ModelCapability { tool_calling: ToolCallingSupport::Parallel, @@ -497,7 +495,7 @@ mod tests { } } - /// cache_control を送らない capability（Ollama 等）。 + /// Capability that does not send cache_control (e.g. Ollama). fn cap_auto() -> ModelCapability { ModelCapability { prompt_caching: CacheStrategy::Auto, @@ -901,8 +899,8 @@ mod tests { #[test] fn cache_auto_does_not_add_cache_control() { - // Ollama のように `CacheStrategy::Auto` のときは cache_control - // マーカーを一切付けない。breakpoint 計算も走らないこと。 + // With `CacheStrategy::Auto` (e.g. Ollama), add no cache_control markers. + // Breakpoint calculation must not run either. let scheme = AnthropicScheme::new(); let mut request = Request::new().user("hello"); request.cache_anchor = Some(0); @@ -933,8 +931,8 @@ mod tests { #[test] fn reasoning_with_signature_projects_thinking_part() { - // Item::Reasoning に signature があれば assistant role の - // `thinking` content_part として送る。 + // Send Item::Reasoning with a signature as an assistant-role + // `thinking` content_part. let scheme = AnthropicScheme::new(); let request = Request::new() .user("hi") @@ -976,21 +974,21 @@ mod tests { #[test] fn reasoning_without_signature_or_encrypted_is_dropped() { - // 他 scheme から流入した素の reasoning は Anthropic に投げない。 + // Do not send plain reasoning from other schemes to Anthropic. let scheme = AnthropicScheme::new(); let request = Request::new() .user("hi") .item(Item::reasoning("plain text")) .item(Item::assistant_message("done")); let req = scheme.build_request("claude-sonnet-4-20250514", &request, &cap_explicit()); - // thinking part は 1 つも乗らない + // No thinking parts are included. assert!(collect_assistant_thinking_parts(&req).is_empty()); } #[test] fn thinking_part_lands_in_assistant_role_message() { - // wire 構造の position 検証: thinking part は assistant role の - // message 配列に並ぶ（user role には絶対に入らない）。 + // Verify wire positions: thinking parts belong in the assistant-role + // message array (never in the user role). let scheme = AnthropicScheme::new(); let request = Request::new() .user("question?") @@ -1000,7 +998,7 @@ mod tests { .user("follow up"); let req = scheme.build_request("claude-sonnet-4-20250514", &request, &cap_explicit()); - // 全 thinking part が assistant role の message に存在すること + // All thinking parts must be in assistant-role messages. let mut thinking_msg_indices = Vec::new(); for (i, msg) in req.messages.iter().enumerate() { if let AnthropicContent::Parts(parts) = &msg.content { @@ -1023,9 +1021,9 @@ mod tests { req.messages, ); - // thinking part を含む assistant message は、それに続く tool_use を含む - // assistant message より前 (= 先頭側) に位置すること - // (Anthropic 仕様: 同一論理ターン内で thinking → tool_use の順) + // Assistant messages containing thinking parts must precede the + // subsequent assistant messages containing tool_use. + // (Anthropic requires thinking → tool_use within one logical turn.) let mut tool_use_msg_indices = Vec::new(); for (i, msg) in req.messages.iter().enumerate() { if let AnthropicContent::Parts(parts) = &msg.content { @@ -1050,7 +1048,7 @@ mod tests { #[test] fn redacted_thinking_part_lands_in_assistant_role_message() { - // RedactedThinking も同様に assistant role に置かれること。 + // RedactedThinking must also be placed in the assistant role. let scheme = AnthropicScheme::new(); let request = Request::new() .user("ask") diff --git a/crates/agen/src/llm_client/scheme/anthropic/scheme_impl.rs b/crates/agen/src/llm_client/scheme/anthropic/scheme_impl.rs index f4b7a0098..7c3ce8be3 100644 --- a/crates/agen/src/llm_client/scheme/anthropic/scheme_impl.rs +++ b/crates/agen/src/llm_client/scheme/anthropic/scheme_impl.rs @@ -1,7 +1,7 @@ //! `impl Scheme for AnthropicScheme` //! -//! Anthropic Messages API の wire 表現に必要な URL・ヘッダ・SSE パース・ -//! リクエスト body 生成を共通 `Scheme` trait にぶら下げる。 +//! Implements the URLs, headers, SSE parsing, and request body generation needed +//! for the Anthropic Messages API wire format through the shared `Scheme` trait. use serde_json::Value; @@ -17,22 +17,20 @@ use crate::llm_client::{ use super::AnthropicScheme; -/// Anthropic の SSE パースで必要な状態。 +/// State required for Anthropic SSE parsing. /// -/// 1. `content_block_stop` イベントは `block_type` を持たない仕様なので、 -/// 直前の `content_block_start` で観測した `block_type` を保持して -/// `BlockStop` に書き戻す。 -/// 2. `thinking` ブロック中の `thinking_delta` テキストと `signature_delta` -/// 署名、および `redacted_thinking` ブロックの `data` を蓄積し、 -/// `content_block_stop` の Thinking block metadata として返す -/// （round-trip 永続化のため）。 +/// 1. `content_block_stop` events lack `block_type`, so retain the type observed +/// in the preceding `content_block_start` and write it back into `BlockStop`. +/// 2. Accumulate `thinking_delta` text and `signature_delta` signatures from +/// `thinking` blocks, plus `data` from `redacted_thinking` blocks. Return them +/// as Thinking block metadata at `content_block_stop` for round-trip persistence. #[derive(Debug, Default)] pub struct AnthropicState { pub(crate) current_block_type: Option, pub(crate) pending_thinking: Option, } -/// 1 つの `thinking` または `redacted_thinking` content_block の蓄積バッファ。 +/// Accumulation buffer for one `thinking` or `redacted_thinking` content_block. #[derive(Debug, Default)] pub(crate) struct PendingThinking { pub(crate) text: String, @@ -64,11 +62,9 @@ impl Scheme for AnthropicScheme { } fn required_auth(&self) -> AuthRequirement { - // Ollama の `/v1/messages` 互換では認証が要らないが、それは - // `AuthRef::None` + `build_headers` 側の「ResolvedAuth::None - // なら何もしない」分岐で吸収する（`accepts` 判定で弾かれない - // よう、現状は XApiKey を要求しつつ、None 側でもパスするよう - // にする戦略）。 + // Anthropic uses XApiKey. Compatible endpoints such as Ollama can opt out + // through ResolvedAuth::None, which the transport accepts for any auth + // requirement and handles without sending authentication headers. AuthRequirement::XApiKey } diff --git a/crates/agen/src/llm_client/scheme/gemini/capability.rs b/crates/agen/src/llm_client/scheme/gemini/capability.rs index 4f0c3a91e..19e6bfee4 100644 --- a/crates/agen/src/llm_client/scheme/gemini/capability.rs +++ b/crates/agen/src/llm_client/scheme/gemini/capability.rs @@ -1,14 +1,14 @@ -//! Gemini scheme の wire-level 既定 capability。 +//! Wire-level default capability for the Gemini scheme. //! -//! モデル ID 固有のテーブル(`gemini-*` バージョン別の reasoning 有無)は -//! client construction layer の責務。ここでは wire の -//! 保守的 default のみ。 +//! Model ID tables (reasoning support by `gemini-*` version) belong to the +//! client construction layer. This module provides only conservative wire +//! defaults. use crate::llm_client::capability::{ CacheStrategy, ModelCapability, StructuredOutput, ToolCallingSupport, }; -/// Scheme 既定の capability(未知モデル / 未明示モデル用)。 +/// Scheme default capability (for unknown or unspecified models). pub(crate) fn default_capability() -> ModelCapability { ModelCapability { tool_calling: ToolCallingSupport::Parallel, diff --git a/crates/agen/src/llm_client/scheme/gemini/events.rs b/crates/agen/src/llm_client/scheme/gemini/events.rs index 06af36a45..c6c4ea0ea 100644 --- a/crates/agen/src/llm_client/scheme/gemini/events.rs +++ b/crates/agen/src/llm_client/scheme/gemini/events.rs @@ -1,6 +1,6 @@ -//! Gemini SSEイベントパース +//! Gemini SSE event parsing. //! -//! Google Gemini APIのSSEイベントをパースし、統一Event型に変換 +//! Parse Google Gemini API SSE events and convert them to the unified Event type. use crate::llm_client::{ ClientError, @@ -11,117 +11,117 @@ use serde::Deserialize; use super::GeminiScheme; // ============================================================================ -// SSEイベントのJSON構造 +// SSE event JSON structures // ============================================================================ -/// Gemini GenerateContentResponse (ストリーミングチャンク) +/// Gemini GenerateContentResponse (streaming chunk). #[allow(dead_code)] #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] pub(crate) struct GenerateContentResponse { - /// 候補 + /// Candidates. pub candidates: Option>, - /// 使用量メタデータ + /// Usage metadata. pub usage_metadata: Option, - /// プロンプトフィードバック + /// Prompt feedback. pub prompt_feedback: Option, - /// モデルバージョン + /// Model version. pub model_version: Option, } -/// 候補 +/// Candidate. #[allow(dead_code)] #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] pub(crate) struct Candidate { - /// コンテンツ + /// Content. pub content: Option, - /// 完了理由 + /// Finish reason. pub finish_reason: Option, - /// インデックス + /// Index. pub index: Option, - /// 安全性評価 + /// Safety rating. pub safety_ratings: Option>, } -/// 候補コンテンツ +/// Candidate content. #[allow(dead_code)] #[derive(Debug, Deserialize)] pub(crate) struct CandidateContent { - /// パーツ + /// Parts. pub parts: Option>, - /// ロール + /// Role. pub role: Option, } -/// 候補パーツ +/// Candidate part. #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] pub(crate) struct CandidatePart { - /// テキスト + /// Text. pub text: Option, - /// 関数呼び出し + /// Function call. pub function_call: Option, } -/// 関数呼び出し +/// Function call. #[derive(Debug, Deserialize)] pub(crate) struct FunctionCall { - /// 関数名 + /// Function name. pub name: String, - /// 引数 + /// Arguments. pub args: Option, } -/// 使用量メタデータ +/// Usage metadata. #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] pub(crate) struct UsageMetadata { - /// プロンプトトークン数 + /// Prompt token count. pub prompt_token_count: Option, - /// 候補トークン数 + /// Candidate token count. pub candidates_token_count: Option, - /// 合計トークン数 + /// Total token count. pub total_token_count: Option, } -/// プロンプトフィードバック +/// Prompt feedback. #[allow(dead_code)] #[derive(Debug, Deserialize)] #[serde(rename_all = "camelCase")] pub(crate) struct PromptFeedback { - /// ブロック理由 + /// Block reason. pub block_reason: Option, - /// 安全性評価 + /// Safety rating. pub safety_ratings: Option>, } -/// 安全性評価 +/// Safety rating. #[allow(dead_code)] #[derive(Debug, Deserialize)] pub(crate) struct SafetyRating { - /// カテゴリ + /// Category. pub category: Option, - /// 確率 + /// Probability. pub probability: Option, } // ============================================================================ -// イベント変換 +// Event conversion // ============================================================================ impl GeminiScheme { - /// SSEデータをEvent型に変換 + /// Convert SSE data to Event. /// /// # Arguments - /// * `data` - SSEイベントデータJSON文字列 + /// * `data` - SSE event data as a JSON string. /// /// # Returns - /// * `Ok(Some(Vec))` - 変換成功 - /// * `Ok(None)` - イベントを無視 - /// * `Err(ClientError)` - パースエラー + /// * `Ok(Some(Vec))` - Conversion succeeded. + /// * `Ok(None)` - Event ignored. + /// * `Err(ClientError)` - Parse error. pub(crate) fn parse_event(&self, data: &str) -> Result>, ClientError> { - // データが空または無効な場合はスキップ + // Skip empty or invalid data. if data.is_empty() || data == "[DONE]" { return Ok(None); } @@ -136,12 +136,12 @@ impl GeminiScheme { let mut events = Vec::new(); - // 使用量メタデータ + // Usage metadata. if let Some(usage) = response.usage_metadata { events.push(self.convert_usage(&usage)); } - // 候補を処理 + // Process candidates. if let Some(candidates) = response.candidates { for candidate in candidates { let candidate_index = candidate.index.unwrap_or(0); @@ -149,26 +149,26 @@ impl GeminiScheme { if let Some(content) = candidate.content { if let Some(parts) = content.parts { for (part_index, part) in parts.iter().enumerate() { - // テキストデルタ + // Text delta. if let Some(text) = &part.text { if !text.is_empty() { - // Geminiは明示的なBlockStartを送らないため、 - // TextDeltaを直接送る（Timelineが暗黙的に開始を処理） + // Gemini sends no explicit BlockStart, so emit TextDelta + // directly (Timeline handles the implicit start). events.push(Event::text_delta(part_index, text.clone())); } } - // 関数呼び出し + // Function call. if let Some(function_call) = &part.function_call { - // 関数呼び出しの開始 - // Geminiでは関数呼び出しは一度に送られることが多い - // ストリーミング引数が有効な場合は部分的に送られる可能性がある + // Function call start. + // Gemini usually sends function calls all at once, but may + // send partial calls when streaming arguments are enabled. - // 関数呼び出しIDはGeminiにはないので、名前をIDとして使用 + // Gemini has no function call IDs, so use the name as the ID. let function_id = format!("call_{}", function_call.name); events.push(Event::BlockStart(BlockStart { - index: candidate_index * 10 + part_index, // 複合インデックス + index: candidate_index * 10 + part_index, // Composite index. block_type: BlockType::ToolUse, metadata: BlockMetadata::ToolUse { id: function_id, @@ -176,7 +176,7 @@ impl GeminiScheme { }, })); - // 引数がある場合はデルタとして送る + // Send arguments as a delta if present. if let Some(args) = &function_call.args { let args_str = serde_json::to_string(args).unwrap_or_default(); if !args_str.is_empty() && args_str != "null" { @@ -191,7 +191,7 @@ impl GeminiScheme { } } - // 完了理由 + // Finish reason. if let Some(finish_reason) = candidate.finish_reason { let stop_reason = match finish_reason.as_str() { "STOP" => Some(StopReason::EndTurn), @@ -200,7 +200,7 @@ impl GeminiScheme { _ => None, }; - // テキストブロックの停止 + // Text block stop. events.push(Event::BlockStop(BlockStop { index: candidate_index, block_type: BlockType::Text, @@ -262,7 +262,7 @@ mod tests { let events = scheme.parse_event(data).unwrap().unwrap(); - // Usageイベントが含まれるはず + // Should contain a Usage event. let usage_event = events.iter().find(|e| matches!(e, Event::Usage(_))); assert!(usage_event.is_some()); @@ -280,7 +280,7 @@ mod tests { let events = scheme.parse_event(data).unwrap().unwrap(); - // BlockStartイベントがあるはず + // Should contain a BlockStart event. let start_event = events.iter().find(|e| matches!(e, Event::BlockStart(_))); assert!(start_event.is_some()); @@ -293,7 +293,7 @@ mod tests { } } - // 引数デルタもあるはず + // Should also contain an argument delta. let delta_event = events.iter().find(|e| { if let Event::BlockDelta(d) = e { matches!(d.delta, DeltaContent::InputJson(_)) @@ -311,7 +311,7 @@ mod tests { let events = scheme.parse_event(data).unwrap().unwrap(); - // BlockStopイベントがあるはず + // Should contain a BlockStop event. let stop_event = events.iter().find(|e| matches!(e, Event::BlockStop(_))); assert!(stop_event.is_some()); diff --git a/crates/agen/src/llm_client/scheme/gemini/mod.rs b/crates/agen/src/llm_client/scheme/gemini/mod.rs index de453156c..52a726f36 100644 --- a/crates/agen/src/llm_client/scheme/gemini/mod.rs +++ b/crates/agen/src/llm_client/scheme/gemini/mod.rs @@ -1,29 +1,29 @@ -//! Google Gemini API スキーマ +//! Google Gemini API scheme. //! -//! - リクエストJSON生成 -//! - SSEイベントパース → Event変換 +//! - Request JSON generation. +//! - SSE event parsing → Event conversion. mod capability; mod events; mod request; mod scheme_impl; -/// Geminiスキーマ +/// Gemini scheme. /// -/// Google Gemini APIのリクエスト/レスポンス変換を担当 +/// Handles request/response conversion for the Google Gemini API. #[derive(Debug, Clone, Default)] pub struct GeminiScheme { - /// ストリーミング関数呼び出し引数を有効にするか + /// Whether to enable streaming function call arguments. pub stream_function_call_arguments: bool, } impl GeminiScheme { - /// 新しいスキーマを作成 + /// Create a new scheme. pub fn new() -> Self { Self::default() } - /// ストリーミング関数呼び出し引数を有効/無効にする + /// Enable or disable streaming function call arguments. pub fn with_stream_function_call_arguments(mut self, enabled: bool) -> Self { self.stream_function_call_arguments = enabled; self diff --git a/crates/agen/src/llm_client/scheme/gemini/request.rs b/crates/agen/src/llm_client/scheme/gemini/request.rs index c1eb6d82c..e1e6b4040 100644 --- a/crates/agen/src/llm_client/scheme/gemini/request.rs +++ b/crates/agen/src/llm_client/scheme/gemini/request.rs @@ -140,12 +140,12 @@ pub(crate) struct GeminiGenerationConfig { /// Stop sequences #[serde(skip_serializing_if = "Vec::is_empty")] pub stop_sequences: Vec, - /// Thinking / reasoning 設定（Gemini 2.5 以降）。 + /// Thinking / reasoning settings (Gemini 2.5 and later). #[serde(skip_serializing_if = "Option::is_none")] pub thinking_config: Option, } -/// Gemini thinking config (gemini-2.5 以降) +/// Gemini thinking config (gemini-2.5 and later). #[derive(Debug, Serialize)] #[serde(rename_all = "camelCase")] pub(crate) struct GeminiThinkingConfig { @@ -193,8 +193,8 @@ impl GeminiScheme { None }; - // Reasoning の投影: capability が BudgetTokens / Both をサポートし、 - // request 側で budget_tokens が指定されているときだけ thinking_config を付ける。 + // Project reasoning: add thinking_config only when the capability + // supports BudgetTokens / Both and the request specifies budget_tokens. let supports_budget = matches!( capability.reasoning, Some(ReasoningSupport::BudgetTokens | ReasoningSupport::Both), diff --git a/crates/agen/src/llm_client/scheme/mod.rs b/crates/agen/src/llm_client/scheme/mod.rs index 0ea78b819..df2f06825 100644 --- a/crates/agen/src/llm_client/scheme/mod.rs +++ b/crates/agen/src/llm_client/scheme/mod.rs @@ -1,11 +1,12 @@ -//! APIスキーマ定義 +//! API scheme definitions. //! -//! 各APIスキーマごとの変換ロジック -//! - リクエスト変換: Request → プロバイダ固有JSON -//! - レスポンス変換: SSEイベント → Event +//! Conversion logic for each API scheme: +//! - Request conversion: Request → provider-specific JSON. +//! - Response conversion: SSE events → Event. //! -//! [`Scheme`] trait により `HttpTransport` から scheme 固有の差分 -//! （パス、ヘッダ、認証要件、body 生成、SSE パース）をすべて委譲する。 +//! The [`Scheme`] trait delegates all scheme-specific differences from +//! `HttpTransport` (paths, headers, authentication requirements, body +//! generation, and SSE parsing). pub mod anthropic; pub mod gemini; @@ -21,42 +22,40 @@ use super::error::ClientError; use super::event::Event; use super::types::{Request, RequestConfig}; -/// wire scheme の抽象。各プロバイダの API 仕様ごとに 1 つ実装する。 +/// Wire scheme abstraction, implemented once per provider API specification. /// -/// `HttpTransport` が URL 組立・認証ヘッダ挿入・SSE パース -/// のループを担い、`Scheme` 実装は各仕様固有の差分のみ提供する。 +/// `HttpTransport` handles URL assembly, authentication header +/// insertion, and the SSE parsing loop; `Scheme` implementations supply only +/// specification-specific differences. /// -/// # 状態 +/// # State /// -/// SSE パースでフレーム間に状態を保つ必要がある scheme（Anthropic の -/// `BlockStop` に `block_type` が載らない仕様の補完など）は -/// [`Scheme::State`] に中間状態を表す型を置く。 -/// 状態を持たない scheme は `type State = ()` とする。 +/// Schemes requiring state across SSE frames (e.g. to compensate for Anthropic +/// omitting `block_type` from `BlockStop`) define an intermediate state type in +/// [`Scheme::State`]. Stateless schemes use `type State = ()`. pub trait Scheme: Clone + Send + Sync + 'static { - /// SSE パースのフレーム間で共有する状態。`HttpTransport` が - /// ストリーム開始時に `Default::default()` を一度だけ作り、 - /// フレームごとに `&mut` で渡す。 + /// State shared across SSE frames. `HttpTransport` creates it once via + /// `Default::default()` at stream start and passes it by `&mut` for each frame. type State: Default + Send + 'static; - /// scheme のベース URL（`ModelConfig::base_url` 未指定時のデフォルト） + /// Default base URL for callers that do not supply an override. fn default_base_url(&self) -> &'static str; - /// リクエスト先の相対パス。Gemini のようにモデル名をパスに埋め込む - /// プロバイダもあるため、モデル ID を受け取る。 + /// Relative request path. Accepts a model ID because some providers, such as + /// Gemini, embed the model name in the path. fn path(&self, model_id: &str) -> String; - /// この scheme が要求する認証形式。呼び出し側は client 構築時に - /// 設定された認証情報と照合する。 + /// Authentication format required by this scheme. The caller checks it + /// against configured credentials when constructing the client. fn required_auth(&self) -> AuthRequirement; - /// `Content-Type` 以外の追加ヘッダ。`anthropic-version` / `anthropic-beta` 等。 + /// Additional headers beyond `Content-Type`, e.g. `anthropic-version` / `anthropic-beta`. fn additional_headers(&self) -> Vec<(&'static str, String)> { Vec::new() } - /// リクエスト body を生成する。`capability` は `CacheStrategy` や - /// `ReasoningSupport` を参照して scheme 側の挙動を分岐させるため - /// に渡される。 + /// Build the request body. `capability` lets the scheme select behavior + /// based on `CacheStrategy` and `ReasoningSupport`. fn build_request_body( &self, model_id: &str, @@ -64,11 +63,11 @@ pub trait Scheme: Clone + Send + Sync + 'static { capability: &ModelCapability, ) -> Value; - /// SSE イベント 1 件を 0 個以上の [`Event`] に変換する。 + /// Convert one SSE event to zero or more [`Event`]s. /// - /// `event_type` は SSE フレームの `event:` フィールド、`data` は - /// `data:` フィールド。`[DONE]` 等の終端マーカーは実装側で判定する。 - /// `state` はストリーム単位で共有される可変状態。 + /// `event_type` is the SSE frame's `event:` field; `data` is its `data:` field. + /// Implementations detect terminal markers such as `[DONE]`. + /// `state` is mutable state shared for the stream. fn parse_sse( &self, event_type: &str, @@ -76,9 +75,9 @@ pub trait Scheme: Clone + Send + Sync + 'static { state: &mut Self::State, ) -> Result, ClientError>; - /// scheme 既定の capability。モデル ID に関係なく、この wire で - /// 安全に送れる最小共通項を返す。既知モデル ID の能力テーブルは - /// 高レベルの client 構築層の責務で、scheme はここには関与しない。 + /// Scheme default capability: the minimum common features safe to send over + /// this wire, regardless of model ID. Known-model capability tables belong + /// to the higher-level client construction layer, not the scheme. fn default_capability(&self) -> ModelCapability; /// Strongest safe completion boundary for streamed tool calls. @@ -86,9 +85,8 @@ pub trait Scheme: Clone + Send + Sync + 'static { ToolCallCompletionSupport::ResponseComplete } - /// scheme 側でサポートしていない `RequestConfig` フィールドを - /// 警告として返す（例: OpenAI Chat は `top_k` 非対応）。 - /// デフォルトは空 Vec。 + /// Return warnings for `RequestConfig` fields unsupported by the scheme + /// (e.g. OpenAI Chat does not support `top_k`). Defaults to an empty Vec. fn validate_config(&self, config: &RequestConfig) -> Vec { let _ = config; Vec::new() diff --git a/crates/agen/src/llm_client/scheme/openai_chat/capability.rs b/crates/agen/src/llm_client/scheme/openai_chat/capability.rs index 69171e3e2..6a7de3509 100644 --- a/crates/agen/src/llm_client/scheme/openai_chat/capability.rs +++ b/crates/agen/src/llm_client/scheme/openai_chat/capability.rs @@ -1,14 +1,14 @@ -//! OpenAI Chat Completions scheme の wire-level 既定 capability。 +//! Wire-level default capability for the OpenAI Chat Completions scheme. //! -//! モデル ID 固有のテーブル(`gpt-5` 系など)は client construction layer -//! の責務。ここでは wire の保守的 default のみ。 +//! Model ID tables (e.g. the `gpt-5` family) belong to the client construction +//! layer. This module provides only conservative wire defaults. use crate::llm_client::capability::{ CacheStrategy, ModelCapability, StructuredOutput, ToolCallingSupport, }; -/// Scheme 既定の capability。OpenAI 互換ルーター系(xAI / Groq / OpenRouter 等) -/// で未知モデル ID を受けたときのフォールバックに使う。 +/// Scheme default capability, used as a fallback for unknown model IDs on +/// OpenAI-compatible routers (xAI / Groq / OpenRouter, etc.). pub(crate) fn default_capability() -> ModelCapability { ModelCapability { tool_calling: ToolCallingSupport::Parallel, diff --git a/crates/agen/src/llm_client/scheme/openai_chat/events.rs b/crates/agen/src/llm_client/scheme/openai_chat/events.rs index 4463dcff4..56af73635 100644 --- a/crates/agen/src/llm_client/scheme/openai_chat/events.rs +++ b/crates/agen/src/llm_client/scheme/openai_chat/events.rs @@ -1,4 +1,4 @@ -//! OpenAI SSEイベントパース +//! OpenAI SSE event parsing. use crate::llm_client::{ ClientError, @@ -61,10 +61,10 @@ pub(crate) struct ChunkUsage { } impl OpenAIScheme { - /// SSEデータのパースとEventへの変換 + /// Parse SSE data and convert it to Event. /// - /// OpenAI APIはBlockStartイベントを明示的に送信しない。 - /// Timeline層が暗黙的なBlockStartを処理する。 + /// The OpenAI API does not explicitly send BlockStart events. + /// The Timeline layer handles implicit BlockStart. pub fn parse_event(&self, data: &str) -> Result>, ClientError> { if data == "[DONE]" { return Ok(None); @@ -94,8 +94,8 @@ impl OpenAIScheme { for choice in chunk.choices { // Text Content Delta if let Some(content) = choice.delta.content { - // OpenAI APIはBlockStartを送らないため、デルタのみを発行 - // Timeline層が暗黙的なBlockStartを処理する + // Emit only deltas because the OpenAI API sends no BlockStart. + // The Timeline layer handles implicit BlockStart. events.push(Event::text_delta(choice.index, content)); } @@ -136,11 +136,11 @@ impl OpenAIScheme { finish_reason == "tool_calls" || finish_reason == "function_call"; if is_tool_finish { - // ツール呼び出し終了 - // Note: OpenAIはどのツールが終了したか明示しないため、 - // Timeline層で適切に処理する必要がある + // Tool call ended. + // Note: OpenAI does not specify which tool ended, so the + // Timeline layer must handle this appropriately. } else { - // テキスト終了 + // Text ended. events.push(Event::text_block_stop(choice.index, stop_reason)); } } @@ -165,7 +165,7 @@ mod tests { let data = r#"{"id":"chatcmpl-123","object":"chat.completion.chunk","created":1694268190,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}"#; let events = scheme.parse_event(data).unwrap().unwrap(); - // OpenAIはBlockStartを発行しないため、デルタのみ + // Deltas only, since OpenAI does not emit BlockStart. assert_eq!(events.len(), 1); if let Event::BlockDelta(delta) = &events[0] { diff --git a/crates/agen/src/llm_client/scheme/openai_chat/mod.rs b/crates/agen/src/llm_client/scheme/openai_chat/mod.rs index f0e1812d2..1de904ec0 100644 --- a/crates/agen/src/llm_client/scheme/openai_chat/mod.rs +++ b/crates/agen/src/llm_client/scheme/openai_chat/mod.rs @@ -1,31 +1,32 @@ -//! OpenAI Chat Completions API スキーマ +//! OpenAI Chat Completions API scheme. //! -//! - リクエストJSON生成 -//! - SSEイベントパース → Event変換 +//! - Request JSON generation. +//! - SSE event parsing → Event conversion. pub(crate) mod capability; mod events; mod request; mod scheme_impl; -/// OpenAIスキーマ +/// OpenAI scheme. /// -/// OpenAI Chat Completions API (および互換API) のリクエスト/レスポンス変換を担当 +/// Handles request/response conversion for the OpenAI Chat Completions API +/// (and compatible APIs). #[derive(Debug, Clone, Default)] pub struct OpenAIScheme { - /// モデル名 (リクエスト時に指定されるが、デフォルト値として保持も可能) + /// Model name (supplied per request, but may also be stored as a default). pub model: Option, - /// レガシーなmax_tokensを使用するか (Ollama互換用) + /// Whether to use legacy max_tokens (for Ollama compatibility). pub use_legacy_max_tokens: bool, } impl OpenAIScheme { - /// 新しいスキーマを作成 + /// Create a new scheme. pub fn new() -> Self { Self::default() } - /// レガシーなmax_tokensを使用するか設定 + /// Set whether to use legacy max_tokens. pub fn with_legacy_max_tokens(mut self, use_legacy: bool) -> Self { self.use_legacy_max_tokens = use_legacy; self diff --git a/crates/agen/src/llm_client/scheme/openai_chat/request.rs b/crates/agen/src/llm_client/scheme/openai_chat/request.rs index e8a87496e..bc493ec4b 100644 --- a/crates/agen/src/llm_client/scheme/openai_chat/request.rs +++ b/crates/agen/src/llm_client/scheme/openai_chat/request.rs @@ -38,7 +38,7 @@ pub(crate) struct OpenAIRequest { pub tools: Vec, #[serde(skip_serializing_if = "Option::is_none")] pub tool_choice: Option, - /// Reasoning effort（o1 / o3 / o4 / gpt-5 系で有効）。 + /// Reasoning effort (supported by the o1 / o3 / o4 / gpt-5 families). #[serde(skip_serializing_if = "Option::is_none")] pub reasoning_effort: Option, } @@ -147,8 +147,8 @@ impl OpenAIScheme { (None, request.config.max_tokens) }; - // Reasoning の投影: capability が Effort / Both をサポートし、 - // request 側で effort が指定されているときだけ reasoning_effort を付ける。 + // Project reasoning: add reasoning_effort only when the capability + // supports Effort / Both and the request specifies an effort. let supports_effort = matches!( capability.reasoning, Some(ReasoningSupport::Effort | ReasoningSupport::Both), diff --git a/crates/agen/src/llm_client/scheme/openai_chat/scheme_impl.rs b/crates/agen/src/llm_client/scheme/openai_chat/scheme_impl.rs index 87c1ef787..2d9edd20e 100644 --- a/crates/agen/src/llm_client/scheme/openai_chat/scheme_impl.rs +++ b/crates/agen/src/llm_client/scheme/openai_chat/scheme_impl.rs @@ -45,7 +45,7 @@ impl Scheme for OpenAIScheme { data: &str, _state: &mut Self::State, ) -> Result, ClientError> { - // `data: [DONE]` は終端マーカー + // `data: [DONE]` is a terminal marker. if data.trim() == "[DONE]" { return Ok(Vec::new()); } @@ -58,7 +58,7 @@ impl Scheme for OpenAIScheme { fn validate_config(&self, config: &RequestConfig) -> Vec { let mut warnings = Vec::new(); - // OpenAI Chat Completions API は top_k を受け付けない + // The OpenAI Chat Completions API does not accept top_k. if config.top_k.is_some() { warnings.push(ConfigWarning::unsupported("top_k", "OpenAI Chat")); } diff --git a/crates/agen/src/llm_client/scheme/openai_responses/capability.rs b/crates/agen/src/llm_client/scheme/openai_responses/capability.rs index ce1df4628..3cf933be0 100644 --- a/crates/agen/src/llm_client/scheme/openai_responses/capability.rs +++ b/crates/agen/src/llm_client/scheme/openai_responses/capability.rs @@ -1,7 +1,7 @@ -//! OpenAI Responses scheme の wire-level 既定 capability。 +//! Wire-level default capability for the OpenAI Responses scheme. //! -//! モデル ID 固有の能力テーブルは高レベルの client 構築層の責務。 -//! ここでは wire の保守的 default のみ。 +//! Model ID capability tables belong to the higher-level client construction +//! layer. This module provides only conservative wire defaults. use crate::llm_client::capability::{ CacheStrategy, ModelCapability, StructuredOutput, ToolCallingSupport, diff --git a/crates/agen/src/llm_client/scheme/openai_responses/events.rs b/crates/agen/src/llm_client/scheme/openai_responses/events.rs index 8b9fbb3ed..27f404b77 100644 --- a/crates/agen/src/llm_client/scheme/openai_responses/events.rs +++ b/crates/agen/src/llm_client/scheme/openai_responses/events.rs @@ -1,9 +1,9 @@ -//! OpenAI Responses API の SSE イベントパース +//! OpenAI Responses API SSE event parsing. //! -//! `response.*` 名前空間の SSE を共通の [`Event`](crate::llm_client::event::Event) -//! に変換する。Responses の (output_index, content_index) 2 次元座標と -//! この crate の 1 次元 `BlockStart/Delta/Stop::index` のマッピングは -//! [`OpenAIResponsesState`] が保持する。 +//! Converts SSE events in the `response.*` namespace to shared [`Event`]s. +//! [`OpenAIResponsesState`] holds the mapping from Responses' two-dimensional +//! (output_index, content_index) coordinates to this crate's one-dimensional +//! `BlockStart/Delta/Stop::index`. use std::collections::{BTreeMap, HashMap}; @@ -18,15 +18,16 @@ use crate::llm_client::{ }, }; -/// SSE パース中の座標 → flat block index マップ。 +/// Coordinate → flat block index map used during SSE parsing. #[derive(Debug, Default)] pub struct OpenAIResponsesState { slots: HashMap, next_index: usize, - /// 蓄積中の reasoning output_item。`output_item.added`(Reasoning) で - /// 確保し、`reasoning_text.delta` / `reasoning_summary_text.delta` で - /// 蓄積、`output_item.done`(Reasoning) で既存 reasoning_text block または - /// metadata-only Thinking block に reasoning persistence material を載せる。 + /// Reasoning output_items being accumulated. Allocated at + /// `output_item.added` (Reasoning), filled by `reasoning_text.delta` / + /// `reasoning_summary_text.delta`, then attached as reasoning persistence + /// material to an existing reasoning_text block or metadata-only Thinking + /// block at `output_item.done` (Reasoning). pending_reasoning: HashMap, /// Tool input deltas and authoritative final values, keyed by output index. pending_tool_inputs: HashMap, @@ -67,19 +68,19 @@ impl ToolInputKind { } } -/// 1 つの reasoning output_item の蓄積バッファ。 +/// Accumulation buffer for one reasoning output_item. #[derive(Debug, Default)] struct PendingReasoning { id: Option, - /// `reasoning_text.delta` の累積。複数 content_part あれば順に concat。 + /// Accumulated `reasoning_text.delta`, concatenating multiple content_parts in order. text: String, - /// `reasoning_summary_text.delta` を summary_index 順に蓄積。 + /// Accumulates `reasoning_summary_text.delta` in summary_index order. summary: Vec, - /// `response.content_part.done` が先に到着した reasoning/thinking block。 + /// Reasoning/thinking blocks whose `response.content_part.done` arrived first. /// - /// `response.output_item.done` まで待たないと encrypted_content や最終 - /// summary が揃わないため、live-visible な余分な synthetic Thinking block - /// を作らず、既存 block の stop に persistence metadata を載せる。 + /// encrypted_content and final summaries are not complete until + /// `response.output_item.done`. Attach persistence metadata to the existing + /// block's stop rather than creating an extra live-visible synthetic Thinking block. deferred_thinking_stops: Vec, } @@ -106,9 +107,9 @@ impl OpenAIResponsesState { info } - /// 既存 slot を取得。無ければ `block_type` で暗黙に確保し、 - /// 新規確保したかを併せて返す。delta 先行 / content_part.added が - /// 抜けたときの防御。 + /// Get an existing slot or implicitly allocate one with `block_type`, also + /// returning whether it was newly allocated. Defends against deltas arriving + /// first or missing content_part.added events. fn get_or_allocate(&mut self, key: SlotKey, block_type: BlockType) -> (SlotInfo, bool) { if let Some(info) = self.slots.get(&key).copied() { (info, false) @@ -157,11 +158,11 @@ impl OpenAIResponsesState { #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] enum SlotKey { - /// tool_use (function_call / custom_tool_call) — output_item 全体で 1 block + /// tool_use (function_call / custom_tool_call) — one block for the entire output_item. OutputItem(usize), - /// message の output_text / reasoning item の reasoning_text + /// Message output_text / reasoning item reasoning_text. ContentPart { output: usize, content: usize }, - /// reasoning item の summary_text (summary_index) + /// Reasoning item summary_text (summary_index). Summary { output: usize, summary: usize }, } @@ -172,7 +173,7 @@ struct SlotInfo { } // ============================================================================ -// SSE イベントの JSON 構造 +// SSE event JSON structures // ============================================================================ #[derive(Debug, Deserialize)] @@ -188,7 +189,7 @@ struct OutputItemDone { item: OutputItem, } -/// `response.output_item.added/done` の `item`。 +/// `item` in `response.output_item.added/done`. #[derive(Debug, Deserialize)] #[serde(tag = "type", rename_all = "snake_case")] enum OutputItem { @@ -199,8 +200,8 @@ enum OutputItem { Reasoning { #[serde(default)] id: Option, - /// `output_item.done` で初めて埋まる。`include=["reasoning.encrypted_content"]` - /// 指定時に opaque blob が乗る。 + /// First populated at `output_item.done`. Carries an opaque blob when + /// `include=["reasoning.encrypted_content"]` is specified. #[serde(default)] encrypted_content: Option, }, @@ -241,7 +242,7 @@ struct ContentPartDone { part: ContentPart, } -/// `response.content_part.added/done` の `part`。 +/// `part` in `response.content_part.added/done`. #[derive(Debug, Deserialize)] #[serde(tag = "type", rename_all = "snake_case")] enum ContentPart { @@ -338,7 +339,7 @@ struct ResponsesUsage { output_tokens: Option, #[serde(default)] total_tokens: Option, - /// `input_tokens` の内訳。`cached_tokens` がプロンプトキャッシュヒット分。 + /// Breakdown of `input_tokens`; `cached_tokens` counts prompt cache hits. #[serde(default)] input_tokens_details: Option, } @@ -399,11 +400,11 @@ struct TopLevelError { // parse entry point // ============================================================================ -/// SSE フレーム 1 件をパースし、0 個以上の [`Event`] に変換する。 +/// Parse one SSE frame and convert it to zero or more [`Event`]s. /// -/// `event_type` は SSE の `event:` フィールド。未対応の event type は -/// [`Event::UnhandledSse`] として観測可能にする。`data` が JSON でない / -/// 必要なフィールドが抜けている等は [`ClientError::Api`] で返す。 +/// `event_type` is the SSE `event:` field. Unsupported event types are observable +/// as [`Event::UnhandledSse`]. Non-JSON `data`, missing required fields, and similar +/// failures return [`ClientError::Api`]. pub(crate) fn parse_sse( event_type: &str, data: &str, @@ -427,7 +428,7 @@ pub(crate) fn parse_sse( cache_read_input_tokens: usage .input_tokens_details .and_then(|d| d.cached_tokens), - // Responses API は cache 書き込みを別計上しない（input_tokens に含まれる） + // The Responses API does not account for cache writes separately (included in input_tokens). cache_creation_input_tokens: None, })); } @@ -466,8 +467,8 @@ pub(crate) fn parse_sse( name, ), OutputItem::Reasoning { id, .. } => { - // wrapper を確保。中身の content_part / summary_part は - // 別 SlotKey で扱われ続ける（Streaming 表示は維持）。 + // Allocate the wrapper. Its content_parts / summary_parts continue + // to use separate SlotKeys (preserving the streaming display). let entry = state.ensure_reasoning(ev.output_index); if id.is_some() { entry.id = id; @@ -480,10 +481,10 @@ pub(crate) fn parse_sse( "response.output_item.done" => { let ev: OutputItemDone = from_json(data)?; - // Reasoning wrapper の done で蓄積分を既存 reasoning_text block の - // stop に載せる。content_part.done が先に来た場合は stop を defer - // しておき、ここで encrypted_content / summary と一緒に完了させる。 - // reasoning_text が無い metadata-only item だけ synthetic block を作る。 + // On Reasoning wrapper done, attach accumulated material to the existing + // reasoning_text block's stop. If content_part.done arrived first, defer + // its stop and complete it here with encrypted_content / summary. + // Create synthetic blocks only for metadata-only items without reasoning_text. if let OutputItem::Reasoning { id, encrypted_content, @@ -638,7 +639,7 @@ pub(crate) fn parse_sse( "response.reasoning_text.delta" => { let ev: ReasoningTextDelta = from_json(data)?; - // round-trip 用に蓄積 + // Accumulate for round-tripping. state .ensure_reasoning(ev.output_index) .text @@ -673,7 +674,7 @@ pub(crate) fn parse_sse( "response.reasoning_summary_text.delta" => { let ev: ReasoningSummaryTextDelta = from_json(data)?; - // round-trip 用に蓄積 + // Accumulate for round-tripping. state.extend_reasoning_summary(ev.output_index, ev.summary_index, &ev.delta); Ok(ensure_and_delta( state, @@ -782,7 +783,7 @@ pub(crate) fn parse_sse( Ok(vec![Event::Error(ErrorEvent { code, message })]) } - // 未対応 / 情報系 event type は生成 semantics からは無視しつつ trace に残す。 + // Ignore unsupported / informational event types for generation semantics, but retain them in traces. _ => Ok(vec![unhandled_sse_event(event_type, data)]), } } @@ -1103,8 +1104,8 @@ fn append_diagnostic(message: String, diagnostic: Map) -> String format!("{} | diagnostic={}", message, Value::Object(diagnostic)) } -/// 対応する BlockStart がまだ発行されていなければ発行しつつ、delta を流す。 -/// content_part.added を取りこぼしても delta 単独で復旧できるようにする。 +/// Emit a delta, first emitting its BlockStart if not already sent. +/// Allows recovery from a delta alone when content_part.added was missed. fn ensure_and_delta( state: &mut OpenAIResponsesState, key: SlotKey, @@ -1219,14 +1220,14 @@ mod tests { assert_eq!(u.output_tokens, Some(50)); assert_eq!(u.total_tokens, Some(12395)); assert_eq!(u.cache_read_input_tokens, Some(11000)); - // OpenAI Responses は cache 書き込みを別計上しない + // OpenAI Responses does not account for cache writes separately. assert_eq!(u.cache_creation_input_tokens, None); } #[test] fn text_stream_start_delta_stop() { let mut state = OpenAIResponsesState::default(); - // output_item.added (message) → 無視 + // output_item.added (message) → ignored. with( &mut state, "response.output_item.added", @@ -1537,7 +1538,7 @@ mod tests { #[test] fn delta_without_prior_start_recovers() { - // 防御: content_part.added が落ちても delta 単独で BlockStart+Delta を発行 + // Defensive: emit BlockStart+Delta from a delta alone if content_part.added was lost. let mut state = OpenAIResponsesState::default(); let ev = with( &mut state, @@ -1551,8 +1552,8 @@ mod tests { #[test] fn parallel_output_items_get_distinct_indices() { - // 2 つの function_call が並列で output_item.added される場合、 - // flat index が別々になる（Parallel tool calling の基本）。 + // Two parallel function_call output_item.added events get distinct + // flat indices (the basis of parallel tool calling). let mut state = OpenAIResponsesState::default(); let ev1 = with( &mut state, @@ -1675,17 +1676,17 @@ mod tests { #[test] fn reasoning_output_item_completes_metadata_thinking_block_with_text_summary_encrypted() { - // 完成済み reasoning wrapper が text + summary[] + encrypted_content を持って - // Thinking BlockStop metadata として届くこと。 + // A completed reasoning wrapper must deliver text + summary[] + + // encrypted_content as Thinking BlockStop metadata. let mut state = OpenAIResponsesState::default(); - // wrapper added (id だけ持つ) + // Wrapper added (with only an id). with( &mut state, "response.output_item.added", r#"{"output_index":0,"item":{"type":"reasoning","id":"r1"}}"#, ); - // 内側の reasoning_text 用 content_part + // Inner content_part for reasoning_text. let start = with( &mut state, "response.content_part.added", @@ -1711,7 +1712,7 @@ mod tests { r#"{"output_index":0,"content_index":0,"item_id":"r1","part":{"type":"reasoning_text","text":"hello world"}}"#, ); assert!(part_done.is_empty()); - // summary 1 件 + // One summary. with( &mut state, "response.reasoning_summary_part.added", @@ -1728,7 +1729,7 @@ mod tests { r#"{"output_index":0,"summary_index":0,"item_id":"r1"}"#, ); - // wrapper done (encrypted_content が乗る) + // Wrapper done (includes encrypted_content). let evs = with( &mut state, "response.output_item.done", @@ -1745,7 +1746,7 @@ mod tests { assert_eq!(reasoning.summary, vec!["sum-A".to_string()]); assert_eq!(reasoning.encrypted_content.as_deref(), Some("ENC-XYZ")); assert!(reasoning.signature.is_none()); - // pending_reasoning は drain されていること + // pending_reasoning must be drained. assert!(state.pending_reasoning.is_empty()); } @@ -1835,8 +1836,8 @@ mod tests { #[test] fn reasoning_wrapper_without_inner_content_emits_empty_text() { - // encrypted_content だけ届く（reasoning_text 無し）ケースでも - // reasoning metadata は届く。 + // Reasoning metadata must arrive even when only encrypted_content + // is received (no reasoning_text). let mut state = OpenAIResponsesState::default(); with( &mut state, diff --git a/crates/agen/src/llm_client/scheme/openai_responses/mod.rs b/crates/agen/src/llm_client/scheme/openai_responses/mod.rs index b56d4ebde..3aea510ff 100644 --- a/crates/agen/src/llm_client/scheme/openai_responses/mod.rs +++ b/crates/agen/src/llm_client/scheme/openai_responses/mod.rs @@ -1,11 +1,11 @@ -//! OpenAI Responses API スキーマ (`/v1/responses`) +//! OpenAI Responses API scheme (`/v1/responses`). //! -//! Chat Completions とは別物の item-based wire format。reasoning item と -//! function_call item が first-class で、SSE イベントも `response.*` 名前空間で -//! 流れる。 +//! An item-based wire format distinct from Chat Completions. Reasoning and +//! function_call items are first-class, and SSE events use the `response.*` +//! namespace. //! -//! - リクエスト JSON 生成: `request` -//! - SSE イベントパース → [`Event`](crate::llm_client::event::Event) 変換: `events` +//! - Request JSON generation: `request`. +//! - SSE event parsing → [`Event`](crate::llm_client::event::Event) conversion: `events`. mod capability; mod events; @@ -14,29 +14,29 @@ mod scheme_impl; pub use scheme_impl::OpenAIResponsesState; -/// OpenAI Responses scheme 本体。 +/// OpenAI Responses scheme. /// /// `store` / `include_encrypted_content` / `send_max_output_tokens` / -/// `send_sampling_params` は scheme 固定の wire 設定で、デフォルトは -/// 公式 OpenAI Responses API 向け (stateless + ZDR + `max_output_tokens` -/// / `temperature` / `top_p` 送出可)。受理パラメータが subset の -/// 互換 backend では client 構築層で `send_max_output_tokens=false` / -/// `send_sampling_params=false` に上書きする。`ModelCapability` には -/// 入れない（モデル能力ではなく wire policy）。 +/// `send_sampling_params` are scheme-level wire settings. Defaults target the +/// official OpenAI Responses API (stateless + ZDR, with `max_output_tokens` / +/// `temperature` / `top_p` allowed). For compatible backends accepting only a +/// subset of parameters, the client construction layer overrides +/// `send_max_output_tokens=false` / `send_sampling_params=false`. +/// These do not belong in `ModelCapability`: they are wire policy, not model capabilities. #[derive(Debug, Clone)] pub struct OpenAIResponsesScheme { - /// サーバ側に response を保存するか。ZDR/stateless 運用では `false`。 + /// Whether to store responses on the server. `false` for ZDR/stateless operation. pub store: bool, - /// `include: ["reasoning.encrypted_content"]` を付けるか。 - /// `store=false` で reasoning を使うなら必須。 + /// Whether to add `include: ["reasoning.encrypted_content"]`. + /// Required when using reasoning with `store=false`. pub include_encrypted_content: bool, - /// `max_output_tokens` を body に載せるか。公式 OpenAI Responses API は - /// 受理するが、互換 backend によっては `Unsupported parameter` で - /// 400 を返すため、その経路では `false` にする。 + /// Whether to include `max_output_tokens` in the body. The official OpenAI + /// Responses API accepts it, but some compatible backends return 400 with + /// `Unsupported parameter`; set this to `false` for those routes. pub send_max_output_tokens: bool, - /// `temperature` / `top_p` を body に載せるか。公式 OpenAI Responses API - /// は受理するが、互換 backend によっては `Unsupported parameter` で - /// 400 を返すため、その経路では `false` にする。 + /// Whether to include `temperature` / `top_p` in the body. The official OpenAI + /// Responses API accepts them, but some compatible backends return 400 with + /// `Unsupported parameter`; set this to `false` for those routes. pub send_sampling_params: bool, } @@ -52,31 +52,31 @@ impl Default for OpenAIResponsesScheme { } impl OpenAIResponsesScheme { - /// デフォルト設定 (`store=false`, `include=["reasoning.encrypted_content"]`, - /// `send_max_output_tokens=true`, `send_sampling_params=true`)。 + /// Default settings (`store=false`, `include=["reasoning.encrypted_content"]`, + /// `send_max_output_tokens=true`, `send_sampling_params=true`). pub fn new() -> Self { Self::default() } - /// `store` を上書き。 + /// Override `store`. pub fn with_store(mut self, store: bool) -> Self { self.store = store; self } - /// `include: ["reasoning.encrypted_content"]` の有無を上書き。 + /// Override whether to include `include: ["reasoning.encrypted_content"]`. pub fn with_include_encrypted_content(mut self, include: bool) -> Self { self.include_encrypted_content = include; self } - /// `max_output_tokens` を body に載せるかを上書き。 + /// Override whether to include `max_output_tokens` in the body. pub fn with_send_max_output_tokens(mut self, send: bool) -> Self { self.send_max_output_tokens = send; self } - /// `temperature` / `top_p` を body に載せるかを上書き。 + /// Override whether to include `temperature` / `top_p` in the body. pub fn with_send_sampling_params(mut self, send: bool) -> Self { self.send_sampling_params = send; self diff --git a/crates/agen/src/llm_client/scheme/openai_responses/request.rs b/crates/agen/src/llm_client/scheme/openai_responses/request.rs index e26b60a51..cd964538c 100644 --- a/crates/agen/src/llm_client/scheme/openai_responses/request.rs +++ b/crates/agen/src/llm_client/scheme/openai_responses/request.rs @@ -1,8 +1,8 @@ -//! OpenAI Responses API リクエスト body 生成 +//! OpenAI Responses API request body generation. //! -//! Chat Completions の `messages` と違い、Responses は `input[]` の -//! item 配列で reasoning / function_call / function_call_output が -//! first-class。`Item` を素に近い形で `input[]` に投影できる。 +//! Unlike Chat Completions `messages`, Responses uses an `input[]` item array +//! with first-class reasoning / function_call / function_call_output items. +//! `Item` can be projected into `input[]` with little transformation. use serde::{Serialize, Serializer}; use serde_json::Value; @@ -32,93 +32,92 @@ pub(crate) enum FunctionCallOutputContentItem { InputImage { image_url: String }, } -/// `/v1/responses` のリクエスト body。 +/// Request body for `/v1/responses`. #[derive(Debug, Serialize)] pub(crate) struct ResponsesRequest { pub model: String, - /// システムプロンプト相当。`input[]` とは別フィールド。 + /// System prompt equivalent, separate from `input[]`. #[serde(skip_serializing_if = "Option::is_none")] pub instructions: Option, pub input: Vec, #[serde(skip_serializing_if = "Vec::is_empty")] pub tools: Vec, - /// 常時 `"auto"` を送る。scheme 固定値。 + /// Always sends `"auto"`, a scheme-level constant. pub tool_choice: &'static str, - /// 常時 `true` を送る。scheme 固定値。 + /// Always sends `true`, a scheme-level constant. pub parallel_tool_calls: bool, #[serde(skip_serializing_if = "Option::is_none")] pub reasoning: Option, - /// ZDR / stateless 運用では `false`。 + /// `false` for ZDR / stateless operation. pub store: bool, - /// 常時 `true`。 + /// Always `true`. pub stream: bool, - /// `["reasoning.encrypted_content"]` 等。 + /// E.g. `["reasoning.encrypted_content"]`. #[serde(skip_serializing_if = "Vec::is_empty")] pub include: Vec<&'static str>, - /// 公式 OpenAI Responses API では受理されるが、互換 backend によっては - /// 400 で弾く。scheme の `send_max_output_tokens` が `false` のときは - /// `None` のまま送る (skip_serializing_if で除外)。 + /// Accepted by the official OpenAI Responses API, but some compatible backends + /// reject it with 400. When the scheme's `send_max_output_tokens` is `false`, + /// leave it as `None` (excluded by skip_serializing_if). #[serde(skip_serializing_if = "Option::is_none")] pub max_output_tokens: Option, - /// 公式 OpenAI Responses API では受理されるが、互換 backend によっては - /// `temperature` / `top_p` を 400 で弾く。scheme の - /// `send_sampling_params` が `false` のときは `None` のまま送る。 + /// Accepted by the official OpenAI Responses API, but some compatible backends + /// reject `temperature` / `top_p` with 400. When the scheme's + /// `send_sampling_params` is `false`, leave them as `None`. #[serde(skip_serializing_if = "Option::is_none")] pub temperature: Option, #[serde(skip_serializing_if = "Option::is_none")] pub top_p: Option, - /// 会話単位の安定キー。明示キーを必要とする backend では、 - /// 呼び出し側が安定した conversation identifier を渡す。 - /// `Request::cache_key` が `None` のときはキー自体を送らない。 + /// Stable conversation key. For backends requiring an explicit key, the caller + /// supplies a stable conversation identifier. The key is omitted entirely + /// when `Request::cache_key` is `None`. #[serde(skip_serializing_if = "Option::is_none")] pub prompt_cache_key: Option, } -/// reasoning 制御。 +/// Reasoning control. #[derive(Debug, Serialize)] pub(crate) struct ReasoningConfig { #[serde(skip_serializing_if = "Option::is_none")] pub effort: Option, - /// summary の出力制御。`"auto"` 固定で summary_text を受け取る。 + /// Summary output control, fixed to `"auto"` to receive summary_text. pub summary: &'static str, } -/// `input[]` の 1 要素。 +/// One element of `input[]`. /// -/// Responses API の item 型を素に近い形で投影する。未対応 type は -/// 無視（reasoning 送信時に `content: []` の場合は `None` として弾く）。 +/// Projects Responses API item types with little transformation. Unsupported types +/// are ignored (reasoning with `content: []` is rejected as `None` when sending). #[derive(Debug, Serialize)] #[serde(tag = "type", rename_all = "snake_case")] pub(crate) enum InputItem { - /// 会話メッセージ。user / assistant / developer のいずれか。 - /// `Role::System` items は `developer` として投影する。OpenAI - /// Responses 互換 backend の一部は `role: "system"` を拒否するため、 - /// system 相当の挿入には `role: "developer"` を使う。 + /// Conversation message: user / assistant / developer. + /// `Role::System` items are projected as `developer`. Some OpenAI Responses + /// compatible backends reject `role: "system"`, so system-equivalent + /// insertions use `role: "developer"`. Message { role: &'static str, content: Vec, }, - /// 過去の function tool 呼び出し（assistant 側）。 + /// Previous function tool call (assistant side). FunctionCall { call_id: String, name: String, - /// JSON 文字列（object でなくても正規化済み）。 + /// JSON string (normalized even if the original was not an object). arguments: String, }, - /// function tool の結果（user 側）。 + /// Function tool result (user side). FunctionCallOutput { call_id: String, output: FunctionCallOutputBody, }, - /// reasoning item。`encrypted_content` があれば必ず添える。 + /// Reasoning item, always including `encrypted_content` when present. Reasoning { #[serde(skip_serializing_if = "Option::is_none")] id: Option, - /// Responses API は reasoning item に `summary` フィールドを必須で - /// 要求する（中身が空でも `[]` として送る必要がある）。GPT-5 など - /// summary を返さないモデル + reasoning effort 指定なしのターンでは - /// summary text が一切付かないので、ここを skip すると 400 - /// "Missing required parameter: 'input[N].summary'" で弾かれる。 + /// The Responses API requires `summary` on reasoning items, even if sent + /// as an empty `[]`. Models such as GPT-5 may return no summary text on + /// turns without a specified reasoning effort. Omitting this field then + /// causes a 400 `"Missing required parameter: 'input[N].summary'"` rejection. summary: Vec, #[serde(skip_serializing_if = "Vec::is_empty")] content: Vec, @@ -127,14 +126,13 @@ pub(crate) enum InputItem { }, } -/// メッセージ content_part。role で input/output を使い分ける。 +/// Message content_part, using input/output according to role. #[derive(Debug, Serialize)] #[serde(tag = "type", rename_all = "snake_case")] pub(crate) enum InputContent { - /// user / developer 側のテキスト + /// User / developer text. InputText { text: String }, - /// user 側の画像 - /// assistant 側のテキスト + /// Assistant text. OutputText { text: String }, } @@ -150,8 +148,8 @@ pub(crate) enum ReasoningContentPart { ReasoningText { text: String }, } -/// Responses 用 tool 定義。Chat と違い function キーでネストせず -/// トップレベルに `name` / `parameters` が載る。 +/// Tool definition for Responses. Unlike Chat, `name` / `parameters` are +/// top-level fields, not nested under a function key. #[derive(Debug, Serialize)] pub(crate) struct ResponseTool { #[serde(rename = "type")] @@ -159,13 +157,12 @@ pub(crate) struct ResponseTool { pub name: String, #[serde(skip_serializing_if = "Option::is_none")] pub description: Option, - /// OpenAI Responses API は `type:"object"` のパラメータスキーマに - /// `properties` が存在することを要求する。schemars は引数なし struct - /// から `properties` を含まない最小スキーマを出すので、serialize - /// 時に空オブジェクトを補う。 + /// The OpenAI Responses API requires `properties` in `type:"object"` parameter + /// schemas. schemars emits minimal schemas without `properties` for structs + /// with no arguments, so serialization supplies an empty object. #[serde(serialize_with = "serialize_parameters")] pub parameters: Value, - /// Structured output モード制御。デフォルト false。 + /// Structured output mode control. Defaults to false. pub strict: bool, } @@ -182,7 +179,7 @@ fn serialize_parameters(value: &Value, s: S) -> Result Vec { let mut out = Vec::with_capacity(items.len()); for item in items { @@ -280,7 +277,7 @@ fn convert_items_to_input(items: &[Item], supports_images: bool) -> Vec { - // 非 object / 旧形式の "null" を "{}" に正規化。 + // Normalize non-objects / legacy "null" to "{}". let normalized = parse_tool_arguments(arguments).to_string(); out.push(InputItem::FunctionCall { call_id: call_id.clone(), @@ -554,11 +551,10 @@ mod tests { #[test] fn reasoning_summary_field_is_always_serialized() { - // Responses API は reasoning item に `summary` を必須で要求する。 - // summary が空でも wire 上に `summary: []` として残らないと、 - // backend によっては missing required parameter として拒否される。 - // reasoning effort 未指定のターンでは summary text が付かないことが - // あるため、空のままでも skip しないこと。 + // The Responses API requires `summary` on reasoning items. Even an empty + // summary must remain as `summary: []` on the wire, or some backends + // reject it as a missing required parameter. Turns without a specified + // reasoning effort may have no summary text; do not skip empty summaries. let scheme = OpenAIResponsesScheme::new(); let item = Item::reasoning("").with_encrypted_content("ENC"); let req = Request::new().user("hi").item(item); @@ -667,9 +663,9 @@ mod tests { #[test] fn tool_schema_without_properties_is_normalized() { - // schemars は引数なし struct から `type:"object"` だけのスキーマを - // 吐く。OpenAI Responses は `properties` 欠落を 400 で拒否するので - // 送る直前に空オブジェクトを補うのを確認。 + // schemars emits only `type:"object"` for structs with no arguments. + // OpenAI Responses rejects missing `properties` with 400, so verify that + // an empty object is supplied just before sending. let scheme = OpenAIResponsesScheme::new(); let raw_schema = serde_json::json!({ "type": "object" }); let req = Request::new().tool( @@ -707,7 +703,7 @@ mod tests { #[test] fn serialized_body_has_expected_shape() { - // wire 形式が崩れていないかのスモークテスト + // Smoke test for an intact wire format. let scheme = OpenAIResponsesScheme::new(); let req = Request::new() .system("sys") diff --git a/crates/agen/src/llm_client/scheme/openai_responses/scheme_impl.rs b/crates/agen/src/llm_client/scheme/openai_responses/scheme_impl.rs index 0c3679c22..18f1e226a 100644 --- a/crates/agen/src/llm_client/scheme/openai_responses/scheme_impl.rs +++ b/crates/agen/src/llm_client/scheme/openai_responses/scheme_impl.rs @@ -20,8 +20,8 @@ impl Scheme for OpenAIResponsesScheme { type State = OpenAIResponsesState; fn default_base_url(&self) -> &'static str { - // `/v1` は base_url 側に寄せる。互換 backend を使う場合も、 - // base URL を差し替えるだけで同じ `/responses` path を使える。 + // Keep `/v1` in base_url. Compatible backends can reuse the same + // `/responses` path by replacing only the base URL. "https://api.openai.com/v1" } diff --git a/crates/agen/src/llm_client/transport.rs b/crates/agen/src/llm_client/transport.rs index 8eca250f1..41058c680 100644 --- a/crates/agen/src/llm_client/transport.rs +++ b/crates/agen/src/llm_client/transport.rs @@ -1,8 +1,8 @@ -//! `HttpTransport`: すべての LLM wire scheme を共通の 1 本の -//! HTTP クライアントで扱う。 +//! `HttpTransport` handles all LLM wire schemes with one shared +//! HTTP client. //! -//! scheme 固有の差分は [`Scheme`] trait 実装に委譲し、backend 固有の -//! HTTP policy は [`TransportPolicy`] で明示的に差し込む。 +//! Scheme-specific differences are delegated to [`Scheme`] implementations; +//! backend-specific HTTP policy is explicitly injected via [`TransportPolicy`]. use std::pin::Pin; use std::sync::Arc; @@ -28,11 +28,11 @@ use super::types::{Request, RequestConfig}; pub const DEFAULT_STREAM_OPEN_TIMEOUT: Duration = Duration::from_secs(20); pub const DEFAULT_FIRST_STREAM_EVENT_TIMEOUT: Duration = Duration::from_secs(30); -/// 認証設定をリクエスト時に使える形へ解決したランタイム表現。 +/// Runtime representation of authentication resolved for use in requests. /// -/// - `None`: 認証ヘッダを送らない（Ollama 等の opt-out） -/// - `ApiKey`: 静的な API key 文字列 -/// - `Custom`: リクエスト毎に動的にヘッダを組み立てる +/// - `None`: send no authentication headers (opt-out for Ollama, etc.). +/// - `ApiKey`: static API key string. +/// - `Custom`: build headers dynamically for each request. #[derive(Debug, Clone)] pub enum ResolvedAuth { None, @@ -41,13 +41,13 @@ pub enum ResolvedAuth { } impl ResolvedAuth { - /// 認証要件と実際の解決値が噛み合うか検査する。構築時検証用。 + /// Check whether resolved authentication matches the requirement at construction. /// - /// - `ResolvedAuth::None` は認証を付けない宣言なので、どの - /// `AuthRequirement` でも受け入れる（Ollama の Anthropic scheme - /// 流用は `required_auth = XApiKey` だが認証ヘッダなしで動く） - /// - `ResolvedAuth::Custom` は「ヘッダ組立を全部こちらで行う」 - /// 宣言なので、scheme が要求する形式によらず受け入れる + /// - `ResolvedAuth::None` declares no authentication, so any `AuthRequirement` + /// is accepted (Ollama reuses the Anthropic scheme with + /// `required_auth = XApiKey`, but works without authentication headers). + /// - `ResolvedAuth::Custom` declares that all header construction is handled + /// externally, so it is accepted regardless of the scheme's required format. pub fn matches(&self, req: AuthRequirement) -> bool { match (self, req) { (Self::None, _) => true, @@ -177,7 +177,7 @@ fn sse_error_context(status: u16, headers: &serde_json::Value, source: &str) -> ) } -/// scheme 共通の HTTP 通信層。 +/// Shared HTTP transport layer for schemes. pub struct HttpTransport { http_client: reqwest::Client, scheme: S, @@ -189,8 +189,8 @@ pub struct HttpTransport { } impl HttpTransport { - /// 新しい transport を作る。`base_url` は末尾スラッシュの有無を - /// どちらでも受け付ける（内部で正規化）。 + /// Create a transport. `base_url` accepts an optional trailing slash + /// (normalized internally). pub fn new( scheme: S, model_id: impl Into, @@ -217,7 +217,7 @@ impl HttpTransport { self } - /// カスタム HTTP クライアントを差し込む（テスト等）。 + /// Inject a custom HTTP client (e.g. for tests). pub fn with_http_client(mut self, client: reqwest::Client) -> Self { self.http_client = client; self @@ -226,7 +226,7 @@ impl HttpTransport { fn build_url(&self) -> String { let path = self.scheme.path(&self.model_id); let url = format!("{}{}", self.base_url, path); - // Gemini のようにクエリパラメータで認証する場合は URL にキーを追記する + // Append the key to the URL for query-parameter authentication, as in Gemini. if let (AuthRequirement::QueryParam { name }, ResolvedAuth::ApiKey(key)) = (self.scheme.required_auth(), &self.auth) { @@ -262,11 +262,11 @@ impl HttpTransport { headers.insert("x-api-key", val); } (_, AuthRequirement::QueryParam { .. }) => { - // クエリパラメータは `build_url` で付与済み + // Query parameters were already added by `build_url`. } (ResolvedAuth::ApiKey(_), AuthRequirement::Custom) => { - // scheme が Custom を要求する組合せに ApiKey は流れてこない想定 - // （`matches()` で弾かれる）。安全側で何もしない + // ApiKey should never reach a scheme requiring Custom (`matches()` + // rejects it). Do nothing as a conservative fallback. } } @@ -459,7 +459,7 @@ impl Clone for HttpTransport { } } -/// エラーレスポンスを `ClientError::Api` に変換する。 +/// Convert an error response to `ClientError::Api`. async fn classify_error_response(resp: reqwest::Response) -> ClientError { let status = resp.status().as_u16(); let retry_after = resp @@ -713,7 +713,7 @@ impl LlmClient for HttpTransport { let byte_stream = response.bytes_stream().map_err(std::io::Error::other); let event_stream = byte_stream.eventsource(); - // scheme 固有のパース状態をストリーム単位で保持する + // Keep scheme-specific parsing state for each stream. let mut state = ::default(); let stream = event_stream diff --git a/crates/agen/src/llm_client/types.rs b/crates/agen/src/llm_client/types.rs index 1a9d3ca24..0a8446d3a 100644 --- a/crates/agen/src/llm_client/types.rs +++ b/crates/agen/src/llm_client/types.rs @@ -151,21 +151,20 @@ pub enum Item { /// Optional item ID #[serde(skip_serializing_if = "Option::is_none")] id: Option, - /// Reasoning text（reasoning body, `reasoning_text.delta` の累積） + /// Reasoning text (reasoning body, accumulated `reasoning_text.delta`). text: String, - /// Reasoning summary（OpenAI Responses の `summary_text[]` を格納。 - /// 他 scheme は空） + /// Reasoning summary (OpenAI Responses `summary_text[]`; empty for other schemes). #[serde(default, skip_serializing_if = "Vec::is_empty")] summary: Vec, - /// サーバから返された暗号化済み reasoning blob。ZDR / `store=false` - /// 運用で stateless に再送するときそのまま添える必要がある。 - /// Anthropic の `redacted_thinking.data` もここに格納する。 + /// Encrypted reasoning blob returned by the server. Must be included unchanged + /// when resending statelessly under ZDR / `store=false`. + /// Also holds Anthropic `redacted_thinking.data`. #[serde(default, skip_serializing_if = "Option::is_none")] encrypted_content: Option, - /// Anthropic extended thinking の `signature`。新世代 Claude - /// (Opus 4.5+/Sonnet 4.6+) では同一論理ターン内の `thinking` - /// ブロックを送り返す際に必須。改ざん検知に使われる。他 scheme - /// では `None`。 + /// Anthropic extended thinking `signature`, used for tamper detection. + /// Newer Claude models (Opus 4.5+/Sonnet 4.6+) require it when sending + /// `thinking` blocks back within the same logical turn. + /// `None` for other schemes. #[serde(default, skip_serializing_if = "Option::is_none")] signature: Option, /// Item status @@ -624,12 +623,11 @@ pub struct Request { /// (Anthropic today) can place a long-lived cache breakpoint there. /// Providers without prompt caching ignore the field. pub cache_anchor: Option, - /// 会話単位の安定キー。`prompt_cache_key` として送られる - /// (OpenAI Responses)。明示キーを必要とする backend では、 - /// 呼び出し側が安定した conversation identifier を渡す。 - /// `cache_anchor` と違い名前空間キーであり、`prefix anchor` とは - /// 別の概念。`cache_anchor` を読まない provider と同じく、 - /// `prompt_cache_key` を持たない provider は無視する。 + /// Stable conversation key, sent as `prompt_cache_key` (OpenAI Responses). + /// For backends requiring an explicit key, the caller supplies a stable + /// conversation identifier. Unlike `cache_anchor`, this is a namespace key, + /// separate from a `prefix anchor`. Just as some providers ignore + /// `cache_anchor`, providers without `prompt_cache_key` ignore this field. pub cache_key: Option, /// Request-local diagnostics sink for transport lifecycle tracing. #[doc(hidden)] @@ -725,7 +723,7 @@ impl Request { /// Set the conversation cache key. /// - /// 詳細は [`Request::cache_key`] のフィールドコメント参照。 + /// See the [`Request::cache_key`] field documentation for details. pub fn cache_key(mut self, key: impl Into) -> Self { self.cache_key = Some(key.into()); self @@ -790,10 +788,11 @@ pub struct RequestConfig { pub top_k: Option, /// Stop sequences pub stop_sequences: Vec, - /// Reasoning / extended-thinking 制御（共通型、scheme 側で各社形式に投影）。 + /// Shared reasoning / extended-thinking control, projected by each scheme + /// into the provider's format. /// - /// `None` のときは何も送らない。`Some` でも scheme の - /// `ModelCapability::reasoning` が `None` なら無視される。 + /// `None` sends nothing. Even `Some` is ignored if the scheme's + /// `ModelCapability::reasoning` is `None`. #[serde(default)] pub reasoning: Option, } @@ -851,7 +850,7 @@ mod parse_tool_arguments_tests { #[test] fn literal_null_normalizes_to_object() { - // 既存セッションに残っている "null" が resume 時に復旧できること + // Ensure "null" retained in existing sessions can be recovered on resume. assert_eq!(parse_tool_arguments("null"), empty_object()); } diff --git a/crates/agen/src/providers/codex/auth_json.rs b/crates/agen/src/providers/codex/auth_json.rs index 374224730..2d6b2b79e 100644 --- a/crates/agen/src/providers/codex/auth_json.rs +++ b/crates/agen/src/providers/codex/auth_json.rs @@ -1,9 +1,9 @@ -//! `~/.codex/auth.json` の読み書き。 +//! Reading and writing `~/.codex/auth.json`. //! -//! Codex CLI と schema を共有するが、知らないフィールドを失わないよう -//! ファイル全体を `serde_json::Value` で保持し、必要箇所 -//! のみアクセスする。書込は `mode 0o600` を再設定（Codex CLI 同様）、 -//! ファイルロックは取らない（manager 側で guarded reload）。 +//! Shares the schema with Codex CLI, but retains the entire file as +//! `serde_json::Value` and accesses only the required fields to preserve +//! unknown fields. Writes reapply `mode 0o600` (as in Codex CLI), +//! without file locking (the manager performs a guarded reload). use std::fs::OpenOptions; use std::io::Write; @@ -16,7 +16,7 @@ use serde_json::Value; use super::error::CodexAuthError; -/// auth.json から取り出した使い回す情報のスナップショット。 +/// Snapshot of reusable information extracted from auth.json. #[derive(Debug, Clone)] pub struct AuthSnapshot { pub access_token: String, @@ -24,12 +24,12 @@ pub struct AuthSnapshot { pub account_id: String, pub id_token: String, pub last_refresh: Option>, - /// 書き戻し時に他のフィールドを失わないため、ファイル全体を保持する。 + /// Retain the entire file to preserve other fields when writing back. pub raw: Value, } impl AuthSnapshot { - /// `Value` から必要フィールドを抽出。欠落・型不一致は `MalformedAuthJson`。 + /// Extract required fields from `Value`. Missing fields or type mismatches yield `MalformedAuthJson`. pub fn from_value(raw: Value) -> Result { let tokens = raw .get("tokens") @@ -55,7 +55,7 @@ impl AuthSnapshot { .ok_or_else(|| CodexAuthError::MalformedAuthJson("missing tokens.id_token".into()))? .to_string(); - // account_id は tokens.account_id を優先、無ければ id_token JWT 由来 + // Prefer tokens.account_id; fall back to the id_token JWT if absent let account_id = tokens .get("account_id") .and_then(Value::as_str) @@ -84,7 +84,7 @@ impl AuthSnapshot { } } -/// auth.json を読む。存在しなければ `NotLoggedIn`。 +/// Read auth.json. Return `NotLoggedIn` if it does not exist. pub async fn load(path: &Path) -> Result { let bytes = match tokio::fs::read(path).await { Ok(b) => b, @@ -103,11 +103,11 @@ pub async fn load(path: &Path) -> Result { AuthSnapshot::from_value(raw) } -/// 既存ファイルを再読込し、`tokens.{id_token,access_token,refresh_token}` と -/// `last_refresh` を更新して書き戻す。Codex CLI の `persist_tokens` 相当。 +/// Reload the existing file, update `tokens.{id_token,access_token,refresh_token}` +/// and `last_refresh`, and write it back. Equivalent to Codex CLI's `persist_tokens`. /// -/// 並行する Codex CLI / 別プロセスが先に refresh していた場合の -/// fields を保護するため、書込前に再 load して merge する。 +/// Reload and merge before writing to preserve fields if a concurrent +/// Codex CLI / another process has already refreshed. pub async fn persist_refreshed( path: &Path, new_id_token: Option, @@ -161,8 +161,8 @@ fn write_atomic(path: &Path, value: &Value) -> Result<(), CodexAuthError> { file.flush() .map_err(|e| CodexAuthError::Io(format!("flush {}: {e}", path.display())))?; - // 既存ファイルが緩いパーミッションだった場合に備えて 0o600 を強制し直す。 - // OpenOptions の `mode()` は新規作成時しか効かないため。 + // Reapply 0o600 in case the existing file had permissive permissions. + // OpenOptions `mode()` only takes effect when creating a new file. #[cfg(unix)] { use std::os::unix::fs::PermissionsExt; @@ -204,7 +204,7 @@ mod tests { assert_eq!(snap.refresh_token, "ref"); assert_eq!(snap.account_id, "acc-1"); assert!(snap.last_refresh.is_some()); - // 未知フィールドが raw に保持されている + // Unknown fields are retained in raw assert_eq!( snap.raw.get("OPENAI_API_KEY").and_then(Value::as_str), Some("sk-extra") @@ -214,7 +214,7 @@ mod tests { #[tokio::test] async fn account_id_falls_back_to_jwt_claim() { let dir = tempfile::tempdir().unwrap(); - // tokens.account_id を欠落させ、id_token JWT 内 claim から拾う + // Omit tokens.account_id and obtain it from a claim in the id_token JWT let id_token = "h.eyJodHRwczovL2FwaS5vcGVuYWkuY29tL2F1dGgiOnsiY2hhdGdwdF9hY2NvdW50X2lkIjoiZnJvbS1qd3QifX0.s"; let path = write_auth_json( dir.path(), @@ -249,7 +249,7 @@ mod tests { .unwrap(); assert_eq!(updated.access_token, "new-acc"); assert_eq!(updated.refresh_token, "new-ref"); - // 未知フィールド agent_identity が保たれる + // The unknown field agent_identity is preserved let on_disk: Value = serde_json::from_slice(&std::fs::read(&path).unwrap()).unwrap(); assert!(on_disk.get("agent_identity").is_some()); assert!(on_disk.get("last_refresh").is_some()); @@ -264,7 +264,7 @@ mod tests { dir.path(), r#"{"tokens":{"id_token":"h.eyJodHRwczovL2FwaS5vcGVuYWkuY29tL2F1dGgiOnsiY2hhdGdwdF9hY2NvdW50X2lkIjoiYSJ9fQ.s","access_token":"a","refresh_token":"r","account_id":"a"}}"#, ); - // 既存ファイルを 644 に変えてから persist → 600 に直るか + // Change the existing file to 644, then persist → check that it returns to 600 std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o644)).unwrap(); persist_refreshed(&path, None, Some("a2".into()), None) .await diff --git a/crates/agen/src/providers/codex/error.rs b/crates/agen/src/providers/codex/error.rs index 7817a03d3..648fce748 100644 --- a/crates/agen/src/providers/codex/error.rs +++ b/crates/agen/src/providers/codex/error.rs @@ -1,8 +1,8 @@ -//! Codex OAuth サブモジュール内部のエラー型。 +//! Internal error types for the Codex OAuth submodule. //! -//! `LlmClient` 境界に渡す際は `to_client_error` で `ClientError` に -//! 変換する。`Permanent` 系（refresh_token 失効）は `codex login` の -//! 再実行案内付きメッセージにする。 +//! Convert to `ClientError` using `to_client_error` at the `LlmClient` +//! boundary. `Permanent` errors (invalidated refresh_token) include +//! a message instructing the user to run `codex login` again. use std::path::PathBuf; @@ -25,7 +25,7 @@ pub enum CodexAuthError { #[error("token refresh failed (transient): {0}")] RefreshTransient(String), - /// refresh_token が永続的に失効。再ログインが必要。 + /// refresh_token is permanently invalidated. Logging in again is required. #[error("token refresh failed permanently ({reason:?}): {message}")] RefreshPermanent { reason: PermanentReason, @@ -45,7 +45,7 @@ pub enum PermanentReason { } impl CodexAuthError { - /// `LlmClient` トランスポート境界向けに変換する。 + /// Convert for the `LlmClient` transport boundary. pub fn to_client_error(self) -> ClientError { match self { CodexAuthError::NotLoggedIn(_) diff --git a/crates/agen/src/providers/codex/jwt.rs b/crates/agen/src/providers/codex/jwt.rs index 16aa82638..890badef8 100644 --- a/crates/agen/src/providers/codex/jwt.rs +++ b/crates/agen/src/providers/codex/jwt.rs @@ -1,13 +1,13 @@ -//! JWT payload の最小限のパース（署名検証なし）。 +//! Minimal JWT payload parsing (without signature verification). //! -//! Codex CLI と同じく、access_token / id_token の payload を base64url -//! デコードして `exp` や ChatGPT 固有 claims を取り出すためだけに使う。 +//! As in Codex CLI, used only to base64url-decode access_token / id_token +//! payloads and extract `exp` and ChatGPT-specific claims. use base64::Engine; use chrono::{DateTime, Utc}; use serde::Deserialize; -/// `Authorization: Bearer` で送る access_token JWT の `exp` を読む。 +/// Read `exp` from the access_token JWT sent in `Authorization: Bearer`. pub fn parse_exp(jwt: &str) -> Option> { #[derive(Deserialize)] struct Claims { @@ -17,7 +17,7 @@ pub fn parse_exp(jwt: &str) -> Option> { DateTime::::from_timestamp(claims.exp?, 0) } -/// id_token JWT から ChatGPT 固有 claims を取り出す。 +/// Extract ChatGPT-specific claims from the id_token JWT. pub fn parse_chatgpt_claims(jwt: &str) -> Option { #[derive(Deserialize)] struct IdClaims { diff --git a/crates/agen/src/providers/codex/mod.rs b/crates/agen/src/providers/codex/mod.rs index a3a5ad2f0..51c98752f 100644 --- a/crates/agen/src/providers/codex/mod.rs +++ b/crates/agen/src/providers/codex/mod.rs @@ -1,17 +1,17 @@ -//! ChatGPT OAuth (`~/.codex/auth.json`) を読んで `Authorization` / -//! `ChatGPT-Account-Id` / `X-OpenAI-Fedramp` ヘッダを組み立てる -//! [`AuthProvider`] 実装。 +//! [`AuthProvider`] implementation that reads ChatGPT OAuth credentials +//! (`~/.codex/auth.json`) and builds `Authorization` / +//! `ChatGPT-Account-Id` / `X-OpenAI-Fedramp` headers. //! -//! 設計: +//! Design: //! -//! - HTTP transport は [`AuthProvider`] trait だけを見て、実体である -//! [`CodexAuthProvider`] はこの optional module に置く -//! - access_token JWT の `exp` を読み、`now` 以下で proactive refresh -//! （Codex CLI と同じバッファなし） -//! - 並行する Codex CLI / 別プロセスの refresh と取り違えないよう、 -//! refresh 直前に再 load して account_id 一致を確認（guarded reload） -//! - ファイルロックは取らず、書込前に再 load + diff merge で吸収 -//! - Codex の Keyring storage は対象外。auth.json 不在ならエラーで案内 +//! - The HTTP transport sees only the [`AuthProvider`] trait; the concrete +//! [`CodexAuthProvider`] lives in this optional module +//! - Read the access_token JWT's `exp` and proactively refresh when `exp` <= `now` +//! (no buffer, as in Codex CLI) +//! - Reload immediately before refreshing and check that account_id matches +//! to avoid confusing concurrent Codex CLI / other-process refreshes (guarded reload) +//! - No file locking; reload and merge differences before writing +//! - Codex Keyring storage is not supported. If auth.json is missing, return an error with guidance mod auth_json; mod error; @@ -32,10 +32,10 @@ use error::CodexAuthError; pub use error::PermanentReason; -/// Codex CLI の `last_refresh` ベース fallback 期限（Codex CLI 準拠で 8 日）。 +/// Codex CLI's `last_refresh`-based fallback interval (8 days, matching Codex CLI). const TOKEN_REFRESH_INTERVAL_DAYS: i64 = 8; -/// `~/.codex/auth.json` を読んで Codex 互換のヘッダを返す provider。 +/// Provider that reads `~/.codex/auth.json` and returns Codex-compatible headers. pub struct CodexAuthProvider { auth_path: PathBuf, refresh_endpoint: String, @@ -58,7 +58,7 @@ struct State { } impl CodexAuthProvider { - /// `CODEX_HOME` → `$HOME/.codex` の順で auth.json の場所を決める。 + /// Locate auth.json using `CODEX_HOME`, then `$HOME/.codex` as a fallback. pub fn from_default_home() -> Result { let codex_home = if let Ok(p) = std::env::var("CODEX_HOME") { PathBuf::from(p) @@ -70,7 +70,7 @@ impl CodexAuthProvider { Ok(Self::new(codex_home)) } - /// 任意の codex_home から構築する。 + /// Construct from a custom codex_home. pub fn new(codex_home: PathBuf) -> Self { Self { auth_path: codex_home.join("auth.json"), @@ -80,14 +80,14 @@ impl CodexAuthProvider { } } - /// テスト用に refresh エンドポイントを差し替える。 + /// Override the refresh endpoint for tests. #[cfg(test)] fn with_refresh_endpoint(mut self, endpoint: impl Into) -> Self { self.refresh_endpoint = endpoint.into(); self } - /// テスト用に HTTP クライアントを差し替える。 + /// Override the HTTP client for tests. #[cfg(test)] fn with_http_client(mut self, client: reqwest::Client) -> Self { self.http = client; @@ -97,29 +97,29 @@ impl CodexAuthProvider { async fn ensure_fresh(&self) -> Result { let mut state = self.state.lock().await; - // 1. 常にディスクから最新を読む（他プロセスの更新を反映） + // 1. Always read the latest from disk (reflect updates by other processes) let snap = auth_json::load(&self.auth_path).await?; - // 2. stale でなければそのまま返す + // 2. Return as-is if not stale if !is_stale(&snap) { state.cached = Some(snap.clone()); return Ok(snap); } - // 3. Refresh 直前に再 load し account_id 一致を確認（guarded reload）。 - // 一致しなければ他プロセスが先に更新済 → 自分は refresh しない + // 3. Reload immediately before refreshing and check that account_id matches (guarded reload). + // If it differs, another process has already updated it → skip refreshing let pre_refresh = auth_json::load(&self.auth_path).await?; if !is_stale(&pre_refresh) { state.cached = Some(pre_refresh.clone()); return Ok(pre_refresh); } if pre_refresh.account_id != snap.account_id { - // account が切り替わった → 新しい auth を採用、refresh はしない + // The account has changed → use the new auth without refreshing state.cached = Some(pre_refresh.clone()); return Ok(pre_refresh); } - // 4. Refresh 実行 + // 4. Perform the refresh let refreshed = refresh::request_refresh( &self.http, &self.refresh_endpoint, @@ -127,7 +127,7 @@ impl CodexAuthProvider { ) .await?; - // 5. 書き戻し（書込前に再 load + merge は persist_refreshed 内で実施） + // 5. Write back (persist_refreshed reloads and merges before writing) let new_snap = auth_json::persist_refreshed( &self.auth_path, refreshed.id_token, @@ -152,9 +152,9 @@ impl CodexAuthProvider { .map_err(|e| CodexAuthError::InvalidHeader(format!("ChatGPT-Account-Id: {e}")))?; out.push((HeaderName::from_static("chatgpt-account-id"), acc_val)); - // Cloudflare WAF は互換 backend アクセス元を `originator` / - // `User-Agent` で識別する。Codex CLI が送る固定値を流用しないと - // HTML challenge (403) を返されて SSE に到達できない。 + // Cloudflare WAF identifies clients accessing the compatible backend by `originator` / + // `User-Agent`. Without the fixed values sent by Codex CLI, + // it returns an HTML challenge (403), preventing access to SSE. out.push(( HeaderName::from_static("originator"), HeaderValue::from_static("codex_cli_rs"), @@ -164,7 +164,7 @@ impl CodexAuthProvider { HeaderValue::from_static("codex_cli_rs/0.60.0"), )); - // FedRAMP 組織は id_token JWT 内の claim で判定 + // Identify FedRAMP organizations using a claim in the id_token JWT if jwt::parse_chatgpt_claims(&snap.id_token) .map(|c| c.is_fedramp) .unwrap_or(false) @@ -190,8 +190,8 @@ impl AuthProvider for CodexAuthProvider { } } -/// `access_token` の JWT `exp` を見て、期限切れなら true。 -/// `exp` が読めない場合は `last_refresh + 8 日` で判定（Codex CLI と同じ）。 +/// Return true if the `access_token` JWT's `exp` indicates expiration. +/// If `exp` cannot be read, use `last_refresh + 8 days` (as in Codex CLI). fn is_stale(snap: &AuthSnapshot) -> bool { if let Some(exp) = jwt::parse_exp(&snap.access_token) { return exp <= Utc::now(); @@ -241,7 +241,7 @@ mod tests { #[tokio::test] async fn returns_headers_for_fresh_token() { let dir = tempfile::tempdir().unwrap(); - // 期限を遠い未来に + // Set expiration far in the future write_auth(dir.path(), Utc::now().timestamp() + 3600, false, "rt"); let provider = CodexAuthProvider::new(dir.path().to_path_buf()); @@ -285,7 +285,7 @@ mod tests { "old-refresh", ); - // refresh エンドポイントを mock。新しい JWT (将来 exp) を返す + // Mock the refresh endpoint to return a new JWT (future exp) let server = MockServer::start().await; let new_access = make_jwt(json!({ "exp": Utc::now().timestamp() + 3600 })); let new_refresh = "new-refresh-token"; @@ -310,7 +310,7 @@ mod tests { .unwrap(); assert_eq!(auth.1.to_str().unwrap(), format!("Bearer {new_access}")); - // ファイルに新しい refresh_token が書き戻されている + // The new refresh_token has been written back to the file let on_disk: serde_json::Value = serde_json::from_slice(&std::fs::read(&path).unwrap()).unwrap(); assert_eq!( diff --git a/crates/agen/src/providers/codex/refresh.rs b/crates/agen/src/providers/codex/refresh.rs index 1b526eeb9..c74e0e829 100644 --- a/crates/agen/src/providers/codex/refresh.rs +++ b/crates/agen/src/providers/codex/refresh.rs @@ -1,7 +1,7 @@ -//! ChatGPT OAuth トークンの refresh HTTP 呼出。 +//! HTTP calls to refresh ChatGPT OAuth tokens. //! -//! Codex CLI と同じ `POST https://auth.openai.com/oauth/token` 形式。 -//! 401 + `error.code` で永続失敗を分類する。 +//! Uses the same `POST https://auth.openai.com/oauth/token` format as Codex CLI. +//! Classifies permanent failures using 401 + `error.code`. use serde::{Deserialize, Serialize}; use std::time::Duration; @@ -29,10 +29,10 @@ pub struct RefreshResponse { pub refresh_token: Option, } -/// refresh_token を使って新しいトークン群を取得する。 +/// Obtain new tokens using refresh_token. /// -/// 永続失敗（401 + `refresh_token_(expired|reused|invalidated)`）は -/// `RefreshPermanent`、それ以外は `RefreshTransient`。 +/// Permanent failures (401 + `refresh_token_(expired|reused|invalidated)`) return +/// `RefreshPermanent`; all others return `RefreshTransient`. pub async fn request_refresh( client: &reqwest::Client, endpoint: &str, diff --git a/crates/agen/src/prune.rs b/crates/agen/src/prune.rs index 37af648da..0fe9c51a5 100644 --- a/crates/agen/src/prune.rs +++ b/crates/agen/src/prune.rs @@ -1,19 +1,17 @@ -//! Prune — context projection for old tool-result content. +//! Request-only omission of details from old tool results. //! -//! LLM 送信時のコンテキストから古い [`Item::ToolResult`] の `content` を -//! 省略して、コンテキスト窓のトークンを回収する。`summary` は残すので -//! 「何が起きたか」の痕跡は保たれる。 +//! [`prunable_indices`] selects candidates while protecting a token budget at the +//! history tail; [`project`] removes detailed text and attachments. Summaries remain +//! visible to the model. Projection operates on a request-history clone, leaving +//! stored history and annotations unchanged. //! -//! # 設計方針 +//! This module does not own token measurements. The host supplies prefix and savings +//! estimates; unavailable estimates or invalid boundaries produce no candidates. +//! The Engine also checks savings to avoid invalidating the cache for little benefit. //! -//! Prune は **コンテキスト射影** であり、history の変換ではない。 -//! この crate が提供するのは pure な候補抽出 [`prunable_indices`] のみで、 -//! 射影の適用は LLM に送る一時コンテキストに対してだけ行う。 -//! Engine の永続履歴は決して変更されない。 -//! -//! 保護境界は末尾 token budget で決めるが、この crate は usage 履歴を -//! 所有しない。prefix ごとの token 推定値と savings 推定は上位層から -//! callback で注入される。 +//! Compaction, which stores a summary as a new segment, is a separate responsibility. +//! Omitting already-consumed images likewise affects only the request projection, +//! never the stored results. use serde::{Deserialize, Serialize}; @@ -36,27 +34,26 @@ pub type TokenEstimator = Box Vec + Send + Syn /// Engine can make `min_savings` decisions without knowing about usage /// measurement sources. Return `0` to signal "no data / refuse to prune". /// -/// 推定対象は「drop する範囲全体」ではなく「content を None にする差分」 -/// であることに注意。item 自体（summary 等）は残るので、この callback は -/// 実際の projection と一致する savings を返す必要がある。 +/// Estimate the savings from removing details, not from dropping entire items. +/// Items and their summaries remain, so this callback must match the actual projection. pub type SavingsEstimator = Box u64 + Send + Sync>; /// Result of one prune evaluation pass, surfaced to the optional /// [`PruneObserver`] for instrumentation. /// -/// Engine は LLM リクエストごとに 1 回 prune の評価をし、その結果を -/// （observer が登録されていれば）この値で通知する。fire/skip の判定 -/// 結果と、判定材料になった候補数 / 推定 savings / 保護領域の先頭 index を持つ。 +/// The Engine evaluates pruning once per LLM request and reports this value if +/// an observer is registered. It includes the decision and its inputs: candidate +/// count, estimated savings, and the start index of the protected suffix. #[derive(Debug, Clone)] pub struct PruneEvaluation { - /// `prunable_indices` の長さ。`Skipped::NoCandidates` の時は 0。 + /// Length of `prunable_indices`; zero for `SkippedNoCandidates`. pub candidate_count: usize, - /// 推定された savings (tokens)。`NoCandidates` の時は 0。 + /// Estimated token savings; zero when there are no candidates. pub estimated_savings: u64, - /// Token budget で保護される suffix の先頭 item index。 - /// usage 推定が `NoData` で境界が決まらない場合は `None`。 + /// First item in the suffix protected by the token budget. + /// `None` when `NoData` estimates prevent determining the boundary. pub protected_start_index: Option, - /// 判定結果。 + /// Evaluation outcome. pub decision: PruneDecision, } @@ -64,13 +61,13 @@ pub struct PruneEvaluation { /// "fire vs skip" decision tree the Engine walks before each LLM request. #[derive(Debug, Clone, PartialEq, Eq)] pub enum PruneDecision { - /// `prunable_indices` が空 → 何もしない。 + /// No projection: `prunable_indices` is empty. SkippedNoCandidates, - /// 候補はあったが推定 savings が `min_savings` 未満 → 何もしない。 + /// No projection: candidates exist, but estimated savings are below `min_savings`. SkippedBelowMinSavings, - /// 候補があり savings >= min_savings → projection を適用した。 - /// `pruned_count` は `project()` が実際に書き換えた item 数 - /// （既に content=None だった候補は 0 計上）。 + /// Applied projection because candidates exist and savings meet `min_savings`. + /// `pruned_count` counts items actually changed by `project()`; + /// candidates that already lack details contribute zero. Fired { pruned_count: usize }, } @@ -513,7 +510,7 @@ mod tests { let estimates = uniform_estimates(&items, 10); let candidates = prunable_indices(&items, 20, &estimates); assert_eq!(project(&mut items, &candidates), 1); - // 2 周目: 候補は一度の prunable_indices 結果を使い回しても 0 件。 + // Second pass: reusing the same prunable_indices result changes zero items. assert_eq!(project(&mut items, &candidates), 0); } diff --git a/crates/agen/src/state.rs b/crates/agen/src/state.rs index 3ca3c14a7..e37da462a 100644 --- a/crates/agen/src/state.rs +++ b/crates/agen/src/state.rs @@ -1,58 +1,34 @@ -//! Engine State +//! Type states that restrict changes to Engine configuration. //! -//! State marker types for cache protection using the Type-state pattern. -//! Engine has state transitions from `Mutable` → `Locked`. +//! Transitioning from [`Mutable`] to [`Locked`] fixes the configuration and history +//! prefix length, restricting changes to system prompts and tool definitions during +//! execution. Changes require an explicit unlock because altering the prefix may +//! prevent prompt-cache reuse. History itself belongs to the caller: these states +//! neither prevent external history mutations nor guarantee cache hits. -/// Marker trait representing Engine state +/// Sealed trait for Engine type states. /// -/// This trait is sealed and cannot be implemented externally. +/// External implementations are prohibited so the crate controls the separation +/// between configuration and execution APIs. pub trait EngineState: private::Sealed + Send + Sync + 'static {} mod private { pub trait Sealed {} } -/// Mutable state (editable) +/// State in which configuration can be edited. /// -/// In this state, the following operations are available: -/// - Setting/changing system prompt -/// - Editing message history (add, delete, clear) -/// - Registering tools and hooks -/// -/// Can transition to [`Locked`] state via `Engine::lock(&history)`. -/// -/// # Examples -/// -/// ```ignore -/// use agen::Engine; -/// -/// let mut engine = Engine::new(client) -/// .system_prompt("You are helpful."); -/// -/// // History can be edited -/// engine.push_message(Message::user("Hello")); -/// engine.clear_history(); -/// -/// // Lock to protected state -/// let locked = engine.lock(); -/// ``` +/// Transitions to [`Locked`] through `Engine::lock` or the first `run`. #[derive(Debug, Clone, Copy, Default)] pub struct Mutable; impl private::Sealed for Mutable {} impl EngineState for Mutable {} -/// Cache locked state (cache protected) +/// State with fixed configuration and history prefix length. /// -/// In this state, the following restrictions apply: -/// - System prompt cannot be changed -/// - Existing message history cannot be modified (only appending to the end) -/// -/// To ensure LLM API KV cache hits, -/// using this state during execution is recommended. -/// -/// Can return to [`Mutable`] state via `Engine::unlock()`, -/// but note that cache protection will be released. +/// `Engine::unlock` returns to [`Mutable`] and permits configuration changes, +/// but requests made after those changes may no longer reuse the previous cache. #[derive(Debug, Clone, Copy, Default)] pub struct Locked; diff --git a/crates/agen/src/timeline/event.rs b/crates/agen/src/timeline/event.rs index 2447d14d4..6ef5c7efd 100644 --- a/crates/agen/src/timeline/event.rs +++ b/crates/agen/src/timeline/event.rs @@ -1,5 +1,5 @@ -//! Timeline層のイベント型 +//! Event types for the Timeline layer. //! -//! llm_client層のイベント型をそのまま使用する。 +//! Uses the event types from the llm_client layer directly. pub use crate::llm_client::event::*; diff --git a/crates/agen/src/timeline/mod.rs b/crates/agen/src/timeline/mod.rs index b016c2592..69a8bbb7d 100644 --- a/crates/agen/src/timeline/mod.rs +++ b/crates/agen/src/timeline/mod.rs @@ -1,14 +1,9 @@ -//! Timeline層 +//! Dispatch normalized LLM events and collect blocks. //! -//! LLMからのイベントストリームを受信し、登録されたHandlerにディスパッチします。 -//! -//! # 主要コンポーネント -//! -//! - [`Timeline`] - イベントストリームの管理とディスパッチ -//! - [`Handler`] - イベントを処理するトレイト -//! - [`TextBlockCollector`] - テキストブロックを収集するHandler -//! - [`ThinkingBlockCollector`] - reasoning material を Thinking block から収集するHandler -//! - [`ToolCallCollector`] - ツール呼び出しを収集するHandler +//! [`Timeline`] dispatches common events to handlers without parsing provider protocols. +//! Block assembly and display callbacks are separate from turn decisions. The Engine +//! uses collected responses to update history and execute tools. Provisional streaming +//! output is distinct from committed history. pub mod event; mod text_block_collector; @@ -16,7 +11,7 @@ mod thinking_block_collector; mod timeline; mod tool_call_collector; -// 公開API +// Public API pub use event::*; pub use text_block_collector::TextBlockCollector; pub use thinking_block_collector::ThinkingBlockCollector; @@ -24,7 +19,7 @@ pub use timeline::Timeline; pub(crate) use tool_call_collector::CollectedToolCall; pub use tool_call_collector::ToolCallCollector; -// 型定義からのre-export +// Re-exports from type definitions pub use crate::handler::{ // Meta Kinds ErrorKind, diff --git a/crates/agen/src/timeline/text_block_collector.rs b/crates/agen/src/timeline/text_block_collector.rs index f7d6f2451..007c0f20f 100644 --- a/crates/agen/src/timeline/text_block_collector.rs +++ b/crates/agen/src/timeline/text_block_collector.rs @@ -1,53 +1,51 @@ -//! TextBlockCollector - テキストブロック収集用ハンドラ +//! TextBlockCollector - a handler for collecting text blocks. //! -//! TimelineのTextBlockHandler として登録され、 -//! ストリーム中のテキストブロックを収集する。 +//! Registered as a TextBlockHandler on Timeline to collect streaming text blocks. use crate::handler::{Handler, TextBlockEvent, TextBlockKind}; use std::sync::{Arc, Mutex}; -/// TextBlockから収集したテキスト情報を保持 +/// Holds text collected from TextBlock events. #[derive(Debug, Default)] pub struct TextCollectorState { - /// 蓄積中のテキスト + /// Text being accumulated. buffer: String, } -/// TextBlockCollector - テキストブロックハンドラ +/// TextBlockCollector - a text block handler. /// -/// Timelineに登録してTextBlockイベントを受信し、 -/// 完了したテキストブロックを収集する。 +/// Register with Timeline to receive TextBlock events and collect completed text blocks. #[derive(Clone)] pub struct TextBlockCollector { - /// 収集されたテキストブロック + /// Collected text blocks. collected: Arc>>, } impl TextBlockCollector { - /// 新しいTextBlockCollectorを作成 + /// Create a new TextBlockCollector. pub fn new() -> Self { Self { collected: Arc::new(Mutex::new(Vec::new())), } } - /// 収集されたテキストを取得してクリア + /// Take the collected text and clear the collection. pub fn take_collected(&self) -> Vec { let mut guard = self.collected.lock().unwrap(); std::mem::take(&mut *guard) } - /// 収集されたテキストの参照を取得 + /// Access the collected text. pub fn collected(&self) -> Vec { self.collected.lock().unwrap().clone() } - /// 収集されたテキストがあるかどうか + /// Whether any text has been collected. pub fn has_content(&self) -> bool { !self.collected.lock().unwrap().is_empty() } - /// 収集をクリア + /// Clear the collection. pub fn clear(&self) { self.collected.lock().unwrap().clear(); } @@ -71,7 +69,7 @@ impl Handler for TextBlockCollector { scope.buffer.push_str(text); } TextBlockEvent::Stop(_) => { - // ブロック完了時にテキストを確定 + // Finalize the text when the block completes. if !scope.buffer.is_empty() { let text = std::mem::take(&mut scope.buffer); self.collected.lock().unwrap().push(text); @@ -87,38 +85,38 @@ mod tests { use crate::timeline::Timeline; use crate::timeline::event::Event; - /// TextBlockCollectorが単一のテキストブロックを正しく収集することを確認 + /// Verify that TextBlockCollector correctly collects a single text block. #[test] fn test_collect_single_text_block() { let collector = TextBlockCollector::new(); let mut timeline = Timeline::new(); timeline.on_text_block(collector.clone()); - // テキストブロックのイベントシーケンスをディスパッチ + // Dispatch a sequence of text block events. timeline.dispatch(&Event::text_block_start(0)); timeline.dispatch(&Event::text_delta(0, "Hello, ")); timeline.dispatch(&Event::text_delta(0, "World!")); timeline.dispatch(&Event::text_block_stop(0, None)); - // 収集されたテキストを確認 + // Check the collected text. let texts = collector.take_collected(); assert_eq!(texts.len(), 1); assert_eq!(texts[0], "Hello, World!"); } - /// TextBlockCollectorが複数のテキストブロックを正しく収集することを確認 + /// Verify that TextBlockCollector correctly collects multiple text blocks. #[test] fn test_collect_multiple_text_blocks() { let collector = TextBlockCollector::new(); let mut timeline = Timeline::new(); timeline.on_text_block(collector.clone()); - // 1つ目のテキストブロック + // First text block timeline.dispatch(&Event::text_block_start(0)); timeline.dispatch(&Event::text_delta(0, "First")); timeline.dispatch(&Event::text_block_stop(0, None)); - // 2つ目のテキストブロック + // Second text block timeline.dispatch(&Event::text_block_start(1)); timeline.dispatch(&Event::text_delta(1, "Second")); timeline.dispatch(&Event::text_block_stop(1, None)); diff --git a/crates/agen/src/timeline/thinking_block_collector.rs b/crates/agen/src/timeline/thinking_block_collector.rs index 0caa776b8..a08f16395 100644 --- a/crates/agen/src/timeline/thinking_block_collector.rs +++ b/crates/agen/src/timeline/thinking_block_collector.rs @@ -20,13 +20,13 @@ impl ThinkingBlockCollector { Self::default() } - /// 収集済み item を取り出してクリア + /// Take the collected items and clear the collection. pub fn take_collected(&self) -> Vec { let mut guard = self.collected.lock().unwrap(); std::mem::take(&mut *guard) } - /// 収集をクリア + /// Clear the collection. pub fn clear(&self) { self.collected.lock().unwrap().clear(); } @@ -107,7 +107,7 @@ mod tests { assert_eq!(items[0].signature.as_deref(), Some("sig1")); assert_eq!(items[1].text.as_deref(), Some("second")); - // take は drain なので 2 度目は空 + // Taking drains the collection, so the second call is empty. assert!(collector.take_collected().is_empty()); } diff --git a/crates/agen/src/timeline/timeline.rs b/crates/agen/src/timeline/timeline.rs index 6dfe80b2c..c830cc526 100644 --- a/crates/agen/src/timeline/timeline.rs +++ b/crates/agen/src/timeline/timeline.rs @@ -1,7 +1,7 @@ -//! Timeline層 +//! Timeline layer. //! -//! LLMからのイベントストリームを受信し、登録されたHandlerにディスパッチします。 -//! 通常はEngine経由で使用しますが、直接使用することも可能です。 +//! Receives an LLM event stream and dispatches it to registered handlers. +//! Usually used through the Engine, but can also be used directly. use std::{collections::HashMap, marker::PhantomData}; @@ -12,11 +12,11 @@ use crate::handler::*; // Helpers // ============================================================================= -/// 1リクエスト内で受信した複数 UsageEvent をマージする。 -/// 各フィールドについて新しい値が `Some` ならそれで上書き。 -/// プロバイダによっては input/cache 系を最初の event だけに載せ、 -/// output_tokens を後続 event で更新するため、最後の値だけを取るのではなく -/// フィールド単位で latest-non-None を取る。 +/// Merge multiple UsageEvents received within one request. +/// Overwrite each field when the new value is `Some`. +/// Some providers send input/cache fields only in the first event and update +/// output_tokens in later events, so use the latest non-None value per field +/// rather than taking only the last event. fn merge_usage(acc: &mut UsageEvent, new: &UsageEvent) { if new.input_tokens.is_some() { acc.input_tokens = new.input_tokens; @@ -39,19 +39,19 @@ fn merge_usage(acc: &mut UsageEvent, new: &UsageEvent) { // Type-erased Handler // ============================================================================= -/// 型消去された`Handler` trait +/// Type-erased `Handler` trait. /// -/// 各Handlerは独自のScope型を持つため、Timelineで保持するには型消去が必要です。 -/// 通常は直接使用せず、`Timeline::on_text_block()`などのメソッド経由で -/// 自動的にラップされます。 +/// Each handler has its own Scope type, so Timeline needs type erasure to store them. +/// Usually wrapped automatically by methods such as `Timeline::on_text_block()` +/// rather than used directly. pub trait ErasedHandler: Send + Sync { - /// イベントをディスパッチ + /// Dispatch an event. fn dispatch(&mut self, event: &K::Event); - /// スコープを開始 + /// Start a scope. fn start_scope(&mut self); } -/// `Handler`を`ErasedHandler`として扱うためのラッパー +/// Wrapper that exposes `Handler` as `ErasedHandler`. pub struct HandlerWrapper where H: Handler, @@ -59,7 +59,7 @@ where { handler: H, scope: Option, - // fn() -> K は常にSend+Syncなので、Kの制約に関係なくSendを満たせる + // fn() -> K is always Send+Sync, so Send does not depend on K's bounds. _kind: PhantomData K>, } @@ -98,7 +98,7 @@ where // Block Handler Registry // ============================================================================= -/// ブロックハンドラーの型消去trait +/// Type-erased trait for block handlers. trait ErasedBlockHandler: Send + Sync { fn dispatch_start(&mut self, start: &BlockStart); fn dispatch_delta(&mut self, delta: &BlockDelta); @@ -106,11 +106,11 @@ trait ErasedBlockHandler: Send + Sync { fn dispatch_abort(&mut self, abort: &BlockAbort); fn start_scope(&mut self); fn end_scope(&mut self); - /// スコープがアクティブかどうか + /// Whether a scope is active. fn has_scope(&self) -> bool; } -/// TextBlockKind用のラッパー +/// Wrapper for TextBlockKind. struct TextBlockHandlerWrapper where H: Handler, @@ -167,7 +167,7 @@ where } fn dispatch_abort(&mut self, _abort: &BlockAbort) { - // TextBlockはabortを特別扱いしない（スコープ終了のみ） + // TextBlock needs no special abort handling (only scope cleanup). } fn start_scope(&mut self) { @@ -183,7 +183,7 @@ where } } -/// ThinkingBlockKind用のラッパー +/// Wrapper for ThinkingBlockKind. struct ThinkingBlockHandlerWrapper where H: Handler, @@ -250,7 +250,7 @@ where } } -/// ToolUseBlockKind用のラッパー +/// Wrapper for ToolUseBlockKind. struct ToolUseBlockHandlerWrapper where H: Handler, @@ -335,10 +335,10 @@ where // Timeline // ============================================================================= -/// イベントストリームの管理とハンドラへのディスパッチ +/// Manages an event stream and dispatches events to handlers. /// -/// LLMからのイベントを受信し、登録されたハンドラに振り分けます。 -/// ブロック系イベントはスコープ管理付きで処理されます。 +/// Receives events from the LLM and routes them to registered handlers. +/// Block events are handled with scope management. /// /// # Examples /// @@ -359,30 +359,30 @@ where /// timeline.on_text_block(MyHandler); /// ``` /// -/// # サポートするイベント種別 +/// # Supported event kinds /// -/// - **メタ系**: Usage, Ping, Status, Error -/// - **ブロック系**: TextBlock, ThinkingBlock, ToolUseBlock +/// - **Meta**: Usage, Ping, Status, Error +/// - **Block**: TextBlock, ThinkingBlock, ToolUseBlock pub struct Timeline { - // Meta系ハンドラー + // Meta handlers usage_handlers: Vec>>, ping_handlers: Vec>>, status_handlers: Vec>>, error_handlers: Vec>>, - // Block系ハンドラー（BlockTypeごとにグループ化） + // Block handlers (grouped by BlockType) text_block_handlers: Vec>, thinking_block_handlers: Vec>, tool_use_block_handlers: Vec>, - // 現在アクティブなブロック。provider は index ごとに複数 scope を - // interleave できるため、単一の current block ではなく全 scope を追跡する。 + // Currently active blocks. Providers can interleave multiple scopes by index, + // so track all scopes rather than a single current block. active_blocks: HashMap<(BlockType, usize), ()>, current_block: Option, - // 1リクエスト内で受信した Usage event の集約バッファ。 - // Anthropic は message_start と message_delta、Gemini は各チャンクと、 - // 多くのプロバイダが複数 Usage を発行するため、リクエスト境界で - // 1度だけ発火するためにここでマージする。flush_usage() で発火する。 + // Aggregation buffer for Usage events received within one request. + // Many providers emit multiple Usage events: Anthropic in message_start and + // message_delta, Gemini in each chunk. Merge them here to dispatch only once + // at the request boundary via flush_usage(). pending_usage: Option, } @@ -412,20 +412,20 @@ impl Timeline { // Handler Registration // ========================================================================= - /// UsageKind用のHandlerを登録 + /// Register a Handler for UsageKind. pub fn on_usage(&mut self, handler: H) -> &mut Self where H: Handler + Send + Sync + 'static, H::Scope: Send + Sync, { - // Meta系はデフォルトでスコープを開始しておく + // Meta handlers start with a scope by default. let mut wrapper = HandlerWrapper::new(handler); wrapper.start_scope(); self.usage_handlers.push(Box::new(wrapper)); self } - /// PingKind用のHandlerを登録 + /// Register a Handler for PingKind. pub fn on_ping(&mut self, handler: H) -> &mut Self where H: Handler + Send + Sync + 'static, @@ -437,7 +437,7 @@ impl Timeline { self } - /// StatusKind用のHandlerを登録 + /// Register a Handler for StatusKind. pub fn on_status(&mut self, handler: H) -> &mut Self where H: Handler + Send + Sync + 'static, @@ -449,7 +449,7 @@ impl Timeline { self } - /// ErrorKind用のHandlerを登録 + /// Register a Handler for ErrorKind. pub fn on_error(&mut self, handler: H) -> &mut Self where H: Handler + Send + Sync + 'static, @@ -461,7 +461,7 @@ impl Timeline { self } - /// TextBlockKind用のHandlerを登録 + /// Register a Handler for TextBlockKind. pub fn on_text_block(&mut self, handler: H) -> &mut Self where H: Handler + Send + Sync + 'static, @@ -472,7 +472,7 @@ impl Timeline { self } - /// ThinkingBlockKind用のHandlerを登録 + /// Register a Handler for ThinkingBlockKind. pub fn on_thinking_block(&mut self, handler: H) -> &mut Self where H: Handler + Send + Sync + 'static, @@ -483,7 +483,7 @@ impl Timeline { self } - /// ToolUseBlockKind用のHandlerを登録 + /// Register a Handler for ToolUseBlockKind. pub fn on_tool_use_block(&mut self, handler: H) -> &mut Self where H: Handler + Send + Sync + 'static, @@ -498,10 +498,10 @@ impl Timeline { // Event Dispatch // ========================================================================= - /// メインのディスパッチエントリポイント + /// Main dispatch entry point. pub fn dispatch(&mut self, event: &Event) { match event { - // Meta系: 即時ディスパッチ（登録順） + // Buffer usage until flush; dispatch other handled meta events immediately. Event::Usage(u) => self.dispatch_usage(u), Event::Ping(p) => self.dispatch_ping(p), Event::Status(s) => self.dispatch_status(s), @@ -509,7 +509,7 @@ impl Timeline { // Observability-only event: stream trace records it before timeline dispatch. Event::UnhandledSse(_) => {} - // Block系: スコープ管理しながらディスパッチ + // Block events: dispatch with scope management. Event::BlockStart(s) => self.handle_block_start(s), Event::BlockDelta(d) => self.handle_block_delta(d), Event::BlockStop(s) => self.handle_block_stop(s), @@ -517,9 +517,9 @@ impl Timeline { } } - /// Usage event を即時には dispatch せず、pending_usage にマージする。 - /// 1リクエスト内で複数の Usage event が来ても、ハンドラには 1 度だけ - /// 最終値を渡したいため。flush_usage() で発火する。 + /// Merge Usage events into pending_usage instead of dispatching immediately. + /// Even if a request emits multiple Usage events, handlers receive the final + /// values only once, when flush_usage() dispatches them. fn dispatch_usage(&mut self, event: &UsageEvent) { match &mut self.pending_usage { Some(acc) => merge_usage(acc, event), @@ -527,9 +527,9 @@ impl Timeline { } } - /// pending_usage を usage_handlers に発火し、バッファをクリアする。 - /// 1リクエスト分のストリーム終了時に1回だけ呼ぶ想定。 - /// pending_usage が空ならば何もしない。 + /// Dispatch pending_usage to usage_handlers and clear the buffer. + /// Intended to be called once at the end of a request's stream. + /// Does nothing if pending_usage is empty. pub fn flush_usage(&mut self) { if let Some(event) = self.pending_usage.take() { for handler in &mut self.usage_handlers { @@ -571,8 +571,8 @@ impl Timeline { fn handle_block_delta(&mut self, delta: &BlockDelta) { let block_type = delta.delta.block_type(); - // OpenAIなどのプロバイダはBlockStartを送らない場合があるため、 - // Deltaが来たときにその index のスコープがなければ暗黙的に開始する + // Providers such as OpenAI may omit BlockStart, so implicitly start a scope + // for this index when a Delta arrives without one. self.active_blocks .entry((block_type, delta.index)) .or_insert(()); @@ -580,7 +580,7 @@ impl Timeline { let handlers = self.get_block_handlers_mut(block_type); for handler in handlers { - // スコープがなければ暗黙的に開始 + // Implicitly start a scope if none exists. if !handler.has_scope() { handler.start_scope(); } @@ -624,19 +624,19 @@ impl Timeline { BlockType::Text => &mut self.text_block_handlers, BlockType::Thinking => &mut self.thinking_block_handlers, BlockType::ToolUse => &mut self.tool_use_block_handlers, - BlockType::ToolResult => &mut self.text_block_handlers, // ToolResultはTextとして扱う + BlockType::ToolResult => &mut self.text_block_handlers, // Treat ToolResult as Text. } } - /// 現在アクティブなブロックタイプを取得 + /// Get the currently active block type. pub fn current_block(&self) -> Option { self.current_block } - /// 現在アクティブな全ブロックを中断する + /// Abort all currently active blocks. /// - /// キャンセルやエラー時に呼び出し、index ごとに進行中の全ブロックへ - /// BlockAbortイベントを発火してスコープをクリーンアップする。 + /// Call on cancellation or error to emit BlockAbort events for all in-progress + /// blocks by index and clean up their scopes. pub fn abort_current_block(&mut self) { let active = self.active_blocks.keys().copied().collect::>(); for (block_type, index) in active { @@ -739,7 +739,7 @@ mod tests { #[test] fn test_meta_event_dispatch() { - // シンプルなテスト用構造体 + // Simple test struct struct TestUsageHandler { calls: Arc>>, } @@ -760,10 +760,10 @@ mod tests { timeline.on_usage(handler); timeline.dispatch(&Event::usage(100, 50)); - // pending_usage に積まれているだけなのでまだ未発火 + // Only buffered in pending_usage; not yet dispatched. assert_eq!(calls.lock().unwrap().len(), 0); - // flush で 1 度だけ発火 + // Flush dispatches once. timeline.flush_usage(); let recorded = calls.lock().unwrap(); assert_eq!(recorded.len(), 1); @@ -788,7 +788,7 @@ mod tests { calls: calls.clone(), }); - // Anthropic 風: message_start で input + 暫定 output + // Anthropic-style: input + provisional output in message_start. timeline.dispatch(&Event::Usage(UsageEvent { input_tokens: Some(409), output_tokens: Some(1), @@ -796,7 +796,7 @@ mod tests { cache_read_input_tokens: Some(0), cache_creation_input_tokens: Some(0), })); - // message_delta で最終 output + // Final output in message_delta. timeline.dispatch(&Event::Usage(UsageEvent { input_tokens: Some(409), output_tokens: Some(71), @@ -805,7 +805,7 @@ mod tests { cache_creation_input_tokens: Some(0), })); - // 未 flush の段階では発火しない + // No dispatch before flushing. assert_eq!(calls.lock().unwrap().len(), 0); timeline.flush_usage(); @@ -814,7 +814,7 @@ mod tests { assert_eq!(recorded[0].input_tokens, Some(409)); assert_eq!(recorded[0].output_tokens, Some(71)); - // flush 後にもう一度 flush しても何も起きない + // Flushing again after a flush does nothing. drop(recorded); timeline.flush_usage(); assert_eq!(calls.lock().unwrap().len(), 1); diff --git a/crates/agen/src/timeline/tool_call_collector.rs b/crates/agen/src/timeline/tool_call_collector.rs index 3663a8545..63a8d9b7f 100644 --- a/crates/agen/src/timeline/tool_call_collector.rs +++ b/crates/agen/src/timeline/tool_call_collector.rs @@ -1,7 +1,6 @@ -//! ToolCallCollector - ツール呼び出し収集用ハンドラ +//! ToolCallCollector - a handler for collecting tool calls. //! -//! TimelineのToolUseBlockHandler として登録され、 -//! ストリーム中のToolUseブロックを収集する。 +//! Registered as a ToolUseBlockHandler on Timeline to collect streaming ToolUse blocks. use crate::{ handler::{Handler, ToolUseBlockEvent, ToolUseBlockKind}, @@ -9,17 +8,17 @@ use crate::{ }; use std::sync::{Arc, Mutex}; -/// ToolUseブロックから収集したツール呼び出し情報を保持 +/// Holds tool call information collected from ToolUse blocks. /// -/// ToolCallCollectorのHandler実装で使用するスコープ型 +/// Scope type for ToolCallCollector's Handler implementation. #[derive(Debug, Default)] pub struct CollectorState { - /// 現在のツール呼び出し情報 (ブロック進行中) + /// Current tool call information (while the block is in progress). current_id: Option, current_name: Option, /// Zero-based order among tool calls in this assistant response. call_index: Option, - /// 蓄積中のJSON入力 + /// JSON input being accumulated. input_json_buffer: String, } @@ -30,19 +29,18 @@ pub(crate) struct CollectedToolCall { pub call: ToolCall, } -/// ToolCallCollector - ToolUseブロックハンドラ +/// ToolCallCollector - a ToolUse block handler. /// -/// Timelineに登録してToolUseブロックイベントを受信し、 -/// 完了したToolCallを収集する。 +/// Register with Timeline to receive ToolUse block events and collect completed ToolCalls. #[derive(Clone)] pub struct ToolCallCollector { - /// 収集されたToolCall + /// Collected ToolCalls. collected: Arc>>, next_call_index: Arc>, } impl ToolCallCollector { - /// 新しいToolCallCollectorを作成 + /// Create a new ToolCallCollector. pub fn new() -> Self { Self { collected: Arc::new(Mutex::new(Vec::new())), @@ -50,7 +48,7 @@ impl ToolCallCollector { } } - /// 収集されたToolCallを取得してクリア + /// Take the collected ToolCalls and clear the collection. pub fn take_collected(&self) -> Vec { self.take_collected_with_index() .into_iter() @@ -64,7 +62,7 @@ impl ToolCallCollector { std::mem::take(&mut *guard) } - /// 収集されたToolCallの参照を取得 + /// Access the collected ToolCalls. pub fn collected(&self) -> Vec { self.collected .lock() @@ -80,12 +78,12 @@ impl ToolCallCollector { *self.next_call_index.lock().unwrap() = 0; } - /// 収集されたToolCallがあるかどうか + /// Whether any ToolCalls have been collected. pub fn has_pending_calls(&self) -> bool { !self.collected.lock().unwrap().is_empty() } - /// 収集をクリア + /// Clear the collection. pub fn clear(&self) { self.collected.lock().unwrap().clear(); } @@ -116,7 +114,7 @@ impl Handler for ToolCallCollector { scope.input_json_buffer.push_str(delta); } ToolUseBlockEvent::Stop(_stop) => { - // ブロック完了時にToolCallを確定 + // Finalize the ToolCall when the block completes. if let (Some(id), Some(name)) = (scope.current_id.take(), scope.current_name.take()) { // A Stop event proves the provider finished the block, but @@ -161,13 +159,13 @@ mod tests { let mut timeline = Timeline::new(); timeline.on_tool_use_block(collector.clone()); - // ToolUseブロックのイベントシーケンスをディスパッチ + // Dispatch a sequence of ToolUse block events. timeline.dispatch(&Event::tool_use_start(0, "tool_123", "get_weather")); timeline.dispatch(&Event::tool_input_delta(0, r#"{"city":"#)); timeline.dispatch(&Event::tool_input_delta(0, r#""Tokyo"}"#)); timeline.dispatch(&Event::tool_use_stop(0)); - // 収集されたToolCallを確認 + // Check the collected ToolCalls. let calls = collector.take_collected(); assert_eq!(calls.len(), 1); assert_eq!(calls[0].id, "tool_123"); @@ -177,7 +175,7 @@ mod tests { #[test] fn test_collect_empty_buffer_returns_object() { - // 引数なしツール呼び出し: input_json_delta が一度も来ないケース + // A tool call without arguments: no input_json_delta arrives. let collector = ToolCallCollector::new(); let mut timeline = Timeline::new(); timeline.on_tool_use_block(collector.clone()); @@ -239,12 +237,12 @@ mod tests { let mut timeline = Timeline::new(); timeline.on_tool_use_block(collector.clone()); - // 1つ目のToolCall + // First ToolCall timeline.dispatch(&Event::tool_use_start(0, "call_1", "tool_a")); timeline.dispatch(&Event::tool_input_delta(0, r#"{"a":1}"#)); timeline.dispatch(&Event::tool_use_stop(0)); - // 2つ目のToolCall + // Second ToolCall timeline.dispatch(&Event::tool_use_start(1, "call_2", "tool_b")); timeline.dispatch(&Event::tool_input_delta(1, r#"{"b":2}"#)); timeline.dispatch(&Event::tool_use_stop(1)); diff --git a/crates/agen/src/token_counter.rs b/crates/agen/src/token_counter.rs index a91649197..0e6325bf9 100644 --- a/crates/agen/src/token_counter.rs +++ b/crates/agen/src/token_counter.rs @@ -1,41 +1,40 @@ -//! Usage 履歴ベースのトークン会計（汎用部分）。 +//! Token estimates based on provider measurements and history growth. //! -//! `UsageRecord` の列（プロバイダ実測値）と現在の history から、 -//! 任意の history index 時点のプロンプト全長トークン数を pure に計算する。 +//! Uses the full input token count from [`UsageRecord`], interpolating between +//! measured boundaries by byte count and extrapolating beyond the latest boundary +//! from text bytes. Input totals include cache reads rather than inventing a +//! system/history breakdown. //! -//! # 方針 -//! -//! - ローカルトークナイザは持たない。実測値があればそれを採用し、 -//! measurement 間はバイト数で按分、最新 measurement より先は byte/4 -//! fallback で外挿する -//! - 推定の出どころは [`EstimateSource`] で呼び出し側に明示する。 -//! 課金判断には使えないが、compact / prune / memory extract trigger 等の -//! 閾値判定には十分な精度 -//! - `records` は `history_len` 昇順を仮定する（呼び出し側がそのように積む） +//! [`EstimateSource`] distinguishes measurements from estimates. Estimates inform +//! threshold decisions; they neither determine billing nor guarantee that a request +//! fits the provider's context limit. Attachment base64 is a storage representation, +//! not text input, and is excluded from byte counts. Provider measurements for +//! requests that actually sent images remain valid. The caller supplies records in +//! ascending `history_len` order. use crate::{Item, UsageRecord}; -/// 推定の出どころ。 +/// Source of a token estimate. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum EstimateSource { - /// measurement の境界にちょうど一致（実測値そのもの） + /// Exact match with a measurement boundary; uses the measured value. Measured, - /// 連続する 2 つの measurement の間をバイト按分で計算 + /// Interpolated between consecutive measurements in proportion to byte count. Interpolated, - /// 最後の measurement より新しい区間を byte/4 fallback で外挿 + /// Extrapolated beyond the latest measurement using the bytes/4 fallback. Extrapolated, - /// measurement が 1 件も無く、バイト数のみのフォールバック + /// Byte-count fallback with no measurements available. NoData, } -/// トークン数の推定値。 +/// Estimated token count. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub struct TokenEstimate { pub tokens: u64, pub source: EstimateSource, } -/// `items[..i]` までの累積バイト数（`prefix[i]`）を返す。長さは `items.len()+1`。 +/// Returns cumulative bytes for `items[..i]` in `prefix[i]`, with length `items.len() + 1`. pub fn prefix_bytes(items: &[Item]) -> Vec { let mut prefix = Vec::with_capacity(items.len() + 1); let mut acc: u64 = 0; @@ -47,7 +46,7 @@ pub fn prefix_bytes(items: &[Item]) -> Vec { prefix } -/// 1 Item の text-shaped size。JSON シリアライズ長を使う粗い近似。 +/// Approximate text size of one Item, using its JSON serialization length. /// /// Binary attachments are deliberately projected out. Their durable base64 encoding is a /// storage representation, not provider text, so charging it at the text `bytes / 4` fallback @@ -63,11 +62,11 @@ pub fn item_bytes(item: &Item) -> u64 { .unwrap_or(0) } -/// `history[..index]` までのトークン数を推定する。 +/// Estimates the token count of `history[..index]`. /// -/// `prefix` は [`prefix_bytes`] で得た `history.len() + 1` 長の累積バイト列。 -/// 呼び出し側が 1 度だけ計算して使い回すことで、線形探索や複数回の推定が -/// O(n) シリアライズで済む（内部で毎回再計算すると O(n²) になる）。 +/// `prefix` contains `history.len() + 1` cumulative byte counts from [`prefix_bytes`]. +/// Computing it once and reusing it keeps serialization for a scan or repeated +/// estimates O(n), rather than O(n²) from recomputing it for every estimate. pub fn tokens_at( history: &[Item], records: &[UsageRecord], @@ -91,7 +90,7 @@ pub fn tokens_at( }; } - // exact match（rev 走査で一番新しい record を採用） + // Exact match: reverse iteration selects the most recent record. if let Some(r) = records.iter().rev().find(|r| r.history_len == index) { return TokenEstimate { tokens: r.input_total_tokens, @@ -153,15 +152,15 @@ pub fn tokens_at( } } -/// 現在の history 全体の推定トークン数。 +/// Estimated token count for the entire current history. pub fn total_tokens(history: &[Item], records: &[UsageRecord]) -> TokenEstimate { let prefix = prefix_bytes(history); tokens_at(history, records, history.len(), &prefix) } -/// 任意の history index 時点でのプロンプト全長推定。 -/// `history_len == 0` で 0 を返す。delta 計算 (extract trigger 等) で -/// `total_tokens_at(now) - total_tokens_at(pointer)` の形で使う。 +/// Estimates the full prompt token count at an arbitrary history boundary. +/// Returns zero for `history_len == 0`. Differences such as +/// `total_tokens_at(now) - total_tokens_at(pointer)` can drive extraction triggers. pub fn total_tokens_at( history: &[Item], records: &[UsageRecord], diff --git a/crates/agen/src/usage_record.rs b/crates/agen/src/usage_record.rs index 948b8cf08..bdb69a6a5 100644 --- a/crates/agen/src/usage_record.rs +++ b/crates/agen/src/usage_record.rs @@ -1,21 +1,20 @@ -//! Per-LLM-request Usage measurement snapshot. +//! Per-LLM-request usage measurement snapshot. //! -//! 1 リクエストの送信時点での「ある history prefix 長で計測した占有量」を -//! 1 件分にまとめたもの。`UsageEvent` (provider stream イベント) を -//! 受けて呼び出し側が組み立て、永続化層に流したり、token accounting -//! (`token_counter`) で履歴として参照したりする。 +//! Records measured token usage for the history prefix included in one request. +//! The caller builds it from provider `UsageEvent`s and passes it to persistence +//! or to token accounting (`token_counter`). -/// LLM リクエスト送信時点での占有量スナップショット。 +/// Usage snapshot for the history prefix sent with an LLM request. #[derive(Debug, Clone, PartialEq, Eq)] pub struct UsageRecord { - /// 送信時の history.len() + /// `history.len()` when the request was sent. pub history_len: usize, - /// history[..history_len] の占有量（プロンプト全長、実測） + /// Measured total prompt tokens for `history[..history_len]`. pub input_total_tokens: u64, - /// 上記のうちキャッシュから読み出された分 + /// Portion of the input tokens read from cache. pub cache_read_tokens: u64, - /// 上記のうちこのリクエストでキャッシュに書かれた分 + /// Portion of the input tokens written to cache by this request. pub cache_write_tokens: u64, - /// このリクエストで生成された出力トークン数 + /// Output tokens generated by this request. pub output_tokens: u64, } diff --git a/crates/agen/tests/reasoning_round_trip_test.rs b/crates/agen/tests/reasoning_round_trip_test.rs index 21e83f2a1..e100b911b 100644 --- a/crates/agen/tests/reasoning_round_trip_test.rs +++ b/crates/agen/tests/reasoning_round_trip_test.rs @@ -1,15 +1,11 @@ -//! Reasoning history round-trip 統合テスト +//! Integration tests for reasoning history round trips. //! -//! Engine のストリーム → history append → 次リクエスト送出までの -//! ライフサイクルで `Item::Reasoning` が脱落せず保持されることを確認する。 +//! Verifies that `Item::Reasoning` survives the lifecycle from Engine streaming +//! through history append to the next outgoing request. //! -//! 検証点: -//! - Anthropic 由来の thinking + signature が `Item::Reasoning::signature` として -//! history に残る -//! - OpenAI Responses 由来の reasoning text + summary + encrypted_content が -//! `Item::Reasoning` の各フィールドに展開される -//! - 直前の reasoning は次の outgoing request の `request.items` の先頭付近に -//! 含まれる（assistant メッセージの先頭、Anthropic 仕様） +//! Covers Anthropic thinking and signatures, OpenAI Responses reasoning text, +//! summaries and encrypted content, and ordering reasoning before other assistant +//! output as required by Anthropic. mod common; @@ -41,8 +37,8 @@ fn reasoning_block(text: impl Into, data: ReasoningBlockData) -> Vec>>, @@ -208,7 +204,7 @@ async fn injected_reasoning_survives_into_outgoing_request() { let mut engine = Engine::new(client); let mut history: History = History::new(); - // resume: 既存 history を流し込む + // Restore existing history before continuing. engine.set_history( &mut history, vec![ @@ -225,7 +221,7 @@ async fn injected_reasoning_survives_into_outgoing_request() { .unwrap() .take() .expect("client should have received a request"); - // Reasoning item が outgoing items に保持されていること + // The reasoning item must remain in the outgoing items. let mut found = false; for item in &req.items { if let Item::Reasoning { diff --git a/crates/agen/tests/transport_retry_test.rs b/crates/agen/tests/transport_retry_test.rs index 72a2fde1d..02d6b924b 100644 --- a/crates/agen/tests/transport_retry_test.rs +++ b/crates/agen/tests/transport_retry_test.rs @@ -1,7 +1,7 @@ -//! HTTP transport の単発 request / error classification テスト。 +//! Tests for single-attempt HTTP requests and error classification. //! -//! Retry/backoff は Engine の lifecycle 管理に属するため、transport は 1 回だけ -//! request を送り、HTTP status / Retry-After を `ClientError` に載せて返す。 +//! Retry and backoff belong to the Engine lifecycle. The transport sends one +//! request and returns the HTTP status and Retry-After information in `ClientError`. use agen::llm_client::LlmClient; use agen::llm_client::auth::AuthRequirement; @@ -17,8 +17,8 @@ use std::time::Duration; use wiremock::matchers::{method, path}; use wiremock::{Mock, MockServer, ResponseTemplate}; -/// SSE 本体は触らないテスト用 scheme。`parse_fail` を立てると -/// stream 消費中で `ClientError::Sse` を返す。 +/// Test scheme that does not inspect SSE payloads. With `parse_fail` set, +/// it returns `ClientError::Sse` while consuming the stream. #[derive(Clone)] struct DummyScheme { parse_fail: bool, diff --git a/crates/api-macros-impl/README.md b/crates/api-macros-impl/README.md new file mode 100644 index 000000000..f55dfb999 --- /dev/null +++ b/crates/api-macros-impl/README.md @@ -0,0 +1,7 @@ +# api-macros-impl + +Procedural macro implementation for `api-macros`. It validates and normalizes trait declarations before generating metadata and adapters. It does not own runtime HTTP handling or application authorization decisions. + +## Main entry points + +- [`api`] — declaration normalization and code generation diff --git a/crates/api-macros-impl/src/lib.rs b/crates/api-macros-impl/src/lib.rs index f012e66b6..655549a7a 100644 --- a/crates/api-macros-impl/src/lib.rs +++ b/crates/api-macros-impl/src/lib.rs @@ -1,4 +1,4 @@ -//! Implementation of `api-macros`. +#![doc = include_str!("../README.md")] use std::collections::{BTreeMap, BTreeSet}; @@ -15,6 +15,13 @@ use syn::{ spanned::Spanned, }; +/// Macro boundary that normalizes a trait before generating each output. +/// +/// Validate routes, operation IDs, and body/response constraints before generation. +/// Generate HTTP, WebSocket, and OpenAPI from one normalized model rather than interpreting +/// them independently, reducing differences between adapters for the same declaration. +/// Validation failures are compile errors; unsupported wire shapes are not implicitly +/// accepted as JSON. #[proc_macro_attribute] pub fn api(attr: TokenStream, item: TokenStream) -> TokenStream { let attr = proc_macro2::TokenStream::from(attr); diff --git a/crates/api-macros/README.md b/crates/api-macros/README.md new file mode 100644 index 000000000..7c8687e21 --- /dev/null +++ b/crates/api-macros/README.md @@ -0,0 +1,9 @@ +# api-macros + +Defines transport-independent API metadata from public traits. The same declarations drive optional HTTP adapters and OpenAPI generation, while callers retain authentication, authorization, and application state. + +## Main entry points + +- [`api`] — API declaration and validation +- [`ApiContract`], [`Operation`], and [`WebSocketOperation`] — typed operation contracts +- `axum`, `reqwest`, and `openapi` — adapters and document generation enabled by their respective features diff --git a/crates/api-macros/src/axum.rs b/crates/api-macros/src/axum.rs index 0376e0c08..e235031d9 100644 --- a/crates/api-macros/src/axum.rs +++ b/crates/api-macros/src/axum.rs @@ -1,9 +1,8 @@ -//! Axum support used by routers generated with `#[api(axum)]`. +//! Transport boundary for generated routers. //! -//! Generated operation routers contain only transport extraction, typed service invocation, and -//! response mapping. Authentication/authorization context, body-limit policy, timeout policy, and -//! other application state remain explicit outer Axum layers. Each operation is also exposed as a -//! separate router so route-specific layers can be applied before routers are merged. +//! Routers only extract inputs, call typed services, and convert responses. +//! Outer layers retain authentication, authorization, body limits, timeouts, and application state. +//! Per-operation routers allow route-specific constraints to be applied before merging. use std::{fmt, str::FromStr}; diff --git a/crates/api-macros/src/lib.rs b/crates/api-macros/src/lib.rs index 5d21d49e2..0c9bfa21f 100644 --- a/crates/api-macros/src/lib.rs +++ b/crates/api-macros/src/lib.rs @@ -1,147 +1,72 @@ -//! Shared API trait declarations for Yoi services. -//! -//! `#[api]` turns a public trait into a single, transport-neutral API definition. The -//! original trait remains usable by Rust code; generated metadata and zero-sized operation -//! markers give adapters and generators a stable view of the same declaration. -//! -//! # Declaration syntax -//! -//! ``` -//! # mod example { -//! use api_macros::api; -//! -//! pub struct CreateWidget; -//! pub struct Widget; -//! pub struct PublicError; -//! -//! #[api] -//! pub trait WidgetApi { -//! #[post( -//! "/widgets", -//! operation_id = "widgets.create", -//! status = 201, -//! error_status = 422 -//! )] -//! async fn create( -//! &self, -//! #[body] request: CreateWidget, -//! #[header("x-request-id")] request_id: String, -//! ) -> Result; -//! -//! #[get("/widgets/{widget_id}", operation_id = "widgets.get")] -//! async fn get(&self, widget_id: u64) -> Result; -//! } -//! # } -//! ``` -//! -//! Supported route attributes are `get`, `post`, `put`, `patch`, `delete`, `head`, and -//! `options`. The first argument is a path template. `operation_id` defaults to the Rust -//! method name but should be set explicitly for contracts which must remain stable while Rust -//! names evolve. Success status defaults to `200`, except an empty (`()`) response defaults to -//! `204`. `alternate_status` opts a JSON response into [`HttpSuccess`] when two successful HTTP -//! outcomes share one schema. Operations whose successful statuses have different body or header -//! shapes use `responses = [(status = 200, body = Widget, headers = [("etag", String)]), -//! (status = 304, headers = [("etag", String)])]`; the macro emits a typed result enum in the -//! `_responses` module. Only explicit body-less `304` declarations extend success -//! beyond 2xx. A public error inferred from `Result` defaults to status `400`; -//! `additional_error_statuses` publishes the same typed error schema for other declared outcomes. -//! `bearer_auth = true` and `browser_auth = true` attach standard bearer and browser-session cookie -//! security schemes. Body operations may opt into typed Axum rejection normalization with -//! typed Axum rejection normalization with `normalize_body_errors = true` and [`HttpRequestError`]. -//! -//! Arguments are classified with `#[body]`, `#[binary]`, `#[query]`, `#[header]`, `#[path]`, or -//! `#[extension]`. Extension values are trusted server-local Axum context: generated clients and -//! OpenAPI omit them. An unannotated argument whose Rust name occurs in the route template is -//! inferred as a path argument. Header attributes may carry a wire name, as in -//! `#[header("x-request-id")]`. Exactly one JSON or binary body is allowed. JSON request, response, -//! and error bodies must be named Rust types; tuples, references, arrays, and other anonymous -//! structural types are rejected. Binary request bodies use [`BinaryBody`] and remain byte-exact. -//! A named success response whose final type identifier is `BinaryBody` is likewise byte-exact, -//! uses `application/octet-stream`, and may include declared response headers. Generated clients -//! collect binary responses under the same configured response byte limit as JSON responses. -//! `openapi = false` explicitly excludes an operation whose wire body cannot yet satisfy the -//! strict OpenAPI schema boundary; Reqwest and Axum adapters are still generated. -//! -//! WebSocket routes use a declaration-only associated type rather than a dummy unary method: -//! -//! ``` -//! # mod websocket_example { -//! # use api_macros::api; -//! # pub struct ClientFrame; -//! # pub struct ServerFrame; -//! #[api] -//! pub trait EventApi { -//! #[websocket( -//! "/events/{stream_id}", -//! operation_id = "events.stream", -//! method = GET, -//! client_to_server = ClientFrame, -//! server_to_client = ServerFrame, -//! path_parameters = [stream_id: u64] -//! )] -//! type EventStream; -//! } -//! # } -//! ``` -//! -//! The declaration is removed from the emitted service trait and becomes a typed marker implementing -//! [`WebSocketOperation`]. It participates in the same operation/route inventory and path validation, -//! but never generates a Reqwest method, unary Axum adapter, or OpenAPI path. Under the `axum` -//! feature, [`axum::websocket_route`] mounts an existing manual upgrade handler at its declared GET -//! path without exposing framework WebSocket types to the contract. -//! -//! # Generated names -//! -//! For `WidgetApi`, the macro emits `WidgetApiMetadata`, which implements [`ApiContract`], -//! plus a `widget_api_operations` module. That module contains one marker per Rust method -//! (`Create`, `Get`, ...), each implementing [`Operation`] and connecting metadata to concrete -//! body types at compile time. Unusual method names which have no valid PascalCase form use the -//! deterministic `Operation` fallback. The `ApiContract::OPERATIONS` inventory is -//! sorted by operation ID, so its ordering is independent of source method order. -//! -//! Empty, JSON, and bounded binary request and success response bodies are supported. [`WireKind`] reserves explicit -//! variants for other future transport work; accepting one requires a deliberate macro and adapter -//! change rather than silently treating it as JSON. -//! -//! # OpenAPI 3.1 export -//! -//! With the `openapi` feature, `#[api(openapi)]` emits a `_openapi` function from the -//! same normalized operation model used by the HTTP adapters. The caller supplies title, version, -//! and an opaque source digest through [`openapi::OpenApiInfo`], then either serves -//! [`openapi::OpenApiDocument::to_json`] or writes exactly those canonical bytes with -//! [`openapi::OpenApiDocument::write_json`] from a CLI. Documents contain no server URLs, -//! environment names, timestamps, or other deployment topology. -//! -//! Every exposed wire type must implement [`openapi::OpenApiSchema`] as an explicit assertion that -//! its `schemars::JsonSchema` contract matches Serde serialization. The trait's naming hook is the -//! public component-name contract. Different schemas claiming one name, unsafe-width integer -//! schemas, and ambiguous non-null `anyOf`/untagged representations fail closed. Use bounded -//! integers (for example `u32`) for JSON numeric fields, and model optionality separately from -//! nullability through object `required` membership and nullable schemas. A normalized -//! `#[header("authorization")]` string (or optional string) is projected as an HTTP bearer -//! security scheme and operation requirement rather than as a raw header parameter. -//! -//! # Optional HTTP adapters -//! -//! `#[api(reqwest)]` generates `TraitNameClient` and enables a typed Reqwest client when this -//! crate's `reqwest` feature is enabled. `#[api(axum)]` generates `TraitNameAxum` plus one public -//! router per operation when the `axum` feature is enabled. `#[api(reqwest, axum)]` emits both -//! from the same normalized definition. A plain `#[api]` remains framework-independent and pulls -//! neither framework into DTO/contract-only builds. -//! -//! Generated clients use a fallible `try_new` constructor and builder. The builder validates an -//! absolute credential-free HTTP(S) base URL and configures request timeout and response byte -//! limit. A prevalidated [`reqwest::BaseUrl`] and policy-configured `reqwest::framework::Client` -//! may be injected instead. Request authorizers receive the final method, encoded path/query, and -//! exact serialized body bytes; credential headers and all body content are redacted from normal -//! diagnostics. -//! -//! Generated Axum adapters keep authentication, authorization, application state, body limits, -//! and timeout layers outside the adapter. Callers may layer the complete router or an individual -//! generated operation router before merging it into their application. +#![doc = include_str!("../README.md")] extern crate self as api_macros; +/// Normalize a public trait into a transport-independent, typed API contract. +/// +/// Preserve the original Rust trait and generate operation markers and metadata from the +/// same declaration. HTTP adapters and OpenAPI use the same normalized result, avoiding +/// manual synchronization of separate route lists. Sort the inventory by operation ID so +/// declaration reordering does not change the external contract order. +/// +/// Validate request/response wire shapes at declaration time; unsupported types and duplicate +/// routes are compile errors. Exclude server-local extensions from clients and OpenAPI; +/// leave authentication, authorization, body limits, and timeouts to the enclosing application. +/// Declarations without adapters do not depend on an HTTP framework. +/// +/// This declaration checks that a Rust trait and typed operation markers can be defined together. +/// +/// ``` +/// # mod example { +/// use api_macros::api; +/// +/// pub struct CreateWidget; +/// pub struct Widget; +/// pub struct PublicError; +/// +/// #[api] +/// pub trait WidgetApi { +/// #[post( +/// "/widgets", +/// operation_id = "widgets.create", +/// status = 201, +/// error_status = 422 +/// )] +/// async fn create( +/// &self, +/// #[body] request: CreateWidget, +/// #[header("x-request-id")] request_id: String, +/// ) -> Result; +/// +/// #[get("/widgets/{widget_id}", operation_id = "widgets.get")] +/// async fn get(&self, widget_id: u64) -> Result; +/// } +/// # } +/// ``` +/// +/// WebSocket associated types are declarations only: remove them from the output trait +/// and turn them into [`WebSocketOperation`] markers. They are not unary HTTP operations +/// and generate neither client methods nor OpenAPI paths. +/// +/// ``` +/// # mod websocket_example { +/// # use api_macros::api; +/// # pub struct ClientFrame; +/// # pub struct ServerFrame; +/// #[api] +/// pub trait EventApi { +/// #[websocket( +/// "/events/{stream_id}", +/// operation_id = "events.stream", +/// method = GET, +/// client_to_server = ClientFrame, +/// server_to_client = ServerFrame, +/// path_parameters = [stream_id: u64] +/// )] +/// type EventStream; +/// } +/// # } +/// ``` pub use api_macros_impl::api; #[cfg(feature = "axum")] diff --git a/crates/api-macros/src/openapi.rs b/crates/api-macros/src/openapi.rs index 6c08815f8..58fc70e82 100644 --- a/crates/api-macros/src/openapi.rs +++ b/crates/api-macros/src/openapi.rs @@ -1,8 +1,10 @@ -//! Deterministic OpenAPI 3.1 projection for `#[api(openapi)]` contracts. +//! Generate deterministic OpenAPI documents from API contracts. //! -//! The projection deliberately has no deployment topology (`servers`) and no -//! generated timestamps. Callers supply an opaque source digest, which is -//! retained as provenance in `info.x-yoi-source-digest`. +//! Omit deployment URLs and generation timestamps; retain the caller's source digest as +//! provenance. This separates contract changes from environment changes and produces the +//! same bytes from the same inputs. Owners explicitly assert wire-schema/Serde agreement +//! through [`OpenApiSchema`]; name collisions, unsafe integers, and ambiguous schemas are +//! not silently published. use std::{ borrow::Cow, diff --git a/crates/api-macros/src/reqwest.rs b/crates/api-macros/src/reqwest.rs index 9c8cf2664..136c953bc 100644 --- a/crates/api-macros/src/reqwest.rs +++ b/crates/api-macros/src/reqwest.rs @@ -1,8 +1,8 @@ -//! Reqwest transport support used by clients generated with `#[api(reqwest)]`. +//! Transport boundary for generated clients. //! -//! The generated builder validates the base URL before it can be stored. Callers that enforce -//! stricter egress policy (custom DNS resolution, proxy policy, TLS roots, or connection pinning) -//! can construct their own [`reqwest::Client`] and inject it with the validated [`BaseUrl`]. +//! Validate the base URL before storing it; apply timeouts and byte limits to requests +//! and response reads. Stricter outbound constraints such as DNS, proxy, and TLS policy +//! remain at the boundary where callers inject their own [`reqwest::Client`] with [`BaseUrl`]. use std::{fmt, time::Duration}; diff --git a/crates/builtin-source/README.md b/crates/builtin-source/README.md new file mode 100644 index 000000000..96b496422 --- /dev/null +++ b/crates/builtin-source/README.md @@ -0,0 +1,8 @@ +# builtin-source + +Catalog of public Profile sources embedded at compile time and shared across native/WASM. It exposes only explicit assets and does not discover sources from host files or the network. + +## Main entry points + +- [`BUILTIN_PROFILE_RESOURCES`] and [`BuiltinProfileResource`] — sources and import relationships +- [`builtin_profile_catalog_snapshot`] and [`BuiltinProfileCatalogSnapshot`] — fixed catalog contents and content identity diff --git a/crates/builtin-source/src/lib.rs b/crates/builtin-source/src/lib.rs index 3165a50d6..194859500 100644 --- a/crates/builtin-source/src/lib.rs +++ b/crates/builtin-source/src/lib.rs @@ -1,4 +1,5 @@ -//! Explicit compile-time public Decodal source catalog, shared by native and WASM hosts. +#![doc = include_str!("../README.md")] + use sha2::{Digest, Sha256}; use std::collections::BTreeMap; @@ -33,6 +34,11 @@ const DEFAULT_IMPORT: &[BuiltinProfileImport] = &[BuiltinProfileImport { }]; const NO_IMPORTS: &[BuiltinProfileImport] = &[]; +/// Fixed catalog of public embedded Profile sources and import relationships. +/// +/// Only the sources listed here are shared across native/WASM, not arbitrary resources. +/// Launch selectors identify entry points; paths identify assets for imports. +/// Publishing sources alone neither launches Workers nor grants permissions. pub const BUILTIN_PROFILE_RESOURCES: &[BuiltinProfileResource] = &[ BuiltinProfileResource { selector: None, @@ -150,6 +156,10 @@ pub const BUILTIN_PROFILE_RESOURCES: &[BuiltinProfileResource] = &[ }, ]; +/// Catalog snapshot fixing source bytes, selector entry points, and import relationships together. +/// +/// [`Self::digest`] identifies all ordered contents and the catalog chosen by the consumer. +/// It is not information for replacing values already evaluated in a Workspace with the current catalog. #[derive(Debug, Clone, PartialEq, Eq)] pub struct BuiltinProfileCatalogSnapshot { pub id: &'static str, diff --git a/crates/client/README.md b/crates/client/README.md index ddeb92fff..5a1dcf248 100644 --- a/crates/client/README.md +++ b/crates/client/README.md @@ -1,30 +1,10 @@ # client -## Role +Shared client for Backend Workspace/Runtime APIs and the Worker protocol. Provides authenticated communication and typed connection targets for CLI/TUI. The standalone crate owns Standalone execution, Runtime and the controller own Worker state and input admission, and higher-level crates own presentation and product commands. -`client` contains reusable socket-client and runtime-command mechanics for talking to Workers from CLI/TUI code. +## Implementation entry points -## Boundaries - -Owns: - -- one-shot Worker socket client behavior -- request/reply delivery mechanics -- runtime command construction below the product façade -- shared attach/status probing helpers used by higher layers - -Does not own: - -- product command names (`yoi`) -- Worker state authority (`worker`, `session-store` worker metadata) -- UI rendering (`tui`) -- Engine turn semantics (`agen`) - -## Design notes - -The client boundary lets `tui` and `yoi` share Worker communication without making library crates depend on the product binary. Socket clients should drain connect-time snapshot/alert traffic before sending a method or deciding status. - -## See also - -- [`../../docs/design/worker-session-state.md`](../../docs/design/worker-session-state.md) -- [`../../docs/design/overview.md`](../../docs/design/overview.md) +- [`BackendApiClient`] — Authenticated HTTP/WS requests bound to an origin +- [`backend_runtime`] / [`backend_workspace`] — Backend resource retrieval and operations +- [`Client`] / [`transport`] — Method/Event encoding and message transport +- [`target`] — Distinct Backend and Standalone targets and operation intents diff --git a/crates/client/src/backend_api.rs b/crates/client/src/backend_api.rs index 125812660..5333ce056 100644 --- a/crates/client/src/backend_api.rs +++ b/crates/client/src/backend_api.rs @@ -1,3 +1,14 @@ +//! Credentials and HTTP/WS requests bound to a Backend origin. +//! +//! BackendOrigin normalizes scheme/host/port and rejects inputs with a path, query, fragment, +//! or userinfo. Stored tokens are selected by origin; cross-origin redirects are rejected +//! to prevent credential leaks. WS requests use the same origin and bearer token. +//! +//! This client uses human Backend API tokens, not Workspace signing capabilities or Runtime +//! source proofs. It returns 401/403 as authentication/authorization failures without falling +//! back to a local Runtime or another token. Callers interpret typed results; transport success +//! does not imply completion of a domain operation. + use chrono::{DateTime, Utc}; use reqwest::{Method, StatusCode, Url, redirect}; use serde::{Deserialize, Serialize}; diff --git a/crates/client/src/backend_runtime.rs b/crates/client/src/backend_runtime.rs index 4f0c0f96c..ee5957e52 100644 --- a/crates/client/src/backend_runtime.rs +++ b/crates/client/src/backend_runtime.rs @@ -1,3 +1,13 @@ +//! Runtime Worker operations and observation through the Backend. +//! +//! Targets are Workspace/Runtime/Worker tuples sent to scoped Backend APIs. Runtime endpoints +//! and display names do not confer execution authority on the client. HTTP management reuses +//! server-api DTOs; protocol connections carry Method/Event messages over Backend WS. +//! +//! Restore sends the caller's fixed observation token/request ID unchanged. Response loss does +//! not imply rollback; there is no token replacement, automatic Restore, or reconnection to +//! the next execution. Initial snapshots and notices are observations, not input admission. + use crate::transport::websocket::{Socket as WebSocket, SocketError as WebSocketError}; use crate::{BackendApiClient, BackendApiClientError, Client}; pub use server_api::{ diff --git a/crates/client/src/backend_workspace.rs b/crates/client/src/backend_workspace.rs index 55f882111..4cdb53b93 100644 --- a/crates/client/src/backend_workspace.rs +++ b/crates/client/src/backend_workspace.rs @@ -1,3 +1,11 @@ +//! Generated client adapter for the Backend Workspace catalog. +//! +//! Workspace creation passes an operation key and optional initial Repository to the Server, +//! without local Git discovery or repository initialization. Empty Repository lists are valid. +//! The Server determines creation idempotency, ownership, and current-state replay; this client +//! keeps no separate creation ledger. It classifies 401/403 as shared Backend authentication +//! errors, distinct from other domain errors. + use crate::{BackendApiClient, BackendApiClientError}; use server_api::{ InitialRepositoryIntent, RepositorySummary, WorkspaceCreateRequest, WorkspaceCreateResponse, diff --git a/crates/client/src/client.rs b/crates/client/src/client.rs index bdd733758..268b9b32a 100644 --- a/crates/client/src/client.rs +++ b/crates/client/src/client.rs @@ -6,7 +6,12 @@ use protocol::{Event, Method}; use crate::transport::Socket; -/// Typed Worker protocol client over an injected message transport. +/// Typed Worker protocol client over a message transport. +/// +/// Encodes and sends Methods and decodes received messages into Events. Successful sends +/// indicate transport delivery, not controller command acknowledgments or durable Submit/Notify +/// receipts. Callers interpret events; connection snapshots are not consumed automatically +/// and state is not inferred. pub struct Client { socket: T, } diff --git a/crates/client/src/lib.rs b/crates/client/src/lib.rs index 53e0392dc..1da118a17 100644 --- a/crates/client/src/lib.rs +++ b/crates/client/src/lib.rs @@ -1,7 +1,4 @@ -//! Backend Workspace/Runtime と既存 Worker protocol へ接続するクライアント。 -//! -//! Standalone execution is owned by the `standalone` crate and does not spawn -//! a Worker subprocess through this crate. +#![doc = include_str!("../README.md")] pub mod backend_api; mod backend_auth; diff --git a/crates/client/src/target.rs b/crates/client/src/target.rs index bc9a861aa..f27953603 100644 --- a/crates/client/src/target.rs +++ b/crates/client/src/target.rs @@ -1,3 +1,10 @@ +//! Distinct Backend and Standalone targets and product operation intents. +//! +//! Backend targets resolve an origin and Workspace and identify Workers by Runtime/Worker pairs. +//! Standalone spawn/list/resume are intents passed to higher-level crates, not execution; +//! this module starts no subprocesses. Target identity is separate from product commands +//! and Worker state authority. + use std::{fmt, path::PathBuf}; use crate::{ diff --git a/crates/client/src/transport/mod.rs b/crates/client/src/transport/mod.rs index 12d5e6d26..bab463ad5 100644 --- a/crates/client/src/transport/mod.rs +++ b/crates/client/src/transport/mod.rs @@ -1,3 +1,8 @@ +//! Message transport for one Worker connection. +//! +//! Separates in-process, Unix socket, and WS framing from protocol Method/Event encoding. +//! Changing transports does not change ownership of Worker execution, input admission, or receipts. + use std::error::Error; use async_trait::async_trait; @@ -6,10 +11,9 @@ pub mod in_process; pub mod unix_socket; pub mod websocket; -/// Message-oriented transport for one Worker protocol connection. +/// UTF-8 message transport for a Worker protocol connection. /// -/// Implementations own physical framing. `client::Client` owns the typed -/// Method/Event protocol encoding layered on top of these UTF-8 messages. +/// Implementations own physical framing; the higher-level Client encodes and decodes Method/Event. #[async_trait] pub trait Socket { type Error: Error + Send + Sync + 'static; diff --git a/crates/config-source-wasm/README.md b/crates/config-source-wasm/README.md new file mode 100644 index 000000000..4058ea322 --- /dev/null +++ b/crates/config-source-wasm/README.md @@ -0,0 +1,9 @@ +# config-source-wasm + +WASM adapter exposing `config-source` snapshot evaluation and editing support to JavaScript. It handles browser-local edit candidates without changing Backend save permissions or persistent state. + +## Main entry points + +- [`set_snapshot`] and [`apply_changes`] — update local edit candidates +- [`evaluate_snapshot`] and [`evaluate_current`] — evaluation through the shared implementation +- [`analyze_snapshot`] and [`complete_current`] — diagnostics and completion diff --git a/crates/config-source-wasm/src/lib.rs b/crates/config-source-wasm/src/lib.rs index df335ab00..eed2a09b4 100644 --- a/crates/config-source-wasm/src/lib.rs +++ b/crates/config-source-wasm/src/lib.rs @@ -1,3 +1,5 @@ +#![doc = include_str!("../README.md")] + use config_source::{ ConfigSchemaContribution, ConfigTreeChange, ConfigTreeSnapshot, EvaluationResult, SnapshotEnvironment, ToolchainContract, VirtualPath, WorkspaceConfigSchemaBundle, @@ -17,6 +19,10 @@ pub fn compose_schema_bundle(contributions: JsValue) -> Result encode(WorkspaceConfigSchemaBundle::compose(contributions).map_err(js_error)?) } +/// Retain the browser's edit candidate as a snapshot. +/// +/// Deserialization uses shared digest validation but neither saves to nor authorizes the Backend. +/// Keep the schema bundle in separate state to distinguish sources from evaluation conditions. #[wasm_bindgen] pub fn set_snapshot(snapshot: JsValue) -> Result<(), JsValue> { let snapshot: ConfigTreeSnapshot = decode(snapshot)?; @@ -85,6 +91,10 @@ struct WasmCompletionItem { priority: i32, } +/// Complete after converting the editor's UTF-16 offset to the shared analyzer's UTF-8 byte offset. +/// +/// Convert the returned start position back to UTF-16 so non-ASCII characters do not +/// misalign editor and analyzer positions. Reject offsets that cannot map to a character boundary. #[wasm_bindgen] pub fn complete_current( entrypoint: String, diff --git a/crates/config-source/README.md b/crates/config-source/README.md new file mode 100644 index 000000000..aa1b345dd --- /dev/null +++ b/crates/config-source/README.md @@ -0,0 +1,9 @@ +# config-source + +Shares the virtual Workspace configuration source tree, Decodal evaluation, diagnostics, and completion across native/WASM. Inputs are content-addressed snapshots, not a mechanism for reading host configuration directories. Callers own persistence, save-time authorization, and whole-tree CAS. + +## Main entry points + +- [`VirtualPath`] and [`ConfigTreeSnapshot`] — virtual paths and source-change validation +- [`WorkspaceConfigSchemaBundle`] and [`ToolchainContract`] — evaluation-condition identity +- [`SnapshotEnvironment`] and [`SnapshotImportLoader`] — evaluation and closed import resolution diff --git a/crates/config-source/src/lib.rs b/crates/config-source/src/lib.rs index b65c2971d..a2d891b24 100644 --- a/crates/config-source/src/lib.rs +++ b/crates/config-source/src/lib.rs @@ -1,3 +1,5 @@ +#![doc = include_str!("../README.md")] + use std::collections::{BTreeMap, BTreeSet}; use std::fmt; use std::sync::{Arc, Mutex}; @@ -37,6 +39,11 @@ impl<'de> Deserialize<'de> for VirtualPath { } } +/// Relative path identifying an entry in the virtual source tree, not a host path. +/// +/// Reject absolute paths, URLs, empty components, `.` and `..`. [`resolve_import`] resolves +/// relative imports against the containing virtual directory, independent of the storing +/// Backend's cwd or OS directory structure. impl VirtualPath { pub fn parse(value: impl AsRef) -> Result { let value = value.as_ref(); @@ -299,6 +306,15 @@ impl<'de> Deserialize<'de> for ConfigTreeSnapshot { } } +/// Virtual tree identified by the contents of author-saved sources. +/// +/// Keep entry points, import targets, and text assets in one tree. Evaluated Profiles and prompts +/// do not replace these sources. The digest identifies paths, content types, and complete bytes, +/// not edit counts or freshness. External deserialization also verifies contents against the digest. +/// +/// [`Self::apply`] creates a candidate without changing the original snapshot. Save permissions, +/// persistence, and conflict detection with the UI or multiple Workers lie outside this type; +/// callers must check the authoritative tree's whole digest and evaluation contract at the save boundary. impl ConfigTreeSnapshot { pub fn empty() -> Self { Self::from_entries(Vec::new()).expect("empty config snapshot is valid") @@ -403,9 +419,11 @@ impl ConfigTreeSnapshot { changes } - /// Apply an atomic batch. Each expected_digest compares the target's complete - /// source bytes only; it is not a tree precondition. Callers needing whole-tree - /// CAS must also compare the authoritative tree digest under their write lock. + /// Check source-byte preconditions only and return a candidate for the entire change batch. + /// + /// The original snapshot remains unchanged even if a change fails. `expected_digest` compares + /// only entry contents, not whole-tree CAS. Saving must separately compare the authoritative + /// whole-tree digest under a write lock or within a transaction. pub fn apply(&self, changes: &[ConfigTreeChange]) -> Result { self.validate()?; if changes.len() > MAX_CHANGE_COUNT { @@ -591,6 +609,11 @@ impl ConfigSchemaContribution { }) } + /// Optional editor shape for completion and non-materializing constraint analysis. + /// + /// Evaluate saved projections with `source`. Do not fill omitted values from this shape or + /// require omitted fields in partial Profiles. Include its contents in the bundle fingerprint + /// to identify the combination of storage schema and editor shape. pub fn with_authoring_source(mut self, source: impl Into) -> Self { self.authoring_source = Some(source.into()); self @@ -750,6 +773,12 @@ pub struct ToolchainContract { pub fingerprint: String, } +/// Contract identifying the schema, toolchain, and import policy used to evaluate a source. +/// +/// Separate source digests from evaluation conditions: a schema update is not the same +/// evaluation even when source bytes are unchanged. Persistence must manage the sources, +/// this contract, and the projection together. This type neither saves configuration nor +/// applies it to existing Workers. impl ToolchainContract { pub fn new( schema_version: u32, @@ -1111,6 +1140,13 @@ fn pop_container( } } +/// Closed environment for evaluating and analyzing virtual source snapshots. +/// +/// Separate input sources from schema contracts; return evaluation results as JSON projections +/// and their digest. Evaluation does not discover configuration sources in host files, +/// environment variables, or the network. Native and WASM share this implementation, so editor +/// analysis does not use a separate storage schema. Constraint analysis of editor shapes is +/// non-materializing and does not add defaults to saved projections. #[derive(Debug, Clone)] pub struct SnapshotEnvironment { snapshot: ConfigTreeSnapshot, @@ -1141,8 +1177,11 @@ impl SnapshotEnvironment { self.evaluate_with_builtin_sources(contract, Arc::default()) } - /// Capture only builtin sources actually loaded by evaluation, including transitive imports. - /// These immutable bytes are provenance for the evaluated value, not Runtime inputs. + /// Retrieve original bytes for transitive builtin dependencies actually loaded during evaluation. + /// + /// This is provenance for evaluated values, not an instruction to reevaluate in the Runtime. + /// Saving and launching can pin values and dependency sources by digest, avoiding implicit + /// replacement with current builtins. Archive construction and validation happen outside this type. pub fn builtin_import_sources( &self, contract: &ToolchainContract, @@ -1597,6 +1636,19 @@ impl HostEnvironment for ConfigAuthoringEnvironment<'_> { } } +/// Import loader returning only snapshot sources and explicitly published builtins. +/// +/// Resolve relative imports against the containing virtual directory; keep relative imports +/// within builtins in the builtin namespace. Do not use host paths to supply missing entries +/// or compensate for namespace escapes. Builtins are read-only; same-name Workspace entries +/// cannot override them. +/// +/// Check import-graph cycles and depth before loading, including imports in lazy fields. +/// Builtins share the snapshot's entry-count and byte budgets. Count repeated loads of the +/// same builtin once and include unused Workspace entries in the budget. +/// Use virtual paths for diagnostics and relative resolution. Engine cache identities also +/// include interpretation and content digests, so changed content or content types cannot +/// be confused with previous cached entries. #[derive(Debug, Clone)] pub struct SnapshotImportLoader { snapshot: ConfigTreeSnapshot, diff --git a/crates/feature-storage/README.md b/crates/feature-storage/README.md index 9defd6a75..1ee4d9680 100644 --- a/crates/feature-storage/README.md +++ b/crates/feature-storage/README.md @@ -1,28 +1,11 @@ # feature-storage -Neutral Host-owned SQLite lifecycle manager for trusted Features. Owns placement, -WAL/connection settings, transaction serialization, migrations, snapshots, -restore/delete fencing, and shutdown. Features receive a managed -`FeatureDatabase` after registration. Worker/model input never selects a database -path or receives SQL access. +Manages the SQLite lifecycle for Features within a trusted Host. +Does not own Feature domain schemas or authentication, or provide SQL access to Workers or models. -Use `FeatureStorage::new(trusted_root)`, `register(FeatureRegistration)`, and -`scope(host_issued_scope_id).open(®istration)`. `ScopedFeatureStorage` and -`FeatureDatabase::scope_id()` are neutral storage isolation APIs: a local Host's -persisted random scope does not create a Workspace record or imply Backend -membership. Existing placement `//features/.sqlite` and -snapshot `workspace=` keys remain unchanged for physical compatibility. +## Main entry points -`WorkspaceFeatureStorage`, `workspace()`, `workspace_id()`, and -`for_server_database()` are Backend compatibility aliases/helpers. They do not -introduce a Server dependency. The Host is responsible for state-root privacy, -symlink/path validation, and external execution leases before constructing this -trusted manager. - -`configure_connection(&rusqlite::Connection)` is reusable by other Host stores -(such as JobStore). It sets the five-second busy timeout, foreign keys ON, WAL, -and synchronous FULL before migrations/transactions. The Feature manager uses -this same helper; WAL lock-contention retries stay within the busy timeout. - -Dependencies: `rusqlite` (existing workspace backup support), `thiserror`, `uuid`; -tests use `tempfile`. +- [`FeatureStorage`], [`ScopedFeatureStorage`]: registration, scope isolation, storage, and shutdown +- [`FeatureDatabase`]: connections and transactions +- [`FeatureMigration`]: Feature-owned schema changes +- [`configure_connection`]: shared SQLite connection policy diff --git a/crates/feature-storage/src/lib.rs b/crates/feature-storage/src/lib.rs index 44d0dc93e..b5f34fe40 100644 --- a/crates/feature-storage/src/lib.rs +++ b/crates/feature-storage/src/lib.rs @@ -1,9 +1,4 @@ -//! Host-owned SQLite storage for trusted Features. -//! -//! The Host owns database placement, connection configuration, migrations, and -//! operational lifecycle. A Feature owns its schema and only receives a scoped -//! [`FeatureDatabase`] after trusted registration; Worker/model inputs never -//! select a path or receive a SQL surface. +#![doc = include_str!("../README.md")] use std::collections::{HashMap, HashSet}; use std::fs; @@ -79,6 +74,9 @@ impl FeatureStorageError { } } +/// Feature-owned schema changes. Commits pending changes and history in one IMMEDIATE transaction. +/// Failure rolls back both changes and history, allowing retry on the next open. +/// Rejects stored versions/names that differ from the plan and schemas newer than the plan. #[derive(Clone, Copy)] pub struct FeatureMigration { version: u32, @@ -135,6 +133,12 @@ impl RegisteredFeature { } } +/// Manages physical layout, connections, migrations, and shutdown within a trusted Host. +/// +/// Features own schemas and domain transactions; the Host owns authentication before calls. +/// This separation is an in-process API boundary, not a sandbox for arbitrary native code. +/// The constructing Host owns state-root protection, path validation, and execution leases. +/// Registration alone creates no database and does not delete existing data for disabled Features. #[derive(Clone)] pub struct FeatureStorage { inner: Arc, @@ -155,8 +159,8 @@ struct FeatureStorageState { } impl FeatureStorage { - /// Creates a lazy manager. No directories or databases are created until a - /// registered Feature is opened, restored, or explicitly backed up. + /// Lazy manager. Creates no directories or databases until a registered + /// Feature is opened, restored, or backed up. pub fn new(root: impl Into) -> Self { Self { inner: Arc::new(FeatureStorageInner { @@ -184,8 +188,9 @@ impl FeatureStorage { Ok(Self::new(parent.join("feature-storage"))) } - /// Binds trusted storage to a Host-issued opaque scope. This is a storage - /// isolation key, not a Workspace record or model authorization credential. + /// Restricts storage to a Host-issued scope. + /// A scope is a physical sharing boundary; it creates no Workspace record or model authorization credentials. + /// The same API supports Backend Workspace IDs and persistent local Host scope IDs. pub fn scope(&self, scope_id: &str) -> Result { Ok(ScopedFeatureStorage { manager: self.clone(), @@ -198,6 +203,8 @@ impl FeatureStorage { self.scope(scope_id) } + /// Registers a schema once per manager. The returned registration is reusable across scopes. + /// Rejects duplicate and foreign-manager registrations, fixing the migration plan used on open. pub fn register(&self, registration: FeatureRegistration) -> Result { let feature_id = validate_identifier("Feature", registration.feature_id)?.to_string(); validate_migrations(&feature_id, registration.migrations)?; @@ -215,8 +222,8 @@ impl FeatureStorage { }) } - /// Closes every managed connection after any operation currently holding a - /// connection lock completes. New opens and operations then fail. + /// Stops admission, waits for operations holding connection locks, and closes all managed connections. + /// Retained handles also fail thereafter, so writes after shutdown cannot be treated as successful. pub fn shutdown(&self) -> Result<()> { let connections = { let mut state = self.lock_state()?; @@ -471,6 +478,7 @@ impl Drop for FeatureStorageInner { } } +/// Feature storage restricted to a Host scope. Scope identity denotes only a storage boundary. #[derive(Clone)] pub struct ScopedFeatureStorage { manager: FeatureStorage, @@ -494,26 +502,33 @@ impl ScopedFeatureStorage { self.manager.register(registration) } + /// Reuses a managed connection, validating and applying migrations on first open. + /// Serializes concurrent opens within a manager; SQLite serializes migration writers across managers. + /// Syncs metadata-directory ancestors before opening SQLite. DB/WAL sync alone cannot + /// persist links to newly created ancestors, so a sync failure prevents connection registration. pub fn open(&self, registration: &RegisteredFeature) -> Result { self.manager.open(&self.scope_id, registration) } - /// Creates a consistent online SQLite snapshot of every Feature database, - /// including data for Features that are currently disabled/unregistered. + /// Backs up each Feature DB, including disabled/unregistered ones, using SQLite online backup. + /// Avoids file copies that omit WAL; checks integrity and foreign keys before publishing by rename. + /// Each DB is internally consistent, but no common-point snapshot across DBs or Host authority is guaranteed. + /// Blobs outside the DB are not saved, so this is not a complete Drive backup. pub fn backup(&self, destination: impl AsRef) -> Result<()> { self.manager .backup_scope(&self.scope_id, destination.as_ref()) } - /// Restores a snapshot only into an absent scope storage directory and - /// only while no Feature database for this scope is open. + /// Restores a snapshot only into an uncreated scope with no connections. + /// Rejects scope mismatches and corrupt DBs; publishes the validated temporary directory by rename. pub fn restore(&self, snapshot: impl AsRef) -> Result<()> { self.manager .restore_scope(&self.scope_id, snapshot.as_ref()) } - /// Closes this scope's Feature connections and removes only its scoped - /// storage tree. This is the operation used by Workspace deletion. + /// Closes connections in the scope and deletes only its storage tree. Retries are idempotent. + /// Keeps the same manager's admission fence after physical deletion, preventing cached tasks + /// from recreating databases before the Host finalizes deletion. Host deletion reservation is a separate responsibility. pub fn delete(&self) -> Result<()> { self.manager.delete_scope(&self.scope_id) } @@ -523,6 +538,9 @@ impl ScopedFeatureStorage { } } +/// Host-managed synchronous SQLite connection. Async callers use a blocking executor. +/// Domain repositories retain this handle and never pass connections or SQL to Workers/Runtimes. +/// Locks serialize operations and shutdown on each connection; deletion/shutdown fences are checked before and after locking. #[derive(Clone)] pub struct FeatureDatabase { inner: Arc, @@ -584,8 +602,9 @@ impl FeatureDatabase { operation(connection) } - /// Executes a Feature-owned multi-table operation atomically. Returning an - /// error rolls the transaction back; successful return commits it. + /// Updates multiple tables within one Feature DB in an IMMEDIATE transaction. + /// Commits after the closure succeeds; closure or commit failures never return success. + /// Does not guarantee atomic writes with the Host control-plane DB or other Feature DBs. pub fn transaction( &self, operation: impl FnOnce(&Transaction<'_>) -> Result, @@ -593,8 +612,8 @@ impl FeatureDatabase { self.try_transaction(operation) } - /// Variant of [`Self::transaction`] that preserves a Feature repository's - /// typed domain errors while still rolling every failure back atomically. + /// Preserves the rollback, connection ownership, and fences of [`Self::transaction`], + /// returning typed Feature domain errors without converting them to storage strings. pub fn try_transaction( &self, operation: impl FnOnce(&Transaction<'_>) -> std::result::Result, @@ -742,11 +761,10 @@ fn sync_directory(path: &Path) -> std::io::Result<()> { fs::File::open(path)?.sync_all() } -/// Applies the common durable Host SQLite connection policy: five-second busy -/// timeout, foreign keys, WAL, and FULL synchronous commits. Call immediately -/// after opening a file-backed database, before transactions or migrations. -/// WAL initialization retries lock contention within the same bounded timeout. -/// This does not grant path authority or select a database schema. +/// Applies the shared SQLite connection policy before migrations or transactions. +/// Enables foreign keys, WAL, and FULL synchronous commits, with bounded lock-contention waits. +/// Retries WAL initialization within the same wait limit and rejects silent fallback to another journal mode. +/// Grants neither path-selection authority nor domain schemas. pub fn configure_connection(connection: &Connection) -> Result<()> { connection.busy_timeout(BUSY_TIMEOUT)?; connection.execute_batch( diff --git a/crates/flow/README.md b/crates/flow/README.md new file mode 100644 index 000000000..517785458 --- /dev/null +++ b/crates/flow/README.md @@ -0,0 +1,10 @@ +# flow + +`flow` compiles DCDL state graphs into typed definitions and evaluates transitions against pinned definitions and verification results. Callers own Workspace source persistence, Worker Session writes, and internal verifier execution. This crate implements neither role-specific work loops nor state-entry side effects. + +## Main entry points + +- [`compile_flow_source`](src/definition.rs): Source evaluation and graph validation. +- [`FlowSelector`](src/selector.rs) / [`ResolvedFlowSource`](src/selector.rs): Explicit source identity and immutable snapshots. +- [`FlowRuntimeState`](src/coordinator.rs): Instances, transition attempts, and ordered events. +- [`builtin_flow_source`](src/builtin.rs): Read-only catalog of embedded sources. diff --git a/crates/flow/src/builtin.rs b/crates/flow/src/builtin.rs index 2514cd30a..5affc50c3 100644 --- a/crates/flow/src/builtin.rs +++ b/crates/flow/src/builtin.rs @@ -1,3 +1,9 @@ +//! Read-only catalog of embedded Flow sources. +//! +//! Provides exact source bytes without copying them into the Workspace DB or +//! maintaining manual update counters. The Runtime pins the compiled digest, +//! so resource updates do not change existing instance definitions. + use crate::{CompiledFlowDefinition, FlowCompileError, compile_flow_source}; pub const CODER_REVIEW_FLOW_SLUG: &str = "coder-review"; diff --git a/crates/flow/src/coordinator.rs b/crates/flow/src/coordinator.rs index 295c5e116..606f53e49 100644 --- a/crates/flow/src/coordinator.rs +++ b/crates/flow/src/coordinator.rs @@ -1,3 +1,18 @@ +//! Builds transition attempts from pinned Flow definitions and chooses destinations +//! from complete verdict sets. +//! +//! Callers do not specify a target state: an attempt pins every outgoing condition +//! of the current state. Each condition needs exactly one result, with no duplicate, +//! missing, or unknown IDs. Entry requires exactly one Met result and no +//! Indeterminate results. Verifier failures, cancellations, and incomplete responses +//! are not successful transitions. +//! +//! Checks the instance, active attempt, definition digest, and captured state so old +//! verifier results cannot apply to new attempts, even after returning to the same +//! state. Event sequences track append order, not freshness or source identity. +//! This module performs typed state transformations; callers own persistence, +//! verifier startup, and business side effects. + use std::collections::{BTreeMap, BTreeSet}; use serde::{Deserialize, Serialize}; @@ -198,11 +213,13 @@ pub struct FlowRuntimeEvent { pub event: FlowEventKind, } -/// Durable Flow authority owned by one Runtime Worker. +/// Flow state durably owned by one Runtime Worker. /// -/// Workspace authority resolves the exact source content, but never mutates -/// this value. Runtime persists the complete snapshot with the Worker session -/// and replaces it only after the corresponding session-log write succeeds. +/// Pins the resolved definition and digest to the instance, so source edits do not +/// change existing instances. Callers prepare new state on a clone and replace live +/// state only after a successful Session write. Recover verifying attempts after +/// restoration rather than creating competing attempts. Events reject reused attempt +/// IDs even after failure/cancellation. Stopping a Worker does not transfer state. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct FlowRuntimeState { pub definition: CompiledFlowDefinition, @@ -253,8 +270,7 @@ impl FlowRuntimeState { )) } - /// Begin a new attempt, or return the persisted active attempt after a - /// Runtime/Worker restore. A recovered attempt is never rewritten. + /// Starts a new attempt or returns the restored active attempt unchanged. pub fn begin_or_recover_transition( &mut self, attempt_id: impl Into, diff --git a/crates/flow/src/definition.rs b/crates/flow/src/definition.rs index 8849e7b8c..f56b11cfe 100644 --- a/crates/flow/src/definition.rs +++ b/crates/flow/src/definition.rs @@ -1,3 +1,19 @@ +//! Compiles DCDL sources into typed Flow definitions. +//! +//! The source document is the graph authority; states and transitions are not split +//! into independently editable records. DCDL evaluation passes through private +//! Serde-compatible values into the source schema, then graph validation produces +//! a [`CompiledFlowDefinition`]. Intermediate JSON is neither a public API type nor +//! a persisted Runtime state type. Imports are rejected, and a single document's +//! content digest pins the execution definition. +//! +//! Validates schemas, identifiers, and targets; rejects unreachable states, closed +//! paths that cannot reach a user-defined terminal, outgoing terminal transitions, +//! and nonterminal states without transitions. Authored `$cancelled` states are +//! forbidden; synthetic cancellation transitions are excluded from normal-terminal +//! reachability checks. Only validated definitions receive a `$cancelled` terminal, +//! so exceptional cancellation cannot make an invalid normal-completion graph pass. + use std::collections::{BTreeMap, BTreeSet, VecDeque}; use std::fmt; use std::fmt::Write as _; diff --git a/crates/flow/src/lib.rs b/crates/flow/src/lib.rs index 2093c090a..72375c04a 100644 --- a/crates/flow/src/lib.rs +++ b/crates/flow/src/lib.rs @@ -1,8 +1,4 @@ -//! Declarative Flow definitions and deterministic transition coordination. -//! -//! DCDL is a source format only. The Flow domain owns the typed source -//! schema, semantic validation, immutable transition snapshots, and state -//! transition rules. +#![doc = include_str!("../README.md")] mod builtin; mod coordinator; diff --git a/crates/flow/src/selector.rs b/crates/flow/src/selector.rs index 6e86c185a..554ffa6dd 100644 --- a/crates/flow/src/selector.rs +++ b/crates/flow/src/selector.rs @@ -1,3 +1,10 @@ +//! Explicit Flow source ownership and immutable source snapshots. +//! +//! `builtin:` and `workspace:` identify distinct logical records and may +//! coexist with the same slug. There is no implicit override order or unqualified +//! selector. Source resolution is separate from execution instance creation and +//! mutation; the Runtime pins the resolved result before execution. + use std::fmt; use std::str::FromStr; @@ -114,11 +121,10 @@ pub struct FlowSourceResolveRequest { pub selector: FlowSelector, } -/// Immutable source snapshot resolved by Workspace authority for one Runtime. +/// Immutable source snapshot resolved by the Workspace authority. /// -/// This is read-only source authority. Starting or mutating a Flow instance is -/// deliberately not part of this response; Runtime persists the snapshot in -/// the target Worker's durable state before execution. +/// This read-only source authority creates no instances, attempts, or events. +/// The Runtime commits the snapshot to the target Worker's Session before execution. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] pub struct ResolvedFlowSource { pub selector: FlowSelector, diff --git a/crates/fs-operation/README.md b/crates/fs-operation/README.md new file mode 100644 index 000000000..b59ac16c2 --- /dev/null +++ b/crates/fs-operation/README.md @@ -0,0 +1,11 @@ +# fs-operation + +Provides provider-side filesystem operations, bounded search, and pure text editing. Access checks and I/O for logical paths go through the provider's `FsAccessPolicy`. Does not own Workdir identity, Runtime transport, or Tool read history. + +## Reference index + +- [`FsAccessPolicy`] — provider access, I/O, and resource limits. +- [`CheckedTarget`] — checked saves retaining descriptors and identity. +- [`run_list`] — bounded live pagination. +- [`run_glob`] / [`run_grep`] — provider-side search. +- [`text`] — shared argument, UTF-8 byte limit, and edit result handling. diff --git a/crates/fs-operation/src/checked.rs b/crates/fs-operation/src/checked.rs index 0e4759353..d7c17de76 100644 --- a/crates/fs-operation/src/checked.rs +++ b/crates/fs-operation/src/checked.rs @@ -1,9 +1,10 @@ -//! Identity-fenced access adapter for the existing read/edit/write engine. +//! Identity-fenced access adapter for existing read/edit/write engines. //! -//! Descriptor-rooted resolution is Linux-only and symlinks/mount crossings fail -//! closed. The final compare and atomic rename are NOT a kernel CAS against an -//! arbitrary external editor: external editors must cooperate with provider -//! serialization. A hostile rename in that last syscall gap is not excluded. +//! Retains target, save-parent, and provider-root descriptors to bind authorization to I/O. +//! Rejects unavailable Linux descriptor primitives, symlinks, and mount crossings. +//! Revalidates names/descriptors until just before save, but the final comparison and rename +//! are not kernel CAS. Does not guarantee against noncooperative external editors replacing +//! entries between those syscalls, or guarantee parent-directory crash durability. use crate::{AtomicWriteMode, FsAccessPolicy}; use std::fs::{File, Metadata}; use std::io::{Seek, SeekFrom}; @@ -41,7 +42,17 @@ pub fn identity_validator(metadata: &Metadata) -> std::io::Result> { } } -/// One pinned target and its save parent. Retained through the shared operation. +/// Target, save-parent, and root descriptors retained during shared operations. +/// +/// Validators encode device/inode and protected metadata, detecting same-content replacements +/// independently of content hashes. Excludes atime so Read does not invalidate its own observation. +/// Saves publish a new temporary inode by rename; updating one hard-link name does not alter +/// other aliases of the old inode. +/// +/// Checks named targets and parents before commit, but cannot exclude external replacements between +/// the final check and rename. Pinned descriptors alone do not guarantee linearizability against +/// arbitrary external editors. Failures after parent creation or save begins report `effects_possible` +/// to callers rather than implying no effects. pub struct CheckedTarget<'a> { policy: &'a dyn FsAccessPolicy, root: File, diff --git a/crates/fs-operation/src/descriptor_ignore.rs b/crates/fs-operation/src/descriptor_ignore.rs index f8089049e..f33fe7fb0 100644 --- a/crates/fs-operation/src/descriptor_ignore.rs +++ b/crates/fs-operation/src/descriptor_ignore.rs @@ -1,5 +1,15 @@ -//! Ignore rules loaded only through the provider's authorized descriptor opens. -//! Never use GitignoreBuilder::add/build_global: those reopen host pathnames. +//! Ignore rules read only through provider-authorized descriptors. +//! +//! .ignore takes precedence over Git rules; nearer directories win within the same source class. +//! Retains only current ancestor matchers, pruning ignored directories before enumeration or reading +//! their own rules. Checks scope before opening rule sources; configuration discovery never expands +//! content authority. +//! +//! Glob requires an authorized .git marker; Grep applies .gitignore even without one. +//! Glob skips unavailable sources; Grep preserves existing explicit errors. Both skip scope-denied +//! sources before opening. UTF-8, file/line/aggregate byte limit, and cancellation violations are +//! not turned into under-filtered success. Does not follow global/ambient Git configuration or +//! linked-worktree gitdir/commondir pointers; .git files serve only as markers. use crate::{FsAccessPolicy, FsError}; use ignore::gitignore::{Gitignore, GitignoreBuilder}; use std::io::{BufRead, BufReader, Read}; diff --git a/crates/fs-operation/src/glob.rs b/crates/fs-operation/src/glob.rs index 739c779b8..22c37819a 100644 --- a/crates/fs-operation/src/glob.rs +++ b/crates/fs-operation/src/glob.rs @@ -5,7 +5,13 @@ use ignore::WalkBuilder; use crate::{FsAccessPolicy, FsError, FsPath, GlobRequest, GlobResult}; -/// Execute a bounded glob entirely inside the provider process. +/// Bounded Glob within the provider process. +/// +/// Descriptor providers check scope before reading ignore rules and prune denied directories +/// before descent. .ignore takes precedence over Git rules; Git rules require an authorized .git +/// marker. Unavailable rule sources are skipped; byte limit or cancellation violations yield errors. +/// Does not follow ambient/global configuration or linked-worktree management pointers. +/// Separate from configuration discovery in regular path-backed walkers. pub fn run_glob( root: &Path, base: &Path, diff --git a/crates/fs-operation/src/lib.rs b/crates/fs-operation/src/lib.rs index 0adc96460..d905c8713 100644 --- a/crates/fs-operation/src/lib.rs +++ b/crates/fs-operation/src/lib.rs @@ -1,8 +1,4 @@ -//! Foundational filesystem operation contracts and provider-side search. -//! -//! This crate deliberately has no dependency on Workdir identity, Runtime -//! transport, or LLM Tool implementations. Paths are logical and root-relative; -//! providers supply host roots and access policy. +#![doc = include_str!("../README.md")] mod checked; pub use checked::{CheckedTarget, identity_validator}; diff --git a/crates/fs-operation/src/local.rs b/crates/fs-operation/src/local.rs index 8885b5587..a93a17a63 100644 --- a/crates/fs-operation/src/local.rs +++ b/crates/fs-operation/src/local.rs @@ -473,6 +473,17 @@ fn list_entry_bytes(entry: &ListEntry) -> usize { entry.path.as_str().len().saturating_add(64) } +/// Bounded live pagination of direct children. +/// +/// Sorts directories first, then logical paths within each group. Cursors are exclusive ordering +/// keys and need not still exist. List applies provider read/enumeration scope, not ignore/dotfile +/// filters. The full scan retains only limit+1 candidates within an aggregate path-byte limit. +/// Enumeration order is unspecified, so a full page does not end scanning. Totals describe current +/// visible entries, including those before the cursor. +/// +/// No cross-page snapshot: insertions, removals, renames, and kind changes may omit or duplicate entries. +/// Cancellation and traversal ceilings apply even to excluded entries; resource exhaustion is not +/// converted into silent success. pub fn run_list( root: &Path, request: ListRequest, diff --git a/crates/fs-operation/src/search.rs b/crates/fs-operation/src/search.rs index 8506b611a..46521d2e4 100644 --- a/crates/fs-operation/src/search.rs +++ b/crates/fs-operation/src/search.rs @@ -220,6 +220,13 @@ struct GrepParams { offset: Option, } +/// Bounded Grep reading content within provider scope. +/// +/// Typed result paths come from reports retained after offset/limit, never reverse-parsed from +/// rendered filenames. Descriptor providers read only authorized ignore sources and apply .gitignore +/// without requiring .git markers. Unlike Glob, unavailable/symlinked ignore opens retain existing +/// explicit errors. Scope-denied sources are skipped before opening; global/ambient configuration +/// is not searched. Search is request-time traversal, not a snapshot. pub fn run_grep( root: &Path, base: PathBuf, diff --git a/crates/fs-operation/src/text.rs b/crates/fs-operation/src/text.rs index c541ef5b5..b2865e53c 100644 --- a/crates/fs-operation/src/text.rs +++ b/crates/fs-operation/src/text.rs @@ -1,7 +1,14 @@ -//! Pure text-operation policy shared by filesystem providers and operation frontends. +//! Pure text operations shared by providers and frontends. //! -//! This module owns no paths, descriptors, version fences, or I/O. Limits count -//! UTF-8 bytes, not characters, and default to unlimited. +//! Owns argument schemas, JSON decoding, edit conditions, and result sizes, not paths, descriptors, +//! authorization, I/O, or version fences. Providers call within existing consistency boundaries; +//! checked edits must not be decomposed into unchecked Read–Write. Adapters own whole-text versus +//! line-range Read selection, history, digests, and canonical multi-file commits. +//! +//! Edit requires a nonempty old string and a different replacement, which may be empty. +//! Unless replace_all is set, the nonoverlapping match must be unique. Providers set UTF-8 byte limits. Arithmetic overflow and capacity are +//! checked before allocating results. Pure validation failures have no side effects; transport +//! failures after dispatch are not converted into validation failures here. use schemars::JsonSchema; use serde::{Deserialize, de::DeserializeOwned}; diff --git a/crates/job/README.md b/crates/job/README.md new file mode 100644 index 000000000..5cd3c7aba --- /dev/null +++ b/crates/job/README.md @@ -0,0 +1,10 @@ +# job + +A shared data contract for immutable Job requests and structured results across Hosts. Validates inputs, Profile selectors, execution limits, and digests. Each Host adapter owns execution, grants, persistence, state transitions, and cleanup. + +## Reference index + +- [`JobRequest`] — request identity and input snapshot. +- [`JobLimits`] — execution limits that Hosts can further restrict. +- [`JobResultSubmission`] — results bound to a Job, attempt, and input. +- [`JobOutcome`] — read projection separating Job state from attempt state. diff --git a/crates/job/src/lib.rs b/crates/job/src/lib.rs index 6c58e4784..e889a413b 100644 --- a/crates/job/src/lib.rs +++ b/crates/job/src/lib.rs @@ -1,7 +1,4 @@ -//! Transport-neutral immutable Job intent and structured-result contract. -//! -//! Hosts own execution, authority, persistence and lifecycle state. This crate -//! deliberately contains no runner, state machine, Runtime or domain grants. +#![doc = include_str!("../README.md")] use serde::{Deserialize, Serialize}; use serde_json::Value; @@ -94,6 +91,11 @@ const fn default_max_attempts() -> u8 { DEFAULT_MAX_ATTEMPTS } +/// Immutable execution intent within a Host namespace. +/// +/// `fingerprint` identifies the entire request; `input_digest` binds results to the input snapshot. +/// Hosts own same-ID replay and attempt creation rules. Profile selector validation checks syntax +/// only; Hosts separately validate registry existence and requirements and grant domain capabilities. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] #[serde(deny_unknown_fields)] pub struct JobRequest { @@ -166,6 +168,11 @@ impl JobRequest { } } +/// Structured result bound to a Job, attempt, and immutable input. +/// +/// Hosts validate bindings and result digests at the acceptance boundary. Structured result +/// acceptance, consumer delivery, and execution resource cleanup are separate facts; +/// Worker termination and final prose do not imply acceptance. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] #[serde(deny_unknown_fields)] pub struct JobResultSubmission { diff --git a/crates/lint-common/README.md b/crates/lint-common/README.md index ddccdb515..fff9747ca 100644 --- a/crates/lint-common/README.md +++ b/crates/lint-common/README.md @@ -1,28 +1,9 @@ # lint-common -## Role +Shared frontmatter splitting and slug validation for Markdown records. Only syntax handling is shared; owning crates retain validation of record-specific required fields and semantics. -`lint-common` contains shared linting primitives for structured project records. +## Main entry points -## Boundaries - -Owns: - -- reusable frontmatter/slug/path validation helpers -- common lint diagnostic shapes used by higher-level crates - -Does not own: - -- memory-specific policy (`memory`) -- workflow-specific policy (`workflow`) -- Ticket backend policy (`ticket`) -- product CLI command shape (`yoi`) - -## Design notes - -Keeping common lint mechanics here avoids duplicating low-level validation while leaving each record type's policy in its owning crate. - -## See also - -- [`../../docs/design/memory-knowledge.md`](../../docs/design/memory-knowledge.md) -- [`../../docs/development/work-items.md`](../../docs/development/work-items.md) +- [`split_frontmatter`] and [`Frontmatter`] — frontmatter/body boundary +- [`Slug`] and [`is_valid_slug`] — identifier syntax validation +- [`RecordLintError`] — shared syntax errors diff --git a/crates/lint-common/src/frontmatter.rs b/crates/lint-common/src/frontmatter.rs index 695484b38..fddec4fbe 100644 --- a/crates/lint-common/src/frontmatter.rs +++ b/crates/lint-common/src/frontmatter.rs @@ -1,4 +1,7 @@ -//! Common frontmatter helpers. +//! Share the frontmatter/body boundary for Markdown records. +//! +//! Splitting returns borrowed slices without interpreting YAML semantics. Owning crates +//! validate required fields and record-specific values, keeping domain policy out of shared code. use chrono::{DateTime, Utc}; diff --git a/crates/lint-common/src/lib.rs b/crates/lint-common/src/lib.rs index b63351004..9af5ce771 100644 --- a/crates/lint-common/src/lib.rs +++ b/crates/lint-common/src/lib.rs @@ -1,4 +1,4 @@ -//! Shared record lint primitives for memory and workflow files. +#![doc = include_str!("../README.md")] mod frontmatter; mod slug; diff --git a/crates/manifest/README.md b/crates/manifest/README.md index 82ccd0aa7..ec5aade50 100644 --- a/crates/manifest/README.md +++ b/crates/manifest/README.md @@ -1,33 +1,10 @@ # manifest -## Role +Validates reusable Worker Profiles and resolves them into Manifests with launch-time names, paths, and permissions. This crate owns configuration types and resolution boundaries, not Worker execution or credential retrieval. -`manifest` resolves reusable profile/configuration inputs into the concrete runtime Manifest used to create or restore Workers. +## Main entry points -## Boundaries - -Owns: - -- Profile and Manifest data structures -- source/partial/resolved configuration layering -- path and prompt-resource resolution -- tool permission and filesystem scope configuration types -- model/provider references as configuration records - -Does not own: - -- provider HTTP clients or secret lookup implementation (`provider`, `secrets`) -- Worker lifecycle (`worker`) -- product CLI parsing (`yoi`) -- generated memory records (`memory`) - -## Design notes - -Profiles are reusable recipes; resolved Manifests are runtime contracts. Keep runtime-bound fields such as Worker name, concrete delegated scope, sockets, session pointers, and raw secrets out of reusable Profiles. - -`feature.workdir_catalog.enabled` defaults to `true` for read-only Repository/Workdir/attachment reference discovery in WIP mode. It is independent of `feature.manage_workdir` (which defaults disabled), registers no normal Tools, and does not grant filesystem, attachment, or Workspace authority. Permissions remain independent. Standalone Tools-mode profiles may retain this default; only WIP installation consumes the flag. Set it to `false` to opt out. - -## See also - -- [`../../docs/design/profiles-manifests-prompts.md`](../../docs/design/profiles-manifests-prompts.md) -- [`../../docs/design/tool-permissions-scope.md`](../../docs/design/tool-permissions-scope.md) +- [`ProfileResolver`] / [`ResolvedProfile`] — Profile selection and resolution results +- [`WorkerManifestConfig`] / [`WorkerManifest`] — Partial configuration merging and execution contracts +- [`Scope`] / [`DelegationScope`] / [`SharedScope`] — Direct access and child delegation checks +- [`plugin`] — Offline package inspection, independent of execution authority diff --git a/crates/manifest/src/builtin_profile.rs b/crates/manifest/src/builtin_profile.rs index a0deb6ada..f954811bf 100644 --- a/crates/manifest/src/builtin_profile.rs +++ b/crates/manifest/src/builtin_profile.rs @@ -1,3 +1,10 @@ +//! Converts an explicit builtin source set into Decodal values. +//! +//! Selects a catalog entry point and loads only relative imports within that catalog. +//! Does not fall back to OS paths or Workspace sources, and distinguishes undefined +//! entry points from evaluation failures. Launching or restoring values already +//! evaluated by the Workspace is owned separately from this reevaluation. + use std::collections::BTreeMap; use decodal::{Data, Engine, ImportLoader, LoadedImport}; diff --git a/crates/manifest/src/config.rs b/crates/manifest/src/config.rs index 50cef53b6..891f835e8 100644 --- a/crates/manifest/src/config.rs +++ b/crates/manifest/src/config.rs @@ -1,10 +1,4 @@ -//! Partial-form of [`crate::WorkerManifest`] used as cascade layers. -//! -//! `WorkerManifestConfig` mirrors `WorkerManifest` but every field is optional -//! so individual layers (builtin defaults, user manifest, project -//! manifest, programmatic overlay) can be partial. Layers are combined -//! via [`WorkerManifestConfig::merge`] and the final config is converted to -//! a validated [`WorkerManifest`] via `TryFrom`. +//! Partial configuration merging and validation. use std::collections::{BTreeSet, HashMap}; use std::num::NonZeroU32; @@ -25,17 +19,21 @@ use crate::{ WorkerManifest, WorkerMeta, }; -/// Partial-form Worker manifest. Every field is optional; one or more -/// instances merge via [`WorkerManifestConfig::merge`] before being -/// converted to a validated [`WorkerManifest`] via `TryFrom`. +/// Merges partial configuration into a validated [`crate::WorkerManifest`]. +/// +/// [`WorkerManifestConfig`] preserves omitted values. Resolve each input's relative +/// paths with [`WorkerManifestConfig::resolve_paths`] before merging; do not later +/// reinterpret them against another input's base. [`WorkerManifestConfig::merge`] +/// handles explicit merging; final conversion validates required fields and +/// semantics. This type does not imply implicit discovery or merging of user/project +/// Manifest files during normal startup. #[derive(Debug, Clone, Default, Serialize, Deserialize)] pub struct WorkerManifestConfig { #[serde(default)] pub worker: WorkerMetaConfig, - /// `[model]` セクションは partial でも完成形でも同じ - /// [`ModelManifest`] を使う。ref / inline の両形を受け入れるための - /// 全 Optional 構造なので、カスケード層と最終マニフェストで型を - /// 分ける必要がない。 + /// Partial and final `[model]` sections share [`ModelManifest`]. All fields + /// are optional to support both ref and inline forms, so cascade layers and + /// final manifests need no separate types. #[serde(default)] pub model: ModelManifest, #[serde(default)] @@ -911,10 +909,12 @@ impl WorkerManifestConfig { self } - /// Merge `upper` into `self`. Fields present in `upper` override - /// fields from `self`. Map entries merge key-wise with `upper` - /// winning on conflict. Scope rules from both layers accumulate - /// (see [`ScopeConfig`] semantics). + /// Merges explicit upper-layer values, retaining lower-layer values when omitted. + /// + /// Maps prefer upper values per key; scope rules and Skills directories accumulate. + /// Resolve each input's paths before merging so relative paths from different + /// inputs are not mistaken for sharing a base. This merge guarantees neither + /// authority containment checks nor persistence CAS. pub fn merge(self, upper: WorkerManifestConfig) -> Self { Self { worker: self.worker.merge(upper.worker), @@ -1135,16 +1135,16 @@ fn ensure_absolute(field: &'static str, path: &Path) -> Result<(), ResolveError> } } -/// `AuthRef::ApiKey { file }` が相対パスのとき `base` を前置する。 +/// Prepends `base` to relative `AuthRef::ApiKey { file }` paths. fn resolve_auth_file(auth: &mut Option, base: &Path) { if let Some(AuthRef::ApiKey { file: Some(p), .. }) = auth.as_mut() { *p = join_if_relative(base, p); } } -/// モデル宣言に含まれる `auth.file` が絶対パスであることを検証する。 -/// ref / scheme / model_id 等の論理的な有効性（ref があるか、inline が -/// 揃っているか）の検証は [`crate::model_catalog`] で行う。 +/// Validates that model declarations use absolute `auth.file` paths. +/// [`crate::model_catalog`] validates logical ref / scheme / model_id requirements, +/// such as the presence of a ref or complete inline fields. fn validate_model_paths(model: &ModelManifest, field: &'static str) -> Result<(), ResolveError> { if let Some(AuthRef::ApiKey { file: Some(p), .. }) = &model.auth { ensure_absolute(field, p)?; diff --git a/crates/manifest/src/defaults.rs b/crates/manifest/src/defaults.rs index 8b5b9619e..4539e8eae 100644 --- a/crates/manifest/src/defaults.rs +++ b/crates/manifest/src/defaults.rs @@ -1,10 +1,8 @@ -//! Single source of truth for manifest default values. +//! Centralizes Manifest defaults. //! -//! Every default that would otherwise be duplicated between serde -//! `#[serde(default = "...")]` attributes (on [`crate::WorkerManifest`]) -//! and the cascade resolution in [`crate::config`] lives here as a -//! `pub const`. Both paths read from this module, so changing a -//! default requires editing exactly one line. +//! Serde defaults in [`crate::WorkerManifest`] and resolution in +//! [`crate::WorkerManifestConfig`] share constants to keep decoding and partial +//! configuration merging consistent. /// Read-only Repository/Workdir/attachment reference discovery in WIP mode. /// This enables a reference provider, not normal Tools or any authority grant. diff --git a/crates/manifest/src/lib.rs b/crates/manifest/src/lib.rs index bb59430ed..88c48473f 100644 --- a/crates/manifest/src/lib.rs +++ b/crates/manifest/src/lib.rs @@ -1,3 +1,5 @@ +#![doc = include_str!("../README.md")] + mod builtin_profile; mod config; pub mod defaults; @@ -40,12 +42,14 @@ use std::path::PathBuf; use serde::de::Error as _; use serde::{Deserialize, Serialize}; -/// Declarative configuration for a Worker. +/// Resolved execution contract for creating and restoring a Worker. /// -/// Parsed from a TOML manifest file. Describes the model, system prompt, -/// and directory scope (required). The Worker's working directory is **not** -/// part of the manifest — it is the process's `std::env::current_dir()` -/// at construction time. +/// Separate from reusable Profiles and partial configuration, this includes +/// launch-time identity, paths, permissions, and prompt references. Persisting +/// resolved values lets restoration reuse the original contract without resolving +/// current configuration or Profiles again. Workdir attachments and their operation +/// authority belong to a separate execution boundary; this value does not pin the +/// process cwd. #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub struct WorkerManifest { @@ -890,10 +894,13 @@ pub struct SessionConfig { pub record_event_trace: bool, } -/// Manifest-level pattern-based tool permission policy. +/// Pattern-based permission policy for tool calls. /// -/// Presence of `[permissions]` enables this layer. Rules are evaluated -/// in declaration order; if none match, [`Self::default_action`] is used. +/// Evaluates rules in declaration order, using [`Self::default_action`] if none +/// match. This is separate from [`Scope`] path authority: allowing a tool does not +/// guarantee access to its target path. `Ask` defers permission; current callers +/// fail closed because there is no approval/resume protocol. The Worker owns +/// enforcement. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] pub struct ToolPermissionConfig { pub default_action: ToolPermissionAction, diff --git a/crates/manifest/src/model.rs b/crates/manifest/src/model.rs index 9cb47de8f..66c1ad3ec 100644 --- a/crates/manifest/src/model.rs +++ b/crates/manifest/src/model.rs @@ -1,72 +1,69 @@ -//! LLM モデル宣言型 +//! LLM model declaration types. //! -//! Worker マニフェストの `[model]` セクションで記述する型。`ref`（プロバイダ -//! とモデルを両方指し示す短縮形）と inline 指定（`scheme` / `model_id` -//! 直書き）の両方を受け入れるため、すべてのフィールドを `Option` として -//! 持つ 1 つの型 [`ModelManifest`] に統合している。実解決（ref をプロバイダ -//! カタログ / モデルカタログから引いて `scheme` や `model_id` を埋める） -//! は [`crate::model_catalog`] の責務で、本モジュールはデータ表現のみを提供する。 +//! The Worker manifest's `[model]` section uses one [`ModelManifest`] with all +//! fields optional to support both `ref` (a provider/model shorthand) and inline +//! `scheme` / `model_id` declarations. [`crate::model_catalog`] resolves refs via +//! provider/model catalogs and fills in `scheme`, `model_id`, and other fields; +//! this module provides only the data representation. //! -//! 同じ型を partial（カスケード層）と完成形（最終マニフェスト）の両方で -//! 使うことで、merge と最終変換の重複を避ける。 +//! Sharing the type between partial cascade layers and final manifests avoids +//! duplicate merging and final conversion logic. use std::path::PathBuf; use serde::{Deserialize, Serialize}; -// `ModelCapability` は `agen` 側に定義される runtime 構造だが、 -// マニフェストで任意に override できるよう型だけ再エクスポートする。 +// `ModelCapability` is a runtime type defined in `agen`; re-export only the type +// to allow optional manifest overrides. pub use agen::llm_client::capability::{ModelCapability, ReasoningControl, ReasoningEffort}; -/// Worker マニフェストの `[model]` セクション。 +/// The Worker manifest's `[model]` section. /// -/// - ref だけ書く: `[model] ref = "anthropic/claude-sonnet-4-6"` -/// - ref + 一部 override: ref で基底を引き、`auth` 等だけ書き換え -/// - 完全 inline: `ref` を省略して `scheme` / `model_id` / `auth` を直書き +/// - Ref only: `[model] ref = "anthropic/claude-sonnet-4-6"` +/// - Ref with overrides: use ref as a base and override fields such as `auth`. +/// - Fully inline: omit `ref` and specify `scheme` / `model_id` / `auth` directly. /// -/// どの形が有効かの判定は [`crate::model_catalog::resolve_model_manifest`] が担う。 -/// 本クレートは「どこから取るか」を表現するだけで、未設定かどうかを -/// 理由にした hard error は出さない。 +/// [`crate::model_catalog::resolve_model_manifest`] determines which forms are valid. +/// This crate only represents configuration sources; it does not raise hard errors +/// merely because fields are unset. #[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq)] pub struct ModelManifest { - /// `/` 形式のカタログ参照。`/` の - /// 最初の 1 文字目で split し provider カタログを引く。 - /// OpenRouter の `anthropic/claude-sonnet-4` のように `/` を含む - /// model_id は `openrouter/anthropic/claude-sonnet-4` と書く - /// （provider 側で最初の `/` のみ split するため）。 + /// Catalog reference in `/` form. Splits at + /// the first `/` to look up the provider catalog. For model IDs containing + /// `/`, such as OpenRouter's `anthropic/claude-sonnet-4`, use + /// `openrouter/anthropic/claude-sonnet-4`; only the first `/` is split. #[serde(default, rename = "ref", skip_serializing_if = "Option::is_none")] pub ref_: Option, - /// wire format の明示指定。ref 未指定時は必須、ref 指定時は override。 + /// Explicit wire format. Required without ref; an override with ref. #[serde(default, skip_serializing_if = "Option::is_none")] pub scheme: Option, - /// API のベース URL。scheme の既定値を override する。 + /// API base URL, overriding the scheme default. #[serde(default, skip_serializing_if = "Option::is_none")] pub base_url: Option, - /// プロバイダが受け付けるモデル ID。ref 未指定時は必須。 + /// Provider-accepted model ID. Required without ref. #[serde(default, skip_serializing_if = "Option::is_none")] pub model_id: Option, - /// 認証方式。ref 未指定時は必須、ref 指定時は override。 + /// Authentication method. Required without ref; an override with ref. #[serde(default, skip_serializing_if = "Option::is_none")] pub auth: Option, - /// モデル能力の明示指定。未指定時はモデルカタログ → provider - /// `default_capability` → scheme 既定の順で解決される。 + /// Explicit model capabilities. If omitted, resolves in order: model catalog, + /// provider `default_capability`, then scheme default. #[serde(default, skip_serializing_if = "Option::is_none")] pub capability: Option, - /// モデルの希望コンテキストウィンドウ（tokens）。カタログ未掲載 / inline - /// モデルでもここで明示 override できる。実効値は `max_context_window` - /// またはカタログ上の backend maximum で clamp される。 + /// Desired model context window in tokens. Can explicitly override uncataloged + /// or inline models. The effective value is clamped by `max_context_window` + /// or the catalog's backend maximum. #[serde(default, skip_serializing_if = "Option::is_none")] pub context_window: Option, - /// backend が実際に受け付けるコンテキストウィンドウ上限（tokens）。 - /// 表示・安全判定に使う実効 context window は `context_window` とこの値の - /// 小さい方になる。 + /// Maximum context window in tokens actually accepted by the backend. + /// The effective window used for display and safety checks is the smaller + /// of `context_window` and this value. #[serde(default, skip_serializing_if = "Option::is_none")] pub max_context_window: Option, } impl ModelManifest { - /// `upper` を `self` に上書きマージする。マニフェスト cascade 向け - /// （builtin → user → project → overlay の優先順位で呼ばれる）。 + /// Merges explicit `upper` values over `self`, retaining lower values when omitted. pub fn merge(self, upper: Self) -> Self { Self { ref_: upper.ref_.or(self.ref_), @@ -81,39 +78,39 @@ impl ModelManifest { } } -/// サポートする wire scheme の種類。 +/// Supported wire scheme kinds. #[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)] #[serde(rename_all = "snake_case")] pub enum SchemeKind { - /// Anthropic Messages API (`/v1/messages`)。Ollama `/v1/messages` もこれで扱う + /// Anthropic Messages API (`/v1/messages`), also used for Ollama `/v1/messages`. Anthropic, - /// OpenAI Chat Completions (`/v1/chat/completions`)。OpenAI 互換ルーター共通枠 + /// OpenAI Chat Completions (`/v1/chat/completions`), shared by compatible routers. OpenaiChat, - /// OpenAI Responses API (`/v1/responses`)。別チケットで scheme 新設予定 + /// OpenAI Responses API (`/v1/responses`). OpenaiResponses, /// Google Gemini (`/v1beta/models/...:streamGenerateContent`) Gemini, } -/// 認証の参照。 +/// Authentication reference. /// -/// 実際のトークン値の解決（local secret store / file 読取、OAuth refresh 等）は -/// worker-side factory で行う。ここはあくまで「どこから取るか」の宣言。 +/// A worker-side factory resolves actual tokens (local secret store / file reads, +/// OAuth refresh, etc.). This is only a declaration of where to obtain them. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq, Default)] #[serde(tag = "kind", rename_all = "snake_case")] pub enum AuthRef { - /// 認証不要（ローカル Ollama 等） + /// No authentication required (e.g. local Ollama). #[default] None, /// API key file reference. Prefer [`AuthRef::SecretRef`] for normal /// provider credentials; this remains an explicit file source for low-level /// manifests and tests. ApiKey { - /// key を書き込んだファイル（絶対パス） + /// File containing the key (absolute path). #[serde(default)] file: Option, }, - /// ChatGPT OAuth（`~/.codex/auth.json`）。 + /// ChatGPT OAuth (`~/.codex/auth.json`). #[serde(rename = "codex_oauth")] CodexOAuth, /// Typed local secret-store reference. The profile resolver preserves this diff --git a/crates/manifest/src/model_catalog.rs b/crates/manifest/src/model_catalog.rs index 0adaa2eaf..c6a9457cb 100644 --- a/crates/manifest/src/model_catalog.rs +++ b/crates/manifest/src/model_catalog.rs @@ -1,17 +1,17 @@ -//! プロバイダ / モデルカタログ。 +//! Provider / model catalogs. //! -//! - builtin プロバイダ: `resources/providers/builtin.toml` -//! - builtin モデル: `resources/models/builtin.toml` +//! - builtin providers: `resources/providers/builtin.toml` +//! - builtin models: `resources/models/builtin.toml` //! - user override: `/{providers,models}.toml` //! -//! `` の解決は [`crate::paths::config_dir`] を参照。 -//! どちらの override も「あれば builtin を置換、無ければ builtin」と -//! いう一方向の差し替え（マージしない）。providers / models は独立に -//! 読み、片方だけ user override も可。 +//! See [`crate::paths::config_dir`] for `` resolution. Each override +//! replaces its builtin catalog if present, otherwise the builtin is used; no +//! merging occurs. Provider and model catalogs load independently, so either +//! can be overridden alone. //! -//! [`resolve_model_manifest`] が [`crate::ModelManifest`]（ref / inline -//! 両形）を最終的な [`ModelConfig`] に解決する単一の入口で、wire 層 -//! に渡す前のバリデーションもここで行う。 +//! [`resolve_model_manifest`] is the single entry point for resolving either ref +//! or inline [`crate::ModelManifest`] declarations into a final [`ModelConfig`], +//! including validation before passing it to the wire layer. use std::path::{Path, PathBuf}; @@ -45,9 +45,8 @@ pub enum CatalogError { BuiltinParse(#[source] toml::de::Error), } -/// マニフェスト解決時のエラー。`ModelManifest` がカタログ参照を満たせ -/// ない、あるいは inline フォームでも必須フィールドが揃わない場合に -/// 返す。 +/// Manifest resolution error: `ModelManifest` cannot satisfy a catalog reference +/// or an inline declaration is missing required fields. #[derive(Debug, thiserror::Error)] pub enum ResolveError { #[error("failed to load provider catalog: {0}")] @@ -62,15 +61,14 @@ pub enum ResolveError { InlineMissing(&'static str), } -/// UI 向けの認証ヒント。 +/// UI authentication hint. /// -/// 「何を表示・要求するか」のメタ情報で、ランタイムの [`AuthRef`] -/// とは責務が別。1:1 の対応関係にあり、[`auth_hint_to_ref`] / [`AuthHint`] -/// 解決経路で相互変換される。 +/// Metadata describing what to display/request, separate from runtime [`AuthRef`]. +/// Maps 1:1 to it via `auth_hint_to_ref` during resolution. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[serde(tag = "kind", rename_all = "snake_case")] pub enum AuthHint { - /// 認証不要（ローカル Ollama 等） + /// No authentication required (e.g. local Ollama). None, /// API key file reference. Normal credential configuration should prefer /// [`AuthHint::SecretRef`] so plaintext credentials stay out of manifests. @@ -82,12 +80,12 @@ pub enum AuthHint { #[serde(rename = "ref")] ref_: String, }, - /// ChatGPT OAuth（`~/.codex/auth.json`） + /// ChatGPT OAuth (`~/.codex/auth.json`) #[serde(rename = "codex_oauth")] CodexOAuth, } -/// プロバイダカタログの 1 エントリ。 +/// One provider catalog entry. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] pub struct ProviderEntry { pub id: String, @@ -96,43 +94,41 @@ pub struct ProviderEntry { #[serde(default)] pub base_url: Option, pub auth_hint: AuthHint, - /// モデルカタログ未登録モデルでこの provider が使われたとき - /// （ref で provider はあるが model 行は無い等）のフォールバック。 - /// 省略時は `Scheme::default_capability()` を最終フォールバックに - /// 使う。 + /// Fallback when this provider serves a model absent from the model catalog + /// (e.g. a ref has a provider but no model entry). If omitted, + /// `Scheme::default_capability()` is the final fallback. #[serde(default)] pub default_capability: Option, - /// モデルカタログ未登録モデルで使う既定の context window。省略時は - /// [`DEFAULT_CONTEXT_WINDOW`] を使う。 + /// Default context window for models absent from the model catalog. + /// Uses [`DEFAULT_CONTEXT_WINDOW`] if omitted. #[serde(default)] pub default_context_window: Option, } -/// モデルカタログの 1 エントリ。 +/// One model catalog entry. /// -/// `id` は **provider 内ユニーク**。同じ `gpt-5` が異なる provider に -/// 存在するのは OK で、ref が必ず `/` を含むため -/// 曖昧性が出ない。 +/// `id` is **unique within a provider**. The same `gpt-5` may appear under different +/// providers without ambiguity, since refs always include `/`. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] pub struct ModelEntry { pub id: String, pub provider: String, - /// モデル単位の capability override。省略時は - /// `ProviderEntry::default_capability` にフォールバックする。 + /// Per-model capability override, falling back to + /// `ProviderEntry::default_capability` if omitted. #[serde(default)] pub capability: Option, - /// モデル単位の context window。省略時は provider default → builtin - /// fallback にフォールバックする。実効値は `max_context_window` で clamp - /// される。 + /// Per-model context window. If omitted, falls back to the provider default, + /// then the builtin fallback. The effective value is clamped by + /// `max_context_window`. #[serde(default)] pub context_window: Option, - /// backend が実際に受け付ける context window の上限。UI や pre-request - /// safety は希望値ではなく clamp 済みの実効値を使う。 + /// Maximum context window actually accepted by the backend. UI and + /// pre-request safety checks use the clamped effective value, not the desired one. #[serde(default)] pub max_context_window: Option, } -/// 解決済みモデル設定。`build_client` が消費する完成形。 +/// Resolved model configuration, ready for `build_client`. #[derive(Debug, Clone, PartialEq)] pub struct ModelConfig { pub scheme: SchemeKind, @@ -158,7 +154,7 @@ struct ModelCatalogFile { model: Vec, } -/// `auth_hint` に対応する [`AuthRef`] のひな型を返す。 +/// Returns an [`AuthRef`] template corresponding to `auth_hint`. fn auth_hint_to_ref(hint: &AuthHint) -> AuthRef { match hint { AuthHint::None => AuthRef::None, @@ -170,13 +166,11 @@ fn auth_hint_to_ref(hint: &AuthHint) -> AuthRef { // --- providers --------------------------------------------------------------- -/// builtin + user override を解決して provider カタログを返す。 +/// Returns the provider catalog after resolving builtin + user overrides. /// -/// user override (`/providers.toml`) が存在すれば builtin -/// を置き換える。存在しなければ builtin のみ。user override が存在 -/// するが壊れている場合はエラーを返す（silent fallback はしない — -/// ユーザーが書いた設定が silent に無視されて builtin に戻る挙動は -/// 気付きにくいため）。 +/// A user override (`/providers.toml`) replaces the builtin if present; +/// otherwise only the builtin is used. An invalid override returns an error rather +/// than silently falling back, since ignoring user configuration is hard to notice. pub fn load_providers() -> Result, CatalogError> { if let Some(path) = crate::paths::user_catalog_override("providers.toml") && path.is_file() @@ -186,14 +180,14 @@ pub fn load_providers() -> Result, CatalogError> { load_builtin_providers() } -/// builtin provider カタログのみを返す。 +/// Returns only the builtin provider catalog. pub fn load_builtin_providers() -> Result, CatalogError> { let parsed: ProviderCatalogFile = toml::from_str(BUILTIN_PROVIDERS).map_err(CatalogError::BuiltinParse)?; Ok(parsed.provider) } -/// 指定パスから provider カタログを読む。 +/// Reads a provider catalog from the given path. pub fn load_providers_from(path: &Path) -> Result, CatalogError> { let text = std::fs::read_to_string(path).map_err(|source| CatalogError::Io { path: path.to_path_buf(), @@ -209,7 +203,7 @@ pub fn load_providers_from(path: &Path) -> Result, CatalogErr // --- models ------------------------------------------------------------------ -/// builtin + user override を解決してモデルカタログを返す。 +/// Returns the model catalog after resolving builtin + user overrides. pub fn load_models() -> Result, CatalogError> { if let Some(path) = crate::paths::user_catalog_override("models.toml") && path.is_file() @@ -219,14 +213,14 @@ pub fn load_models() -> Result, CatalogError> { load_builtin_models() } -/// builtin model カタログのみを返す。 +/// Returns only the builtin model catalog. pub fn load_builtin_models() -> Result, CatalogError> { let parsed: ModelCatalogFile = toml::from_str(BUILTIN_MODELS).map_err(CatalogError::BuiltinParse)?; Ok(parsed.model) } -/// 指定パスからモデルカタログを読む。 +/// Reads a model catalog from the given path. pub fn load_models_from(path: &Path) -> Result, CatalogError> { let text = std::fs::read_to_string(path).map_err(|source| CatalogError::Io { path: path.to_path_buf(), @@ -239,12 +233,12 @@ pub fn load_models_from(path: &Path) -> Result, CatalogError> { Ok(parsed.model) } -// --- ref 解決 / マニフェスト → ModelConfig --------------------------------- +// --- ref resolution / manifest → ModelConfig --------------------------------- -/// `/` の最初の `/` で 1 回だけ split する。 -/// OpenRouter の `openrouter/anthropic/claude-sonnet-4.6` のように -/// model_id に `/` を含むケースは、provider=`openrouter`、 -/// model_id=`anthropic/claude-sonnet-4.6` として通る。 +/// Splits `/` once, at the first `/`. +/// Model IDs containing `/`, such as OpenRouter's +/// `openrouter/anthropic/claude-sonnet-4.6`, resolve as provider=`openrouter`, +/// model_id=`anthropic/claude-sonnet-4.6`. fn split_ref(s: &str) -> Option<(&str, &str)> { let (provider, rest) = s.split_once('/')?; if provider.is_empty() || rest.is_empty() { @@ -253,30 +247,30 @@ fn split_ref(s: &str) -> Option<(&str, &str)> { Some((provider, rest)) } -/// `ModelManifest` をカタログ込みで解決し、最終 [`ModelConfig`] を返す。 +/// Resolves `ModelManifest` against catalogs into a final [`ModelConfig`]. /// -/// - **`ref` あり** → provider カタログを引き、未登録なら hard error。 -/// model カタログは未登録でも warn ログだけに留め、`provider.default_capability` -/// にフォールバック（タイポで動く可能性は API 側 `model not found` で -/// 結果的に検出されるため）。 -/// - **`ref` なし** → `scheme` / `model_id` / `auth` の 3 つが揃って -/// いることを検証し、そのまま `ModelConfig` を組む。 +/// - **With `ref`**: looks up the provider; an unknown provider is a hard error. +/// A missing model entry only logs a warning and falls back to +/// `provider.default_capability` (typos are eventually caught by the API's +/// `model not found` error). +/// - **Without `ref`**: requires `scheme` / `model_id` / `auth` and builds +/// `ModelConfig` directly. /// -/// 各フィールドの解決順は ticket の表に準拠: -/// scheme/base_url は manifest 明示 > provider、model_id は manifest 明示 > ref、 -/// auth は manifest 明示 > provider.auth_hint 由来、capability は -/// manifest 明示 > model catalog > provider.default_capability > -/// （`build_client` 側で）`Scheme::default_capability()`。 -/// context_window は manifest 明示 > model catalog > provider default > -/// [`DEFAULT_CONTEXT_WINDOW`]。実効 context_window は manifest/model の -/// max_context_window で clamp される。 +/// Field precedence: +/// scheme/base_url: explicit manifest > provider; model_id: explicit manifest > ref; +/// auth: explicit manifest > provider.auth_hint; capability: +/// explicit manifest > model catalog > provider.default_capability > +/// `Scheme::default_capability()` (in `build_client`). +/// context_window: explicit manifest > model catalog > provider default > +/// [`DEFAULT_CONTEXT_WINDOW`]. The effective context_window is clamped by the +/// manifest/model max_context_window. pub fn resolve_model_manifest(manifest: &ModelManifest) -> Result { let providers = load_providers().map_err(ResolveError::LoadProviders)?; let models = load_models().map_err(ResolveError::LoadModels)?; resolve_with_catalogs(manifest, &providers, &models) } -/// テスト等で in-memory カタログを差し込む解決経路。 +/// Resolution path accepting in-memory catalogs, e.g. for tests. pub fn resolve_with_catalogs( manifest: &ModelManifest, providers: &[ProviderEntry], @@ -290,7 +284,7 @@ pub fn resolve_with_catalogs( .find(|p| p.id == provider_id) .ok_or_else(|| ResolveError::UnknownProvider(provider_id.to_string()))?; - // model 行は無くても続行可（warn ログ + provider.default_capability）。 + // Missing model entries may proceed with a warning + provider.default_capability. let model_entry = models .iter() .find(|m| m.provider == provider_id && m.id == ref_model_id); @@ -637,10 +631,9 @@ auth_hint = { kind = "none" } assert!(matches!(err, CatalogError::Parse { .. })); } - /// `YOI_CONFIG_DIR` を tempdir に向けるテストガード。 - /// `paths::config_dir` は他の env (YOI_HOME / XDG_CONFIG_HOME) - /// より高優先で `YOI_CONFIG_DIR` を尊重するため、これだけで - /// 開発機の env 設定に左右されないテストになる。 + /// Test guard pointing `YOI_CONFIG_DIR` to a temporary directory. + /// `paths::config_dir` prioritizes it over YOI_HOME / XDG_CONFIG_HOME, + /// isolating tests from the development machine's environment settings. struct ConfigDirGuard { prev: Option, } @@ -648,8 +641,8 @@ auth_hint = { kind = "none" } impl ConfigDirGuard { fn new(path: &Path) -> Self { let prev = std::env::var("YOI_CONFIG_DIR").ok(); - // SAFETY: serial_test の `#[serial]` 属性で env を弄るテスト - // 同士は直列化される。 + // SAFETY: serial_test's `#[serial]` attribute serializes tests that + // modify the environment. unsafe { std::env::set_var("YOI_CONFIG_DIR", path) }; Self { prev } } @@ -692,7 +685,7 @@ auth_hint = { kind = "none" } #[serial] fn load_falls_back_to_builtin_when_override_absent() { let dir = tempfile::tempdir().unwrap(); - // override ファイルは作らない + // Do not create an override file. let _g = ConfigDirGuard::new(dir.path()); let entries = load_providers().unwrap(); assert!(!entries.is_empty()); diff --git a/crates/manifest/src/paths.rs b/crates/manifest/src/paths.rs index 619820789..abd79e343 100644 --- a/crates/manifest/src/paths.rs +++ b/crates/manifest/src/paths.rs @@ -1,36 +1,36 @@ -//! Yoi のホームディレクトリ配下のパス解決を一元化するモジュール。 +//! Centralized path resolution under Yoi's home directory. //! -//! 用途別に三つの base directory を持つ: +//! Separates base directories by purpose: //! -//! - **`config_dir`** — 人が手で書く / 編集する設定。`profiles.toml`, -//! `providers.toml`, `models.toml` 等 -//! - **`data_dir`** — プログラムが書く永続データ。`sessions/` 等 -//! - **`secret_data_dir`** — local secret store の読み書き base。既存 -//! secret store は path-derived key を使うため、通常 data とは別に -//! legacy `$HOME/.yoi` を既定にする -//! - **`runtime_dir`** — 再起動で消えてよいランタイム状態。socket, -//! `workers.json`, `pid` ファイル等 +//! - **`config_dir`** — Handwritten/edited configuration: `profiles.toml`, +//! `providers.toml`, `models.toml`, etc. +//! - **`data_dir`** — Program-written persistent data, such as `sessions/`. +//! - **`secret_data_dir`** — Base for local secret-store reads/writes. The existing +//! store uses path-derived keys, so it defaults to legacy `$HOME/.yoi`, separate +//! from regular data. +//! - **`runtime_dir`** — Runtime state disposable on restart: sockets, +//! `workers.json`, `pid` files, etc. //! -//! ## 解決順 (優先順位高 → 低) +//! ## Resolution order (highest to lowest priority) //! -//! | base | 1. `YOI__DIR` | 2. `YOI_HOME` | 3. `XDG_*` | 4. 既定 | +//! | base | 1. `YOI__DIR` | 2. `YOI_HOME` | 3. `XDG_*` | 4. default | //! |---|---|---|---|---| //! | config | `YOI_CONFIG_DIR` | `$YOI_HOME/config` | `$XDG_CONFIG_HOME/yoi` | `$HOME/.config/yoi` | //! | data | `YOI_DATA_DIR` | `$YOI_HOME` | `$XDG_DATA_HOME/yoi` | `$HOME/.local/share/yoi` | //! | secrets | `YOI_DATA_DIR` | `$YOI_HOME` | — | `$HOME/.yoi` | //! | runtime | `YOI_RUNTIME_DIR` | `$YOI_HOME/run` | `$XDG_RUNTIME_DIR/yoi` | `$HOME/.yoi/run` | //! -//! `YOI_HOME=$X` のとき config は `$X/config`、data は `$X` 直下、 -//! runtime は `$X/run` に集約される。テストや sandbox 利用ではこれ一本 -//! で全部 tempdir に向けられる。 +//! With `YOI_HOME=$X`, config uses `$X/config`, data uses `$X` directly, and +//! runtime uses `$X/run`. Tests and sandboxes can redirect all of them to a +//! temporary directory with this single variable. //! -//! 解決された各 base が存在するか / ディレクトリかは保証しない — -//! 呼び出し側がファイル操作の前に作成 / 検査する。 +//! Resolved bases are not guaranteed to exist or be directories; callers must +//! create/check them before file operations. use std::path::PathBuf; -/// 設定ディレクトリ。`profiles.toml`, `providers.toml`, `models.toml`, -/// `prompts/` などが置かれる。 +/// Configuration directory for `profiles.toml`, `providers.toml`, `models.toml`, +/// `prompts/`, etc. pub fn config_dir() -> Option { resolve_config_dir_from_parts( env_path("YOI_CONFIG_DIR"), @@ -40,8 +40,7 @@ pub fn config_dir() -> Option { ) } -/// データディレクトリ。`sessions/` などプログラムが書く永続データの -/// 置き場。 +/// Data directory for program-written persistent data, such as `sessions/`. pub fn data_dir() -> Option { resolve_data_dir_from_parts( env_path("YOI_DATA_DIR"), @@ -51,11 +50,11 @@ pub fn data_dir() -> Option { ) } -/// Secret store 用の data directory。 +/// Data directory for the secret store. /// -/// SecretStore は base path から復号 key を導出するため、通常 data dir の -/// XDG fallback には追従させず、明示 override が無い場合は legacy -/// `$HOME/.yoi` を読み書きする。 +/// SecretStore derives decryption keys from the base path, so it does not follow +/// the regular data directory's XDG fallback. Without an explicit override, it +/// reads/writes legacy `$HOME/.yoi`. pub fn secret_data_dir() -> Option { resolve_secret_data_dir_from_parts( env_path("YOI_DATA_DIR"), @@ -64,8 +63,8 @@ pub fn secret_data_dir() -> Option { ) } -/// ランタイムディレクトリ。socket, `workers.json`, Worker ごとの `pid` / -/// `status.json` 等が置かれる。再起動で消えて構わない。 +/// Runtime directory for sockets, `workers.json`, per-Worker `pid` / +/// `status.json`, etc. Its contents may be discarded on restart. pub fn runtime_dir() -> Option { resolve_runtime_dir_from_parts( env_path("YOI_RUNTIME_DIR"), @@ -85,33 +84,33 @@ pub fn user_profiles_path() -> Option { user_profiles_path_from_config_dir(config_dir()) } -/// `/` — providers.toml / models.toml 等の -/// user override ファイル。 +/// `/` — user override files such as providers.toml / +/// models.toml. pub fn user_catalog_override(file_name: &str) -> Option { user_catalog_override_from_config_dir(config_dir(), file_name) } -/// `/sessions/` — session store のデフォルト位置。 +/// `/sessions/` — default session store location. pub fn sessions_dir() -> Option { sessions_dir_from_data_dir(data_dir()) } -/// `/workers.json` — machine-wide Worker allocation table。 +/// `/workers.json` — machine-wide Worker allocation table. pub fn worker_allocation_path() -> Option { worker_allocation_path_from_runtime_dir(runtime_dir()) } -/// `//` — Worker ごとのランタイムディレクトリ。 +/// `//` — per-Worker runtime directory. pub fn worker_runtime_dir(worker_name: &str) -> Option { worker_runtime_dir_from_runtime_dir(runtime_dir(), worker_name) } -/// `//sock` — Worker の Unix socket パス。 +/// `//sock` — Worker's Unix socket path. /// -/// Worker プロセス内で実際に socket を作成するのは `worker` crate の -/// `RuntimeDir::socket_path()` で、Worker 名が分かっている外部 (TUI の -/// attach フロー等) からの**予測**はこの関数で行う。両者は同じパス -/// を返すことが期待される。 +/// The `worker` crate's `RuntimeDir::socket_path()` creates the socket inside the +/// Worker process. This function **predicts** the path for external callers that +/// know the Worker name, such as the TUI attach flow. Both should return the same +/// path. pub fn worker_socket_path(worker_name: &str) -> Option { worker_socket_path_from_runtime_dir(runtime_dir(), worker_name) } @@ -219,9 +218,9 @@ fn worker_socket_path_from_runtime_dir( Some(worker_runtime_dir_from_runtime_dir(runtime_dir, worker_name)?.join("sock")) } -/// 空文字列の env は未設定として扱う。`std::env::var` は `Ok("")` と -/// `Err(NotPresent)` を区別するが、パス解決においては両者を未設定と -/// 同等に扱うのが直感的。 +/// Treats empty environment values as unset. `std::env::var` distinguishes +/// `Ok("")` from `Err(NotPresent)`, but path resolution treats both as unset +/// for intuitive behavior. fn env_path(name: &str) -> Option { let value = std::env::var(name).ok()?; path_from_env_value(Some(value.as_str())) diff --git a/crates/manifest/src/plugin.rs b/crates/manifest/src/plugin.rs index 1a14d8f2e..9686f3979 100644 --- a/crates/manifest/src/plugin.rs +++ b/crates/manifest/src/plugin.rs @@ -1,3 +1,19 @@ +//! Offline reading, validation, and writing of the `.yoi-plugin` format. +//! +//! Inspects only explicitly supplied directories or archives; does not implicitly +//! discover catalogs, register with Workers, or instantiate components. Validating +//! the format grants no execution authority. Worker Manifests/Profiles reject dynamic +//! Plugin configuration; package declarations and local paths do not grant Worker +//! capabilities. +//! +//! Archives use a restricted stored-ZIP subset. Unsafe paths, links, unsupported +//! entries, size/count limit violations, and inconsistent headers are rejected; +//! the file set is digested in order. Directory inputs also reject internal links +//! and undergo the same validation, so distribution formats share the package +//! contract. Component Model metadata is required; legacy core-Wasm runtime +//! declarations are rejected. Validation covers the offline format, not safety +//! established by running components. + use serde::{Deserialize, Serialize}; use sha2::{Digest, Sha256}; use std::collections::{BTreeMap, BTreeSet}; @@ -398,12 +414,11 @@ impl PluginPackageManifest { const LEGACY_PLUGIN_RUNTIME_WASM_KIND: &str = "wasm"; const LEGACY_PLUGIN_RUNTIME_WASM_ABI: &str = "yoi-plugin-wasm-1"; -/// Manifest runtime kind for current WebAssembly Component Model packages. +/// Component Model declaration kind accepted in offline packages. /// -/// Component runtime manifests must set `component` to the packaged component -/// artifact path and `world` to [`PLUGIN_COMPONENT_TOOL_WORLD`] or -/// [`PLUGIN_COMPONENT_INSTANCE_WORLD`]. Legacy raw core-Wasm manifests are -/// intentionally rejected by validation; `wasm-component` is the public runtime. +/// Validates component paths and corresponding worlds; rejects legacy core-Wasm +/// declarations. This constant implies neither runtime registration nor verified +/// component execution. pub const PLUGIN_RUNTIME_COMPONENT_KIND: &str = "wasm-component"; pub const PLUGIN_COMPONENT_TOOL_WORLD: &str = "yoi:plugin/tool@1.0.0"; pub const PLUGIN_COMPONENT_INSTANCE_WORLD: &str = "yoi:plugin/instance@1.0.0"; diff --git a/crates/manifest/src/profile.rs b/crates/manifest/src/profile.rs index 454643df7..60aabad97 100644 --- a/crates/manifest/src/profile.rs +++ b/crates/manifest/src/profile.rs @@ -1,9 +1,4 @@ -//! Profile discovery and resolution. -//! -//! Profiles are reusable, human-authored recipes. They are intentionally not -//! complete runtime manifests: runtime-bound and authority-bearing fields such -//! as `worker.name` and concrete `scope.allow` rules are supplied by the resolver -//! from launch context. +//! Profile selection and resolution. use serde::{Deserialize, Serialize}; use std::collections::{BTreeMap, BTreeSet}; @@ -461,6 +456,20 @@ impl ProfileResolveOptions { } } +/// Resolves a launch-time Manifest from a reusable Profile. +/// +/// Names and concrete scope rules come from the launch context, not the Profile, +/// so reusing configuration for another Worker does not inherit the original +/// Worker's identity or path authority. Scope intent and configured secret references +/// are retained; input validation rejects runtime-bound fields and Manifest-style +/// authority rules. +/// +/// Profiles resolve as values, not executable scripts. Omitted values use +/// [`crate::WorkerManifestConfig::resolution_defaults`], not an implicit builtin +/// parent role. Source-relative paths resolve against their own source before +/// Manifest conversion. [`ResolvedProfile`] keeps source values and the Manifest +/// snapshot separate, so restoration after launch need not reselect the current +/// Profile. #[derive(Debug, Clone, Default)] pub struct ProfileResolver { workspace_base: Option, diff --git a/crates/manifest/src/scope.rs b/crates/manifest/src/scope.rs index 270cf61ae..98a30918a 100644 --- a/crates/manifest/src/scope.rs +++ b/crates/manifest/src/scope.rs @@ -1,11 +1,4 @@ -//! Runtime representation of a Worker's access scope. -//! -//! Built from [`crate::ScopeConfig`] via [`Scope::from_config`]. Every -//! rule `target` must already be an absolute path — per-layer path -//! resolution runs earlier, inside [`crate::WorkerManifestConfig::resolve_paths`]. -//! All rule targets retain both their lexically normalized logical identity and -//! their provider-resolved identity. Allow rules select one identity explicitly; -//! deny rules always inspect both so aliases cannot bypass a restriction. +//! Path authority and delegation scope checks. use std::ffi::OsString; use std::path::{Component, Path, PathBuf}; @@ -15,10 +8,25 @@ use arc_swap::{ArcSwap, Guard}; use crate::{Permission, ScopeConfig, ScopeRule, SymlinkPolicy}; -/// Parsed, pwd-resolved set of allow/deny rules for a Worker. +/// Checks Worker path authority and authority delegable to children. /// -/// Read/write access decisions are pure functions of the path being -/// queried and these rules — see [`Scope::permission_at`]. +/// Rule targets must already be absolute paths resolved by +/// [`crate::WorkerManifestConfig::resolve_paths`]. Each rule retains a lexically +/// normalized logical identity and a symlink-resolved identity. Allows default to +/// `resolved`; explicit `logical` rules allow logical paths even when link targets +/// lie outside the logical subtree. Denies check both identities, preventing aliases +/// or symlinks from bypassing restrictions. +/// +/// [`Scope`] grants direct access; [`DelegationScope`] grants authority transferable +/// to children. Direct access alone does not permit delegation. Delegated rules must +/// fit existing allows, including permission, recursion, and symlink policy, and +/// must not overlap denies. A `resolved` allow cannot expand into a `logical` rule. +/// +/// Parent write restrictions during delegation can be additional denies. On return, +/// remove added rules one at a time by exact match, without lifting preexisting +/// denies in the same region. [`SharedScope`] serializes mutations and exposes the +/// same latest scope to checks. File-operation path validation and reclaiming +/// authority when children stop remain with their respective owners. #[derive(Debug, Clone)] pub struct Scope { allow: Vec, diff --git a/crates/mcp/README.md b/crates/mcp/README.md new file mode 100644 index 000000000..4d8b6faa7 --- /dev/null +++ b/crates/mcp/README.md @@ -0,0 +1,7 @@ +# mcp + +`mcp` implements local stdio MCP server startup, JSON-RPC communication, discovery, and shutdown. The `worker` Feature owns decisions to register MCP Tools, resources, and prompts for models; this crate handles communication and process lifetime. + +## Main entry points + +- [`stdio`](src/stdio.rs): Server launch configuration resolution, bounded responses, diagnostics, and shutdown. diff --git a/crates/mcp/src/lib.rs b/crates/mcp/src/lib.rs index f382de19e..0b8fdd618 100644 --- a/crates/mcp/src/lib.rs +++ b/crates/mcp/src/lib.rs @@ -1,7 +1,3 @@ -//! Model Context Protocol client foundations. -//! -//! This crate intentionally only owns protocol/lifecycle plumbing. It does not -//! register MCP tools, resources, or prompts into Yoi's model-visible tool -//! surface. +#![doc = include_str!("../README.md")] pub mod stdio; diff --git a/crates/mcp/src/stdio.rs b/crates/mcp/src/stdio.rs index 7379d65e8..25eb0c049 100644 --- a/crates/mcp/src/stdio.rs +++ b/crates/mcp/src/stdio.rs @@ -1,3 +1,20 @@ +//! Local stdio MCP server communication and process lifecycle. +//! +//! The Host resolves launch specs, clears the environment, and passes only explicit +//! values to child processes. Values resolved from SecretStore or inherited from +//! the environment are redacted in diagnostics. This is not a sandbox: child +//! execution capabilities depend on Host and OS user boundaries. +//! +//! Handles stdout as JSON-RPC and stderr as bounded diagnostics, separately. +//! Limits line lengths, response bytes, discovery pages, and request deadlines; +//! malformed communication is not registered as valid descriptors. Requests are +//! accepted only after the initialize/initialized handshake. Shutdown closes stdin, +//! waits for exit within a deadline, and kills/waits to reap the process if needed. +//! Startup failures also trigger cleanup; killing on Drop is a last resort. +//! +//! This module preserves Tool/resource/prompt wire values. `worker::feature::mcp` +//! owns Yoi naming, schema acceptance, model output limits, and Feature registration. + use std::collections::{BTreeMap, BTreeSet, VecDeque}; use std::env; use std::fmt; diff --git a/crates/memory/README.md b/crates/memory/README.md index ef11f2ec1..3f3cb1ab1 100644 --- a/crates/memory/README.md +++ b/crates/memory/README.md @@ -1,23 +1,11 @@ # memory -## Role +Defines shared Memory schemas, extraction candidates, evidence, and typed communication. +Does not own persistence or implicitly convert between legacy single-document Workspace Memory and per-subject Memory. -`memory` owns the shared types used by Workspace-backed Memory operations: +## Main entry points -- Memory API request/response DTOs -- extraction candidate and evidence schemas -- audit event schemas -- pure Memory document frontmatter schemas and parsing helpers - -## Boundaries - -The crate does **not** own persistence. Workspace Memory records, staging candidates, and audit observations are stored by the Workspace Server control plane and accessed by Workers through typed API operations. - -The crate intentionally contains no repository-local layout, cwd/ancestor discovery, resident-file reader, staging writer, audit/usage log writer, generic filesystem scope helper, or local backend executor. The product CLI does not expose `yoi memory lint`. - -Memory is useful context, not implementation authority. Tickets, Objectives, repository files, Git history, and append-only session records remain the exact evidence surfaces for implementation state. - -## See also - -- [`../../docs/design/memory-knowledge.md`](../../docs/design/memory-knowledge.md) -- [`../../docs/development/work-items.md`](../../docs/development/work-items.md) +- [`backend`]: legacy Workspace Memory operation contracts and resident responses +- [`extract`]: candidate extraction and progress pointers +- [`schema`]: shared document and evidence types +- [`audit`]: audit-event transport types diff --git a/crates/memory/src/audit.rs b/crates/memory/src/audit.rs index eff4f9b13..876b71ffc 100644 --- a/crates/memory/src/audit.rs +++ b/crates/memory/src/audit.rs @@ -1,7 +1,7 @@ -//! Typed Memory audit events sent to the Workspace authority. +//! Transport types for Memory audit events. //! -//! Runtime Workers construct these DTOs and submit them through the Memory -//! backend operation boundary. Repository-local audit logs are not an authority. +//! Runtime Workers pass these DTOs to backend operations. This crate does not store audit logs, +//! and DTOs alone do not guarantee persistence. Session storage owns Session records. use chrono::{DateTime, Utc}; use schemars::JsonSchema; diff --git a/crates/memory/src/backend.rs b/crates/memory/src/backend.rs index c8186bf97..36fe18149 100644 --- a/crates/memory/src/backend.rs +++ b/crates/memory/src/backend.rs @@ -1,8 +1,11 @@ -//! Typed transport boundary for Workspace-backed Memory operations. +//! Typed communication boundary for legacy single-document Workspace Memory. //! -//! Runtime Workers serialize these DTOs to the Workspace API. Persistence and -//! validation are owned by the Workspace authority; this crate deliberately has -//! no repository-local filesystem executor. +//! Workspace authority owns persistence and validation of documents, candidates, and dispositions. +//! Does not reinterpret existing input as subject Memory, implicitly assign subjects, or dual-write. +//! This module has no storage or fallback implementation that searches Repositories or ancestor directories. +//! Resident response types are shared with subject Memory, but do not imply identical storage destinations. +//! [`MemoryResidentSummaryOutput`] represents availability separately from body presence. +//! `Ready` without content is a successfully generated empty result, distinct from ungenerated, stale, or failed. use schemars::JsonSchema; use serde::{Deserialize, Serialize}; diff --git a/crates/memory/src/extract/input.rs b/crates/memory/src/extract/input.rs index e2169df46..e68880e3f 100644 --- a/crates/memory/src/extract/input.rs +++ b/crates/memory/src/extract/input.rs @@ -1,13 +1,12 @@ -//! extract sub-Engine への入力テキスト組み立て。 +//! Builds input text for the extract sub-Engine. //! -//! `crates/worker/src/worker.rs::build_summary_prompt` と同じ方針で -//! Item 列を flat な行に落とす（reasoning は省く、tool call は名前のみ、 -//! tool result は summary のみ）。conversation 全体を Markdown の単一 -//! セクションとして渡し、抽出指示は system prompt 側に寄せる。 +//! Flattens the supplied Items into lines (omitting reasoning, keeping only tool-call names +//! and tool-result summaries). Passes the conversation slice as a single Markdown +//! section, leaving extraction instructions to the system prompt. use agen::Item; -/// 与えられた `items` を extract sub-Engine の最初の user 入力に整形する。 +/// Formats `items` as the extract sub-Engine's initial user input. pub fn build_extract_input(items: &[Item]) -> String { let mut out = String::new(); out.push_str( diff --git a/crates/memory/src/extract/mod.rs b/crates/memory/src/extract/mod.rs index 2b934804f..e66481644 100644 --- a/crates/memory/src/extract/mod.rs +++ b/crates/memory/src/extract/mod.rs @@ -1,8 +1,8 @@ -//! Memory candidate extraction domain types and Worker tool boundary. +//! Boundary for Memory extraction candidates and Worker-facing operations. //! -//! Worker-side extraction produces typed candidates that are submitted to the -//! Workspace backend. Staging persistence is owned by that backend; this module -//! does not write repository-local files. +//! Extraction creates typed candidates and passes them to Host storage. Saving candidates and +//! committing the extraction pointer are separate responsibilities; generating candidates alone does not update committed Memory. +//! This module does not store data in Repository files. mod input; mod payload; @@ -17,6 +17,6 @@ pub use payload::{ pub use pointer::{ExtractPointerPayload, fold_pointer}; pub use tool::{ExtractWorkerContext, write_extracted_tool}; -/// session-store `LogEntry::Extension` で使う domain 名。 -/// pointer の永続化と読み出しはこの定数を使う側が一致している必要がある。 +/// Domain name used in session-store `LogEntry::Extension`. +/// Callers persisting and reading pointers must agree on this constant. pub const EXTRACT_DOMAIN: &str = "memory.extract"; diff --git a/crates/memory/src/extract/pointer.rs b/crates/memory/src/extract/pointer.rs index e3400bca1..aaa9532e5 100644 --- a/crates/memory/src/extract/pointer.rs +++ b/crates/memory/src/extract/pointer.rs @@ -1,28 +1,29 @@ -//! `LogEntry::Extension { domain: "memory.extract", payload }` の payload 形式と -//! restore 時の fold ヘルパー。memory crate がドメインを所有するので、 -//! session-store / Worker は payload 構造を知らない。 +//! Payload format for `LogEntry::Extension { domain: "memory.extract", payload }` +//! and fold helper for restore. The memory crate owns the domain, so +//! session-store / Worker do not know the payload structure. use serde::{Deserialize, Serialize}; use super::EXTRACT_DOMAIN; -/// extract 完了境界の永続化 payload。session log の Extension entry -/// として 1 回ずつ書かれ、最新の 1 件が現行 pointer として有効になる。 +/// Payload recording an extraction completion boundary as a Session Extension entry. +/// Only the latest entry is the current pointer; the Host lifecycle checks storage completion conditions. +/// This commits separately from candidate storage, so saved candidates may remain even without a committed pointer. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] pub struct ExtractPointerPayload { - /// 直近 extract が処理した最後の session-store LogEntry の index。 - /// 次回の `source.range.start` はこの値 + 1。 + /// Index of the last session-store LogEntry processed by the latest extraction. + /// The next `source.range.start` is this value + 1. pub processed_through_entry: usize, - /// 直近 extract 時点の `history.len()`。次回入力は - /// `history[processed_through_history_len..]` を切り出す。 + /// `history.len()` at the latest extraction. The next input slices + /// `history[processed_through_history_len..]`. pub processed_through_history_len: usize, - /// 書き出した staging file の UUIDv7 文字列。LLM が空 payload を返した - /// 場合は staging file を作らず空文字列で記録する（pointer は前進する）。 + /// Staging record ID returned by storage. Completion without candidates may use an empty value. + /// Does not denote a local file's existence or storage path. pub staging_id: String, } -/// `RestoredState.extensions` から最新の extract pointer を取り出す。 -/// 未抽出セッションでは `None`。 +/// Returns the latest extraction pointer, or `None` if absent or the latest payload is invalid. +/// No fallback: ignoring a corrupt latest entry and using an older pointer would yield the wrong re-extraction range. pub fn fold_pointer(extensions: &[(String, serde_json::Value)]) -> Option { extensions .iter() @@ -74,8 +75,8 @@ mod tests { EXTRACT_DOMAIN.to_string(), serde_json::json!({ "wrong_shape": true }), )]; - // 現状は最新を取り出して JSON 不一致なら None。古いものに fallback - // しないのは、壊れた最新を黙って無視すると意図しない再抽出を招くため。 + // Takes the latest entry and returns None if its JSON does not match. + // No fallback: silently ignoring a corrupt latest entry could cause unintended re-extraction. assert!(fold_pointer(&exts).is_none()); } } diff --git a/crates/memory/src/extract/tool.rs b/crates/memory/src/extract/tool.rs index 202953e8d..892b8e843 100644 --- a/crates/memory/src/extract/tool.rs +++ b/crates/memory/src/extract/tool.rs @@ -1,9 +1,8 @@ -//! `write_extracted` ツール実装と sub-Engine 用 context。 +//! Temporary storage receiving `write_extracted` results for the legacy extraction engine. //! -//! sub-Engine からは extract worker が出した [`ExtractedPayload`] を -//! 受け取って `Mutex` 越しに [`ExtractWorkerContext`] に置くだけ。 -//! Worker 側はランループ完了後に `take_payload()` で取り出して -//! [`super::staging::write_staging`] に渡す。 +//! Validates [`ExtractedPayload`] and holds it in [`ExtractWorkerContext`]'s Mutex. +//! Callers retrieve the result with [`ExtractWorkerContext::take_payload`]. +//! This module neither persists candidates nor decides whether to adopt them. use std::sync::{Arc, Mutex}; @@ -17,12 +16,12 @@ Pass an object with a `candidates` array. Each candidate must have `kind`, `clai `staleness` and `evidence_ids` are optional. Call this exactly once and end the turn. Do not include \ record ids, source anchors, session metadata, or free-form prose — the wrapper attaches staging metadata mechanically."; -/// extract sub-Engine の出力受け口。`ExtractedPayload` 1 件をホストする。 +/// Output receiver for the extract sub-Engine. Holds one `ExtractedPayload`. #[derive(Debug, Default)] pub struct ExtractWorkerContext { payload: Mutex>, - /// `write_extracted` が複数回呼ばれた回数（debug 用）。 - /// 後勝ちで上書きするが、Worker 側で warn を出したい場合に参照する。 + /// Count of `write_extracted` calls (for debugging). + /// Later calls overwrite earlier payloads; Workers can use this to emit a warning. call_count: Mutex, } @@ -31,8 +30,8 @@ impl ExtractWorkerContext { Self::default() } - /// sub-Engine 終了後に Worker が呼んで payload を取り出す。 - /// 一度も `write_extracted` が呼ばれなければ `None`。 + /// Called by the Worker after the sub-Engine exits to retrieve the payload. + /// Returns `None` if `write_extracted` was never called. pub fn take_payload(&self) -> Option { self.payload .lock() @@ -90,7 +89,7 @@ impl Tool for WriteExtractedTool { } } -/// sub-Engine に register する `write_extracted` ツール定義を返す。 +/// Returns the `write_extracted` tool definition to register with the sub-Engine. pub fn write_extracted_tool(ctx: Arc) -> ToolDefinition { Arc::new(move || { let schema = schemars::schema_for!(ExtractedPayload); diff --git a/crates/memory/src/lib.rs b/crates/memory/src/lib.rs index 4afa340aa..8f67d9494 100644 --- a/crates/memory/src/lib.rs +++ b/crates/memory/src/lib.rs @@ -1,8 +1,4 @@ -//! Shared Memory domain and Workspace API transport types. -//! -//! Normal Memory persistence is owned by the Workspace control plane. This crate -//! intentionally contains no repository-local `.yoi` layout, discovery, reader, -//! writer, or fallback implementation. +#![doc = include_str!("../README.md")] pub mod audit; pub mod backend; diff --git a/crates/merge-request/README.md b/crates/merge-request/README.md new file mode 100644 index 000000000..1d6e8aefb --- /dev/null +++ b/crates/merge-request/README.md @@ -0,0 +1,10 @@ +# merge-request + +Handles SQLite Merge Request records, review evidence, and integration results. +Does not observe Repository sources, execute Git integration, or decide Ticket completion. + +## Main entry points + +- [`MergeRequestStore`]: records and state updates +- [`MergeRequest::integration_approval`]: approval evidence for stored integration results +- [`requirement_approval`]: approval evidence for current Ticket requirements and the related source set diff --git a/crates/merge-request/src/lib.rs b/crates/merge-request/src/lib.rs index 21054a748..d6da1151d 100644 --- a/crates/merge-request/src/lib.rs +++ b/crates/merge-request/src/lib.rs @@ -1,3 +1,5 @@ +#![doc = include_str!("../README.md")] + use chrono::{DateTime, Utc}; use rusqlite::{Connection, OptionalExtension, Transaction, params}; use serde::{Deserialize, Serialize}; @@ -316,7 +318,10 @@ impl MergeRequest { self.thread.iter().rev().find_map(|e|match e{MergeRequestThreadEvent::Review(v)if v.subject_ref==subject&&!self.thread.iter().any(|x|matches!(x,MergeRequestThreadEvent::ReviewRevoked(r)if r.review_event_id==v.event_id))=>Some(v),_=>None}) } - /// Returns the recorded merged result, without resolving selectors or mutating state. + /// Returns the stored immutable integration result without resolving selectors again. + /// Pins the approved source and approval event; branch movement/deletion or later requirement reviews never replace them. + /// Post-integration evidence evaluation uses the pinned source, independent of current branch/provider availability. + /// Do not infer approval from `merged` state alone; verify evidence with [`Self::integration_approval`]. pub fn merged_result(&self) -> Result<&MergeEvent, MergeRequestEvidenceError> { if self.state != MergeRequestState::Merged { return Err(MergeRequestEvidenceError::NotMerged); @@ -331,8 +336,9 @@ impl MergeRequest { .ok_or(MergeRequestEvidenceError::MergeResultMissing) } - /// Validates the specific approval recorded by the merged result. A later - /// requirement review does not replace this integration evidence. + /// Verifies source equality, ordering, non-revocation, and the matching review request for the approval event named by the integration result. + /// Later requirement approval does not replace integration-time approval. Validation requires the full thread; + /// a merge event in a page alone cannot prove approval exists. pub fn integration_approval(&self) -> Result<&ReviewEvent, MergeRequestEvidenceError> { let merge = self.merged_result()?; let review = self @@ -393,12 +399,15 @@ impl MergeRequest { } } -/// Finds an effective approval of the authoritative current Ticket digest and -/// exact linked source snapshot. Callers supply all linked requests and resolved -/// source refs (recorded merged refs for completion); this function does not read -/// authority, resolve selectors, require merged state, or mutate lifecycle state. -/// An explicit event id validates that approval only; `None` discovers any valid -/// attestation. Snapshot ordering is immaterial, but duplicate ids are rejected. +/// Finds a valid review approving the current Ticket digest and the exact set of all related MR/sources. +/// The source of the MR holding the review and its corresponding ReviewRequested must also match. +/// Comments, recent timestamps, or one representative MR cannot substitute for approval of all requirements. +/// +/// The caller supplies all MRs and observed sources, using stored sources for integrated MRs. +/// Does not resolve selectors, require merged state, or authorize/change Ticket completion. +/// A post-integration review may approve a new requirement set while integration results remain immutable. +/// With an event ID, validates only that approval; otherwise searches for a valid approval. +/// Set order does not matter, but duplicate IDs are rejected. pub fn requirement_approval<'a>( requests: &'a [MergeRequest], content_digest: &str, @@ -656,6 +665,9 @@ pub enum MergeRequestError { Storage(#[from] rusqlite::Error), } +/// SQLite storage for Workspace/Repository-scoped MR records and thread events. +/// External authority owns Git operations and source observation; this store checks supplied results, +/// current assignments, and review evidence. Ticket completion decisions are not this store's responsibility. pub struct MergeRequestStore { conn: Arc>, assignments: Arc, @@ -1174,6 +1186,9 @@ impl MergeRequestStore { Ok(()) } + /// Records externally performed integration results with the approved source. + /// Rechecks assignments, Ticket content/state, and valid approval before commit. + /// Replay with the same operation ID and all inputs returns the same result; reuse with different inputs is rejected. pub fn complete(&self, i: CompleteMergeRequest) -> Result { self.validate_completion(&i)?; let mut connection = self.lock()?; diff --git a/crates/plugin-pdk/README.md b/crates/plugin-pdk/README.md new file mode 100644 index 000000000..50e2ee049 --- /dev/null +++ b/crates/plugin-pdk/README.md @@ -0,0 +1,9 @@ +# plugin-pdk + +Guest-side JSON/WIT helpers for offline Component Model packages. Without depending on host/runtime crates, it converts Tool and Service inputs/outputs and bounds Tool output size. Using the PDK or validating a package does not authorize installation or execution in a Worker. + +## Main entry points + +- [`run_json_tool`] and [`export_component_tool!`] — typed Tool input/output boundary +- [`ToolOutput`] and [`ToolError`] — bounded output and errors +- [`ServiceOutput`] and [`export_plugin_instance!`] — instance output contracts diff --git a/crates/plugin-pdk/src/lib.rs b/crates/plugin-pdk/src/lib.rs index f1aa137d5..592a45687 100644 --- a/crates/plugin-pdk/src/lib.rs +++ b/crates/plugin-pdk/src/lib.rs @@ -1,36 +1,4 @@ -//! Guest-side helpers for Yoi Component Model Tool plugins. -//! -//! This crate is intentionally small and guest-only: it depends on JSON/WIT -//! binding support, but not on Yoi host/runtime crates. It grants no authority; -//! package manifests and host-side Plugin grants decide whether a Tool or host -//! API can run. -//! -//! Component authors still generate the WIT bindings in their guest crate, then -//! delegate the exported `call` function to [`run_json_tool`] or to the -//! [`export_component_tool!`] macro: -//! -//! ```ignore -//! use yoi_plugin_pdk::wit_bindgen; -//! -//! wit_bindgen::generate!({ -//! world: "tool", -//! path: "../../../../resources/plugin/wit", -//! generate_all, -//! runtime_path: "yoi_plugin_pdk::wit_bindgen::rt", -//! }); -//! -//! fn echo( -//! ctx: yoi_plugin_pdk::ToolContext, -//! input: EchoInput, -//! ) -> Result { -//! yoi_plugin_pdk::ToolOutput::json( -//! format!("{} ok", ctx.tool_name()), -//! EchoOutput { text: input.text }, -//! ) -//! } -//! -//! yoi_plugin_pdk::export_component_tool!(Plugin, echo); -//! ``` +#![doc = include_str!("../README.md")] use serde::Serialize; use serde::de::DeserializeOwned; @@ -51,9 +19,9 @@ pub const TOOL_WIT: &str = include_str!("../../../resources/plugin/wit/yoi-plugi pub const HOST_WIT: &str = include_str!("../../../resources/plugin/wit/deps/yoi-host/yoi-host-v1.wit"); -/// Maximum serialized ToolOutput JSON accepted by Yoi's current Plugin runtime. +/// Serialized JSON limit defined by the offline Tool world's output contract. pub const MAX_TOOL_OUTPUT_BYTES: usize = 64 * 1024; -/// Maximum summary bytes accepted by Yoi's current Plugin runtime. +/// Byte limit for summaries in the offline Tool world. pub const MAX_SUMMARY_BYTES: usize = 1024; /// Conservative content cap that leaves room for JSON framing and escaping. pub const MAX_CONTENT_BYTES: usize = MAX_TOOL_OUTPUT_BYTES - MAX_SUMMARY_BYTES - 4096; @@ -80,7 +48,11 @@ impl ToolContext { } } -/// ToolOutput JSON shape accepted by the current Yoi Plugin runtime. +/// Bounded Tool output formatted on the guest side. +/// +/// Shorten text at character boundaries and recheck the serialized JSON limit, including escapes. +/// Typed JSON through [`Self::json`] errors on overflow rather than truncating and breaking +/// its structure. Providing this output format does not provide Plugin execution by Workers. #[derive(Clone, Debug, PartialEq, Eq, Serialize)] pub struct ToolOutput { summary: String, @@ -308,11 +280,11 @@ where }) } -/// Execute a typed JSON Tool handler and return ToolOutput JSON for the runtime. +/// Guest-side boundary that runs a typed JSON handler and converts the result to ToolOutput JSON. /// -/// Handler errors and parse/serialization failures are not panics or host-side -/// authority decisions. They are rendered into ordinary ToolOutput JSON so the -/// runtime can route them through the normal Tool result path. +/// Convert parse, handler, and serialization failures to the normal output form. These errors +/// are not host authorization decisions, and catching handler panics is not guaranteed. +/// The PDK does not depend on host/runtime crates or grant permissions through input conversion. pub fn run_json_tool(tool_name: &str, input_json: &str, handler: F) -> String where I: DeserializeOwned, @@ -328,16 +300,11 @@ where } } -/// Implement the generated Component Model `Guest` trait for a typed JSON -/// handler and export it with the `wit-bindgen` generated `export!` macro. +/// Connect a typed JSON handler to a WIT `Guest` implementation and component export. /// -/// The caller must import the PDK's `wit_bindgen` re-export and invoke -/// `wit_bindgen::generate!` for the `tool` world first, with -/// `runtime_path: "yoi_plugin_pdk::wit_bindgen::rt"`. That defines the -/// `Guest` trait and `export!` macro in the current module. The generated -/// component still imports only WIT-declared -/// host APIs; this macro does not grant filesystem, network, or environment -/// authority. +/// Use guest-generated bindings and centralize handler conversion in [`run_json_tool`]. +/// This macro grants no host API permissions and is not a mechanism for registering or +/// executing dynamic Plugins in the current Worker. #[macro_export] macro_rules! export_component_tool { ($adapter:ident, $handler:path) => { @@ -642,6 +609,11 @@ impl ServiceOutputCommand { } } +/// Instance processing results and output commands requested from the host. +/// +/// The guest does not send anything here; it returns commands as data tied to the original +/// ingress event. Command creation/serialization and host authorization/execution are separate +/// boundaries. This type provides neither network permissions nor the current Worker execution mechanism. #[derive(Clone, Debug, PartialEq, serde::Serialize, serde::Deserialize)] pub struct ServiceOutput { #[serde(default)] diff --git a/crates/plugin-pdk/tests/fixtures/component-tool-source.txt b/crates/plugin-pdk/tests/fixtures/component-tool-source.txt new file mode 100644 index 000000000..4311cc2f0 --- /dev/null +++ b/crates/plugin-pdk/tests/fixtures/component-tool-source.txt @@ -0,0 +1,38 @@ +//! Minimal Component Model Tool plugin authoring sketch using `yoi-plugin-pdk`. +//! +//! Build this as a `wasm32-unknown-unknown` cdylib with Component Model tooling +//! and package the adapted component as `plugin.component.wasm`. + +use serde::{Deserialize, Serialize}; +use yoi_plugin_pdk::wit_bindgen; +use yoi_plugin_pdk::{ToolContext, ToolError, ToolOutput}; + +wit_bindgen::generate!({ + world: "tool", + path: "../../../resources/plugin/wit", + generate_all, + runtime_path: "yoi_plugin_pdk::wit_bindgen::rt", +}); + +#[derive(Debug, Deserialize)] +struct EchoInput { + text: String, +} + +#[derive(Debug, Serialize)] +struct EchoOutput<'a> { + tool: &'a str, + text: String, +} + +fn handle_echo(ctx: ToolContext, input: EchoInput) -> Result { + ToolOutput::json( + format!("{} ok", ctx.tool_name()), + EchoOutput { + tool: ctx.tool_name(), + text: input.text, + }, + ) +} + +yoi_plugin_pdk::export_component_tool!(Plugin, handle_echo); diff --git a/crates/plugin-pdk/tests/template.rs b/crates/plugin-pdk/tests/template.rs index c972f99e9..29d15b684 100644 --- a/crates/plugin-pdk/tests/template.rs +++ b/crates/plugin-pdk/tests/template.rs @@ -10,17 +10,14 @@ const TEMPLATE_LIB: &str = include_str!("../../../resources/plugin/templates/rust-component-tool/src/lib.rs"); const TEMPLATE_PLUGIN: &str = include_str!("../../../resources/plugin/templates/rust-component-tool/plugin.toml"); -const TEMPLATE_README: &str = - include_str!("../../../resources/plugin/templates/rust-component-tool/README.md"); const SERVICE_TEMPLATE_CARGO: &str = include_str!("../../../resources/plugin/templates/rust-component-instance/Cargo.toml"); const SERVICE_TEMPLATE_LIB: &str = include_str!("../../../resources/plugin/templates/rust-component-instance/src/lib.rs"); const SERVICE_TEMPLATE_PLUGIN: &str = include_str!("../../../resources/plugin/templates/rust-component-instance/plugin.toml"); -const SERVICE_TEMPLATE_README: &str = - include_str!("../../../resources/plugin/templates/rust-component-instance/README.md"); -const SAMPLE_LIB: &str = include_str!("../../../docs/examples/plugin-component-tool/lib.rs"); +// Preserve the removed document's source as a byte-identical fixture to keep the test input unchanged. +const SAMPLE_LIB: &str = include_str!("fixtures/component-tool-source.txt"); const PDK_CARGO: &str = include_str!("../Cargo.toml"); #[test] @@ -50,7 +47,6 @@ fn rust_component_tool_template_has_expected_files() { assert!(TEMPLATE_LIB.contains("runtime_path: \"yoi_plugin_pdk::wit_bindgen::rt\"")); assert!(TEMPLATE_LIB.contains("yoi_plugin_pdk::export_component_tool!")); assert!(TEMPLATE_LIB.contains("ToolOutput::json")); - assert!(TEMPLATE_README.contains("Component Model Tool Plugin")); } #[test] @@ -114,12 +110,11 @@ fn rust_component_service_template_has_event_output_pattern() { assert!(SERVICE_TEMPLATE_LIB.contains("PluginIngressEvent")); assert!(SERVICE_TEMPLATE_LIB.contains("ServiceOutput::websocket_send")); assert!(!SERVICE_TEMPLATE_LIB.contains("recv(timeout")); - assert!(SERVICE_TEMPLATE_README.contains("output command")); assert!(!SERVICE_TEMPLATE_PLUGIN.contains("kind = \"wasm\"")); } #[test] -fn documented_sample_uses_pdk_component_path() { +fn component_tool_fixture_uses_pdk_bindings() { assert!(SAMPLE_LIB.contains("use yoi_plugin_pdk::wit_bindgen")); assert!(SAMPLE_LIB.contains("wit_bindgen::generate!")); assert!(SAMPLE_LIB.contains("generate_all")); diff --git a/crates/project-record/README.md b/crates/project-record/README.md new file mode 100644 index 000000000..c144c09e0 --- /dev/null +++ b/crates/project-record/README.md @@ -0,0 +1,9 @@ +# project-record + +Provides timestamp encoding, validation, and collision-aware candidate allocation for record IDs. +The caller owns record persistence and concurrency control for writes. + +## Main entry points + +- [`encode_unix_epoch_millis`], [`decode_unix_epoch_millis`]: fixed-width ID conversion +- [`allocate_record_id`]: candidate selection avoiding existing IDs diff --git a/crates/project-record/src/lib.rs b/crates/project-record/src/lib.rs index 76a3712b0..84ac74484 100644 --- a/crates/project-record/src/lib.rs +++ b/crates/project-record/src/lib.rs @@ -1,8 +1,4 @@ -//! Shared path-safe project-record identifiers. -//! -//! Record IDs are fixed-width Crockford base32 encodings of Unix epoch -//! milliseconds. The fixed width keeps lexicographic order aligned with -//! chronological order. +#![doc = include_str!("../README.md")] use std::fmt; use std::time::{SystemTime, UNIX_EPOCH}; @@ -54,6 +50,9 @@ pub fn unix_epoch_millis_now() -> Result { u64::try_from(duration.as_millis()).map_err(|_| RecordIdError::TimestampOverflow) } +/// Converts Unix epoch milliseconds to fixed-width, path-component-safe Crockford base32. +/// Fixed width makes lexical order match encoded timestamp order. +/// Allocation may shift timestamps to avoid collisions, so an ID does not prove the actual creation time. pub fn encode_unix_epoch_millis(millis: u64) -> String { let mut value = millis; let mut out = [b'0'; RECORD_ID_WIDTH]; @@ -85,6 +84,9 @@ pub fn validate_record_id(id: &str) -> Result<(), RecordIdError> { decode_unix_epoch_millis(id).map(|_| ()) } +/// Selects a candidate avoiding existing IDs by incrementing milliseconds from the base timestamp. +/// Keeps the same encoding without a suffix and stops at the probe limit. +/// This is not a storage reservation. The caller owns concurrency control between checking and saving. pub fn allocate_record_id(base_millis: u64, mut exists: F) -> Result where F: FnMut(&str) -> bool, diff --git a/crates/protocol/README.md b/crates/protocol/README.md index e85403e15..ef1f9d782 100644 --- a/crates/protocol/README.md +++ b/crates/protocol/README.md @@ -1,31 +1,13 @@ # protocol -## Role +`protocol` defines input, control, event, and public history types shared by Workers +and clients. It does not own transport, stored-log replay, or rendering. It separates +live state from committed history so clients can reconnect without depending on storage formats. -`protocol` defines the JSONL message boundary between Worker clients and Worker servers. +## Design entry points -## Boundaries - -Owns: - -- transport-neutral method/event/result types -- request/reply and broadcast event shapes -- protocol error categories shared by clients and servers - -Does not own: - -- Unix socket implementation details (`client`, `worker`) -- TUI rendering (`tui`) -- Engine history semantics (`agen`) -- durable storage (`session-store`, `session-store` worker metadata) - -## Design notes - -The exact enum variants are code authority. The README should describe the boundary, not duplicate every message shape. - -Protocol events can inform UI and orchestration, but durable state changes still need to flow through Worker/session/metadata records. - -## See also - -- [`../../docs/design/worker-session-state.md`](../../docs/design/worker-session-state.md) -- [`../../docs/design/context-history.md`](../../docs/design/context-history.md) +- [`Method`] / [`Event`]: input acceptance versus live updates and history commits +- [`WorkerStateSnapshot`]: controller state versus Runtime catalog state +- [`SessionSnapshot`]: public representation of committed history +- [`subscription`]: ordering of snapshots and subsequent events +- [`invocation`]: validation of typed Feature input diff --git a/crates/protocol/src/lib.rs b/crates/protocol/src/lib.rs index dfa3a53d1..9bd93d407 100644 --- a/crates/protocol/src/lib.rs +++ b/crates/protocol/src/lib.rs @@ -1,3 +1,5 @@ +#![doc = include_str!("../README.md")] + pub mod identity; pub mod invocation; #[cfg(feature = "stream")] @@ -211,6 +213,10 @@ pub struct WorkerStateSnapshot { pub state: WorkerState, } +/// Handles current controller state; input admission is not inferred from past history. +/// +/// Distinguishes Run from maintenance. The Runtime catalog's `Stopped` state persists +/// after shutdown and is not part of this controller snapshot. impl WorkerStateSnapshot { pub fn initial() -> Self { Self { @@ -445,6 +451,11 @@ pub enum WorkerEvent { }, } +/// Separates child reports into control handling and model-facing notifications. +/// +/// Scope registration updates are control-only and do not wake the parent's model. +/// Messages may arrive out of order; child notifications prompt state checks, +/// not proof of completed artifacts or validation. impl WorkerEvent { /// Whether this event should become an agent-visible notification/history item. /// @@ -732,6 +743,13 @@ impl Segment { } } +/// Treats input acceptance and model-history inclusion as separate boundaries. +/// +/// Submit/Notify receipts do not imply execution success; the corresponding Event +/// reports committed history. Human Submit requests use a waiting FIFO, while agents +/// use SubmitIfIdle only when Idle, never implicitly scheduling the next Run. +/// Pause/Resume suspend and resume a Run independently of new Submit acceptance; +/// Resume does not create a new user message. impl Method { /// Convenience: a `Submit` carrying a single `Segment::Text`. /// Used by dumb clients, inter-Worker tools, and tests that only have diff --git a/crates/protocol/src/subscription.rs b/crates/protocol/src/subscription.rs index 5a59175ca..9a7dee696 100644 --- a/crates/protocol/src/subscription.rs +++ b/crates/protocol/src/subscription.rs @@ -1,9 +1,19 @@ -//! Shared subscription-multiplexer wire contract. +//! Transport contract for multiple subscriptions over one connection. //! -//! This protocol is distinct from the single-Worker [`crate::Method`] / [`crate::Event`] -//! protocol. A connection carries many independently managed subscriptions. The -//! connection endpoint owns authorization and maps the typed selectors below to -//! visible resources; clients cannot provide Workspace scope as selector input. +//! Manages subscriptions separately from a single Worker's [`crate::Method`] and +//! [`crate::Event`]. The endpoint owns authorization and selector resolution; +//! it does not accept Workspace scope from clients. Each subscription snapshot +//! anchors subsequent events; reconnecting replaces public state. +//! +//! Worker snapshots use public DTOs, not stored logs, keeping storage replay rules +//! out of clients. Committed history and uncommitted output stay separate; +//! subsequent history-commit events are matched by entry ID. Reasoning and internal +//! extension payloads are not exposed in conversation views. +//! +//! Input acceptance, stream deltas, and history commits are distinct boundaries; +//! live notifications alone do not establish durable state or completed results. +//! Turn boundaries also differ from individual generation requests. Transport +//! retries before streaming are retry notifications for the same generation request. use std::collections::HashSet; use std::fmt; diff --git a/crates/protocol/src/typescript.rs b/crates/protocol/src/typescript.rs index 1aab7686d..458c1c257 100644 --- a/crates/protocol/src/typescript.rs +++ b/crates/protocol/src/typescript.rs @@ -173,11 +173,13 @@ pub fn generated_protocol_types() -> String { normalize_typescript(output) } -/// Render the strict runtime validator paired with the generated declarations. +/// Generates the Browser receive validator paired with TypeScript declarations. /// -/// `schemars` projects the reachable Serde shape from [`SubscriptionFrame`]. The -/// normalization closes every object and expands flattened enum payloads before -/// embedding that schema into the dependency-free Browser validator template. +/// Normalizes schemas reachable from [`SubscriptionFrame`] to Rust's actual wire shape. +/// Corrects required/nullable fields and integer bounds rather than adopting Serde's +/// decode-time omission rules as the receive contract. Expands common flattened-enum +/// fields into each variant before closing objects, then embeds the schema in a +/// validator template that rejects unknown fields. pub fn generated_protocol_validator() -> String { let mut schema = serde_json::to_value(schemars::schema_for!(SubscriptionFrame)) .expect("SubscriptionFrame schema must serialize"); @@ -365,6 +367,8 @@ fn close_object_shapes(schema: &mut Value) { } } +/// Makes schemas with `default: null` accept null in their types as well. +/// A null default with a string-only or similar type would reject null sent by Rust. fn normalize_null_default(schema: &mut Map) { if schema.get("default") != Some(&Value::Null) || schema.contains_key("anyOf") { return; @@ -386,6 +390,10 @@ fn normalize_null_default(schema: &mut Map) { } } +/// Limits JSON integers to the intersection of Rust's type range and JavaScript's +/// safe-integer range for TypeScript `number`. Never relaxes explicit bounds; +/// rejects unknown formats during generation rather than emitting integer types +/// with unknown ranges as if they could be validated. fn normalize_integer_format(schema: &mut Map) { let is_integer = match schema.get("type") { Some(Value::String(value)) => value == "integer", @@ -426,6 +434,9 @@ fn tighten_numeric_bound( ); } +/// Serde decode-time defaults do not imply omitted fields on the wire. +/// Restores fields with schema defaults to required to match the serialized shape. +/// Does not fill missing fields with defaults in the Browser. fn require_serialized_defaults(schema: &mut Map) { if schema.get("type").and_then(Value::as_str) != Some("object") { return; diff --git a/crates/runtime-api/README.md b/crates/runtime-api/README.md new file mode 100644 index 000000000..79ebfd971 --- /dev/null +++ b/crates/runtime-api/README.md @@ -0,0 +1,10 @@ +# runtime-api + +HTTP contract for Worker Runtime management. Owns REST DTOs and generated client/server adapters. Runtime persistence, authentication, body limits, and operation permissions are implemented in their owning crates, such as worker-runtime. + +## Contract entry points + +- [`RuntimeApi`] — Runtime management operations +- [`WorkerRestoreRequest`] / [`WorkerRestoreCoordinationRequest`] — Restore intent and Backend preparation contracts +- [`LogicalWorkdirAttachment`] — Logical Workdir attachments determined by Workspace +- [`BackendJobExecutionBinding`] — Job/attempt execution identity fixed by the Host diff --git a/crates/runtime-api/src/lib.rs b/crates/runtime-api/src/lib.rs index 5254f07e0..88250d8e1 100644 --- a/crates/runtime-api/src/lib.rs +++ b/crates/runtime-api/src/lib.rs @@ -1,8 +1,4 @@ -//! Typed Worker Runtime management HTTP contract. -//! -//! This crate owns the public REST DTOs and generated client/server adapters only. Runtime domain -//! state, authorization, body limits, and operation permission policy remain in their existing -//! authorities. +#![doc = include_str!("../README.md")] use std::collections::BTreeMap; @@ -340,8 +336,10 @@ pub struct WorkingDirectoryRequest { pub materialization: Option, } -/// Workspace-authoritative alias-to-Workdir mapping, independent of any -/// Runtime-local materialized binding. +/// Workspace-defined alias-to-Workdir mapping, independent of Runtime materialized bindings. +/// +/// Logical aliases and session capabilities are stored as Workspace attachments; Runtime resolves +/// host paths and provider sessions. Workdir write scope alone does not grant command execution. #[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] pub struct LogicalWorkdirAttachment { /// Stable Worker-local routing key. This is not a Workdir id or display name. @@ -414,7 +412,10 @@ pub struct WorkerInput { pub segments: Option>, } -/// Host-authored Job/attempt identity, independent of Profile selection. +/// Job/attempt identity fixed by the Host at creation or Restore. +/// +/// Profile, instruction, and display name do not imply Job authority. The Backend accepts structured +/// results by matching the submitting Worker and this binding's current attempt and input digest. #[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] #[serde(deny_unknown_fields)] pub struct BackendJobExecutionBinding { @@ -494,8 +495,11 @@ pub struct WorkerResponse { pub worker: WorkerDetail, } -/// Explicit observed Restore intent. Preparation is authored by the Workspace -/// Backend, not browser/Tool JSON, and applied only after Runtime admission. +/// Restore intent with a fixed observation token and request ID. +/// +/// Replay and recovery use the same pair; Runtime owns the durable operation and its settlement. +/// preparation is created by the Workspace Backend, not chosen as browser/Tool input. +/// Preparation side effects must not run before Runtime admission. #[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] #[serde(deny_unknown_fields)] pub struct WorkerRestoreRequest { @@ -526,6 +530,11 @@ pub struct RepositoryAccessWorkdirReference { pub working_directory_id: String, } +/// Request tying Restore admission and Workspace preparation to the same Runtime operation. +/// +/// Queries without preparation only recover an existing owner/result; they admit no new intent. +/// Prepared requests verify and bind the observation token to the owner, retaining the original +/// preparation on replay. #[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] #[serde(deny_unknown_fields)] pub struct WorkerRestoreCoordinationRequest { @@ -550,8 +559,11 @@ pub struct WorkingDirectoryRepositoryAccessRequest { pub materialization: RepositoryMaterializationContext, } -/// Explicit restore outcome returned for every reachable Runtime restore attempt, -/// including preflight rejection and uncertain post-side-effect reconciliation. +/// Explicit result of a Restore attempt that could be reached. +/// +/// Distinguishes preflight rejection from outcomes requiring state checks after side effects. +/// HTTP response loss does not imply rollback; check the Runtime-owned receipt/reconciliation +/// using the same Restore intent. #[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] pub struct WorkerRestoreResponse { pub state: WorkerRestoreState, diff --git a/crates/secrets/README.md b/crates/secrets/README.md index af71d2da1..5276ad705 100644 --- a/crates/secrets/README.md +++ b/crates/secrets/README.md @@ -1,30 +1,9 @@ # secrets -## Role +Local store retrieving secrets by provider-independent logical IDs. It owns the storage format and validation that keep secrets out of scattered configuration files, not provider-specific authentication protocols or caller logging controls. -`secrets` provides the local secret reference store used by provider and tool configuration. +## Main entry points -## Boundaries - -Owns: - -- provider-independent secret id to value lookup -- modest plaintext-at-rest reduction and integrity checks -- secret store file format and validation - -Does not own: - -- provider-specific auth protocol (`provider`) -- Codex OAuth local integration shape (`provider`) -- prompting or model context -- work item or diagnostic redaction policy outside its API surface - -## Design notes - -The store is not a high-assurance keychain. It exists to avoid scattering plaintext credentials through config files and logs, not to provide strong local adversary protection. - -Secret values must stay out of diagnostics, Debug output, CLI/TUI output, work items, docs, session logs, model context, and persisted plaintext files. - -## See also - -- [`../../docs/design/provider-model-boundary.md`](../../docs/design/provider-model-boundary.md) +- [`SecretId`] — logical ID validation +- [`SecretValue`] — secret values with redacted Debug output +- [`SecretStore`] — local storage, integrity checks, and protection limits diff --git a/crates/secrets/src/lib.rs b/crates/secrets/src/lib.rs index de722e3d0..18dfc1074 100644 --- a/crates/secrets/src/lib.rs +++ b/crates/secrets/src/lib.rs @@ -1,3 +1,5 @@ +#![doc = include_str!("../README.md")] + use std::collections::BTreeMap; use std::fmt; use std::fs::{self, OpenOptions}; @@ -59,6 +61,10 @@ pub enum Error { pub type Result = std::result::Result; +/// Logical secret ID with no interpretation of provider semantics. +/// +/// `/` separates name components, not references to individual host files. +/// Reject absolute references and traversal components to prevent escape even if misused as a path. #[derive(Clone, Eq, PartialEq, Ord, PartialOrd, Hash)] pub struct SecretId(String); @@ -86,6 +92,11 @@ impl fmt::Display for SecretId { } } +/// Secret value that omits plaintext from Debug output. +/// +/// [`Self::expose_secret`] and [`Self::into_string`] return plaintext; callers remain responsible +/// for keeping it out of diagnostics, history, and model context. This wrapper does not +/// guarantee arbitrary serialization, memory erasure, or redaction of external logs. #[derive(Clone, Eq, PartialEq)] pub struct SecretValue(String); @@ -109,6 +120,16 @@ impl fmt::Debug for SecretValue { } } +/// Local store retrieving secrets by logical ID. +/// +/// Store obfuscated values and integrity tags to keep plaintext out of scattered configuration files. +/// Normal constructors derive the key from the data directory path, so this is neither a strong +/// keychain nor protection against local attackers. It does not guarantee secrecy from anyone +/// who knows that path; moving the store also changes the conditions for decrypting existing values. +/// +/// Saving writes and syncs a temporary file, then replaces the store by rename. Parent-directory +/// sync is best-effort; CAS or locks do not serialize multiple writers' read-modify-write cycles. +/// Replacement by rename does not guarantee protection against lost concurrent updates. #[derive(Debug, Clone)] pub struct SecretStore { path: PathBuf, diff --git a/crates/server-api/README.md b/crates/server-api/README.md new file mode 100644 index 000000000..ec5e41c1a --- /dev/null +++ b/crates/server-api/README.md @@ -0,0 +1,10 @@ +# server-api + +HTTP contract shared by Workspace Server and Rust clients. Owns public DTOs, API declarations, and generated client/server adapters. workspace-server determines persistent resource authorization and state transitions; worker-runtime determines Runtime execution state. + +## Contract entry points + +- [`ServerApi`] — HTTP operations and transport metadata +- [`ServerRequestContext`] — Request context passed to Server authentication adapters +- [`RequestActor`] — Human identity and authentication method +- [`WorkerRestoreRequest`] / [`WorkerRestoreResult`] — Observation-based Restore intent and result diff --git a/crates/server-api/src/lib.rs b/crates/server-api/src/lib.rs index 7c512b2b3..dc2e7567e 100644 --- a/crates/server-api/src/lib.rs +++ b/crates/server-api/src/lib.rs @@ -1,8 +1,4 @@ -//! Shared Workspace HTTP resource contracts. -//! -//! This crate owns transport DTOs exposed by the Workspace Server and consumed -//! by Rust clients. Runtime-internal projections remain in their owning crates; -//! callers must explicitly construct these Server-authoritative resources. +#![doc = include_str!("../README.md")] use std::collections::BTreeMap; @@ -472,6 +468,17 @@ pub struct WorkspaceWorkerSessionHistoryResponse { pub observation: runtime_api::WorkerSessionHistoryAvailability, } +/// Shared Workspace HTTP contract. Server/Runtime determine resource authorization and settlement. +/// +/// # Restore intent and results +/// +/// Replay keeps the [`WorkerRestoreRequest`] observation token and request ID fixed. The token +/// identifies a Runtime observation, not a controller command ID or cleanup plan digest. GET +/// comparisons do not replace admission checks under the Runtime operation lock; conflicts do +/// not trigger automatic token replacement. +/// [`WorkerRestoreState`] Accepted means Restore committed, not that the subsequent run succeeded. +/// Distinguish read-only preflight Rejected, confirmed RolledBack after side effects, and +/// ReconciliationRequired needing settled-state checks. Response loss does not imply rollback. #[api(reqwest, axum, openapi)] pub trait ServerApi { #[get("/health", status = 200, error_status = 400)] @@ -3605,6 +3612,11 @@ pub struct ServerRuntimeSource { pub worker_id: Option, } +/// Request context built by Server authentication adapters, not authority selected via public JSON. +/// +/// actor and Runtime/Worker sources are middleware verification results; operation services recheck +/// current registrations, grants, and Accounts as needed. Transport headers are a Server-internal +/// extension reusing existing authentication, excluded from generated client and OpenAPI inputs. #[derive(Clone)] pub struct ServerRequestContext { pub actor: Option, @@ -3685,6 +3697,8 @@ pub struct RequestActor { pub auth_method: ActorAuthMethod, } +/// user_id identifies a human; account_id identifies the Workspace owner's namespace. +/// Actor authentication is separate from Workspace resource authorization, checked by resource services. impl RequestActor { pub fn user(&self) -> AuthenticatedUser { AuthenticatedUser { diff --git a/crates/server-api/src/repository_openapi_typescript.rs b/crates/server-api/src/repository_openapi_typescript.rs index cd0dcf3cb..9dde1f3d8 100644 --- a/crates/server-api/src/repository_openapi_typescript.rs +++ b/crates/server-api/src/repository_openapi_typescript.rs @@ -1,3 +1,18 @@ +//! Converts schemas reachable from canonical OpenAPI operations to TypeScript. +//! +//! Unsupported keywords, references, and ambiguous unions cause generation errors: ignoring +//! them would make types missing Rust constraints appear valid. Only the target operations' +//! schema closure is validated and emitted; unused domains' schemas are outside the contract. +//! +//! Integer bounds are checked against format ranges and JavaScript's safe-number range; +//! out-of-range or non-integer bounds are rejected. Type markers distinguish narrower ranges, +//! but are not runtime validators. Mapping all Rust integers to number would hide precision +//! loss and differing constraints. +//! +//! Defaults generated by Schemars from Serde defaults are annotations, not replacements for +//! referenced types or constraints. required determines omission; schemas determine nullability. +//! The generator does not fill defaults or make TypeScript types perform Serde normalization. + use std::collections::{BTreeMap, BTreeSet}; use serde_json::{Map, Value}; diff --git a/crates/session-analytics/README.md b/crates/session-analytics/README.md new file mode 100644 index 000000000..08f433a26 --- /dev/null +++ b/crates/session-analytics/README.md @@ -0,0 +1,10 @@ +# session-analytics + +`session-analytics` reads Session JSONL and aggregates observations of tool use and +context changes. It neither runs Workers nor restores history, and reports omit +message bodies and raw tool arguments/results. + +## Design entry points + +- [`analyze_session`]: reading and aggregation tolerant of storage-format differences +- [`SessionReport`]: observation scope and limits on interpretation diff --git a/crates/session-analytics/src/lib.rs b/crates/session-analytics/src/lib.rs index 1b4a8c64a..6d50456f4 100644 --- a/crates/session-analytics/src/lib.rs +++ b/crates/session-analytics/src/lib.rs @@ -1,10 +1,4 @@ -//! Read-only analytics for Yoi session JSONL logs. -//! -//! This crate intentionally parses the persisted JSON shape tolerantly with -//! `serde_json::Value` rather than depending on Worker runtime or TUI crates. The -//! report contains counts, paths, sizes, line/turn indexes, and bounded -//! diagnostics; raw user messages, tool arguments, and tool output snippets are -//! not emitted. +#![doc = include_str!("../README.md")] use std::collections::{BTreeMap, BTreeSet, HashMap}; use std::fs::File; @@ -34,11 +28,24 @@ pub enum AnalyzeError { Metadata { path: PathBuf, source: io::Error }, } -/// Analyze one explicit session JSONL path. +/// Aggregates usage from the specified Session JSONL. +/// +/// Reads JSON as `serde_json::Value`, independent of Worker and TUI. +/// Counts unknown formats and invalid lines as diagnostics while continuing to +/// analyze readable records. This API does not verify correct history replay. pub fn analyze_session(path: impl AsRef) -> Result { Analyzer::analyze(path.as_ref()) } +/// Usage summaries and diagnostics without message bodies. +/// +/// Retains paths, sizes, and line/turn positions, but omits raw user messages and +/// tool arguments/results. The diagnostic count is capped. +/// +/// Excludes initial history from per-response aggregation because its original +/// response boundaries are unknown. Repeated reads/execution after context changes +/// indicate correlation, not causation. Repeated edits are observations, not proof +/// that they could be batched. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] pub struct SessionReport { pub input: InputSummary, diff --git a/crates/session-metrics/README.md b/crates/session-metrics/README.md index 07284cef2..f6e26c070 100644 --- a/crates/session-metrics/README.md +++ b/crates/session-metrics/README.md @@ -1,39 +1,12 @@ # session-metrics -Session 単位の append-only な観測値を既存 session-log に記録し、明示的な -metrics 読取 / JSONL export 経路で取り出すための小さなヘルパークレートです。 +`session-metrics` records measurements in append-only Session logs and retrieves them +through dedicated read/export paths. It handles measurement types and Segment lineage; +callers choose what to measure and calculate charges. Measurements are not exposed in +ordinary conversation snapshots. -- 保存先は `session-store` の `LogEntry::Extension` -- extension domain は `metrics` -- metric は `name / ts / dimensions / value / correlation_id` の最小 envelope -- `record_metric` で指定した Session / Segment に append する -- `read_segment_metrics` は 1 Segment、`read_session_metrics` は Session 内の全 - Segment を読み、各 metric に `segment_id` と `compacted_from` を付ける -- `export_metrics_jsonl` はその located metric を newline-delimited JSON にする -- 通常の Session snapshot / Worker list / Worker detail は Extension を公開しない +## Design entry points -compaction は `compact.start` を source Segment、`compact.finish` と -`compact.post_request` を結果 Segment に記録する。同じ `correlation_id` と -`SegmentStart.compacted_from` により、Segment をまたぐ attempt と次の通常 LLM -request を結合できる。 - -```rust,ignore -use session_metrics::{ - Metric, export_metrics_jsonl, read_session_metrics, record_metric, -}; - -let metric = Metric::now("compact.start") - .with_value(12_345.0) - .with_dimension("trigger", "pre_run") - .with_correlation_id("018f6f8a-9822-7b11-8b35-706f30313700"); -record_metric( - &store, - location.session_id, - location.segment_id, - &metric, -)?; - -let records = read_session_metrics(&store, location.session_id)?; -let jsonl = export_metrics_jsonl(&records)?; -# Ok::<(), Box>(()) -``` +- [`Metric`]: missing values and correlation with later measurements +- [`LocatedMetric`]: storage locations and compaction lineage +- [`read_session_metrics`]: read boundary for aggregation across Segments diff --git a/crates/session-metrics/src/lib.rs b/crates/session-metrics/src/lib.rs index 5f0d2f468..9291eae00 100644 --- a/crates/session-metrics/src/lib.rs +++ b/crates/session-metrics/src/lib.rs @@ -1,20 +1,4 @@ -//! Session metrics — generic ad-hoc measurement lane on top of -//! `LogEntry::Extension { domain: "metrics" }`. -//! -//! セッション中に積み上げて後で引きたい値（prune の発火頻度・Hook の実行 -//! 時間・ツールリトライ回数 等）を session-log に乗せるための薄い層。 -//! session-store は payload を不透明な `serde_json::Value` として扱うので、 -//! このクレートは型と読み書きヘルパーだけを提供する。 -//! -//! # 設計 -//! -//! - 厳格な label set は持たない。次元は sparse な `BTreeMap`、 -//! 観測できない値は `None` で明示する -//! - 「後から埋まる値」（例: prune 発火直後の `cache_read_tokens`）は前 entry に -//! 書き戻さず、`correlation_id` を共有する別 metric として流す。集計は読み手で join -//! - 集計 / 可視化には [`read_session_metrics`] / [`read_segment_metrics`] / -//! [`export_metrics_jsonl`] の明示的な metrics 専用経路を使う。通常の -//! Session snapshot は `Extension` を公開しない。 +#![doc = include_str!("../README.md")] use std::collections::BTreeMap; @@ -26,28 +10,33 @@ use session_store::{ /// Domain tag used in `LogEntry::Extension` for all metrics records. pub const DOMAIN: &str = "metrics"; -/// 単発の計測値。`name` は `namespace.metric` 形式の自由文字列 -/// （例: `"prune.fire"`、`"hook.duration"`）。 +/// A single measurement appended to a Session log. +/// +/// Stored as an extension payload and read separately from ordinary conversation +/// snapshots. Labels are unrestricted; unobservable dimensions are omitted and +/// missing values use `None`. Values learned later are appended as separate +/// measurements with the same `correlation_id`, not written back to old records. +/// This allows more measurements without changing conversation bodies or existing records. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] pub struct Metric { - /// `namespace.metric` 形式の名前。 + /// Name in `namespace.metric` form. pub name: String, - /// epoch ms。 + /// Epoch milliseconds. pub ts: u64, - /// sparse な次元（label）。観測できないものはキー自体を入れない。 + /// Sparse dimensions (labels). Omit keys for unobservable dimensions. #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] pub dimensions: BTreeMap, - /// 主スカラ値。dimension では表現したくない数値を載せる場所。 + /// Primary scalar value for numbers not represented as dimensions. #[serde(default, skip_serializing_if = "Option::is_none")] pub value: Option, - /// 関連 metric を join するためのキー。同一 ID を持つ複数 metric は - /// 「同じ事象を多面的に観測している」という意味付けで読まれる。 + /// Key for joining related metrics. Metrics with the same ID represent + /// different observations of the same event. #[serde(default, skip_serializing_if = "Option::is_none")] pub correlation_id: Option, } impl Metric { - /// 最小コンストラクタ。`ts` は呼び出し時刻（epoch ms）で埋める。 + /// Minimal constructor; sets `ts` to the call time in epoch milliseconds. pub fn now(name: impl Into) -> Self { Self { name: name.into(), @@ -74,10 +63,10 @@ impl Metric { } } -/// `LogEntry::Extension { domain: "metrics", payload: }` を append する。 +/// Appends `LogEntry::Extension { domain: "metrics", payload: }`. /// -/// `save_extension` の薄い wrapper。書き込み失敗は呼び出し側に返す -/// （メトリクスのために本体処理を止めるかは呼び出し側の判断）。 +/// Thin wrapper around `save_extension`. Returns write failures to the caller, +/// who decides whether a metrics failure should stop the main operation. pub fn record_metric( store: &impl Store, session_id: SessionId, @@ -88,10 +77,10 @@ pub fn record_metric( save_extension(store, session_id, segment_id, DOMAIN, payload) } -/// `RestoredState.extensions` から metrics domain の payload を順に取り出し、 -/// `Metric` 列に fold する。 +/// Collects metrics-domain payloads from `RestoredState.extensions` in order +/// into a sequence of `Metric` values. /// -/// schema 変更で deserialize できない payload は無視する（後方互換）。 +/// Skips payloads that cannot deserialize after schema changes for backward compatibility. pub fn metrics_from_extensions(extensions: &[(String, serde_json::Value)]) -> Vec { extensions .iter() @@ -100,11 +89,11 @@ pub fn metrics_from_extensions(extensions: &[(String, serde_json::Value)]) -> Ve .collect() } -/// A metric together with its durable Session/Segment origin. +/// A measurement with its storage location and Segment lineage. /// -/// `compacted_from` is copied from the Segment start record so readers can -/// reconstruct compaction lineage without inferring relationships from metric -/// names or timestamps. +/// Takes `compacted_from` from the Segment's initial record rather than inferring +/// compaction order from names or timestamps. Use `correlation_id` to relate the +/// same attempt, and `compacted_from` to relate source and resulting Segments. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub struct LocatedMetric { pub session_id: SessionId, @@ -214,7 +203,12 @@ pub fn read_segment_metrics( .collect()) } -/// Read every metric for a Session across all of its Segments. +/// Reads measurements from all Segments with lineage. +/// +/// Sorts by timestamp, measurement phase, Segment, and log position for reproducible +/// exports under the same conditions. This display order does not prove causality; +/// use `correlation_id` and `compacted_from` for relationships. Rejects Segments +/// without initial records and Session mismatches. pub fn read_session_metrics( store: &dyn Store, session_id: SessionId, @@ -277,7 +271,7 @@ mod tests { #[test] fn metric_serializes_minimal_form_compactly() { - // dimensions が空 / value/correlation_id が None の時は出力に含めない。 + // Omit empty dimensions and None values for value/correlation_id. let metric = Metric { name: "x".into(), ts: 1, @@ -410,7 +404,7 @@ mod tests { #[test] fn fold_skips_undeserializable_payloads() { - // 将来 schema が変わって読めない payload も skip して落ちない。 + // Skip unreadable payloads from future schema changes without failing. let extensions = vec![ (DOMAIN.into(), serde_json::json!({ "garbage": true })), ( diff --git a/crates/session-store/README.md b/crates/session-store/README.md index 87f631740..1aecdad4a 100644 --- a/crates/session-store/README.md +++ b/crates/session-store/README.md @@ -1,32 +1,15 @@ # session-store -## Role +`session-store` stores replayable Session history and current Worker restoration data. +Session history uses append-only JSONL; current Segment references and child/peer +registrations live separately in Worker metadata. It does not own process liveness +checks or Runtime catalog lifecycle management. -`session-store` owns replayable append-only session logs. +## Design entry points -## Boundaries - -Owns: - -- session identifiers and segment lineage -- JSONL log entries for replayable conversation/runtime history -- restoring Engine/session state from committed records -- schema surfaces that should make drift compile-visible - -Does not own: - -- current Worker-name metadata (`session-store` worker metadata) -- live process/socket discovery (`worker-allocation`, `client`) -- UI state (`tui`) -- generated memory summaries (`memory`) - -## Design notes - -A session log records what happened. It is not the current Worker registry and should not be queried as the only source of "what does Worker X mean now?" - -Prefer explicit current log variants over broad legacy compatibility when schema changes; hidden compatibility can make future replay bugs silent. - -## See also - -- [`../../docs/design/worker-session-state.md`](../../docs/design/worker-session-state.md) -- [`../../docs/design/context-history.md`](../../docs/design/context-history.md) +- [`segment_log`] / [`segment`]: history replay and fork/compaction lineage +- [`worker_session_store`]: storage boundary for a Session bound to a Worker +- [`worker_metadata`]: current Worker state and active Segment switching +- [`system_item`]: records of system input presented to the model +- [`public_snapshot`] / [`public_index`]: public projections of stored history +- [`event_trace`]: diagnostic records independent of history replay diff --git a/crates/session-store/src/event_trace.rs b/crates/session-store/src/event_trace.rs index 465015120..e08e592aa 100644 --- a/crates/session-store/src/event_trace.rs +++ b/crates/session-store/src/event_trace.rs @@ -1,11 +1,9 @@ -//! Debug-only LLM request/stream trace recording. +//! Diagnostic records of LLM transport, independent of history replay. //! -//! [`TraceEntry`] captures stream lifecycle markers and normalized provider -//! stream events for debugging stalls. It is not a byte-for-byte raw SSE -//! capture. Written to a separate `.trace.jsonl` file, -//! completely independent of the segment log used for state restoration. -//! -//! Disabled by default. Enable via `SessionConfig::record_event_trace`. +//! [`TraceEntry`] records normalized events and lifecycle boundaries, not a complete +//! copy of raw SSE. Stored separately from Segment logs, it is not a source for +//! restored model input. This separates observed transport state from input/output +//! committed to history. use agen::llm_client::event::Event; use serde::{Deserialize, Serialize}; diff --git a/crates/session-store/src/lib.rs b/crates/session-store/src/lib.rs index 89ceb9a89..a240ed7b1 100644 --- a/crates/session-store/src/lib.rs +++ b/crates/session-store/src/lib.rs @@ -1,35 +1,4 @@ -//! Session persistence via append-only JSONL logs. -//! -//! # Architecture -//! -//! A [`Session`](SessionId) is a fork-tree of [`Segment`](SegmentId)s -//! belonging to the same logical conversation. Each Segment is recorded -//! as a sequence of [`LogEntry`] values, one per line in a `.jsonl` -//! file. Reading a segment log and collecting entries reconstructs the -//! Engine state at that segment — no separate snapshots or checkpoints -//! needed. Compaction and fork operations mint a fresh Segment within -//! the same Session. -//! -//! This crate provides free functions for persistence operations. -//! The caller (typically Worker) holds the Engine directly and calls these -//! functions after state-mutating operations. -//! -//! Debug-mode [`TraceEntry`] records capture raw stream events in a separate -//! `.trace.jsonl` file, independent of the segment log. -//! -//! # Quick start -//! -//! ```ignore -//! use session_store::{create_segment, restore, save_delta, FsStore, SegmentStartState}; -//! -//! let store = FsStore::new("./sessions")?; -//! let (session_id, segment_id) = create_segment(&store, SegmentStartState { -//! system_prompt: None, -//! config: &config, -//! history: Vec::new(), -//! user_segments: Vec::new(), -//! })?; -//! ``` +#![doc = include_str!("../README.md")] use std::io; use std::path::Path; @@ -141,11 +110,10 @@ pub use worker_metadata::{ }; pub use worker_session_store::WorkerSessionStore; -/// Session identifier — the fork-tree root. UUID v7 (time-ordered). +/// UUID v7 identifying Segments belonging to the same conversation. /// -/// All Segments belonging to the same Session share this ID. Compaction -/// and fork operations create a new Segment within the same Session, so -/// `WHERE session_id = ?` retrieves the full lineage. +/// Fork/compaction preserves this ID and only adds Segments. Switching the current +/// history projection therefore does not change the conversation-wide reference. pub type SessionId = uuid::Uuid; /// Segment identifier. UUID v7 (time-ordered, lexicographically sortable). diff --git a/crates/session-store/src/logged_item.rs b/crates/session-store/src/logged_item.rs index 546110baa..b927be5d2 100644 --- a/crates/session-store/src/logged_item.rs +++ b/crates/session-store/src/logged_item.rs @@ -82,8 +82,8 @@ pub enum LoggedItem { summary: Vec, #[serde(default, skip_serializing_if = "Option::is_none")] encrypted_content: Option, - /// Anthropic extended thinking signature。新世代 Claude で round-trip - /// 必須。OpenAI Responses など他 scheme では `None`。 + /// Anthropic extended thinking signature, required for round-trips with + /// newer Claude generations. `None` for other schemes such as OpenAI Responses. #[serde(default, skip_serializing_if = "Option::is_none")] signature: Option, }, @@ -418,13 +418,13 @@ mod tests { #[test] fn round_trip_reasoning_preserves_signature() { - // 新世代 Claude の thinking signature が history.json に永続化され、 - // resume 後の Item::Reasoning に復元されること。 + // A newer Claude thinking signature persists in history.json and is + // restored to Item::Reasoning after resume. let original = Item::reasoning("inner thought").with_signature("SIG-OPUS-XYZ"); let logged: LoggedItem = (&original).into(); let json = serde_json::to_string(&logged).unwrap(); - // wire 形式に signature キーが乗ること（古い形式との互換のため - // 値が None のときは省略される。Some の値は載る） + // The wire format includes Some signatures; None is omitted for + // compatibility with older formats. assert!( json.contains("SIG-OPUS-XYZ"), "serialised JSON must carry signature: {json}", @@ -443,8 +443,8 @@ mod tests { #[test] fn legacy_reasoning_without_signature_field_deserializes() { - // signature フィールドが無い旧形式の history.json を読み込んでも - // None としてロードできる（後方互換性）。 + // Legacy history.json without a signature field loads it as None + // for backward compatibility. let legacy_json = r#"{"kind":"reasoning","text":"old","summary":[],"encrypted_content":null}"#; let parsed: LoggedItem = serde_json::from_str(legacy_json).unwrap(); diff --git a/crates/session-store/src/public_snapshot.rs b/crates/session-store/src/public_snapshot.rs index 838f394a1..6d1c1a4d7 100644 --- a/crates/session-store/src/public_snapshot.rs +++ b/crates/session-store/src/public_snapshot.rs @@ -1,3 +1,14 @@ +//! Projects stored history for clients. +//! +//! Converts committed entry IDs and provenance to public DTOs, keeping JSONL replay +//! rules out of clients. Model reasoning, system prompts, internal response boundaries, +//! and extension payloads are not exposed in conversation views. Measurements use +//! dedicated readers and do not enter ordinary snapshots. +//! +//! Retained-history reads validate active references and Session IDs and cap read volume. +//! Observation neither migrates old formats nor repairs corruption; required migration +//! or repair returns an error. Matches live output by entry ID, not timestamps alone. + mod run_stats; use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD}; diff --git a/crates/session-store/src/segment.rs b/crates/session-store/src/segment.rs index cbd1a12b4..3078c298f 100644 --- a/crates/session-store/src/segment.rs +++ b/crates/session-store/src/segment.rs @@ -1,8 +1,13 @@ -//! Free functions for segment persistence operations. +//! Segment creation, append, and restoration. //! -//! These functions record and restore segment state without owning a Engine. -//! The caller (typically Worker) holds the Engine directly and calls these -//! functions after state-mutating operations. +//! Records host-committed history and execution boundaries in Store without owning +//! Engine. Fork and compaction create a Segment with initial history in the same +//! Session without rewriting the source. Initial records retain the source Segment +//! and adopted boundary so replacement and branching decisions remain traceable. +//! +//! Saving a new Segment does not change the active reference. The host coordinates +//! compaction activation with Worker metadata and the live append target. On failure, +//! a saved Segment alone does not establish that current history has switched. use crate::segment_log::{self, LogEntry, SegmentOrigin}; use crate::store::{Store, StoreError}; @@ -400,12 +405,11 @@ pub fn save_run_errored( ) } -/// Log an `LlmUsage` entry — 1 LLM リクエスト分の Usage スナップショット。 +/// Logs an `LlmUsage` entry: a usage snapshot for one LLM request. /// -/// `history_len` は送信時の `history.len()`。`input_total_tokens` は -/// その prefix をプロバイダが実測した占有量（プロンプト全長）で、 -/// プロバイダ別の正規化（Anthropic では `input + cache_read + cache_creation`）を -/// 済ませた値を渡す。 +/// `history_len` is `history.len()` at send time. `input_total_tokens` is the +/// provider-measured size of that prefix (the full prompt). Pass a value normalized +/// for the provider: `input + cache_read + cache_creation` for Anthropic. pub fn save_usage( store: &impl Store, session_id: SessionId, diff --git a/crates/session-store/src/segment_log.rs b/crates/session-store/src/segment_log.rs index 8c093d3e3..543c3752b 100644 --- a/crates/session-store/src/segment_log.rs +++ b/crates/session-store/src/segment_log.rs @@ -1,13 +1,13 @@ -//! Segment log types for append-only JSONL persistence. +//! Replays Segment state from append-only JSONL. //! -//! Each [`LogEntry`] represents a single state transition within one -//! segment, serialized as one line in a `.jsonl` file. Reading all -//! entries and collecting them via [`collect_state`] reconstructs the -//! full [`Engine`] state at that segment. +//! [`collect_state`] restores model input, configuration, and execution interruption +//! state from [`LogEntry`] position order and initial history. Fork/compaction seeds +//! retain lineage to the old Segment and history item IDs and provenance. //! -//! The on-disk format is one `LogEntry` per line — entries are positionally -//! ordered. Fork lineage references between segments use turn-number indices -//! (`SegmentOrigin.at_turn_index`) rather than per-entry hashes. +//! An execution-start record alone does not imply success. Runs without terminal +//! records restore as interrupted; uncommitted tool execution is not treated as +//! successful. Yield/resume records mark intermediate logical-Run boundaries, +//! preserving the turn budget even when resuming after compaction. use agen::llm_client::types::{Item, RequestConfig}; use agen::{EngineResult, UsageRecord}; @@ -17,20 +17,10 @@ use serde::{Deserialize, Serialize}; use crate::history::{LoggedHistoryEntry, LoggedSystemHistoryEntry}; use crate::logged_item::LoggedItem; -/// A single segment log entry, serialized as one JSONL line. +/// Host-owned extension data. /// -/// Variants correspond to specific mutation points in `Engine`: -/// - `SegmentStart` — always the first entry; captures initial state -/// - `Invoke` — IDLE → active marker (start of a new self-driving cycle) -/// - `UserInput` / `AssistantItem` / `ToolResult` / `SystemItem` — history appends -/// - `TurnEnd` — AgentTurn boundary marker; carries the post-increment -/// `turn_count`. With retry unimplemented today this fires once per -/// `run()`/`resume()` (current callers persist a single TurnEnd at -/// run completion); the fork-point seq for `at_turn_index` is the -/// preceding `Invoke` entry, not the TurnEnd. -/// - `RunCompleted` / `RunCancelled` / `RunErrored` — terminal run outcomes -/// - `PausedTurnAbandoned` — explicit abandon/cancel of a paused interrupted turn -/// - `ConfigChanged` — `RequestConfig` mutation +/// Excluded from public conversation-history projections; interpreted by +/// dedicated domain readers, such as metrics readers. #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] pub struct SessionExtension { pub domain: String, @@ -46,6 +36,11 @@ impl SessionExtension { } } +/// JSONL record of history additions and state transitions within a Segment. +/// +/// History items and provenance commit in the same record, separately from execution +/// boundaries and extension data. These records do not establish current Worker +/// catalog state or process liveness. #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(tag = "kind", rename_all = "snake_case")] pub enum LogEntry { @@ -123,7 +118,7 @@ pub enum LogEntry { /// Turn boundary. Records the turn count after increment. TurnEnd { ts: u64, turn_count: usize }, - /// `run()` / `resume()` が `EngineResult` で正常終了した。 + /// `run()` / `resume()` returned normally with an `EngineResult`. /// Replay restores both interruption state and any resumable logical-run /// turn budget. RunCompleted { @@ -172,9 +167,9 @@ pub enum LogEntry { entry_id: Option, }, - /// `run()` / `resume()` が `EngineError` で終了した。 - /// `EngineError` は `Serialize` 不可なので `message` のみ lossy 保持する。 - /// Audit-only metadata: replay は `interrupted` のみ反映する。 + /// `run()` / `resume()` ended with an `EngineError`. + /// `EngineError` is not `Serialize`, so only `message` is retained lossily. + /// Audit-only metadata: replay applies only `interrupted`. RunErrored { ts: u64, #[serde(default, skip_serializing_if = "Option::is_none")] @@ -204,19 +199,19 @@ pub enum LogEntry { /// `RequestConfig` changed. ConfigChanged { ts: u64, config: RequestConfig }, - /// LLM リクエスト 1 件分の Usage スナップショット。 + /// Usage snapshot for one LLM request. /// - /// `history_len` は送信時の `history.len()`。`input_total_tokens` は - /// その prefix をプロバイダが実測した占有量（プロンプト全長）。 - /// このリクエスト 1 件で新しく追加された分ではない。 + /// `history_len` is `history.len()` at send time. `input_total_tokens` is the + /// provider-measured size of that prefix (the full prompt), not the amount + /// newly added by this request. /// - /// プロバイダ別の正規化（呼び出し側で行う想定）: + /// Provider-specific normalization (expected from the caller): /// - Anthropic: `input_tokens + cache_read + cache_creation` /// - OpenAI: `prompt_tokens` /// - Gemini: `promptTokenCount` /// - Ollama: `prompt_eval_count` /// - /// `cache_read_tokens` / `cache_write_tokens` は上記の内訳で、料金会計用。 + /// `cache_read_tokens` / `cache_write_tokens` break down the above for billing. LlmUsage { ts: u64, history_len: usize, @@ -226,14 +221,13 @@ pub enum LogEntry { output_tokens: u64, }, - /// 汎用拡張点。ドメイン名で名前空間を切って任意 JSON を載せる。 - /// session-store は payload を不透明扱いし、replay 時は - /// `RestoredState.extensions` に `(domain, payload)` を順に積むだけ。 - /// 各ドメイン側が自前で fold して最新値を取り出す前提。 + /// Generic extension point for arbitrary JSON namespaced by domain. + /// session-store treats payloads as opaque; replay only appends + /// `(domain, payload)` to `RestoredState.extensions` in order. + /// Each domain folds its own records to obtain the latest value. /// - /// 想定用途: memory subsystem の extract 処理境界 pointer 等、 - /// 「session 寿命に縛りたいが session-store の型を汚したくない」 - /// メタデータ。 + /// Intended for metadata tied to the Session lifetime without adding + /// session-store types, such as memory extraction boundary pointers. Extension { ts: u64, domain: String, @@ -317,12 +311,12 @@ pub struct RestoredState { /// Writers track their own append count via the same counter so /// `ensure_head_or_fork` can compare it with the on-disk count. pub entries_count: usize, - /// LLM リクエストごとの Usage スナップショット時系列。 - /// `LogEntry::LlmUsage` を replay して時系列順に積まれる。 - /// 任意位置のトークン数推定に使う。 + /// Per-request usage snapshots in chronological order. + /// Populated by replaying `LogEntry::LlmUsage`. + /// Used to estimate token counts at arbitrary positions. pub usage_history: Vec, - /// `LogEntry::Extension` を replay 順に積んだもの。`(domain, payload)`。 - /// session-store は domain を不透明扱いし、各ドメインが自前で fold する。 + /// `(domain, payload)` values from `LogEntry::Extension` in replay order. + /// session-store treats domains as opaque; each domain folds its own records. pub extensions: Vec<(String, serde_json::Value)>, /// User submissions in original typed form, in submit order. /// One entry per `LogEntry::AnnotatedUserInput`; the K-th entry corresponds to @@ -766,9 +760,9 @@ mod tests { output_tokens: 5, }, ]); - // history は LlmUsage で変化しない + // LlmUsage does not change history. assert_eq!(state.history.len(), 2); - // usage_history は時系列順 + // usage_history is in chronological order. assert_eq!(state.usage_history.len(), 2); assert_eq!(state.usage_history[0].history_len, 1); assert_eq!(state.usage_history[0].input_total_tokens, 50); @@ -1114,7 +1108,7 @@ mod tests { payload: serde_json::json!({ "x": 1 }), }, ]); - // 順序保持で全件積まれる。fold は呼び出し側の責務。 + // All records retain their order. Folding is the caller's responsibility. assert_eq!(state.extensions.len(), 3); assert_eq!(state.extensions[0].0, "memory.extract"); assert_eq!(state.extensions[1].1["processed_through_entry"], 12); diff --git a/crates/session-store/src/system_item.rs b/crates/session-store/src/system_item.rs index b5ccb41e0..2fc902090 100644 --- a/crates/session-store/src/system_item.rs +++ b/crates/session-store/src/system_item.rs @@ -1,22 +1,14 @@ -//! Typed system-message items injected by the agent system. +//! Typed records of system input a Worker presents to the model. //! -//! Items in worker history with `role:system` are never produced by the -//! LLM — they are always inserted by the Worker itself (notifications, -//! file ref resolutions, child-worker lifecycle events, reminders, …). -//! [`SystemItem`] carries the -//! typed shape of each such injection so clients can dispatch on -//! `kind` instead of parsing text prefixes like `[Notification] …` or -//! `[File: …]`. +//! Notifications, child events, reference resolution, and reminders are Worker-added +//! input, not model output. [`SystemItem`] retains the display kind and model-visible +//! body, representing the same input in history and live events. Stored bodies allow +//! old input to be reproduced even if prompt rendering rules later change. //! -//! Persisted as the payload of [`crate::LogEntry::AnnotatedSystemItem`] (one -//! entry per item), and broadcast live as the payload of -//! `Event::SystemItem` on the wire. -//! -//! For LLM context replay, each `SystemItem` reduces to an -//! `Item::system_message(...)` whose body matches the legacy free-text -//! shape (see [`SystemItem::history_text`]). The kind metadata is -//! preserved only on the log/wire side; the LLM still sees plain -//! system-message text. +//! Input affecting the model is committed via [`crate::LogEntry::AnnotatedSystemItem`] +//! before use. Request-only transient insertion would leave later turns unable to +//! explain decisions from stored history. The model projection uses only the body; +//! clients can render using type information without parsing body prefixes. use agen::llm_client::types::Item; use protocol::WorkerEvent; diff --git a/crates/session-store/src/worker_metadata.rs b/crates/session-store/src/worker_metadata.rs index 279562129..40a9f3895 100644 --- a/crates/session-store/src/worker_metadata.rs +++ b/crates/session-store/src/worker_metadata.rs @@ -1,16 +1,19 @@ -//! Durable Worker-name metadata/state persistence. +//! Stores current Worker restoration data separately from Session history. //! -//! This crate owns the name-keyed Worker state surface under a Worker-state root, -//! e.g. `{data_dir}/workers/{worker_name}/metadata.json`. Session JSONL replay stays -//! in `session-store`; Worker metadata may point at a `(SessionId, SegmentId)` but -//! does not own or replay session logs. +//! [`WorkerAggregateStore`] handles metadata for one Runtime-selected Worker. +//! The active Segment, resolved manifest, and child/peer visibility are current +//! state, separate from state replayed from past conversation. Session model input +//! is restored from Segment-log initial history and appended records. //! -//! `resolved_manifest_snapshot` is authority only for Worker-name restore before -//! loading the session log. Existing segment replay still uses `SegmentStart` -//! entries from `session-store`. `spawned_children` is durable current parent -//! Worker state for child registry/reclaim; child lifecycle messages shown to the -//! model remain session JSONL history. Socket and callback paths are last-known -//! runtime hints, not proof of liveness. +//! Active-reference compare-and-swap replaces only a matching source Segment. +//! Updates preserve unrelated metadata and prevent stale compaction results from +//! overwriting newer references. Callers coordinate switching with append targets +//! and allocation. +//! +//! Child delegation/reclamation state is current restoration data; model-facing +//! child notifications are recorded separately in Session history. Peer registration +//! includes no delegated scope. Stored socket/callback locations are last-known +//! execution data, not proof of liveness. use crate::{SegmentId, SessionId}; use serde::{Deserialize, Serialize}; diff --git a/crates/session-store/src/worker_session_store.rs b/crates/session-store/src/worker_session_store.rs index 92b5b2f32..f8ef36d45 100644 --- a/crates/session-store/src/worker_session_store.rs +++ b/crates/session-store/src/worker_session_store.rs @@ -1,13 +1,19 @@ -//! Filesystem store for the canonical `1 Worker = 1 Session` aggregate. +//! File storage for a Session bound to one Worker. //! -//! Layout under one Worker aggregate: -//! - `session/session.json` — immutable Session identity -//! - `session/segments/.jsonl` -//! - `session/segments/.trace.jsonl` +//! The manifest fixes the Session ID; requests to switch Sessions are rejected. +//! Fork/compaction adds Segments within the same Session, so Worker restoration and +//! history observation need not change the conversation ID. Runtime execution attempts +//! and sockets are not part of this ID. //! -//! Unlike [`crate::FsStore`], this store cannot enumerate or switch between -//! arbitrary Sessions. The first segment materializes the sole Session identity; -//! every later operation must use that same ID. +//! Validates complete JSONL records before append; only a trailing fragment without +//! a closing newline is discarded as uncommitted. Corrupt complete records are rejected. +//! Failed appends attempt rollback to the previous boundary; if rollback also fails, +//! both failures are returned. Whole-Segment and manifest saves sync temporary files +//! and rename them. +//! +//! Retained-history observation uses a read path without creation or migration. +//! Reads are rejected where access time cannot be preserved, avoiding observation-induced +//! changes to stored state. use crate::event_trace::TraceEntry; use crate::paste_artifact::{read_from_dir, write_to_dir}; diff --git a/crates/standalone/README.md b/crates/standalone/README.md new file mode 100644 index 000000000..2425ec0fe --- /dev/null +++ b/crates/standalone/README.md @@ -0,0 +1,11 @@ +# standalone + +A Host that starts existing Workers in the same process. Assembles interactive Workers, local persistence, Host-owned Jobs, and explicitly selected Subjects. Does not create Runtime/Workspace registries or a separate execution system. + +## Reference index + +- [`host`] — bootstrap and confirmed shutdown. +- [`launch`] — Profile, cwd, and Subject validation before launch. +- [`store`] — Worker records and leases. +- [`jobs`] — Job execution and independent result and cleanup lifetimes. +- [`subjektiv`] — exclusive ownership of local Subjects and bounded capabilities. diff --git a/crates/standalone/src/host.rs b/crates/standalone/src/host.rs index 09cd555e5..85ee39453 100644 --- a/crates/standalone/src/host.rs +++ b/crates/standalone/src/host.rs @@ -1,3 +1,15 @@ +//! Assembly of in-process Workers and Host-owned resources. +//! +//! Fresh launches use `WorkerBootstrap`; restores use saved Sessions. Both follow the existing +//! controller lifecycle. Transport is InProcess; normal launches create no subprocess or listener. +//! Canonical cwd supplies filesystem authority; the controller plans/installs Features. +//! Injected model clients replace transport without skipping Profile or capability validation. +//! +//! The Host owns the Worker lease and, when selected, the Subject lease. It waits for Job cleanup +//! and SQLite closure, then releases leases after confirmed controller shutdown, final record +//! commit, and Subject closure. Timeout, lost confirmation, and drop do not prove cleanup; +//! leases remain held to prevent reopening. + use std::path::PathBuf; use std::time::Duration; diff --git a/crates/standalone/src/job_store.rs b/crates/standalone/src/job_store.rs index 2a1110be3..49846eefe 100644 --- a/crates/standalone/src/job_store.rs +++ b/crates/standalone/src/job_store.rs @@ -1,9 +1,18 @@ -//! Host-owned durable Jobs. Opening performs recovery and therefore requires the -//! StandaloneHost's Worker lease; this is not a user-facing database constructor. +//! Persistence of Job intents, attempts, and results under a Host lease. //! -//! A result is committed here before the service releases its Internal Worker. -//! Worker termination, final prose, and consumer acknowledgement are not success -//! authority. The service owns the connection mutex and live-resource cleanup. +//! Opening performs recovery, so the Worker lease (Subject lease for Subject-shared stores) +//! is held for the connection lifetime. SQLite transactions and immutability constraints fix +//! requests and results; the service's connection mutex serializes operations. +//! Job state is not stored in Memory or interactive Sessions. +//! +//! Identical intent under the same ID returns the current snapshot; changes conflict. +//! Only known failures can create new attempts through explicit retry. Reopening preserves +//! Reserved and makes unaccepted Dispatched attempts Unknown. Unknown does not prove no effects; +//! cancellation does not erase that attempt's uncertainty or serialization fence. +//! +//! Results commit before releasing the Internal Worker. Replaying the same result is idempotent; +//! different results and stale attempts are rejected. Later cancellation/execution failure does +//! not erase accepted results; consumer acknowledgement is persisted independently. use std::path::Path; diff --git a/crates/standalone/src/jobs.rs b/crates/standalone/src/jobs.rs index 795aa2d50..56c22d2f4 100644 --- a/crates/standalone/src/jobs.rs +++ b/crates/standalone/src/jobs.rs @@ -1,4 +1,19 @@ -//! Host-provided local Jobs. No Runtime registry, Workspace client or dialog-session state. +//! Host-owned local Job execution service. +//! +//! Each attempt uses an existing private Internal Worker, separate from interactive Sessions +//! and model-created SubWorkers. Profile names do not grant result/domain authority. +//! The shared runner validates Profile requirements; the Host supplies a sink bound to the +//! Job, attempt, input digest, and deadline, plus limited domain grants when needed. +//! Parent filesystem and Workspace connections are not implicitly inherited. +//! +//! Success rests on the structured result's SQLite commit, not final prose or `wait` completion. +//! Concurrency slots and serialization keys stay held after result acceptance until cleanup. +//! Dropping a wait future does not release Host cancel/cleanup ownership. Consumer acknowledgement +//! is separate from result lifetime; delivery failures never trigger model reexecution. +//! +//! Timeout/shutdown makes unaccepted attempts Unknown before cancellation. Unconfirmed cleanup +//! keeps Host leases held. Clones cannot reopen the store after service closure or carry +//! authority over to another Host. #[cfg(test)] mod tests; use std::collections::HashMap; diff --git a/crates/standalone/src/launch.rs b/crates/standalone/src/launch.rs index 2220a7c30..d9fa4aff8 100644 --- a/crates/standalone/src/launch.rs +++ b/crates/standalone/src/launch.rs @@ -1,3 +1,9 @@ +//! Standalone launch validation before side effects. +//! +//! Resolves Standalone Profiles from built-in/user registries and canonicalizes cwd. +//! Repository-local Profiles and path selectors do not authorize launch. Selecting a Subject +//! requires a compatible body policy, but policy alone never creates or selects a Subject. + use std::path::{Path, PathBuf}; use manifest::{ diff --git a/crates/standalone/src/lib.rs b/crates/standalone/src/lib.rs index 2d8c6fa5d..a5daf2821 100644 --- a/crates/standalone/src/lib.rs +++ b/crates/standalone/src/lib.rs @@ -1,8 +1,4 @@ -//! In-process standalone host for one top-level Yoi Worker. -//! -//! The crate composes existing `worker`, `manifest`, `session-store`, and -//! `workdir` contracts. It intentionally owns no TUI, Runtime, Workspace -//! Server, HTTP, WebSocket, subprocess Worker, or alternative execution path. +#![doc = include_str!("../README.md")] pub mod host; mod job_store; diff --git a/crates/standalone/src/store.rs b/crates/standalone/src/store.rs index 6b2137ee8..747f2a5e9 100644 --- a/crates/standalone/src/store.rs +++ b/crates/standalone/src/store.rs @@ -1,3 +1,11 @@ +//! Worker records and process leases independent of the Runtime registry. +//! +//! Separates WorkerId and internal storage keys from display names, storing Session pointers, +//! cwd identity, and optional Subject bindings. Checks canonical cwd paths and available device/inode +//! identity, so replacements at the same path are not restore targets. Checks kernel locks against +//! saved lease evidence; owners with uncertain liveness are not reclaimed. Uncertain record commits +//! do not imply that execution stopped or leases were released. + use std::fs::{self, File, OpenOptions}; use std::io::{self, Write}; use std::path::{Path, PathBuf}; diff --git a/crates/standalone/src/subjektiv.rs b/crates/standalone/src/subjektiv.rs index 357f2679f..12ad57587 100644 --- a/crates/standalone/src/subjektiv.rs +++ b/crates/standalone/src/subjektiv.rs @@ -1,5 +1,19 @@ -//! Operator-managed local Subject storage and explicit Host capabilities. -//! Profile policy, identifiers and repository files are never execution grants. +//! Subjects in operator-managed state scopes and explicit Host capabilities. +//! +//! Issues durable storage-scope identity in a private state directory; the shared domain issues +//! Subject IDs. Cwd, Worker names, and Profiles are not sharing boundaries. Rejects symlinks and +//! non-private nodes; delegates schema/migration to `subjektiv` and connection policy to `feature-storage`. +//! +//! The Host holds each selected Subject's exclusive kernel lease through body execution and +//! confirmed delegated Job cleanup. Process death releases kernel locks; uncertain in-process +//! cleanup retains leases. Restore validates saved Subject/storage bindings and reacquires the +//! same Subject. Launches without a selected Subject do not implicitly connect this store. +//! +//! The selected Subject's Job store is shared across Workers; the generic Jobs service owns execution +//! and cleanup. Consolidation grants allow only immutable candidate batches and limited operations, +//! not body leases or parent filesystems. Startup pending processing follows Host connections. +//! Unknown attempts are not rerun; delivery failures for accepted results do not trigger model +//! reexecution. Memory application and surface repair use shared-domain CAS/receipts. use std::fs::{self, File, OpenOptions}; use std::io::Write; diff --git a/crates/standalone/src/subjektiv_sessions.rs b/crates/standalone/src/subjektiv_sessions.rs index 0c510aa57..5adf1ece5 100644 --- a/crates/standalone/src/subjektiv_sessions.rs +++ b/crates/standalone/src/subjektiv_sessions.rs @@ -1,6 +1,11 @@ -//! Subject-scoped observation of standalone Sessions. This module owns no -//! Session body or projection: it uses the shared strict committed public index -//! reader for the FsStore layout, never a raw snapshot projector. +//! Read adapter for standalone Sessions attributed to a Subject. +//! +//! Checks saved storage-scope/Subject/Worker bindings and attribution, reading only the shared +//! strict committed public index. Does not copy Session content into the Memory DB or expose +//! system prompts, reasoning, uncommitted Runs, or partial JSONL tails. The Host establishes +//! attribution even for Sessions without candidates. Cursor continuation revalidates filters, +//! bindings, and committed generations. Local storage namespaces are not execution Runtime +//! identities; public evidence does not fabricate Workspace/Account/Runtime identity. use std::path::{Path, PathBuf}; diff --git a/crates/subjektiv/README.md b/crates/subjektiv/README.md index ec717a162..b120b95e8 100644 --- a/crates/subjektiv/README.md +++ b/crates/subjektiv/README.md @@ -1,97 +1,11 @@ # subjektiv -Shared Subject-scoped Memory domain and model-facing operations. This crate has -no Workspace Server dependency. `server-api` is the existing transport-neutral -request/response contract crate, not an authentication or Server implementation. +Shared Host domain implementation for per-Subject Memory, change history, candidate dispositions, and derived surfaces. +`feature-storage` manages physical SQLite storage; the calling Host handles authentication and execution authority. -## Storage scope +## Main entry points -The trusted Host creates `feature_storage::FeatureStorage`, selects -`FeatureStorage::scope(scope_id)`, registers `REGISTRATION` (or uses -`SubjektivStore::register`), and opens `SubjektivStore::open`. A trusted managed -`FeatureDatabase` can also be opened with `SubjektivStore::from_database`. - -`scope_id` identifies the physical storage sharing boundary. Backend uses its -existing Workspace ID. Standalone uses a persisted random local storage-scope -identity under its operator-managed state directory, **not a fake Workspace**. -Subject IDs are independently issued random domain identities. An ID is a lookup -key, never an authorization credential. - -Historical migration plans remain frozen. Migration 7 replaces state counters -with opaque Memory change identities and explicit predecessor history; the -original serialized records are retained in an immutable audit archive. -`store_scope.workspace_id` still stores the physical storage scope, under its -historical column name. Existing Backend placement and snapshot manifest keys -are unchanged. Optional historical `EvidenceOrigin.workspace_id` values continue -to be checked against the physical scope; local provenance need not invent a -Workspace value. `workspace_id()` is a Backend compatibility alias for -`scope_id()`. - -## Host operation boundary - -`api::execute(store, &HostOperationContext, operation)` is the common dispatcher -used by Backend and local Hosts. It owns query/cursor rules, bounded body and -nested provenance projections, immutable change reads, proposal validation, -explicit staging/receipt rules, resident context, candidate decisions and -surface generation/publication/failure projections. - -The Host context is not serializable or model input. Trusted Rust Hosts construct -it only after validating their external execution authority: - -- `HostOperationContext::validated_body` binds one Subject and optionally an - existing immutable `SubjectSessionAttribution`. Explicit staging requires the - request Session to match that attribution. -- `HostOperationContext::validated_consolidation` binds one Subject, an optional - immutable candidate batch, and an optional `JobAttemptBinding`. Attenuated Jobs - must supply `Some(batch)`, including an empty batch; `None` is legacy non-Job - Host consolidation authority. Body-only operations are unavailable to - consolidation contexts, and candidate reads/decisions cannot escape the batch. - -These constructors validate store/domain bindings, not external authentication. -Backend retains authenticated Runtime/Worker identity, current singleton and -live Job attempt checks, candidate grant checks, and public Session validation. -Local Hosts retain exclusive Subject execution leases, generic Job admission, -Worker/Session ownership, and committed public Session evidence validation. -Before explicit staging, the Host resolves evidence using its committed public -Session projection; the common API does not inspect Session files, hidden -reasoning, system prompts, or arbitrary paths. Lower-level repository/projection -functions are trusted Rust APIs, not model capabilities. - -## Common consolidation Job rules - -`job::bounded_candidate_batch(ids, max_result_bytes)` selects the exact T-704 -ordered-prefix budget (100 candidates maximum, 2,048-byte envelope, twice each -JSON-escaped ID size plus 128 bytes per disposition). -`job::consolidation_required(count, bytes, force, availability)` owns the shared -trigger thresholds and empty-ready/surface-retry rules. - -`job::validate_result(store, subject_id, batch, job_id, attempt_id, value)` checks -the strict result shape, exact admitted Subject/batch, durable dispositions, and -the persisted terminal surface outcome from that attempt. After validation, -`job::with_dispositions(store, subject_id, batch, value)` adds the deterministic -compact disposition projection. Hosts retain Job result binding/admission; these -helpers never run or replay a Job. The store's `prepare_job_surface_generation`, -`require_job_surface_generation`, and `validate_job_surface_outcome` are public -trusted-domain methods for both Backend and local Hosts. - -## Content conditions and history - -Memory updates require the exact `expected_change_id` read by the caller. -Each committed change receives a new opaque ID and retains `previous_change_id`; -IDs are never sorted or incremented to infer history. Body/evidence continuations -pin that immutable ID, and history cursors pin a predecessor-chain head. - -Subject behavior updates compare `expected_behavior_md` byte-for-byte and do not -participate in Memory freshness. Surface runs, publications and query cursors use -`memory_fingerprint`, a SHA-256 digest of the length-framed, ordered current -`(memory_id, change_id)` identities. The Subject stores no redundant freshness -counter. Publication remains first-wins for the captured Memory input set. - -Old Surface input sets cannot be proven from historical counters. Migration 7 -retains those snapshots and runs as archival history with deliberately detached -input identities, marks existing surfaces stale, and requires regeneration. -Old Flow provenance keeps its exact original bytes in the archive; a definition -fingerprint is not fabricated from an old numeric label. - -Dependencies: `feature-storage`, `chrono`, `memory`, `manifest`, `rusqlite`, -`serde`, `serde_json`, `server-api`, `sha2`, `thiserror`, `uuid`; tests use `tempfile`. +- [`SubjektivStore`]: scope-bound records and transactions +- [`MemoryRecord`], [`SurfaceSnapshot`]: change history and derived context +- [`api`]: shared Host operations and bounded responses +- [`job`]: consolidation selection rules and result validation diff --git a/crates/subjektiv/src/api.rs b/crates/subjektiv/src/api.rs index 8295aa7ab..185d55b5d 100644 --- a/crates/subjektiv/src/api.rs +++ b/crates/subjektiv/src/api.rs @@ -1,6 +1,13 @@ -//! Shared model-facing Memory operations and bounded projections. -//! Hosts retain authentication, execution leases, Job admission and committed -//! Session evidence validation; this module never derives ambient authority. +//! Shared Host Memory operations and bounded responses. +//! +//! Applies the Subject and consolidation candidate batch pinned by [`HostOperationContext`] to operations. +//! The Host owns authentication, execution leases, Job admission, and validation of committed Session evidence. +//! This module reads neither Session files nor arbitrary paths, and never infers authority from model input. +//! +//! Query cursors pin filters and the Memory fingerprint, rejecting intervening changes as stale. +//! Body/provenance continuation pins immutable change IDs; history pages pin the predecessor-chain head. +//! Pages respect the whole-response serialization budget and UTF-8 boundaries, allowing continuation +//! offsets to resume even with JSON escapes or long lines. Returns only stored evidence anchors, never copies raw Session bodies. mod context; mod operation; diff --git a/crates/subjektiv/src/api/context.rs b/crates/subjektiv/src/api/context.rs index 6e1a95633..af88e3581 100644 --- a/crates/subjektiv/src/api/context.rs +++ b/crates/subjektiv/src/api/context.rs @@ -19,11 +19,9 @@ enum Authority { }, } -/// A capability issued by trusted Rust Host code after authentication/lease, -/// delegation, and committed Session evidence checks. Not a request DTO. -/// Constructors check domain binding, not external identity or execution leases. -/// Backend must retain its authenticated Worker and active Job grant checks; -/// standalone must hold the selected Subject lease and attenuate Job batches. +/// Operation context constructed by a Host after checking external authentication, leases, delegation, and Session evidence. +/// Constructors validate only store domain bindings, not external execution authority. +/// Not a request DTO; models never select the Subject or Job batch. #[derive(Debug, Clone)] pub struct HostOperationContext { scope_id: String, @@ -32,10 +30,9 @@ pub struct HostOperationContext { } impl HostOperationContext { - /// The Host has validated body execution authority. Attribution, when supplied, - /// must already be immutable history in this store. Before StageExplicit the - /// Host must resolve/validate all evidence against committed public Session - /// entries; neither this constructor nor the operation reads Session files. + /// Pins the Host-verified execution context for the Subject body. + /// Supplied attribution must match immutable store history. The Host resolves and validates + /// explicit staging evidence against committed public Sessions; neither this constructor nor operations read Session files. pub fn validated_body( store: &SubjektivStore, subject_id: &str, @@ -62,10 +59,10 @@ impl HostOperationContext { }) } - /// The Host has validated consolidation authority and (for a Job) its active - /// attempt. `Some(candidate_ids)` is the immutable admitted batch; no candidate - /// outside it is accessible. `None` retains Backend's non-Job consolidation - /// authority, and must not be issued to an attenuated Job. + /// Pins Host-verified consolidation authority and, when needed, the current Job attempt. + /// `Some(candidate_ids)` is an immutable allowed set, including empty batches; candidates outside it cannot be operated on. + /// `None` is for non-Job Host authority, never issued to a restricted Job. + /// This context cannot perform Subject-body-only operations. pub fn validated_consolidation( store: &SubjektivStore, subject_id: &str, diff --git a/crates/subjektiv/src/api/operation.rs b/crates/subjektiv/src/api/operation.rs index 857909349..6b48371d1 100644 --- a/crates/subjektiv/src/api/operation.rs +++ b/crates/subjektiv/src/api/operation.rs @@ -3,8 +3,8 @@ use server_api::{ SubjektivMemoryBackendOperation as Op, SubjektivMemoryBackendResponse as Response, }; -/// Executes the same domain operations and bounded projections for every Host. -/// The caller authenticates and validates evidence before issuing `context`. +/// Executes the same domain operations and bounded projections on each Host. +/// The caller validates authentication/evidence and issues context; this function checks its scope and operation kind. pub fn execute( store: &crate::SubjektivStore, context: &HostOperationContext, diff --git a/crates/subjektiv/src/job.rs b/crates/subjektiv/src/job.rs index 7b29c4e0e..656d341d2 100644 --- a/crates/subjektiv/src/job.rs +++ b/crates/subjektiv/src/job.rs @@ -1,7 +1,8 @@ -//! Common consolidation Job policy and verified result projections. +//! Shared consolidation Job selection rules and validation against stored results. //! -//! Hosts authenticate/admit Jobs and bind their immutable batch and attempt. -//! These helpers do not dispatch, replay, or grant Job execution authority. +//! The Host owns authentication/admission and pins immutable batches/attempts. +//! This module neither dispatches nor reruns Jobs, and grants no execution authority from Profile names or model claims. +//! Validates Memory candidate processing and surface preparation separately; failure of the latter never revokes the former. use serde::Deserialize; use crate::api::{OperationError, OperationResult}; @@ -11,9 +12,9 @@ pub const CANDIDATE_BATCH_LIMIT: usize = 100; pub const CONSOLIDATION_THRESHOLD_FILES: usize = 5; pub const CONSOLIDATION_THRESHOLD_BYTES: u64 = 50_000; -/// Selects an ordered prefix using the T-704 complete-result budget. Reserve -/// 2,048 bytes for the Subject/Host surface envelope, then twice the JSON-escaped -/// ID size plus 128 bytes per candidate for its compact verified disposition. +/// Selects an ordered prefix fitting the result budget. +/// Estimates the Subject/surface envelope and JSON-escaped IDs appearing in both +/// batch declarations and validated dispositions. Overflow candidates stay for the next batch; existing batches never expand. pub fn bounded_candidate_batch( ids: impl IntoIterator, max_result_bytes: u32, @@ -34,8 +35,8 @@ pub fn bounded_candidate_batch( Ok(selected) } -/// Empty ready Subjects need no work, even when forced. Missing/stale/failed -/// surfaces bypass candidate thresholds; confirmed Memory is never discarded. +/// With no candidates and a ready surface, skips even when forced; ungenerated/stale/failed surfaces need generation regardless of candidate thresholds. +/// This is not a decision to delete committed Memory. pub fn consolidation_required( candidate_count: usize, total_bytes: u64, @@ -72,9 +73,9 @@ struct SurfaceResult { reason_code: Option, } -/// Verifies the exact Subject/batch, every immutable disposition, and the actual -/// terminal surface outcome begun by this Job attempt. Model claims alone never -/// establish completion. Same admission semantics as the Backend T-704 adapter. +/// Validates the exact Subject/batch, immutable dispositions for all candidates, and that Job attempt's stored terminal surface outcome. +/// Rejects unresolved candidates, stale input, generation from another attempt, and fabricated results. +/// Model final text or Idle is not completion evidence; this validation also does not replace Host Job-result acceptance. pub fn validate_result( store: &SubjektivStore, subject_id: &str, @@ -117,9 +118,9 @@ pub fn validate_result( }) } -/// Enriches a previously verified result with compact durable dispositions. Full -/// reasons, affected changes and provenance remain in the receipt store. This -/// deterministic projection neither applies candidates nor creates a new ledger. +/// Adds a compact projection of stored dispositions to a validated result. +/// Reasons, change references, and provenance stay in the receipt store, not duplicated in bounded Job results. +/// Neither reapplies candidates nor creates a new ledger. pub fn with_dispositions( store: &SubjektivStore, subject_id: &str, diff --git a/crates/subjektiv/src/legacy_migrations.rs b/crates/subjektiv/src/legacy_migrations.rs index f20e21694..614f79e05 100644 --- a/crates/subjektiv/src/legacy_migrations.rs +++ b/crates/subjektiv/src/legacy_migrations.rs @@ -1,5 +1,8 @@ -//! Frozen migrations 1–6 and the one-time conversion of their data. -//! Do not expose these old fields through current domain operations. +//! Frozen migrations and one-time conversion of existing data. +//! +//! Preserves original serialized records in an immutable archive; current changes receive independent IDs and predecessors. +//! Legacy counters cannot prove the complete surface input set, so snapshots and runs remain as history, +//! detached from current input identity and stale until regeneration. Legacy representations are not exposed to current operations. use super::*; pub(super) fn create_schema(transaction: &Transaction<'_>) -> feature_storage::Result<()> { diff --git a/crates/subjektiv/src/lib.rs b/crates/subjektiv/src/lib.rs index b461e8d8a..b8e176940 100644 --- a/crates/subjektiv/src/lib.rs +++ b/crates/subjektiv/src/lib.rs @@ -1,10 +1,4 @@ -//! Subject-scoped Memory storage with immutable change history. -//! -//! This module owns subjektiv's domain schema and typed repository. Physical -//! SQLite placement, connection configuration, migration execution, backup, -//! restore, and teardown remain owned by [`feature_storage`]. A store is -//! already bound to one Host storage scope; a subject identifier is therefore a lookup -//! key, not an authorization credential. +#![doc = include_str!("../README.md")] pub mod api; pub mod job; @@ -181,9 +175,9 @@ pub fn subject_worker_singleton_key(subject_id: &str) -> Result { Ok(format!("{SUBJECT_WORKER_SINGLETON_PREFIX}{subject_id}")) } -/// Immutable host-recorded history that attributes one durable Session to the -/// subject and Worker that produced it. This is archival provenance only: it is -/// deliberately not a current-Worker link or execution ownership record. +/// Immutable historical attribution of a Session to its Subject/Worker, recorded by the Host. +/// Not a current Worker link or execution ownership. Never infer or repair attribution from candidate presence or bodies. +/// Session content stays in Session storage; this store saves only attribution. #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub struct SubjectSessionAttribution { @@ -230,10 +224,10 @@ pub enum ChangeProposalIntent { Reopen, } -/// Host-owned metadata describing how a staged candidate is expected to change -/// one existing Memory. It is validated against the current scoped Memory when -/// the candidate is staged; it is not model-authored and does not itself apply -/// the change. +/// Host-owned metadata specifying how a staging candidate proposes to change existing Memory. +/// Checked against current scoped Memory at staging; registering the candidate does not itself change Memory. +/// Application rechecks the same Memory ID, expected change ID, intent, and reason, +/// never retargeting an old proposal to the latest change or inferring the target from claim text. #[derive(Debug, Clone, PartialEq, Eq, Serialize)] #[serde(deny_unknown_fields)] pub struct ChangeProposal { @@ -353,15 +347,14 @@ impl SubjectStagingRecord { #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "snake_case")] +/// Memory recall eligibility, not proof of truth or execution authority. +/// New Memory starts Active; Retracted is terminal. pub enum MemoryState { - /// Current and eligible for ordinary recall. This does not assert truth and - /// grants no authority. + /// Currently eligible for ordinary recall. Active, - /// No longer current because its question, condition, or applicability has - /// ended. A later change may reopen it when circumstances change. + /// The question, condition, or applicability period has ended. A later change can restore Active. Resolved, - /// Invalidated and no longer eligible for recall. Retraction is terminal; - /// corrected follow-up experience is a separate Memory linked by derivation. + /// Invalidated. A distinct experience after correction creates new Memory and may reference the old change through derivation. Retracted, } @@ -374,6 +367,9 @@ pub struct MemoryChangeRef { #[derive(Debug, Clone, Serialize, Deserialize)] #[serde(deny_unknown_fields)] +/// Immutable change record appended for each update to the same Memory. +/// Opaque change IDs and previous change IDs link history; neither ID magnitude nor timestamps imply order. +/// The current projection points to the history head. Candidate/derivation references also pin individual changes. pub struct MemoryRecord { pub schema_version: u32, pub id: String, @@ -461,6 +457,8 @@ pub struct StagingResolution { #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)] +/// Immutable startup context derived from committed Memory, not another authoritative copy of Memory. +/// Pins the body, exact change references, and source fingerprint; never used as current context after Memory changes. pub struct SurfaceSnapshot { pub schema_version: u32, pub id: String, @@ -609,14 +607,21 @@ static MIGRATIONS: &[FeatureMigration] = &[ pub const REGISTRATION: FeatureRegistration = FeatureRegistration::new(SUBJEKTIV_FEATURE_ID, MIGRATIONS); +/// Subject Memory repository restricted to a Host storage scope. +/// Owns schemas, validation, queries, and domain transactions; delegates physical layout +/// and connection lifecycle to [`feature_storage`]. Synchronous API; async Hosts use a blocking executor. +/// +/// Scope and subject IDs select storage boundaries; the Host authenticates before calls. +/// Independent of the legacy Workspace Memory DB, with no implicit import or dual writes. +/// This store saves no current Worker links; the Host manages execution ownership. #[derive(Clone)] pub struct SubjektivStore { database: FeatureDatabase, } impl SubjektivStore { - /// Registers the trusted schema with the neutral Host storage manager. - /// One registration can be reused across all scopes served by that manager. + /// Registers the schema with neutral Host storage. + /// Registration is separate from opening the DB and is reusable across scopes in the same manager. pub fn register(storage: &ScopedFeatureStorage) -> Result { storage.register(REGISTRATION).map_err(Into::into) } @@ -631,10 +636,10 @@ impl SubjektivStore { Self::from_database(storage.open(registration)?) } - /// Opens a trusted managed database. `store_scope.workspace_id` retains its - /// historical SQL name: it stores the physical Host storage-scope identity. - /// Backend uses its Workspace ID; local Hosts use a persisted random scope, - /// without creating a Workspace. Subject IDs are separate domain identities. + /// Validates the managed DB's Feature ID and stored scope match. + /// `store_scope.workspace_id` is a compatibility SQL name denoting the physical storage scope. + /// Backends use Workspace IDs; local Hosts use persistent independent scopes. + /// Subjects have separate identities within a scope; a local scope creates no Workspace. pub fn from_database(database: FeatureDatabase) -> Result { if database.feature_id() != SUBJEKTIV_FEATURE_ID { return Err(SubjektivError::InvalidRecord( @@ -736,8 +741,8 @@ impl SubjektivStore { }) } - /// Replaces behavior only if its exact previously-read content still matches. - /// Memory changes do not participate in this condition. + /// Updates only when the observed behavior body matches byte-for-byte. + /// Behavior is a document independent of Memory, excluded from Memory-change and surface-freshness comparison conditions. pub fn update_subject_behavior( &self, subject_id: &str, @@ -759,8 +764,9 @@ impl SubjektivStore { }) } - /// Derived content condition over all current Memory identities, not stored - /// on the Subject and never incremented. + /// SHA-256 hashes the sorted, length-delimited current `(memory_id, change_id)` set across all states. + /// Content/state/provenance changes issue new change IDs, preventing equality even in ABA returns to prior content. + /// Saves no redundant Subject counter; staging alone does not change this fingerprint. pub fn memory_fingerprint(&self, subject_id: &str) -> Result { self.database.try_transaction(|tx| { require_subject(tx, subject_id)?; @@ -847,8 +853,8 @@ impl SubjektivStore { }) } - /// Subject retirement is distinct from stopping any current Worker. It does - /// not edit or delete Memory and cannot be reversed by this baseline schema. + /// Retires a Subject independently of stopping Workers. Memory/history remain unedited, undeleted, and readable. + /// Retirement is irreversible; new candidates and Memory changes require an active Subject. pub fn retire_subject(&self, subject_id: &str) -> Result { self.database.try_transaction(|transaction| { let mut subject = require_subject(transaction, subject_id)?; @@ -868,6 +874,9 @@ impl SubjektivStore { }) } + /// Saves an immutable candidate without changing committed Memory. + /// Replay with the same candidate identity and payload returns the original record; different content conflicts. + /// Extraction failure after saving never revokes the candidate. The Host commits the extraction pointer separately. pub fn stage_candidate(&self, record: SubjectStagingRecord) -> Result { validate_staging_record(self.workspace_id(), &record)?; validate_staging_admission(&record)?; @@ -878,9 +887,8 @@ impl SubjektivStore { }) } - /// Atomically verifies immutable Host attribution already completed at the - /// Session lifecycle boundary and stages its extracted candidate. Candidate - /// staging never creates or repairs Session attribution. + /// Validates immutable Host attribution already recorded by the Session lifecycle and saves the candidate in the same transaction. + /// Candidate registration never creates or repairs Session attribution. pub fn stage_candidate_with_attribution( &self, mut record: SubjectStagingRecord, @@ -1154,6 +1162,9 @@ impl SubjektivStore { }) } + /// Appends an immutable change only when the observed expected change ID matches the current head. + /// Comparison, history append, and current projection update share a transaction, preventing late stale decisions from overwriting. + /// This direct update cannot add candidate provenance; use [`Self::apply_candidates`]. pub fn revise_memory( &self, subject_id: &str, @@ -1173,8 +1184,8 @@ impl SubjektivStore { }) } - /// Atomically writes a Memory change and one candidate's immutable - /// resolution. Any conflict or invalid reference rolls both changes back. + /// Saves a Memory change and one candidate's immutable disposition together. + /// Conflicts or invalid references roll back both. pub fn apply_candidate( &self, subject_id: &str, @@ -1189,9 +1200,11 @@ impl SubjektivStore { Ok((memory, resolutions.remove(0))) } - /// Atomically applies one or more candidates to exactly one new Memory - /// change_id. The repository derives `source_candidate_ids` from this list so - /// every applied resolution has a reciprocal immutable provenance edge. + /// Atomically applies unresolved candidates from the same Subject to one new Memory change. + /// Derives provenance from the candidate set, committing the change, reference edges, current projection, + /// surface staleness, and each immutable disposition in one transaction. Standalone close cannot record Applied. + /// Mismatched comparison conditions or proposal intents roll back everything, leaving candidates unresolved. + /// Atomicity covers only this Feature DB, not the Host DB or subsequent surface generation. pub fn apply_candidates( &self, subject_id: &str, @@ -1223,6 +1236,10 @@ impl SubjektivStore { }) } + /// Saves candidate decisions with stable request IDs and immutable receipts. + /// Exact replay returns the committed receipt; ID reuse with different input is rejected. + /// Applied Memory is never revoked by subsequent model-execution or surface-generation failure. + /// Failures/empty responses are not themselves candidate dispositions; only explicit decisions are saved. pub fn decide_candidate( &self, subject_id: &str, @@ -1502,9 +1519,11 @@ impl SubjektivStore { }) } - /// Captures one deterministic, bounded generation input from only current - /// active Memory changes. Kind order is fixed and selection is round-robin - /// across kinds; within each kind, updated_at desc then memory id asc. + /// Selects generation materials only from current active Memory, with fixed ordering and limits. + /// Uses round-robin across kinds; within each kind, descending update time then ascending Memory ID. + /// Reserves equal budgets for nonempty kinds before redistributing surplus, preventing large kinds from dominating. + /// Uses neither staging, Session history, nor old surfaces. The material budget leaves room for fixed prompts, etc.; + /// the caller checks the complete editor-request budget. pub fn prepare_surface_generation(&self, subject_id: &str) -> Result { self.prepare_job_surface_generation(subject_id, None) } @@ -1547,9 +1566,9 @@ impl SubjektivStore { }) } - /// Validate a Job's structured surface claim against the existing generation - /// and publication/failure records. This does not create another receipt or - /// roll back Memory. The generation must have begun inside this attempt. + /// Checks a structured Job surface result against generation started by the same attempt and the stored terminal outcome. + /// Rejects mismatches in the current Memory fingerprint or ready/failed content. + /// Creates no new receipt and never rolls back committed Memory. #[allow(clippy::too_many_arguments)] pub fn validate_job_surface_outcome( &self, @@ -1596,8 +1615,10 @@ impl SubjektivStore { }) } - /// Validates structured editor output against the exact material snapshot and - /// atomically publishes it only while the subject store generation is current. + /// Checks editor output shape, budget, and exact material references, publishing in a transaction where the source fingerprint matches current input. + /// Reference validation checks existence/freshness, not semantic correctness of generated text. + /// Replay with the same fingerprint/output returns the same snapshot; different concurrent output never replaces the first committed result. + /// If active Memory exists but no material fits, fails rather than publishes a false empty result. pub fn publish_surface_generation( &self, subject_id: &str, @@ -1665,8 +1686,9 @@ impl SubjektivStore { }) } - /// Records a bounded generator failure without changing confirmed Memory. A - /// late failure cannot replace a ready surface or dirty a newer generation. + /// Saves a bounded failure reason without changing committed Memory. + /// Late failures never replace a ready surface or the state of newer Memory input. + /// Separate from candidate application, so generation failure never revokes committed decisions. pub fn fail_surface_generation( &self, subject_id: &str, @@ -1718,9 +1740,9 @@ impl SubjektivStore { }) } - /// Returns only a current ready snapshot for injection. Other states are - /// explicit so callers never report missing/stale/failed generation as empty - /// confirmed Memory. + /// Returns only ready snapshots matching the current fingerprint for resident injection. + /// Ready with an empty body is a normal empty result, distinct from ungenerated/stale/failed. + /// Returns availability separately so an absent injected body implies neither absent committed Memory nor successful generation. pub fn resident_surface(&self, subject_id: &str) -> Result { self.database.try_with_connection(|connection| { let subject = require_subject_in_connection(connection, subject_id)?; @@ -1728,8 +1750,7 @@ impl SubjektivStore { }) } - /// Atomically reads user-managed behavior and the independently-fresh - /// generated Memory surface from one SQLite connection snapshot. + /// Returns independent behavior and the generated Memory surface as observations from the same SQLite transaction. pub fn resident_context(&self, subject_id: &str) -> Result { self.database.try_transaction(|transaction| { let subject = require_subject(transaction, subject_id)?; diff --git a/crates/subjektiv/src/schema.rs b/crates/subjektiv/src/schema.rs index fdf87ef0a..424a35f80 100644 --- a/crates/subjektiv/src/schema.rs +++ b/crates/subjektiv/src/schema.rs @@ -1,4 +1,8 @@ -//! Current schema creation. Frozen data conversion lives with the old migrations. +//! Creates the current schema and connects frozen migrations. +//! +//! Foreign keys for Memory history and candidate/derivation/surface references include the Subject, +//! rejecting cross-Subject references. Seals/triggers prevent adding references after immutable JSON and edges commit; +//! retention triggers reject deletion of history and referenced records. The repository updates current projections and search columns together. use super::*; pub(super) fn migrate(tx: &Transaction<'_>) -> feature_storage::Result<()> { diff --git a/crates/ticket/README.md b/crates/ticket/README.md new file mode 100644 index 000000000..f9f9ec978 --- /dev/null +++ b/crates/ticket/README.md @@ -0,0 +1,11 @@ +# ticket + +Handles Ticket requirements, states, relations, audit events, and per-Workspace SQLite persistence. +Does not search Repository documents or handle Git integration. + +## Main entry points + +- [`TicketBackend`], [`SqliteTicketBackend`]: operation boundary and persistence +- [`TicketBackend::complete`], [`TicketBackend::update_state`]: completion decisions and state updates +- [`ticket_content_digest`]: requirement-content comparison condition +- [`tool`]: Worker-facing operation definitions diff --git a/crates/ticket/src/lib.rs b/crates/ticket/src/lib.rs index ee50132b9..af5a3594f 100644 --- a/crates/ticket/src/lib.rs +++ b/crates/ticket/src/lib.rs @@ -1,8 +1,4 @@ -//! Ticket domain types and SQLite Workspace authority backend. -//! -//! Repository-local Ticket trees are not an authority. Runtime Workers access -//! Tickets through the Workspace API, whose control-plane implementation uses -//! the SQLite backend defined here. +#![doc = include_str!("../README.md")] use std::collections::{BTreeMap, BTreeSet, HashMap}; use std::fmt; @@ -516,10 +512,9 @@ impl From for TicketStateUpdate { } } -/// SHA-256 of the actual editable Ticket requirements: identity, title, body, -/// and ordered repository targets. Comments, timestamps, workflow state, and -/// event order are not requirements. Restoring identical content restores its -/// digest; state CAS and operation receipts are independent preconditions. +/// SHA-256 digest comparing Ticket identity, title, body, and ordered Repository targets. +/// Comments, timestamps, workflow state, and event order are not requirement content. +/// Returning to identical content restores the digest, so state comparison conditions and operation receipts remain separately necessary. pub fn ticket_content_digest(ticket: &Ticket) -> String { content_digest( &ticket.meta.id, @@ -1804,11 +1799,23 @@ pub trait TicketBackend { change: TicketStateChange, ) -> Result<()>; fn set_workflow_state(&self, id: TicketIdOrSlug, change: TicketStateChange) -> Result<()>; + /// State update comparing requirement content and workflow state in the same write transaction. + /// Compares both even when only requirements change, preventing late decisions based on old content from committing. + /// Saves state, status, resolution, and the audit event together; the event is an immutable operation receipt. + /// + /// Exact replay returns the current Ticket even after later edits or reopening. + /// Does not reapply past state or append events. Rejects different input with the same operation key. + /// Updating to Closed also stores the reason as resolution. Reopening updates Ticket metadata; + /// it does not revive released work identities. fn update_state(&self, _ticket: &str, _request: TicketStateUpdate) -> Result { Err(TicketError::Conflict( "backend does not support audited state updates".to_owned(), )) } + /// Ticket completion decision with a reason and references, independent of Repository integration. + /// Uses the same content/state comparison conditions and operation receipts as [`Self::update_state`]. + /// Requires neither MR nor Reviewer approval; completion neither creates nor changes their evidence. + /// Missing MR evidence on the read side is not grounds to revoke a recorded completion decision. fn complete(&self, ticket: &str, request: TicketCompletion) -> Result { self.update_state(ticket, request.into()) } @@ -2058,6 +2065,9 @@ pub struct SqliteTicketMutationEvent { pub type SqliteTicketMutationHook = dyn Fn(&Connection, &SqliteTicketMutationEvent) -> Result<()> + Send + Sync; +/// SQLite persistence implementation for per-Workspace Ticket records. +/// Uses rows in the specified Workspace as authoritative, without searching Repository Ticket trees. +/// The Host supplies authenticated operation context; this implementation commits state changes and audit in one transaction. #[derive(Clone)] pub struct SqliteTicketBackend { db_path: PathBuf, diff --git a/crates/tools/README.md b/crates/tools/README.md index dd046a3e0..5003eb332 100644 --- a/crates/tools/README.md +++ b/crates/tools/README.md @@ -1,100 +1,10 @@ # tools -## Role +`tools` implements file and command operations through Host-owned `WorkdirSession`s, plus Web fetching and image output. Owns input schemas, output formatting, and change detection after reads, but not Workdir creation or connection lifetimes, Manifest permission definitions, or the Engine execution loop. -`tools` implements built-in tools and shared tool execution helpers used by Workers. +## Main entry points -## Boundaries - -Owns: - -- built-in filesystem, web, memory, and Worker-management tool implementations where applicable -- bounded tool output formatting -- scope-aware file operation helpers -- tool-facing diagnostics suitable for history/model consumption - -Does not own: - -- manifest permission policy definition (`manifest`) -- Engine tool-loop semantics (`agen`) -- Worker lifecycle decisions (`worker`) -- UI presentation (`tui`) - -## Design notes - -A tool implementation must assume model input is untrusted. Permission policy, scope checks, output bounding, and redaction are part of the safety boundary, not optional UI behavior. - -## Checkout-native processing - -`execute_checkout_tool` (also exported through `tools::checkout`) accepts the -original `WorkdirSessionRouter`, shared `Tracker`, route-bound alias/generation, -logical target and provider validator. It does not create an attachment or router. -The Host owns native publication and permission identity; provider scope and -capabilities remain authoritative. `Create` uses the `Write` capability/identity. - -| Operation | Arguments (only these keys accepted) | Target | -| --- | --- | --- | -| Read | `offset?`, `limit?` | Existing file; same numbered lines, defaults and byte bound as Tool Read | -| Edit | `old_string`, `new_string`, `replace_all?` | Existing file; prior read/hash and provider validator required | -| Write | `content` | Existing file only; prior read/hash and provider validator required | -| Create | `path`, `content` | Bound parent directory; relative destination strictly below it; create-new, missing parents allowed | -| List | `limit?` (1..1000, default 100), `after?` (`{kind, path}`) | Bound directory; Read capability; all typed `EntryKind` cursor values accepted, path is a direct child in checkout-root-relative provider coordinates | -| Glob | `pattern`, `path?` | Bound directory; optional path relative beneath it | -| Grep | `pattern`, `path?`, `glob?`, `type?`, `case_insensitive?`, `-B?`, `-A?`, `-C?`, `multiline?`, `output_mode?`, `head_limit?`, `offset?` | Bound directory; same provider search/options/rendering as Tool Grep | - -Native arguments cannot override `file_path`, `target_workdir`, generation or -validator. Absolute/parent-traversing directory-relative paths are rejected. -Checkout file calls share Tool processing through an optional checked `FileTarget` -and use `checkout_execute`, never a separate Worker-side filesystem engine. -Read records the provider's full content hash **and total line count**, including -partial reads, in the original alias/generation namespace. Normal and native -observations/mutations interoperate. Only successful mutations update tracking. - -Glob/Grep share normal request defaults/options and rendering through `SearchTarget`. -Only native dispatch uses `CheckoutSearchRequest::new` and `checkout_search`. -The constructor's empty scope layers/root output are refined by provider wrappers; -Tools does not synthesize authority or parse/rebase rendered filenames. Ordinary -Tool dispatch remains `session.glob`/`session.grep`, with its existing error mapping. - -List dispatches one bounded `checkout_search` page without Bash or entry prefetch. -`CheckoutToolOutput.listing` preserves the typed `ListResult`, including -`next_after`, for Host projection into canonical Worldspace entries and cursors. -Its normal Tool output is a summary only, and `paths` is empty; every other -Operation returns `listing: None`. The Host decodes canonical cursor paths before -calling Tools, mapping the public `{kind, entry}` cursor into `{kind, path}`. -Both returned entries and the continuation must stay directly inside the bound -directory. Cursor kinds retain all typed values (`directory`, `file`, `symlink`, -`other`); ordering groups directories first and all non-directories together, -then compares paths. A truncated page must supply `next_after` matching its last -entry's ordering key; an untruncated page has no continuation. Special typed -listing entries are path coordinates, not checkout observations: direct checkout -observation remains provider File/Directory-only. Pagination is live, not a -cross-page snapshot; the same before/after bound-directory validator checks -used for search apply to List. - -`CheckoutToolOutput.paths` contains typed checkout-root-relative search paths -(from `GlobResult.paths` / `GrepResult.paths`) and the created path for Create. -Rendered Grep text is never parsed to construct links. The optional validator is -the provider's returned observation validator, unchanged. Create's result -observation describes the bound parent target, while returned paths and tracker -updates describe the destination. Search checks the bound observation before and -after the bounded checkout provider search, but does **not** guarantee an atomic tree snapshot -or prevent external edits to descendants. File validator/hash checks are inside -the provider's checked execution boundary. Checked native provider conflicts and -native generation/search fences use `ToolError::StructuredConflict` with code -`checkout_stale`; ordinary Tool conflicts retain `ExecutionFailed`. Checked-only -known provider refusals use `checkout_denied` (scope/capability/read-only policy), -`checkout_invalid` (invalid arguments/paths/target kind/search syntax), or -`checkout_unavailable` (closed session/unavailable capacity). A disappeared bound -target is `checkout_stale`. These codes preserve known no-effect rejection for the -Host; ordinary Tool error mappings are unchanged. Ambiguous transport/IO and -operation failures remain `ExecutionFailed`, as does explicit provider -`OutcomeUnknown`; a failure after mutation must not -be automatically retried. A malformed post-mutation provider result is likewise -reported explicitly as outcome unknown. Explicit mismatched Read/Glob/Grep results -are readonly failures classified as `checkout_unavailable`, not stale or unknown. - -## See also - -- [`../../docs/design/tool-permissions-scope.md`](../../docs/design/tool-permissions-scope.md) -- [`../../docs/design/memory-knowledge.md`](../../docs/design/memory-knowledge.md) +- [`core_builtin_tools`](src/lib.rs) / [`routed_builtin_tools`](src/lib.rs): registration by provider capability and connection resolution. +- [`checkout`](src/checkout.rs): shared handling for route-bound native operations and regular Tools. +- [`tracker`](src/tracker.rs): read hashes and change tracking per connection generation. +- [`view_image`](src/view_image.rs): bounded byte retrieval and image detection. diff --git a/crates/tools/src/bash.rs b/crates/tools/src/bash.rs index 46d9d3b98..801507c0b 100644 --- a/crates/tools/src/bash.rs +++ b/crates/tools/src/bash.rs @@ -1,3 +1,11 @@ +//! Bash Tool using WorkdirSession command capability. +//! +//! The selected session owns cwd, process launch, bounded output, timeout, and cancellation. +//! Bash is not a sandbox; the Runtime process and OS user define the execution boundary. +//! Per-execution handles and nonces prevent cancellation of another command. Dropping a future +//! does not confirm termination; active registrations remain until cleanup requests cancellation +//! and observes terminal output. + use std::collections::{HashMap, HashSet}; use std::path::PathBuf; use std::sync::{Arc, Mutex}; diff --git a/crates/tools/src/checkout.rs b/crates/tools/src/checkout.rs index be2fdf19f..24e2dde31 100644 --- a/crates/tools/src/checkout.rs +++ b/crates/tools/src/checkout.rs @@ -1,11 +1,36 @@ -//! Native checkout entrypoint using the same processing as ordinary filesystem Tools. +//! Native checkout entry points sharing regular filesystem Tool handling. //! -//! The Host binds alias, generation, path and validator from the checkout route; none -//! can be overridden by Operation arguments. File reads/mutations dispatch through -//! the provider's checked execution boundary. Search shares normal request and -//! rendering processing but dispatches through checkout_search so provider scope -//! layers and output rebasing are preserved; checks bracket the bound directory -//! observation, not an atomic snapshot of the searched tree. +//! The Host pins aliases, connection generations, paths, and validators from routes; operation +//! arguments cannot override them. The original router and Tracker record regular Tool and native +//! read/change observations under the same connection generation. Checks provider capabilities +//! without creating new routers or attachments. Create uses Write capability and accepts only +//! relative paths below the bound directory. +//! +//! # Checked execution and observation +//! +//! File operations dispatch through checked FileTarget to `checkout_execute`; hash and validator +//! comparisons occur at the provider execution boundary. Partial Read also records the whole-file +//! hash and total line count; tracking updates only after successful mutation. Post-operation +//! validators are returned unchanged from provider responses. Create observes the parent directory, +//! while its created path and tracking target are the new file; these remain distinct. +//! +//! Glob/Grep share regular Tool defaults and formatting, but native operations use `checkout_search` +//! to preserve provider wrapper scope layers and output coordinates. Only the bound directory is +//! checked before/after search; this neither prevents concurrent descendant edits nor guarantees +//! an atomic snapshot of the whole tree. Paths come from typed provider results, not links +//! reconstructed from rendered Grep content. +//! +//! # live directory page +//! +//! List returns one bounded page and a typed continuation without prefetching entries. +//! Entries and cursors are limited to direct children of the bound directory; kind/path ordering +//! is checked. Special entries are path coordinates, not substitutes for checked file/directory +//! observations. Pagination is live, not a cross-page snapshot. The Host owns conversion to +//! Worldspace Entry/cursors; Tools retain checkout-root-relative provider coordinates. +//! +//! file_target classifies known checked-operation rejections into structured codes. Ambiguous +//! transport/IO failures and invalid post-mutation responses are not downgraded to stale/no-execution. +//! Regular Tool error mapping stays unchanged; diagnostic strings are not parsed to classify denial. use std::sync::Arc; use agen::tool::{ToolError, ToolExecutionContext, ToolOutput}; diff --git a/crates/tools/src/file_target.rs b/crates/tools/src/file_target.rs index 424b005e1..f090431ac 100644 --- a/crates/tools/src/file_target.rs +++ b/crates/tools/src/file_target.rs @@ -1,4 +1,11 @@ -//! Shared provider dispatch for Tool and checkout file processing. +//! File dispatch shared by regular Tools and native checkout. +//! +//! Targets with validators use provider checked execution; others use regular session operations. +//! Post-operation validators come from the same execution response, not a later stat. +//! Only known checked rejections map to `checkout_stale`, `checkout_denied`, `checkout_invalid`, +//! or `checkout_unavailable`. Ambiguous IO/transport failures and explicit OutcomeUnknown do not +//! imply nonexecution. Invalid read-only responses mean retrieval failure; invalid mutation +//! responses mean unknown outcomes. use agen::tool::ToolError; use workdir::{ CheckoutOperation, CheckoutOutput, CheckoutRequest, ContentHash, EditRequest, EditResult, diff --git a/crates/tools/src/lib.rs b/crates/tools/src/lib.rs index 8df330c17..3bdb410d5 100644 --- a/crates/tools/src/lib.rs +++ b/crates/tools/src/lib.rs @@ -1,13 +1,4 @@ -//! Built-in tools for the Yoi LLM agent. -//! -//! Read / Write / Edit / Glob / Grep / Bash operate through a host-owned -//! [`workdir::WorkdirSession`] handle. This crate owns tool schemas, rendering, and -//! read-before-edit tracking; it does not own Workdir identity/materialization or -//! WorkdirSession lifecycle. -//! -//! Bash is intentionally not sandboxed. The WorkdirSession supplies its initial cwd -//! and command capability, while the Runtime process and OS user remain the -//! trusted execution boundary. +#![doc = include_str!("../README.md")] pub mod checkout; pub mod error; diff --git a/crates/tools/src/search_target.rs b/crates/tools/src/search_target.rs index 59110e57d..4829a593a 100644 --- a/crates/tools/src/search_target.rs +++ b/crates/tools/src/search_target.rs @@ -1,6 +1,9 @@ -//! Provider dispatch shared by native and ordinary search processing. -//! Request construction/defaults and rendering remain in Glob/Grep; this only -//! selects the provider surface and verifies the returned readonly result kind. +//! Provider dispatch shared by regular search and native checkout search. +//! +//! Glob/Grep own request defaults, input handling, and formatting; this module checks provider +//! routing and read-only response kinds. Only native search uses `checkout_search`; provider +//! wrappers restrict its request scope and output coordinates. Tools neither create scopes +//! nor reinterpret rendered paths. use agen::tool::ToolError; use workdir::{ CheckoutSearchOperation, CheckoutSearchRequest, CheckoutSearchResult, GlobRequest, GlobResult, diff --git a/crates/tools/src/tracker.rs b/crates/tools/src/tracker.rs index 52cecabf9..ce41a67bf 100644 --- a/crates/tools/src/tracker.rs +++ b/crates/tools/src/tracker.rs @@ -1,46 +1,16 @@ -//! Worker-lifetime tracker for file operations performed by the builtin -//! file-manipulation tools. +//! Tracker retaining file Tool read hashes, mutation coordination, and recent operation paths. //! -//! A `Tracker` serves two orthogonal purposes: +//! Read and successful Write/Edit record content hashes and line counts. Before changing existing +//! files, checks against the last observation. This verifies mutation preconditions, not scope +//! authorization, and complements hash/validator checks at the provider execution boundary. +//! Regular Tools and native operations on the same attachment share observations. Aliases and +//! connection generations separate namespaces, preventing stale reads from applying to a different +//! session when an alias is reused after detach. //! -//! 1. **Read-before-edit policy.** It records a SHA-256 hash of each -//! file's contents at the moment it was observed via `Read` (or -//! mutated via `Write` / `Edit`), and lets `Write` / `Edit` later -//! verify that the file has not been externally modified since then. -//! -//! 2. **Recency of touched files.** It keeps an LRU-ordered list of -//! files that have been touched by any of the tools, so the Worker -//! layer can ask "which files did the agent recently look at?" — -//! used e.g. as a default reference set passed to context compaction. -//! -//! Despite its historic name, the Tracker already watches all three of -//! Read / Write / Edit; the rename away from `ReadTracker` reflects this. -//! -//! # Lifetime -//! -//! A `Tracker` is **Worker-process scoped**: the Worker layer creates a fresh -//! instance at the start of each Worker run (including resume) and discards -//! it when the process exits — it is not persisted, so a resumed -//! conversation starts with an empty read/edit history. The local WorkdirSession -//! scope boundary is likewise Worker-process scoped (derived from the -//! manifest). The two are orthogonal and the Worker wires them together -//! when registering builtin tools. -//! -//! ```no_run -//! # use std::path::PathBuf; -//! # use std::sync::Arc; -//! # use manifest::Scope; -//! # use tools::{Tracker, core_builtin_tools}; -//! # use workdir::{LocalWorkdirSession, WorkdirSessionHandle}; -//! let scope = Scope::writable("/workspace").unwrap(); -//! let session: WorkdirSessionHandle = Arc::new(LocalWorkdirSession::new( -//! scope, -//! PathBuf::from("/workspace"), -//! )); -//! let tracker = Tracker::new(); // session lifetime -//! let bash_outputs = PathBuf::from("/run/yoi/bash-output"); -//! let defs = core_builtin_tools(session, tracker, bash_outputs); -//! ``` +//! Per-path mutexes coordinate mutations within a Worker only; they do not prevent external process +//! edits. A bounded LRU of recently touched paths supplies reference candidates for compaction, etc. +//! Tracker is transient state of a running Worker, never persisted/restored. Past Reads in a restored +//! conversation do not establish mutation preconditions for a new Tracker. use std::collections::{HashMap, VecDeque}; use std::path::{Component, Path, PathBuf}; @@ -215,7 +185,7 @@ impl Tracker { /// subsequent edits see a clean history). /// /// Also bumps `path` to the front of the recency list. If the list - /// grows past [`RECENCY_CAPACITY`], the oldest entry is evicted. + /// grows past `RECENCY_CAPACITY`, the oldest entry is evicted. pub fn record(&self, path: &Path, bytes: &[u8]) { let key = self.key(canonicalize_or_owned(path)); let hash = hash_bytes(bytes); diff --git a/crates/tui/README.md b/crates/tui/README.md index b4e8e3b15..34d916092 100644 --- a/crates/tui/README.md +++ b/crates/tui/README.md @@ -1,61 +1,10 @@ # tui -## Role +`tui` is a terminal client for a single Worker's Console and the Workspace Dashboard. It handles input editing, display state, and connection-specific screen transitions. It does not own Session persistence or Worker lifecycle decisions. -`tui` implements terminal UI clients for the single-Worker Console and workspace Dashboard surfaces. +## Implementation entry points -## Boundaries - -Owns: - -- terminal rendering and input handling -- local composer state and UI affordances -- single-Worker Console attach/restore/chat screens -- workspace Dashboard presentation and role-action UI - -Does not own: - -- durable transcript authority (`session-store`) -- Worker current state (`session-store` worker metadata) -- Worker lifecycle policy (`worker`) -- product CLI ownership (`yoi`) - -## Design notes - -The TUI should display committed events and Worker snapshots rather than inventing durable state. Local input history and optimistic UI affordances are editing conveniences; they must not become hidden model context. - -## See also - -- [`../../docs/design/context-history.md`](../../docs/design/context-history.md) -- [`../../docs/design/worker-session-state.md`](../../docs/design/worker-session-state.md) - -## Feature invocation editing - -`/` discovers enabled Feature declarations with descriptions and usage. Choose a -name or alias with Tab/Enter, complete its parenthesized arguments, and press -Enter to confirm that selected occurrence as a typed chip. A later Enter sends -the message. Raw slash text and paste content are never dispatched as commands. -Use Alt+Enter beside a Feature chip to rediscover its declaration and reopen its -arguments; Backspace/Delete removes a chip atomically. Typed chips survive local -history, draft restoration, transport retry, and rewind. - -Every completion and chip-re-edit query carries a fresh `request_id`. Replies -must echo that ID and match the active query context and Worker target. The first -input/cursor edit cancels the pending input watch; returning to identical text -cannot revive it. Authority snapshots and Worker/view switches discard pending -queries and visible candidates, rather than advancing another counter. -Uncorrelated replies are ignored, even for File queries where the wire field -remains optional for legacy callers. - -The declared attachment adapter stages a client-local file in place, replacing -its staging chip with an `UploadedFile` reference when ready. Local file argument -completion runs on the client, not Worker. Alt+Enter beside a failed staging chip -retries with the same upload ID; deleting an unsent chip cancels only its owned -staging resource. Accepted, restored, or history attachments are never deleted by -composer cleanup. Staging blocks Send/Notify and history navigation until it is -ready, retried, or removed. There is no raw `/attach` or `/clear-attachments` -command lane. - -Attachment staging remains Backend-only: the existing Standalone transport has -no client-local file-upload staging API. The TUI reports that limitation and -preserves the draft invocation; it never falls back to raw path dispatch. +- [`launch`] / [`LaunchMode`] — typed launch requests and terminal restoration. +- `src/app.rs`: display state built from protocol snapshots and events. +- `src/input.rs` / `src/composer_history.rs`: typed input and local editing history. +- `src/console/`: connection, input-event, and attachment-staging management. diff --git a/crates/tui/src/app.rs b/crates/tui/src/app.rs index da103d04b..5698d7d67 100644 --- a/crates/tui/src/app.rs +++ b/crates/tui/src/app.rs @@ -1,3 +1,11 @@ +//! Builds terminal display state from Worker protocol snapshots and events. +//! +//! WorkerStateSnapshot updates current run state; connection state and streaming +//! display do not replace it. Input and run control still target the parent Worker +//! when a SubWorker view is selected. Completion replies apply only when request_id, +//! input watch, connection target, and displayed target match. Editing cancels the +//! watch, so returning to the same text does not revive stale replies. + use std::collections::{HashMap, VecDeque}; use std::path::Path; use std::time::{Duration, Instant}; diff --git a/crates/tui/src/composer_history.rs b/crates/tui/src/composer_history.rs index fac7ac37f..831832e10 100644 --- a/crates/tui/src/composer_history.rs +++ b/crates/tui/src/composer_history.rs @@ -1,3 +1,9 @@ +//! Stores per-Workspace local input history as typed Segments. +//! +//! This history supports editing reuse; it is not a Worker's durable Session or model +//! context. Feature invocations and UploadedFile references stay typed during re-editing +//! rather than reverting to strings. + use std::collections::VecDeque; use std::fs; use std::io; diff --git a/crates/tui/src/console/mod.rs b/crates/tui/src/console/mod.rs index 65d8ae931..d254c8230 100644 --- a/crates/tui/src/console/mod.rs +++ b/crates/tui/src/console/mod.rs @@ -1,3 +1,11 @@ +//! Manages connection-specific Console loops and client-local attachment staging. +//! +//! Staging uses Backend transport; completion replaces the corresponding input chip +//! with an UploadedFile reference. Connections own only unsent resources. Deleting +//! drafts does not remove accepted, restored, or historical attachments. Standalone +//! has no upload API, so it retains typed drafts and reports the unsupported operation +//! rather than falling back to sending raw paths. + use std::collections::HashMap; use std::fmt; use std::future::Future; diff --git a/crates/tui/src/input.rs b/crates/tui/src/input.rs index 51ffa7b43..487b6bdd1 100644 --- a/crates/tui/src/input.rs +++ b/crates/tui/src/input.rs @@ -1,15 +1,9 @@ -//! Multi-line input buffer with paste placeholders. +//! Multiline input buffer storing characters and typed references as an Atom sequence. //! -//! The buffer stores a sequence of [`Atom`]s — each either a single -//! character (including `\n`) or an atomic paste reference. The cursor -//! is an index in `0..=atoms.len()` marking the insertion point between -//! atoms. Paste atoms are indivisible: Backspace deletes the whole -//! placeholder, the cursor can't land "inside" one. -//! -//! Display form: paste atoms render as -//! `[Clipboard #N | X chars, Y lines]`. Submit form: paste atoms expand -//! back to their original captured content so the Worker sees the full -//! pasted text (without the placeholder label). +//! Paste, Feature invocation, and attachment chips are indivisible; the cursor sits +//! between Atoms. Display placeholders are separate from outgoing Segments; pasted +//! content expands to its original body on send. Raw slash text and paste are not +//! inferred to be Feature calls; only confirmed invocations are sent as typed input. use std::{ cell::RefCell, @@ -985,7 +979,7 @@ impl InputBuffer { } } - /// Delete one word backward — the same span [`move_word_left`] would + /// Delete one word backward — the same span [`Self::move_word_left`] would /// jump over. pub fn delete_word_before(&mut self) { let end = self.cursor; @@ -1021,7 +1015,7 @@ impl InputBuffer { } } - /// Move forward by one word. Mirror of [`move_word_left`]. + /// Move forward by one word. Mirror of [`Self::move_word_left`]. pub fn move_word_right(&mut self) { self.cancel_completion_on_edit(); while self.cursor < self.atoms.len() @@ -2050,7 +2044,7 @@ mod word_motion_tests { #[test] fn script_switch_is_a_word_boundary() { - // 漢字 | ひらがな | ASCII + // Han | Hiragana | ASCII let mut buf = buf_from("日本語のtest"); buf.cursor = 0; buf.move_word_right(); @@ -2070,7 +2064,7 @@ mod word_motion_tests { #[test] fn halfwidth_katakana_is_treated_as_katakana() { - // 半角カナ「ｱｲｳｴｵ」は5 atom、すべて Katakana 種別。 + // Halfwidth kana "ｱｲｳｴｵ" has 5 atoms, all of kind Katakana. let mut buf = buf_from("ｱｲｳｴｵfoo"); buf.cursor = 0; buf.move_word_right(); @@ -2078,7 +2072,7 @@ mod word_motion_tests { buf.move_word_right(); assert_eq!(cursor(&buf), 8); // after "foo" - // 全角と半角のカタカナは同じ Katakana 種別なので1単語につながる。 + // Fullwidth and halfwidth katakana share kind Katakana, forming one word. let mut buf2 = buf_from("カタｶﾅ"); buf2.cursor = 0; buf2.move_word_right(); @@ -2154,7 +2148,7 @@ mod word_motion_tests { #[test] fn delete_word_respects_script_boundary() { - // 「日本語の」末尾から1回削除すると、ひらがな部分「の」だけ消える + // One deletion from the end of "日本語の" removes only the hiragana "の". let mut buf = buf_from("日本語の"); buf.delete_word_before(); assert_eq!(as_text(&buf), "日本語"); diff --git a/crates/tui/src/lib.rs b/crates/tui/src/lib.rs index d72dffc1d..8810d2f71 100644 --- a/crates/tui/src/lib.rs +++ b/crates/tui/src/lib.rs @@ -1,3 +1,5 @@ +#![doc = include_str!("../README.md")] + mod app; mod backend_dashboard; mod backend_spawn; @@ -129,6 +131,10 @@ impl Drop for TerminalModeGuard { } } +/// Opens Console or Dashboard from a typed launch request. +/// +/// Restores raw mode and the alternate screen on both success and error. +/// Emits exit diagnostics after screen restoration, not inside the alternate screen. pub async fn launch(options: LaunchOptions) -> ExitCode { let LaunchOptions { target, diff --git a/crates/workdir/CHECKOUT_PROVIDER.md b/crates/workdir/CHECKOUT_PROVIDER.md deleted file mode 100644 index 119702002..000000000 --- a/crates/workdir/CHECKOUT_PROVIDER.md +++ /dev/null @@ -1,199 +0,0 @@ -# Request-time checkout provider boundary - -`workdir::checkout` (also crate-root reexports) exposes `CheckoutObservation`, -`CheckoutRequest`, `CheckoutOperation`, `CheckoutOutput`, and `CheckoutResult`. -`WorkdirSession::checkout_observe(path)` defaults to unsupported; execution also -fails closed by default. Observation is metadata-only, with no recursive tree -walk. Directory discovery/search remains the ordinary bounded List/Glob/Grep -provider engine. GrepResult.paths carries exact typed provider paths appearing -in the returned report, respecting its offset/limit/truncation; filenames are -never parsed from rendered text. List/Glob/Grep skip unrepresentable non-UTF8 -paths, preserving valid Unicode exactly without normalization. Native callers supply their own attachment generation/namespace -fence; this provider validator is not a substitute for that fence. - -* Read: line `offset`, `limit`, `max_bytes`; same `ReadResult` (bytes, start line, - total lines, full-source content hash, truncation) as normal Tools. Formatting - line numbers and recording the returned hash in caller read-history are caller - responsibilities. The validator is **not** permission to bypass read-before-write. -* Write: replace existing file with `content` and required `expected_hash`. -* Edit: existing UTF-8 file, unique `old_string` unless `replace_all`, required - `expected_hash`; same match/replacement/result-size checks as Tools. -* Create: target must be an observed directory; `path` is **checkout-root - relative**, strictly below target, not target-relative. No existing file can - be overwritten. Intermediate directories and destination must be writable. -* Every result observation describes the bound request target. Create returns - the updated **parent directory** observation from the retained pinned target, - not the created destination. Read returns the read state's observation. - -## Shared implementation - -`fs_operation::CheckedTarget` is an `FsAccessPolicy` adapter around pinned target, -save-parent, and provider-root descriptors. It calls the ordinary -`run_read_bounded`, `run_edit`, and `run_write` functions, including full-source -hashing, UTF-8/string matching, bounds, and atomic temp-save primitive; there is -no second content-operation engine. Shared text reads now check descriptor -metadata before/after reading. Validator bytes encode device/inode/size/mode/ -mtime/ctime with nanoseconds, excluding atime and host paths; same-content inode -replacement is distinguishable. Successful native Read rechecks the named target -and pinned descriptor before returning hash and observation. - -Local sessions serialize checked observe/execute and ordinary trait Read/Write/ -Edit using one nonqueueing session lock. Expected validator comparison happens -inside that boundary. The adapter retains handles across reading and save, -rechecks named target + save-parent identity at save boundary, and writes a new -temporary inode then atomically renames it, isolating existing hardlink aliases. -External sessions retain the approved descriptor root after pathname rename or -replacement. Checkout resolution rejects symlinks, special files, mount crossings, -absolute/traversal paths and unavailable descriptor primitives, rather than -weakening confinement. This currently requires Linux `openat2` and `/proc/self/fd`. -Ordinary Tools preserve their existing explicit symlink-policy behavior. - -Scoped sessions use their existing scope mutex, current validity/source -capability checks, provider authorization and write leases; Create also checks -intermediate paths. Observation capabilities are attenuated per path and live -write leases, never broadened to the whole attachment. ReadOnly removes mutation -capabilities and rejects mutations. Routed operations use existing active-op -accounting and detach fencing. Remote HTTP forwards exact operation/result types -and validates returned operation/path pairing. External DTOs validate checkout -path/validator/content/result bounds. No provider endpoint or host root is added -to the observation. - -## Effects and limits - -Absent paths during checkout observation return `NotFound` (including a missing -ancestor or a non-directory ancestor); native discovery can return no route. -An execution against a previously observed target that disappeared instead -returns `Conflict`, consistently at pinning, shared-engine, and final-check -boundaries. Conflict means stale validator/hash/name or existing Create destination before -save effects. `WorkdirError::OutcomeUnknown` has a separate sanitized wire code -`outcome_unknown` (HTTP 500), never an automatic retry. Partial parent creation, -possible commit failure, or failed post-save observation is outcome unknown. -Remote lost/malformed/mismatched mutation responses are also outcome unknown. -Create failures conservatively report unknown once parent creation may begin, -even if a particular failure happened without a persisted effect. - -Bounds: path <=4096 UTF-8 bytes/128 components, validator <=256 bytes; Read source -64 MiB, retained response <=1 MiB (or stricter provider read limits), Write/Edit -preimage and result <=8 MiB (or stricter provider limit), Edit replacements -<=10,000 (or stricter provider limit). One checked operation per Local session, -with bounded retained handles and no waiting queue. Cancellation/deadline checks -use the existing provider guard, or a 30-second cooperative guard for Local -checkout operations. Blocking kernel/filesystem calls are not preempted; this -is **not** a hard wall-time guarantee for hung storage. Transport request-body -bounds before serde allocation remain the hosting transport's responsibility. - -### External-editor limitations (not a magic advisory-lock guarantee) - -No advisory `flock` is claimed to exclude arbitrary editors. Provider-session -serialization does not lock other sessions/processes. Metadata checks detect -concurrent reads/edits up to each check, but are not a transactional snapshot -against privileged actors forging metadata. Linux offers no atomic -compare-identity-and-replace-by-name primitive here: an uncooperative external -rename/change between the final validator check and rename can still race the -save. The pinned handles prevent following replacements/symlinks into arbitrary -objects, and deterministic substitution before the save boundary is rejected, -but the final syscall gap is an explicit limitation. Callers must not interpret -this as linearizability against all external filesystem mutations. Likewise, -post-save observation is the then-current named state, not an exclusion of an -external edit after save. Crash durability of parent directories is not claimed. - -Host generation reuse, native permission identity, read-history updates, model -Operation publication, transport hosting (Runtime/Backend/External), and durable -Ticket/MR review remain the parent integration's responsibilities. - -## Bounded live List pagination - -`ListRequest { path, limit, after: Option }` accepts an exclusive -ordering key `ListCursor { kind: EntryKind, path: FsPath }`. Directories sort -first, then paths sort lexically within the directory/non-directory groups; -File, Symlink and Other share the non-directory group. `ListResult.next_after` -is the last returned key when another page remains. Reuse that key unchanged -with the same directory and output root. Cursor paths must name a direct child -in the result coordinate frame; they are not statted and need not still exist. -The optional fields default to absent and are omitted from JSON when absent. - -The shared provider engine retains at most `limit + 1` candidates, additionally -bounded by the 1 MiB aggregate path-response ceiling (including continuation), -while scanning the directory once. It never stops merely because the page is -full: enumeration order cannot determine the retained prefix. `total_entries` -and `total_bytes` describe all visible entries in the current scan, including -keys at/before `after`, not only the page or remaining suffix. A byte-bounded -page may contain fewer than `limit` entries and still provide continuation. -A zero-limit request returns totals and truncation but no continuation. -List uses Read capability and read/enumeration scope, and **does not ignore-filter** -`.gitignore`/`.ignore` entries or hide dotfiles; Glob/Grep keep their own policy. -Cancellation and the 100,000-entry traversal ceiling apply to the complete scan, -including excluded entries. External DTOs retain their existing item/path bounds -and also validate cursor paths and continuation response bytes. - -Each page observes a live directory, not a snapshot, and no cross-page -consistency is promised. Insertions after the cursor may appear; earlier keys -are not revisited. Removed entries disappear, including the cursor entry. -Renames or kind changes can move an entry across the cursor and cause omission -or repetition. Even one scan is not transactional against external editors. - -## Scoped checkout search boundary - -`CheckoutSearchRequest::new(CheckoutSearchOperation::{List, Glob, Grep}(request))` -uses the current session's logical root as its result frame. The typed result is -`CheckoutSearchResult::{List, Glob, Grep}`. The request's internal context fields -are `scope_layers: Vec>` and `output_root: WorkdirPath`. -Rules union within each layer; layers intersect with each other, the current -provider authority, and the output-root boundary. Wrappers contribute their own -scope; caller-supplied layers can only narrow, never replace it. Each wrapper -translates incoming provider coordinates once and forwards the composed output root. -List cursors remain output-root-relative, just like result paths. -Provider-rendered Grep text and every typed result path are already relative to -that final root; wrappers must not prefix cwd onto returned paths again. - -Local/External use the same shared `run_list`, `run_glob`, `run_grep` engines. -`SearchAccess` gates metadata/content opens and enumeration. Shared List checks -`can_enumerate_directory`; Glob/Grep use the shared bounded descriptor walk and -owned `traversal_filter` policy to prune before descending/opening directories. -The walk anchors at the actual search directory, not at a broader provider -root. It counts rejected entries too, checks cancellation while scanning them, -and limits descent to 128 levels (bounded retained directory handles). Existing -provider-owned ignore loading, glob/type selection, grep context/offset/limit, -rendering, and source/result limits remain in their shared engines. Unreadable -ignore files are not opened to discover out-of-scope contents. - -Descriptor search loads `.ignore`, `.gitignore` and an authorized directory's -`.git/info/exclude` through the same `FsAccessPolicy` opens as other sources. -It preserves anchored rules, negation, nearest-directory precedence within a -source class, `.ignore` precedence over Git rules, and nested repository -boundaries. Ignored directories are pruned before enumeration or loading their -own rules; explicitly selected ignored search roots are still traversable. -Only the current ancestor matchers are retained, never a second tree/store. -Glob requires an authorized `.git` directory/file marker for Git rules. Grep -retains the existing descriptor-provider contract of applying `.gitignore` -without a marker and rejecting unavailable/symlinked ignore opens; Glob skips -unavailable ignore sources like its ordinary walker. Scope-denied sources are -skipped by both before open. Bounds/cancellation failures are never retried. - -Configuration discovery cannot expand provider authority. Ambient ancestors -outside the search result/provider root, global Git configuration/ignore files, -and linked-worktree `.git`/`commondir` administration pointers are not followed. -A `.git` file activates repository matching without treating it as a directory -or opening the host path it contains. In-root readable ignore rules remain -active, including inherited rules above a nested base. Descriptor ignore text -must be UTF-8, bounded to 1 MiB/file, 16 KiB/line and 8 MiB aggregate per search; -limit violations fail rather than returning silently incomplete filtering. -Grep's separate candidate-content budget remains 64 MiB. Ordinary unscoped -path-backed Tools keep their original discovery policy. - -Scoped ordinary trait List/Glob/Grep use this boundary as well, returning -session-logical coordinates. Unscoped normal Tools continue using their original -trait/provider behavior. A provider without checkout search support returns -UnsupportedOperation; there is no broad traversal fallback. Native/scoped -searches intentionally fail closed on symlink paths, including logical-policy -aliases which normal unscoped Tools may follow. A visible symlink may therefore -cause a native List failure rather than expanding authority. Descriptor support -is currently Linux-only. Search remains request-time traversal, not a snapshot -or transactional exclusion of external editors. - -HTTP operation/result variant: `CheckoutSearch`. External DTO validation rejects -absolute/out-of-root paths, empty scope layers, oversized paths/patterns/context/ -results and invalid result paths. Generic request ceilings are 16 layers and 256 -aggregate rules; External transport further limits aggregate rules to 64. Search -result limits follow the existing shared List/Glob/Grep wire bounds. Runtime and -Backend forwarding outside this crate remain parent integration work. diff --git a/crates/workdir/README.md b/crates/workdir/README.md new file mode 100644 index 000000000..4793e832c --- /dev/null +++ b/crates/workdir/README.md @@ -0,0 +1,12 @@ +# workdir + +Provides contracts for persistent Workdir identity and Worker-bound operation sessions. Handles local, HTTP, and External providers through shared scope and capability boundaries. Providers own materialization and deletion; caller Tools do not take over those responsibilities. + +## Reference index + +- [`WorkdirSession`] — provider operation and close boundaries. +- [`WorkdirSessionRouter`] — aliases, connection generations, and detach coordination. +- [`WorkdirToolBroker`] — scope delegation and write leases for child sessions. +- [`checkout`] — identity-validated operations and search coordinates. +- [`external`] / [`http`] — typed transport, unknown outcomes, and denial reasons. +- [`workspace`] — inventory projection without host information. diff --git a/crates/workdir/src/checkout.rs b/crates/workdir/src/checkout.rs index b9458748c..f61c29210 100644 --- a/crates/workdir/src/checkout.rs +++ b/crates/workdir/src/checkout.rs @@ -1,6 +1,27 @@ -//! Request-time checkout contracts. Validators are opaque provider observations, -//! not content hashes. Callers must separately retain Read's content hash for -//! read-before-mutation. Create destinations are checkout-root relative. +//! Contracts for request-time checkout operations and provider observations. +//! +//! Observations fetch metadata only, not recursive trees or file caches. Validators are opaque +//! identity/protected-metadata values, separate from Read content hashes and caller attachment +//! generations. Write/Edit require both a validator and a previously read content hash; +//! hashes alone do not accept same-content inode replacements. Callers own Read line numbering +//! and per-attachment read history. +//! +//! Create paths are checkout-root-relative and must be strict descendants of the observed target +//! directory. Existing destinations are never overwritten; results return the updated observation +//! of the retained target directory, not the created file. Read results also return the validator +//! of the state actually read, never a replacement from a later independent stat. +//! +//! Missing targets at observation yield NotFound; disappearance of observed targets or validator/hash +//! mismatches yield Conflict. Once parent creation starts, uncertain commit or post-save observation +//! yields OutcomeUnknown, not a no-effect claim. Providers/transports do not automatically retry +//! uncertain mutations. +//! +//! Search delegates to shared List/Glob/Grep engines. Scope rules form unions within layers and +//! intersections across layers, provider authority, and output roots. Wrappers transform coordinates +//! once; returned paths and Grep content are relative to the final session's logical root. +//! Paths are not reverse-parsed from rendered filenames; non-UTF-8 names are not lossily aliased. +//! Unsupported providers reject search rather than fall back to broader trait searches. Search is +//! live traversal, not an atomic snapshot of all descendants or exclusion of external editors. use crate::{ ContentHash, EditResult, EntryKind, ReadResult, WorkdirError, WorkdirPath, WorkdirSessionCapabilities, WorkdirSessionCapability, WriteResult, diff --git a/crates/workdir/src/external.rs b/crates/workdir/src/external.rs index 76186cbb8..203701bc0 100644 --- a/crates/workdir/src/external.rs +++ b/crates/workdir/src/external.rs @@ -1,9 +1,15 @@ -//! Versioned websocket DTOs for a client-hosted External Workdir provider. +//! Bounded WebSocket DTOs for client-hosted External providers. //! -//! Authentication belongs to the websocket upgrade and Backend grant -//! authority. These frames intentionally contain only opaque identities, -//! logical Workdir operations, generation fences, and bounded control data. -//! They must never carry a provider host path, credential, or tunnel token. +//! Upgrade/Backend owns authentication and grant issuance. Frames contain only opaque identities, +//! logical operations, and generations, not host paths, credentials, or tunnel tokens. Reconnection +//! creates a new generation; old handles/results do not enter the new connection's command namespace. +//! Provider free-text errors are not forwarded; only closed codes/reasons are used. +//! +//! External grants follow COMMAND ⇒ WRITE ⇒ READ. WRITE includes READ for previously read hashes; +//! COMMAND remains separate explicit authority. General session capability intersections may form +//! subsets independently of this grant hierarchy; the codec never implicitly expands nonhierarchical +//! grants. Commands run with the shared host's OS permissions, not a sandbox confining processes +//! to a directory. Output is bounded and inline; provider-local spill paths are not exposed. use std::path::{Component, Path}; diff --git a/crates/workdir/src/http.rs b/crates/workdir/src/http.rs index 940506ca7..22f0f14b6 100644 --- a/crates/workdir/src/http.rs +++ b/crates/workdir/src/http.rs @@ -1,9 +1,12 @@ -//! HTTP transport contract and client for remote Workdir sessions. +//! HTTP contracts and an optional client for remote Workdir sessions. //! -//! The protocol keeps filesystem/search/process operations provider-side: one -//! [`WorkdirSessionOperation`] is one bounded HTTP request. The HTTP client is -//! optional so Runtime servers can share these DTOs without depending on a -//! client stack. +//! Keeps filesystem/search/process operations provider-side and pairs typed operations with results. +//! The optional client lets Runtime share DTOs without the client stack. Lost mutation responses, +//! decode failures, or pairing mismatches after dispatch yield OutcomeUnknown; no automatic resend. +//! +//! Denial reasons travel as an optional closed enum; provider free text maps to safe vocabulary. +//! Missing reasons stay unknown. Diagnostic wrappers preserve original classifications such as +//! Conflict/OutcomeUnknown and do not infer meaning from unknown reason strings. use schemars::JsonSchema; use serde::{Deserialize, Serialize}; diff --git a/crates/workdir/src/lib.rs b/crates/workdir/src/lib.rs index c77ee7d81..de9e7f8e5 100644 --- a/crates/workdir/src/lib.rs +++ b/crates/workdir/src/lib.rs @@ -1,9 +1,4 @@ -//! Persistent Workdir identity and Worker-bound operation sessions. -//! -//! A [`Workdir`] identifies a materialized repository execution context across -//! Worker lifetimes. A [`WorkdirSession`] is the live operation attachment -//! bound to one Worker. Tools consume sessions; they do not own Workdir -//! materialization or cleanup. +#![doc = include_str!("../README.md")] pub mod checkout; pub use checkout::{ @@ -187,12 +182,13 @@ impl WorkdirSessionCapabilities { pub type WriteOutcome = WriteResult; -/// Live, Worker-bound operations for one persistent [`Workdir`]. +/// Worker-bound operation session for a persistent [`Workdir`]. /// -/// Implementations execute filesystem search and command work on the host -/// that owns the materialization. Structured requests and results never -/// contain the raw materialized root. Closing a session is terminal and does -/// not delete the persistent Workdir or its materialization. +/// Filesystem search and commands run on the host owning materialization. +/// Session closure is terminal but does not delete the Workdir or files. +/// Unsupported checkout/search implementations explicitly reject requests, with no ambient +/// filesystem fallback. Providers own identity validators; callers own read hash history +/// and attachment generations. #[async_trait] pub trait WorkdirSession: std::fmt::Debug + Send + Sync { fn workdir(&self) -> &Workdir; @@ -294,8 +290,13 @@ pub trait WorkdirSession: std::fmt::Debug + Send + Sync { pub type WorkdirSessionHandle = Arc; -/// Closed diagnostic vocabulary: no request text, paths or secrets. -/// A reason describes a refusing branch, not an authorization grant. +/// Closed diagnostic vocabulary for denial reasons, excluding paths and free text. +/// +/// Assigned by the branch deciding denial, never inferred from error text. Missing reasons from +/// legacy providers remain unknown. Distinguishes PermissionDenied with an OS error code from +/// synthetic PermissionDenied of unknown origin, without asserting the specific OS-level cause. +/// Remote provider reasons are reported data, not authority or independent attestation. +/// `DenialContext` adds diagnostics without changing error classification or retry/cleanup rules. #[derive( Clone, Copy, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize, schemars::JsonSchema, )] diff --git a/crates/workdir/src/local.rs b/crates/workdir/src/local.rs index 7ff202283..f750bf0ee 100644 --- a/crates/workdir/src/local.rs +++ b/crates/workdir/src/local.rs @@ -1,12 +1,15 @@ -//! Scope-aware filesystem primitive. +//! Local sessions retaining scope and provider resources. //! -//! `LocalWorkdirSession` is the write/read gate layered on top of a [`manifest::Scope`] -//! and a Worker's working directory. The scope decides which paths are -//! readable and writable; the cwd is carried alongside for convenience -//! (Glob/Grep default their search base to it). +//! Scope determines read/write bounds; cwd is the reference for searches and operations. +//! Clones share resources, but Tools own per-attachment read history. Checked observe/execute/search +//! and regular trait Read/Write/Edit share a nonqueueing session lock, serializing validator +//! comparison through I/O. This does not exclude other sessions/processes or external editors. //! -//! `LocalWorkdirSession` is cheap to clone (`Arc` inside). Tool-specific session -//! state, such as read-before-edit tracking, remains owned by the tool layer. +//! Checkout uses Linux openat2 and /proc/self/fd, rejecting symlinks, special files, mount crossings, +//! absolute/traversal paths, and missing primitives. It does not implicitly expand regular Tools' +//! explicit symlink policies. Hashes bounded sources and uses shared fs-operation engines for +//! editing and temporary-file saves. Cooperative deadlines cannot forcibly interrupt kernel I/O, +//! so they provide no hard wall-time guarantee for hung storage. use std::collections::{BTreeMap, HashMap}; use std::fmt::Debug; @@ -670,12 +673,11 @@ impl Drop for LocalWorkdirSessionInner { pub trait WorkdirSessionResource: Debug + Send + Sync {} impl WorkdirSessionResource for T where T: Debug + Send + Sync {} -/// An atomically validated, descriptor-pinned External Workdir root. +/// External provider root with identity checked and retained before approval. /// -/// Pinning compares the selected path identity with the opened descriptor, so -/// a rename or replacement between approval and open fails closed. The -/// descriptor, rather than the pathname, remains the filesystem authority for -/// the lifetime of every session created from this root. +/// Compares selected-path metadata with the opened descriptor, rejecting renames/replacements +/// during acquisition. Sessions from this root use the descriptor as filesystem authority; +/// replacing the ambient pathname after approval does not transfer authority to another tree. #[derive(Debug, Clone)] pub struct ExternalWorkdirRoot { canonical_path: PathBuf, diff --git a/crates/workdir/src/router.rs b/crates/workdir/src/router.rs index 54e5c538b..9eb190121 100644 --- a/crates/workdir/src/router.rs +++ b/crates/workdir/src/router.rs @@ -1,9 +1,9 @@ -//! Alias-keyed Worker Workdir session routing. +//! Live session routing through Worker-local aliases. //! -//! The router owns live provider sessions without exposing whether a session is -//! local or transported over HTTP. Attachment aliases are Worker-local stable -//! identities; provider Workdir ids and human display names are deliberately not -//! routing keys. +//! Pins targets by alias and connection generation, not provider Workdir IDs or display names. +//! Reusing an alias on detach/reattach does not direct old observations, read history, or operations +//! to the new connection. Routed operations retain active-operation accounting coordinated with +//! begin/finish/cancel-detach. Detach ends a session connection; it does not delete Workdirs or files. use std::collections::{BTreeMap, HashSet}; use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering}; @@ -376,7 +376,11 @@ impl WorkdirSession for RoutedWorkdirSession { } } -/// Worker-owned map of stable attachment aliases to provider-neutral sessions. +/// Maps Worker-owned aliases to live provider sessions. +/// +/// Issues a generation per connection to distinguish alias reuse. Coordinates active operation +/// and command accounting with detach; old connections never retarget new sessions. +/// Detach does not delete the Workdir. #[derive(Debug, Default)] pub struct WorkdirSessionRouter { sessions: RwLock>>, diff --git a/crates/workdir/src/scope.rs b/crates/workdir/src/scope.rs index 8c59354ad..6fd71cfd3 100644 --- a/crates/workdir/src/scope.rs +++ b/crates/workdir/src/scope.rs @@ -1,3 +1,15 @@ +//! Scope brokers and write leases that attenuate parent session authority. +//! +//! Checks provider capabilities, attachment ceilings, parent scopes, and delegated scopes per operation. +//! Child write leases protect their ranges from conflicting parent/sibling writes; commands require +//! explicit grants and writable scope. Cwd sets coordinate transformations, not wider authority. +//! Rejects operations when the provider cannot compare scopes. +//! +//! Close revokes new operations before awaiting command cleanup; write leases remain held while +//! cleanup is uncertain. Checkout search composes rule layers and output roots, checking visibility +//! provider-side before name enumeration, descent, or content reads. Returned coordinates are not +//! prefixed with cwd again. + use std::collections::{BTreeMap, HashMap, HashSet}; use std::path::Path; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; @@ -64,6 +76,11 @@ pub struct WorkdirToolScope { pub command: bool, } +/// Broker mediating operations within parent provider and attachment ceilings. +/// +/// Delegation only attenuates scope; cwd and command grants do not expand file authority. +/// Child write leases suppress conflicting writes and remain held if command cleanup on close +/// is uncertain. #[derive(Clone)] pub struct WorkdirToolBroker { authority: Arc, diff --git a/crates/workdir/src/workspace.rs b/crates/workdir/src/workspace.rs index 8bb21fe06..33e54475e 100644 --- a/crates/workdir/src/workspace.rs +++ b/crates/workdir/src/workspace.rs @@ -1,8 +1,9 @@ -//! Shared Workdir inventory domain contracts. +//! Shared domain projection of Workdir inventory. //! -//! These types are a neutral lower-level authority shared by Server and Runtime -//! transport projections. They intentionally do not expose provider/session -//! handles, host paths, Runtime URLs, or credentials. +//! Types shared by Server and Runtime expose no provider/session handles, host paths, Runtime URLs, +//! or credentials. A Workdir's inventory presence is separate from content access through a Worker's +//! live attachment. Status and past materialization evidence alone do not grant current scope +//! or cleanup authority. use schemars::JsonSchema; use serde::{Deserialize, Serialize}; diff --git a/crates/worker-runtime/README.md b/crates/worker-runtime/README.md index e14d955fc..22b93b028 100644 --- a/crates/worker-runtime/README.md +++ b/crates/worker-runtime/README.md @@ -1,45 +1,10 @@ # worker-runtime -`worker-runtime` owns the Runtime authority surface for Worker management. A Runtime process bundles Worker lifecycle management, the HTTP/WebSocket control API, and the Worker execution backend. +Manages Worker creation, stopping, and Restore, execution connections within the Runtime, and Workdir materialization. The Backend owns Workspace records and grants; the Worker controller owns model execution and input acceptance. Optional features provide HTTP/WebSocket and filesystem persistence. -## Run the local Runtime server +## Reference index -From the repository root: - -```bash -cargo run -p worker-runtime \ - --bin yoi-runtime \ - -- --bind 127.0.0.1:38800 -``` - -By default the server listens on: - -```text -127.0.0.1:38800 -``` - -To bind another address explicitly: - -```bash -cargo run -p worker-runtime \ - --bin yoi-runtime \ - -- --bind 0.0.0.0:38800 -``` - -The REST server is intended for a trusted Backend/proxy, not direct browser access. -Workspace requests authenticate the actual HTTP request with the Workspace's current -signing identity. The Runtime verifies signatures against its explicitly enrolled -Workspace public key and binds the Runtime audience, Workspace/Worker scope, -operation, method, path/query, body digest, expiry, and single-use token ID. A -connection test is an ordinary signed `/v1/ping`; no prior handshake or saved -verification evidence is required or consulted. - -Keep `trust-workspace add`, `replace`, and `revoke` as explicit operator actions. -A self-reported public key cannot grant trust. Trust record IDs are administrative -only and must not be copied into Backend registration or request claims. Replay -protection uses Workspace plus the trusted key fingerprint, so re-enrolling the -same key or rotating away and back cannot reset consumed requests. Reaccepting a -key does permit fresh, unconsumed requests signed by that key until their short -expiry; it does not create a separate sender-controlled enrollment namespace. - -For authenticated remote Runtime setup, see [`../../docs/development/server-runtime-auth.md`](../../docs/development/server-runtime-auth.md). +- [`Runtime`](crate::Runtime): lifecycle operation acceptance, Workspace boundaries, and execution connections. +- [`execution`] — execution backends and operation-specific results. +- [`working_directory`] — repository materialization, session resources, and safe deletion. +- [`workspace_issuer`] — Workspace signature verification and replay prevention. diff --git a/crates/worker-runtime/src/execution.rs b/crates/worker-runtime/src/execution.rs index 42b82e216..7c4682dbc 100644 --- a/crates/worker-runtime/src/execution.rs +++ b/crates/worker-runtime/src/execution.rs @@ -1,3 +1,11 @@ +//! Operation and result boundaries between Runtime and execution backends. +//! +//! Lifecycle operation IDs identify durable reconciliation; they are not interchangeable with +//! controller command IDs or Submit/Notify request IDs. Catalog Idle/Stopped status alone does +//! not determine input acceptance. Submit/Notify success rests on their durable receipts, +//! not transport enqueueing or connection closure. Lost responses do not imply no effects; +//! operation-specific unknown/settlement rules still apply. + use crate::catalog::{ ConfigBundleRef, ProfileSelector, RepositoryRefObservation, RepositoryRefObservationRequest, WorkingDirectoryRepositoryAccessRequest, WorkingDirectoryRequest, WorkingDirectoryStatus, @@ -224,6 +232,13 @@ enum WorkerExecutionPublicationState { Discarded, } +/// Publication boundary for protocol observations bound to Worker identity. +/// +/// Restore candidates buffer events until the execution commit to the catalog succeeds. +/// Events during activation join the same queue, so publication preserves event order. +/// Discarding a candidate on rollback also suppresses its later events. This isolation prevents +/// a newly started controller from exposing a live candidate inconsistent with the settled +/// catalog; it does not prove execution cleanup. #[derive(Clone)] pub struct WorkerExecutionContext { worker_ref: WorkerRef, diff --git a/crates/worker-runtime/src/fs_store.rs b/crates/worker-runtime/src/fs_store.rs index dba335e13..5bc656a77 100644 --- a/crates/worker-runtime/src/fs_store.rs +++ b/crates/worker-runtime/src/fs_store.rs @@ -1,3 +1,21 @@ +//! Filesystem persistence for the Runtime catalog and Worker aggregates. +//! +//! Atomically publishes settled Worker status, execution binding, and pending lifecycle operations +//! as one Worker record. Session content and metadata projections use separate stores; +//! sockets and run paths do not substitute for durable identity. +//! +//! Legacy conversion copies the Runtime tree into staging under the OS owner lock, validates +//! Worker record/metadata/Manifest identity consistency, then syncs and switches trees. +//! Partially converted trees are not opened as normal authority. Duplicate identities, existing +//! targets, missing Workspace bindings, and metadata identity mismatches are rejected, not guessed away. +//! After interruption, staging/backup and validation determine which tree to publish; +//! writes do not continue into an incomplete tree. +//! +//! Directories without Worker records are archived, not adopted into the normal catalog. +//! Archive target collisions are rejected; data of unknown ownership is never overwritten or deleted. +//! Symlinks/special files are rejected except recognizable stopped sockets. Preserving historical +//! data and deciding whether execution hints can migrate are separate judgments. + use crate::catalog::{ ConfigBundleRef, CreateWorkerRequest, LogicalWorkdirAttachment, ProfileSelector, WorkerRestoreIntent, WorkerStatus, WorkingDirectoryAttachmentClaim, diff --git a/crates/worker-runtime/src/lib.rs b/crates/worker-runtime/src/lib.rs index 202cde59e..2c3abf7e3 100644 --- a/crates/worker-runtime/src/lib.rs +++ b/crates/worker-runtime/src/lib.rs @@ -1,10 +1,4 @@ -//! Embedded Runtime domain API for Worker management. -//! -//! `worker-runtime` owns the Runtime authority surface and, for the standalone -//! process, wires that Runtime to the real Worker host. Filesystem persistence is -//! available only through the optional `fs-store` feature, and the REST/WebSocket -//! server is available only through the optional `http-server` / `ws-server` -//! features. +#![doc = include_str!("../README.md")] pub mod auth; pub mod catalog; diff --git a/crates/worker-runtime/src/runtime.rs b/crates/worker-runtime/src/runtime.rs index fd23e5f8c..6f97ce2e5 100644 --- a/crates/worker-runtime/src/runtime.rs +++ b/crates/worker-runtime/src/runtime.rs @@ -300,13 +300,25 @@ impl Drop for RuntimeEventSelectorSubscription { } } -/// Concrete embedded Runtime domain entity. +/// Domain entity managing Worker lifecycles and Runtime-local resources. /// -/// The default implementation is memory-backed and tools/provider-less by -/// design. An optional `fs-store` feature adds filesystem persistence while -/// preserving the same typed authority boundary. It can later be adapted by -/// backend registries or web servers without making sockets, sessions, or paths -/// public authority. +/// Memory-backed by default, with persistence provided by `fs-store`. Workspace registry +/// projections do not authorize controller input admission. Scoped operations validate current +/// Workspace membership and delegate to execution backends/controllers. Per-Worker operation +/// locks serialize lifecycle changes. +/// +/// Workdir cleanup reserves target IDs under the state mutex, then inspects and deletes files +/// outside it. Conflicting use/cleanup of the same target is rejected without blocking unrelated +/// Worker observations on long I/O. Transient reservations replace neither durable deletion +/// evidence nor Backend attachment authority. +/// +/// Restore/Stop separate settled status from pending operations, persisting intent before backend +/// side effects. Pending operations reject conflicting input, lifecycle, deletion, and retention +/// even when status appears Idle/Stopped. Startup reconciles pending Stop first and does not +/// auto-restore Workers whose stop is confirmed and committed. Restore reuses the same operation +/// ID; candidates with uncertain commit/cleanup are not replaced by new executions. Successful +/// commit publishes backend observation registration and buffered protocol events; rollback +/// discards the candidate. #[derive(Clone, Debug)] pub struct Runtime { inner: Arc>, @@ -1858,6 +1870,18 @@ impl Runtime { .into_legacy_result() } + /// Admits Restore with a fixed observation token and request ID. + /// + /// Revalidates the token under the Worker operation lock and saves the request owner and + /// `pending_restore` operation ID in the same Worker record before preparation/execution. + /// The same request reuses its stored payload; replaying a terminal receipt never restarts + /// a Worker stopped later. New intent with an old token conflicts. This token is an admission + /// boundary distinct from controller command IDs and cleanup-plan digests. + /// + /// If a saved pending operation has no public owner, binds recovery to that operation without + /// substituting another request or preparing a new candidate. Uncertain commits/responses + /// retain ReconciliationRequired, never implying no effect or rollback. Accepted denotes + /// Restore commit, not success of a subsequent Run. pub fn restore_worker_operation_scoped( &self, scope: &RuntimeWorkspaceScope, @@ -3533,6 +3557,12 @@ impl Runtime { } #[cfg(feature = "ws-server")] + /// Checks the authorized Worker against the captured execution endpoint before operating. + /// + /// Validates scope, pending lifecycle settlement, and endpoint freshness before durable mutation. + /// Never forwards send/Shutdown to the next execution of the same WorkerRef, even if this + /// connection has not observed the old execution ending. Continued registry observation + /// and the operational connection have separate lifetimes. pub fn send_connected_protocol_method_scoped( &self, scope: &RuntimeWorkspaceScope, diff --git a/crates/worker-runtime/src/worker_backend.rs b/crates/worker-runtime/src/worker_backend.rs index 83dd75626..2f384dd36 100644 --- a/crates/worker-runtime/src/worker_backend.rs +++ b/crates/worker-runtime/src/worker_backend.rs @@ -1,11 +1,19 @@ -//! Adapter from `worker-runtime` execution backend boundary to the real -//! `worker` crate controller/run lifecycle. +//! Adapter connecting Runtime execution contracts to the existing Worker controller. //! -//! The adapter intentionally owns real `WorkerHandle`s internally and exposes -//! operations addressed by `WorkerRef` to callers. Browser/API -//! projections therefore keep the existing runtime redaction boundary: no raw -//! socket paths, session paths, manifests, credentials, or handles leave this -//! module. +//! Separates durable Worker/Session identity from a single controller launch. Fresh runs use unique +//! IDs; Restore runs use lifecycle operation IDs. Reuses registered controllers for the same Restore +//! operation. Even after restart, exclusive creation of operation-bound run directories prevents +//! overwriting existing runs. Run artifacts or sockets alone prove neither controller liveness nor cleanup. +//! +//! Opens Session and metadata stores within the aggregate through trusted WorkerRef, with no +//! process-global root fallback. Restore preserves identity while rebuilding Workspace connections +//! and Workdir bindings. Candidate controller observation registration waits for activation after +//! catalog commit. Cleanup checks the actual controller and session resources, deleting only +//! operation-owned disposable run artifacts. It implies neither Session/metadata deletion nor +//! retention of all past runs. +//! +//! WorkerHandle is owned internally; external operations take WorkerRef. Socket/session paths, +//! Manifests, credentials, and handles are not passed to browser/API projections. use std::collections::{BTreeMap, HashMap}; use std::future::Future; diff --git a/crates/worker-runtime/src/working_directory.rs b/crates/worker-runtime/src/working_directory.rs index 956d1c181..749d32903 100644 --- a/crates/worker-runtime/src/working_directory.rs +++ b/crates/worker-runtime/src/working_directory.rs @@ -1,3 +1,20 @@ +//! Ownership of Runtime-local repository materialization and session resources. +//! +//! Independently of Backend inventory, Runtime checks duplicate primary Workdir ownership and +//! current bindings. Usability depends on materialization identity, current state, and occupancy, +//! not creation provenance alone. Per-target provider leases exclude competing creation, access, +//! and cleanup; the admission bookkeeping mutex is not held during filesystem work. +//! Fresh roots are created exclusively; failures do not delete another creator's tree. +//! +//! Binding clones are passive; only activating session resources inhibits cleanup. +//! Resource clones held by sessions and commands maintain that protection. Deletion invalidates +//! the materialization generation, so recreating the same ID cannot activate old bindings. +//! +//! Deletion is a provider operation checking current identity, mounts, and remaining content. +//! Partial-deletion witnesses are stored outside the tree. CleanupPending/unknown inventory status +//! does not guarantee current cleanliness. Detailed inspection and remaining-content checks +//! are limited to cleanup retries. + use crate::catalog::{ MaterializerKind, RepositoryRefObservation, RepositoryRefObservationRequest, RepositorySshMaterializationAccess, WorkingDirectoryCleanupTarget, diff --git a/crates/worker-runtime/src/working_directory/cleanup.rs b/crates/worker-runtime/src/working_directory/cleanup.rs index e13906edf..345c219ee 100644 --- a/crates/worker-runtime/src/working_directory/cleanup.rs +++ b/crates/worker-runtime/src/working_directory/cleanup.rs @@ -1,8 +1,14 @@ -//! Ordinary removal authority, not a recovery lifecycle. The content witness is -//! outside the tree being removed so a crash/partial unlink cannot erase it. -//! Every retry checks current identities, mounts and remaining content. Missing -//! entries are expected after partial deletion. The same intact clean checkout -//! may renew its witness after owner recovery, never discarding new ignored content. +//! Mount-safe deletion and content checks after partial deletion. +//! +//! Content witnesses are stored outside the deletion tree, surviving partial unlinking. +//! Witnesses describe past facts; every retry revalidates current identity, mounts, and remaining +//! content. Missing entries count as partial deletion, but new or changed content is not discarded. +//! Only the same intact clean checkout can refresh an old witness. List/detail check saved identity +//! only; they infer neither remaining file hashes nor current cleanliness. +//! +//! Rejects deletion if Linux openat2 or /proc mount checks are unavailable; no unchecked recursive +//! deletion fallback. Does not implicitly unmount resources or delete other owners' data. +//! OS errors stay in Runtime-private logs; public diagnostics contain only closed causes and safe messages. use super::*; use std::io; diff --git a/crates/worker-runtime/src/workspace_issuer.rs b/crates/worker-runtime/src/workspace_issuer.rs index 1ae60192f..0756a495a 100644 --- a/crates/worker-runtime/src/workspace_issuer.rs +++ b/crates/worker-runtime/src/workspace_issuer.rs @@ -1,3 +1,14 @@ +//! Runtime request authentication through explicitly registered Workspace issuers. +//! +//! Matches actual HTTP method, path/query, body digest, operation, Runtime audience, Workspace/Worker +//! scope, expiry, and single-use token ID against signed claims. Ping is an ordinary signed request; +//! prior handshakes or saved connection tests do not authorize it. No fallback to self-reported +//! public keys, Server-global trust, or static bearers. +//! +//! Trust IDs are administrative, not namespaces for sender claims. Replay tracking uses Workspace +//! and trusted key fingerprints, so re-registering or rotating the same key cannot revive consumed +//! requests. This does not guarantee invalidation of unconsumed, unexpired requests from a reaccepted key. + use std::collections::HashMap; use std::fmt; use std::fs; diff --git a/crates/worker-runtime/src/workspace_request.rs b/crates/worker-runtime/src/workspace_request.rs index 1eaa95490..13beb9814 100644 --- a/crates/worker-runtime/src/workspace_request.rs +++ b/crates/worker-runtime/src/workspace_request.rs @@ -1,3 +1,10 @@ +//! Runtime-originated HTTP requests bound to Workspace boundaries. +//! +//! Signs the actual request method/path/body with Runtime identity, using the Backend URL from +//! an explicit Workspace issuer bundle as audience. Connection hints alone do not create +//! Account/Worker authority. Reads bounded responses with an async client; no blocking HTTP +//! clients on async Worker/runtime paths. + use std::error::Error as _; use std::io::Read; use std::time::{Duration, SystemTime, UNIX_EPOCH}; diff --git a/crates/worker/README.md b/crates/worker/README.md index 46f8fa2a3..5e55bb88e 100644 --- a/crates/worker/README.md +++ b/crates/worker/README.md @@ -1,107 +1,11 @@ # worker -## Role +`worker` builds an `agen` Engine from a resolved Manifest and connects Worker input handling, Feature registration, Session persistence, and protocol handling. It uses scoped capabilities supplied by the Host rather than owning Workspace business records or managing physical Workdirs. -`worker` turns an `agen` Engine into a named runtime entity with manifest configuration, scoped tools, session persistence, protocol handling, and Worker metadata integration. +## Main entry points -## Boundaries - -Owns: - -- Worker lifecycle and socket protocol serving -- Engine construction around a resolved Manifest -- session-store and session-store worker metadata coordination -- built-in tool registration under scope/policy -- spawned-child orchestration hooks - -Does not own: - -- provider-specific wire formats (`provider` / `agen` clients) -- product CLI parsing (`yoi`) -- TUI display authority (`tui`) -- current-state storage schema outside Worker metadata (`session-store` worker metadata) - -## Design notes - -A Worker is runtime authority, not UI state. It should commit model-visible events through history/session paths and keep current Worker-name state in Worker metadata rather than in transient runtime files. - -## Workspace config WIP Feature - -Enable the opt-in surface in a Worker profile/manifest: - -```toml -[feature.workspace_config] -enabled = true -``` - -This setting accepts **only** `enabled` and defaults to false. Controller -registration additionally requires Workspace identity, an available injected -Workspace Client and WIP mode. It creates no grant. An owner must explicitly -create a Backend grant bound to Workspace/Runtime/Worker and read-only or -read-write access; standalone/Tools-mode config compatibility tools do not exist. - -`feature::builtin::workspace_config` registers `/workspace-config` through the -common structured invocation registry (T696). Its optional `access` argument is -`effective` (default), `read_only`, or `read_write`; explicit write never silently -downgrades. Feature preparation attaches before the request reaches the assistant -and reports the logical target and effective access. Native self-attach uses the -same `WorkspaceConfigFeature::attach` path. Backend alias attachment is idempotent -under the session lock; neither entry point blindly retries uncertain outcomes. - -The canonical alias is `workspace-config`; the content entrance is WIP -`/workspace-config`. Attachment identity and conditional detach use the existing -T697 Backend ledger. No OS mount, Workdir session, Bash capability, local config -copy or second attachment inventory is created. The registry source is logical, -not a Runtime filesystem target. - -The request-time provider maps `/workspace-config` to the Backend's empty root -path and descendants to canonical relative source paths, for example -`/workspace-config/main.dcdl` -> `main.dcdl`. Observations return structure, -Backend validators and current operations, **not bodies**. Files publish `read`; -read-write grants additionally publish `write`, unique-substring `edit`, and -`delete`. Directories and explicit absence observations publish `create` where -Backend metadata permits it. The root publishes `apply_changes` for atomic -create/update/delete/rename sets, allowing new imported -sources and their references to be saved together. Empty directory creation, -commands and generic POSIX operations are not exposed. The adapter uses the typed -`workers/self/workspace-config` GET/attach/observe/read/commit API; it never calls -an owner config API as a bypass. - -Client-managed PATH validators are compared against freshly resolved Host -metadata. Each mutation captures the ROOT validator, whole-tree content digest, -canonical entrypoints and node state from that same observation. Backend resolves -the live grant/link and validates canonical changes/CAS under its execution lock. -No operation accepts LLM-supplied validators or `expected_digest` fields. Returned -post-call path validators are refreshed from the exact committed content, not -substituted with root validators or later unrelated state. Content changes and -reattachment to a new connection lifetime reject stale calls, even when the new -attachment exposes the same bytes. Old descriptors cannot -bypass read-only restrictions, detach, revocation or Workspace boundaries. - -Bounds: depth 0..8, 256 observed nodes/changes, 512-byte relative paths, 256 KiB -aggregate changed text/read content, and 30-second retrieval/operation deadlines. -Path traversal and malformed parameters fail before content calls. Typed -`NotCommitted` failures remain distinct from `Unknown`; timeouts/lost mutation -responses are unknown and never automatically retried. Raw transport/provider -errors are not copied into model/UI output. Retrieval protocol failures are -reported as failures after recording their Client cache state, not as fresh -successful observations. Saving commits the canonical Workspace configuration -and existing projection/activation pipeline; it does **not** rewrite the resolved -manifests of already-running Workers. - -Public integration seams: `WorkspaceConfigFeature::for_workspace`, -`workspace_config::wip::mount_workspace_config_wip`, and -`WipRuntime::from_mounts` plus `tree`/`inspect`/`invoke`. Worker tests exercise the -real Feature registry, native mount and WIP Client/Host through a typed in-process -contract router: deep discovery, slash/native attach, read/edit/write, -create/delete, atomic multi-source updates, grant restrictions/revocation, -detach, cross-Workspace isolation, concurrent/same-bytes stale observations, -invalid/oversize/save/unknown failures and no retries. Those contract-router tests -are not a substitute for Backend canonical persistence, semantic validation, -projection activation or full Web-to-Backend acceptance tests. - -## See also - -- [`../../docs/design/worker-session-state.md`](../../docs/design/worker-session-state.md) -- [`../../docs/design/context-history.md`](../../docs/design/context-history.md) -- [`../../docs/design/tool-permissions-scope.md`](../../docs/design/tool-permissions-scope.md) +- [`Worker`](src/worker.rs): Input admission, execution, and Session restoration. +- [`controller`](src/controller.rs): Worker control and protocol intake. +- [`feature`](src/feature.rs): Feature declarations, registration, and structured invocation. +- [`wip`](src/wip.rs): WIP namespaces, native/compatibility projections, and the Client/Host boundary. +- [`spawn`](src/spawn/mod.rs): Delegation and control of parent-owned Internal SubWorkers. diff --git a/crates/worker/src/checkout.rs b/crates/worker/src/checkout.rs index ce2896062..752c28688 100644 --- a/crates/worker/src/checkout.rs +++ b/crates/worker/src/checkout.rs @@ -1,5 +1,22 @@ -//! Provider-owned request-time checkout projection. No filesystem path is opened -//! by the Worker: every observation and operation goes through the live router. +//! Provider-owned, request-time checkout projection entered through current attachments. +//! +//! Worker delegates observation and execution to the live router and WorkdirSession without opening +//! host pathnames. This surface is separate from Repository/Workdir inventory; it neither copies +//! file trees nor maintains resident registrations. Only `/checkouts` and attachment entrances are +//! indexable; descendants are merely addressable. Deeper observation does not List content, and +//! empty children mean "no indexable children", not "empty directory". List/Glob/Grep discover +//! content; Inspect checks selected coordinates. +//! +//! Interfaces/validators bind the projection incarnation, attachment generation, and provider state +//! so alias reuse after restore or detach cannot transfer old observations to a new connection. +//! Publication also checks the generation recorded at acquisition. Possessing an Interface grants +//! no authority: route-reconstructed permission inputs and current provider capability/scope are +//! rechecked, as are the existing Write/Edit prior-Read and content-hash preconditions. +//! +//! Search-result entry coordinates come from typed provider paths without reobserving every result, +//! so later disappearance does not undo successful discovery. Losing the connection after a +//! successful Write/Edit/Create, or exceeding a mutation deadline, can leave the outcome unknown. +//! Interrupting a future does not roll back side effects. use std::collections::BTreeMap; use std::sync::Arc; diff --git a/crates/worker/src/compact/prune.rs b/crates/worker/src/compact/prune.rs index 6be34fa10..171edebda 100644 --- a/crates/worker/src/compact/prune.rs +++ b/crates/worker/src/compact/prune.rs @@ -1,15 +1,13 @@ //! Prune integration — wires the Engine's prune projection to the Worker's //! usage-history-backed token accounting. //! -//! Engine 自身がコンテキスト射影を行う（`worker.rs` の `request_context` 構築 -//! 直後）。Engine は usage 履歴を知らないので、`min_savings` 判定に使う savings -//! の見積もりはコールバックで外部から注入する。このモジュールはそのコールバック -//! を組み立てて Engine に差し込むための `impl Worker` を提供する。 +//! The Engine projects context itself, immediately after constructing `request_context` in +//! `worker.rs`. Since it has no usage history, a callback supplies the savings estimate used +//! to evaluate `min_savings`. This module provides `impl Worker` to assemble and install it. //! -//! 同じ経路で `PruneObserver` も install し、評価のたびに `prune.fire` / -//! `prune.skip` metric を `MetricsTracker` に積む。`Fired` 時は uuid を -//! `UsageTracker` にも stash しておき、後続の `LlmUsage` と組で -//! `prune.post_request` を吐けるようにする。 +//! The same path installs a `PruneObserver` that adds `prune.fire` / `prune.skip` metrics to +//! `MetricsTracker` on each evaluation. On `Fired`, it also stashes a uuid in `UsageTracker` +//! to pair with the subsequent `LlmUsage` and emit `prune.post_request`. use agen::Item; use agen::llm_client::client::LlmClient; @@ -42,9 +40,9 @@ impl Worker { /// prevents the prune projection from firing until usage data exists. /// /// Also installs a [`PruneObserver`] that pushes `prune.fire` / - /// `prune.skip` metrics into the shared [`MetricsTracker`]. On `Fired` + /// `prune.skip` metrics into the shared `MetricsTracker`. On `Fired` /// the observer additionally stashes a fresh correlation_id in - /// [`UsageTracker`] so the next `LlmUsage` can be paired with a + /// `UsageTracker` so the next `LlmUsage` can be paired with a /// `prune.post_request` metric carrying the same id. pub fn attach_prune(&mut self, config: PruneConfig) { let usage_history_for_tokens = self.usage_history_handle(); diff --git a/crates/worker/src/compact/token_counter.rs b/crates/worker/src/compact/token_counter.rs index 5dc7c0e46..60fd8fea7 100644 --- a/crates/worker/src/compact/token_counter.rs +++ b/crates/worker/src/compact/token_counter.rs @@ -1,21 +1,19 @@ -//! Compact / prune 専用のトークン会計補助。 +//! Token-accounting helpers specific to compact / prune. //! -//! 汎用部分（`prefix_bytes`, `tokens_at`, `total_tokens`, `total_tokens_at`）は -//! [`agen::token_counter`] にあり、`UsageRecord` の列と現在の history から -//! pure に推定する。本モジュールは compact / prune 固有のロジック -//! （`split_for_retained`, `savings_for_prune`）と、Worker 上の公開 API に -//! 限定する。 +//! Generic functions (`prefix_bytes`, `tokens_at`, `total_tokens`, `total_tokens_at`) live in +//! [`agen::token_counter`] and compute pure estimates from `UsageRecord` sequences and current +//! history. This module is limited to compact / prune logic (`split_for_retained`, +//! `savings_for_prune`) and public Worker APIs. //! -//! # 方針 +//! # Policy //! -//! - ローカルトークナイザは持たない。実測値があればそれを採用し、 -//! measurement 間はバイト数で按分、最新 measurement より先は byte/4 で外挿する -//! - Compact の retained split では、request-time pruning / projection 後の -//! `UsageRecord` を persisted history prefix の単調系列として扱わない。 -//! 現在の prompt occupancy 推定を raw serialized bytes に配分し、末尾の -//! persisted tail サイズで cut を決める。 -//! - 推定の出どころは [`EstimateSource`] で呼び出し側に明示する。 -//! 課金判断には使えないが、compact / prune の閾値判定には十分な精度 +//! - No local tokenizer. Use measurements when available, interpolate between them by byte count, +//! and extrapolate beyond the latest measurement at byte/4. +//! - Compact's retained split does not treat request-time pruned/projected `UsageRecord`s as a +//! monotonic series of persisted history prefixes. It distributes current prompt-occupancy +//! estimates over raw serialized bytes and chooses the cut by persisted tail size. +//! - [`EstimateSource`] exposes estimate provenance to callers. Accuracy is sufficient for +//! compact / prune thresholds, not billing decisions. use agen::llm_client::client::LlmClient; use agen::token_counter::{item_bytes, prefix_bytes, tokens_at}; @@ -26,9 +24,9 @@ pub use agen::token_counter::{EstimateSource, TokenEstimate}; use crate::Worker; -/// history を分割する位置。 +/// Position at which to split history. /// -/// `items[..index]` が捨てる／要約される側、`items[index..]` が残る側。 +/// `items[..index]` is discarded/summarized; `items[index..]` is retained. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub struct SplitPoint { pub index: usize, @@ -101,25 +99,19 @@ fn ceil_div_u128(n: u128, d: u128) -> u128 { if n == 0 { 0 } else { ((n - 1) / d) + 1 } } -/// `history[cut..]` が `ToolCall` / `ToolResult` のペア境界を尊重するよう -/// `cut` を後退させる。 +/// Move `cut` backward so `history[cut..]` respects `ToolCall` / `ToolResult` pair boundaries. /// -/// LLM API は「`ToolResult` を送るならその `ToolCall` も同じ request に -/// 含まれていなければならない」というバリデーションを持つ。トークン数 -/// だけで切った `cut` は並列 tool 呼び出しの途中に落ちうるので、retained -/// 側の先頭に対応 `ToolCall` を持たない `ToolResult`（orphan）が残ると -/// 次セッション初回 request が API バリデーションで弾かれる。 +/// LLM APIs require each `ToolResult`'s `ToolCall` in the same request. Token-only cuts can fall +/// inside parallel tool calls, leaving an orphan `ToolResult` at the retained head and causing +/// API validation to reject the next session's first request. /// -/// 対策は「retained に入る `ToolResult` について、対応 `ToolCall` も -/// retained に含まれる位置まで `cut` を引き下げる」こと。retained_tokens -/// 予算は超えうるが、ここでは直接 LLM に投げる訳ではなく次の -/// `pre_llm_request` で再評価されるだけなので safe。 +/// Lower `cut` until every retained `ToolResult`'s matching `ToolCall` is also retained. This may +/// exceed the retained_tokens budget, but is safe: nothing is sent directly to the LLM here; +/// the next `pre_llm_request` reevaluates it. /// -/// アルゴリズム: history を末尾から走査し、retained 範囲内の `ToolResult` -/// に出会うたびに対応 `ToolCall` の位置で `cut` を min 更新する。`cut` が -/// 下がると以前は要約側だった位置が retained に入るので、後続走査で連鎖的 -/// に正しい位置まで引き下がる。`ToolCall` の `call_id` はユニークなので -/// 事前にマップ化して O(n) で済ます。 +/// Scan history backward, lowering `cut` to each retained `ToolResult`'s matching `ToolCall` position. +/// Lowering the cut retains positions previously on the summary side; the remaining scan cascades +/// backward to the correct boundary. Unique `ToolCall` `call_id`s allow a precomputed map for O(n). fn balance_to_pair_boundary(history: &[Item], cut: usize) -> usize { let mut idx = cut.min(history.len()); if idx == 0 { @@ -151,9 +143,8 @@ fn balance_to_pair_boundary(history: &[Item], cut: usize) -> usize { idx } -/// 1 つの ToolResult 項目について、`content` を `None` に射影したとき -/// 減少するシリアライズ後バイト数。ToolResult 以外や既に content=None -/// の item は 0 を返す。 +/// Serialized bytes saved by projecting one ToolResult's `content` to `None`. +/// Returns 0 for non-ToolResult items or items already having content=None. fn tool_result_content_bytes(item: &Item) -> u64 { if !matches!( item, @@ -186,13 +177,12 @@ pub(crate) fn token_estimates_for_prune_impl( .collect() } -/// Prune 射影（`ToolResult.content = None`）で節約されるトークン数の推定。 +/// Estimate token savings from prune projection (`ToolResult.content = None`). /// -/// `indices` は [`agen::prune::prunable_indices`] が返す候補列を -/// 想定する。各候補の content バイト差分を合算し、usage 履歴由来の -/// tokens/byte レートでトークン数に換算する。範囲を「丸ごと drop」する -/// のではなく、item 自体（summary 等）は残したままの値を返す点が -/// `tokens_at` ベースの計算と異なる。 +/// `indices` is expected to contain candidates from [`agen::prune::prunable_indices`]. Sum their +/// content-byte differences and convert using the usage-history tokens/byte rate. Unlike +/// `tokens_at`-based calculations, this retains the items themselves (summaries, etc.) rather +/// than dropping the entire range. pub(crate) fn savings_for_prune_impl( history: &[Item], records: &[UsageRecord], @@ -218,9 +208,9 @@ pub(crate) fn savings_for_prune_impl( }; } - // 最新の measurement を使って tokens/byte を求め、バイト差分を換算する。 - // 実測値そのものではなく比率しか使わないので、history_len と - // record.history_len が一致しなくても rate は正しい。 + // Convert byte differences using the latest measurement's tokens/byte rate. + // Only the ratio, not the measurement itself, is used, so the rate remains + // valid even when history_len differs from record.history_len. let prefix = prefix_bytes(history); let last = records.last().expect("records non-empty"); let ref_bytes = prefix[last.history_len.min(history.len())]; @@ -240,31 +230,30 @@ pub(crate) fn savings_for_prune_impl( TokenEstimate { tokens, source } } -// ── Worker に生やす公開 API ─────────────────────────────────────────────── +// ── Public Worker APIs ─────────────────────────────────────────────── impl Worker { - /// 現在の history 全体の推定トークン数。 + /// Estimated tokens for the entire current history. /// - /// 最後の measurement と、その後に追加された未測定分の byte/4 外挿。 + /// Uses the last measurement plus byte/4 extrapolation for subsequent unmeasured additions. pub fn total_tokens(&self) -> TokenEstimate { let usage = self.usage_history(); agen::token_counter::total_tokens(&self.history(), &usage) } - /// 任意の history index 時点でのプロンプト全長推定。 + /// Estimate full prompt length at any history index. /// - /// `total_tokens()` と同じ accounting を任意位置で評価する版。 - /// memory extract trigger が - /// `total_tokens_at(now) - total_tokens_at(pointer)` で - /// pointer 以降に増えたプロンプト長を測るのに使う。 + /// Evaluates the same accounting as `total_tokens()` at an arbitrary position. + /// Memory extraction triggers use `total_tokens_at(now) - total_tokens_at(pointer)` + /// to measure prompt growth since the pointer. pub fn total_tokens_at(&self, history_len: usize) -> TokenEstimate { let usage = self.usage_history(); agen::token_counter::total_tokens_at(&self.history(), &usage, history_len) } - /// 末尾から `retained` トークン以上を残すための分割位置。 + /// Split position retaining at least `retained` tokens from the tail. /// - /// `history[..cut.index]` が要約／破棄される側、`history[cut.index..]` が残る側。 + /// `history[..cut.index]` is summarized/discarded; `history[cut.index..]` is retained. pub fn split_for_retained(&self, retained: u64) -> SplitPoint { let usage = self.usage_history(); split_for_retained_impl(&self.history(), &usage, retained) @@ -496,10 +485,10 @@ mod tests { #[test] fn balance_retreats_from_inside_parallel_tool_results() { // [Msg, TC_a, TC_b, TC_c, TR_a, TR_b, TR_c] - // cut=5 → retained=[TR_b, TR_c]。TR_c の TC は idx=3、TR_b は idx=2 → - // idx=2 まで後退。だが retained に TR_a (idx=4) が新たに入り、その TC_a - // は idx=1 でまだ外 → 連鎖後退で最終的に idx=1。retained は - // [TC_a, TC_b, TC_c, TR_a, TR_b, TR_c]。 + // cut=5 -> retained=[TR_b, TR_c]. TR_c's TC is at idx=3, TR_b's at idx=2, + // so retreat to idx=2. This now retains TR_a (idx=4), whose TC_a at idx=1 + // is still outside. Cascading retreat ends at idx=1, retaining + // [TC_a, TC_b, TC_c, TR_a, TR_b, TR_c]. let history = vec![ msg("u"), tc("a"), @@ -514,47 +503,47 @@ mod tests { #[test] fn balance_retreats_between_call_and_result() { - // [TC_a, TR_a, TC_b, TR_b]。cut=3 → retained=[TR_b] orphan。 - // TC_b は idx=2 → cut=2。retained=[TC_b, TR_b]。 + // [TC_a, TR_a, TC_b, TR_b]. cut=3 -> retained=[TR_b], an orphan. + // TC_b is at idx=2 -> cut=2, retained=[TC_b, TR_b]. let history = vec![tc("a"), tr("a"), tc("b"), tr("b")]; assert_eq!(balance_to_pair_boundary(&history, 3), 2); } #[test] fn balance_cascades_through_nested_pairs() { - // [TC_a, TC_b, TR_b, TR_a, TC_c, TR_c]。cut=3 → retained=[TR_a, TC_c, TR_c]。 - // TR_a の TC は idx=0 → cut=0。retained=full。 + // [TC_a, TC_b, TR_b, TR_a, TC_c, TR_c]. cut=3 -> retained=[TR_a, TC_c, TR_c]. + // TR_a's TC is at idx=0 -> cut=0, retaining everything. let history = vec![tc("a"), tc("b"), tr("b"), tr("a"), tc("c"), tr("c")]; assert_eq!(balance_to_pair_boundary(&history, 3), 0); } #[test] fn balance_noop_when_cut_at_pair_boundary() { - // [TC_a, TR_a, Msg, TC_b, TR_b]。cut=2 → retained=[Msg, TC_b, TR_b] balanced。 + // [TC_a, TR_a, Msg, TC_b, TR_b]. cut=2 -> retained=[Msg, TC_b, TR_b], balanced. let history = vec![tc("a"), tr("a"), msg("u"), tc("b"), tr("b")]; assert_eq!(balance_to_pair_boundary(&history, 2), 2); } #[test] fn balance_handles_orphan_result_without_matching_call() { - // ToolCall がそもそも存在しない ToolResult は触らない（壊れた history は - // ここでは直しようがない）。cut=1 → そのまま 1 を返す。 + // Leave ToolResults with no matching ToolCall untouched; broken history cannot + // be repaired here. cut=1 -> return 1 unchanged. let history = vec![msg("u"), tr("zombie")]; assert_eq!(balance_to_pair_boundary(&history, 1), 1); } #[test] fn balance_keeps_cut_when_call_is_inside_retained() { - // [Msg, TC_a, TR_a]。cut=1 → retained=[TC_a, TR_a]。TR_a の call_pos=1 >= idx=1。OK。 + // [Msg, TC_a, TR_a]. cut=1 -> retained=[TC_a, TR_a]. TR_a's call_pos=1 >= idx=1: OK. let history = vec![msg("u"), tc("a"), tr("a")]; assert_eq!(balance_to_pair_boundary(&history, 1), 1); } #[test] fn split_for_retained_aligns_to_pair_boundary() { - // 並列 TC*3 / TR*3 ターン後に Msg を 1 件足し、retained=Msg のサイズ相当に - // 設定。トークン的には cut=末尾近くだが、orphan を避けるため TC 群の手前 - // まで後退するはず。 + // Add one Msg after a parallel TC*3 / TR*3 turn and set retained to its size. + // The token-only cut is near the tail, but should retreat before the TC group + // if needed to avoid orphans. let history = vec![ msg("user"), tc("a"), @@ -567,16 +556,15 @@ mod tests { ]; let total_bytes: u64 = history.iter().map(item_bytes).sum(); let records = vec![record(history.len(), total_bytes)]; // rate = 1 tok/byte - // tail の item_bytes 相当のみ retain したい。 + // Retain only the tail's item_bytes equivalent. let tail_tokens = item_bytes(&history[7]); let cut = split_for_retained_impl(&history, &records, tail_tokens); - // token 単独だと cut は 7（tail のみ retained）になるが、retained 先頭が - // Msg なら balance しなくて OK。balance helper の no-op を確認する意味も込めて - // index == 7 を期待する。 + // Tokens alone give cut=7 (only the tail retained). A retained head of Msg needs + // no balancing. Expect index == 7, also checking the balance helper's no-op. assert_eq!(cut.index, 7); - // 逆に retained をやや増やしてトークン的に cut=6（TR_c のみ retained）に - // させると、TR_c は orphan なので balance が 1 まで後退するはず。 + // Slightly increasing retained gives token-only cut=6 (TR_c alone from the tool turn). + // TR_c is an orphan, so balancing should retreat to 1. let big_retain = tail_tokens + item_bytes(&history[6]); let cut = split_for_retained_impl(&history, &records, big_retain); assert_eq!(cut.index, 1); diff --git a/crates/worker/src/compact/usage_tracker.rs b/crates/worker/src/compact/usage_tracker.rs index b0d068f5c..937b42f5d 100644 --- a/crates/worker/src/compact/usage_tracker.rs +++ b/crates/worker/src/compact/usage_tracker.rs @@ -151,9 +151,8 @@ impl UsageTracker { pending.usage_recorded = true; let request_id = pending.id; let history_len = pending.history_len; - // UsageEvent.input_tokens は scheme 層で「占有量（プロンプト全長）」に - // 正規化済みである前提（Anthropic は cache_read + cache_creation を - // 加算して emit する）。 + // Assumes the scheme layer has normalized UsageEvent.input_tokens to occupancy + // (full prompt length). Anthropic adds cache_read + cache_creation before emitting. let input_total = event.input_tokens.unwrap_or(0); let cache_read = event.cache_read_input_tokens.unwrap_or(0); let cache_write = event.cache_creation_input_tokens.unwrap_or(0); diff --git a/crates/worker/src/compact/worker.rs b/crates/worker/src/compact/worker.rs index 2e450a2fd..96dbc2ffd 100644 --- a/crates/worker/src/compact/worker.rs +++ b/crates/worker/src/compact/worker.rs @@ -1,21 +1,14 @@ -//! Compact worker state and the four tools that drive it. +//! Summarization and reference-file selection by the compaction Internal Worker. //! -//! The compact worker is a disposable `Engine` instance spun up by -//! [`Worker::compact`]. It receives the history to summarise plus a list of -//! default reference files (from the session-lifetime `Tracker`) and runs -//! a tool-driven LLM loop. The tools here let it: +//! Summary input is a bounded history projection; it does not rewrite the original history. +//! Results collect in [`CompactWorkerContext`]; [`crate::Worker::compact`] validates, saves, +//! and activates them. Missing summaries or hard-limit violations fail compaction, unlike +//! targets or warning-only thresholds. //! -//! - `read_file` — inspect referenced files (reuses `tools::read_tool`) -//! - `mark_read_required(path, offset?, limit?)` — nominate a file whose -//! contents should be injected into the compacted context as an -//! auto-read system message -//! - `add_reference(path)` — nominate a file the next session should -//! know about by name only (contents not included) -//! - `write_summary(text)` — deliver (or overwrite) the structured summary -//! -//! Everything the worker decides ends up in [`CompactWorkerContext`], -//! which `Worker::compact` drains after the loop and turns into the -//! compacted session's opening system messages. +//! Auto-read accepts only UTF-8 text through the current WorkdirSession, checking scope and +//! the aggregate budget. Images, binary data, and truncated content stay out of the body; +//! paths can remain as references for the next request. Naming a reference grants no read +//! authority; incorporating it into the compacted result still follows actual read conditions. use std::path::PathBuf; use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; diff --git a/crates/worker/src/controller.rs b/crates/worker/src/controller.rs index 801af0c74..816810571 100644 --- a/crates/worker/src/controller.rs +++ b/crates/worker/src/controller.rs @@ -3790,8 +3790,8 @@ where St: Store, { let manifest = worker.manifest(); - // `build_client` がここに到達する前に同じマニフェストで成功している - // ため、カタログ解決も必ず通る。念のため失敗時は "unknown" に落とす。 + // `build_client` already succeeded with this manifest, so catalog resolution + // must also succeed. Fall back to "unknown" on failure as a precaution. let resolved = manifest::model_catalog::resolve_model_manifest(&manifest.model).ok(); let context_window = resolved .as_ref() diff --git a/crates/worker/src/feature/builtin/drive.rs b/crates/worker/src/feature/builtin/drive.rs index d1f2e9477..36e7307fb 100644 --- a/crates/worker/src/feature/builtin/drive.rs +++ b/crates/worker/src/feature/builtin/drive.rs @@ -1,5 +1,31 @@ -//! Worker Drive operations use only the injected Workspace API. Observations -//! below are bounded preconditions, never a hierarchy replica or a grant cache. +//! Drive operations through the Workspace API and bounded mutation-precondition observations. +//! +//! The Backend owns Drive hierarchy, blobs, and grants. Worker neither reads the Server's physical +//! blob root nor treats Drive as a Workdir or Session attachment store. Entries are identified by +//! Workspace and canonical decimal-string node ID; large IDs are never converted to JSON numbers. +//! +//! # Mutation-precondition observations +//! +//! Regular Tool read/metadata keeps a bounded record of entries and their last commit request IDs. +//! List/search is discovery, not a substitute for mutation-precondition observation. Explicit +//! observation is required after cache eviction; successful mutations remove old observations. +//! This is neither an authority cache nor a persistent hierarchy copy. Edit validates the full +//! preimage and previously observed commit ID, never editing truncated text. String replacement +//! shares `fs_operation::text` rules; Backend CAS decides the final winner. +//! +//! # Workdir sources and images +//! +//! Saving from a Workdir reads bytes through the selected alias's scoped session and verifies each +//! chunk's path, offset, size, and hash plus the overall SHA256. Source reads and Drive destination +//! writes are independent capabilities; completing blob transfer alone does not prove publication. +//! Images are assembled from chunks bound to the entry's commit request; bytes from mixed +//! generations are never attached. Image type is detected from bytes, not URLs or filenames. +//! `ImageAttachment` returns through existing ToolOutput. Later Drive updates/deletions do not +//! change observations already recorded in the Session; Engine/Session owns retention, pruning, +//! and replay. +//! +//! [`wip`] owns native publication, [`backend`] request receipt checks, and delegation Drive blocking +//! for internal children. Shared WIP references, permissions, and unknown outcomes follow [`crate::wip`]. pub mod backend; pub(crate) mod delegation; #[cfg(test)] diff --git a/crates/worker/src/feature/builtin/drive/backend.rs b/crates/worker/src/feature/builtin/drive/backend.rs index eea66bc12..c4ec3e986 100644 --- a/crates/worker/src/feature/builtin/drive/backend.rs +++ b/crates/worker/src/feature/builtin/drive/backend.rs @@ -1,5 +1,14 @@ -//! Identity-bound Drive API adapter. The Backend owns grants, hierarchy, CAS and -//! receipts. In particular, a lost mutation response is queried, never resent. +//! Identity-bound Drive API adapter and mutation receipt checks. +//! +//! The Backend owns grants, hierarchy, CAS, and receipts, applying current authorization to every +//! read/chunk/status request. The adapter validates Workspace, node ID, commit ID, and size, and +//! never copies raw diagnostics that may contain provider paths, SQL, or credentials into output. +//! +//! Mutations use Host-generated request IDs and observed node/commit IDs. Conflicts do not trigger +//! retries against substituted current state. For an unknown mutation response, check the same +//! request's receipt once; only a committed receipt with valid IDs and metadata proves DB success. +//! An uncommitted snapshot cannot rule out a later commit by an in-flight request, so all other +//! cases retain an unknown outcome with the original request ID. Never blindly resend with a new ID. use std::{sync::Arc, time::Duration}; use crate::worker::{ diff --git a/crates/worker/src/feature/builtin/drive/delegation.rs b/crates/worker/src/feature/builtin/drive/delegation.rs index 2c32aa625..0162833c2 100644 --- a/crates/worker/src/feature/builtin/drive/delegation.rs +++ b/crates/worker/src/feature/builtin/drive/delegation.rs @@ -1,10 +1,11 @@ -//! Fail-closed Drive boundary for parent-owned Internal SubWorkers, including Reviewers. +//! Boundary that blocks Drive capabilities for parent-owned Internal SubWorkers. //! -//! Internal children have no Backend Drive identity. Explicit internal Drive delegation is not -//! supported until the Backend can approve scope bound to that child identity. A parent's current -//! grant, enabled profile, or Reviewer attestation is not a child Drive grant. Registered Runtime -//! Workers must use their own identity-bound, current Backend grants; this wrapper creates neither -//! identities nor grants and keeps no local authorization cache. +//! Internal children have no independent Backend Drive identity/grant. Parent grants, effective +//! Profiles, and Reviewer attestations therefore grant no child authority. Spawn disables Drive +//! after resolving every Profile selector; this wrapper also rejects JSON/binary Drive operations +//! through the shared Workspace client. Non-Drive operations and Reviewer attestations remain; +//! no identity, grant, or authorization cache is created. Future internal delegation will require +//! scope no broader than the parent's, bound to a child identity, and Backend reauthorization. use std::{sync::Arc, time::Duration}; diff --git a/crates/worker/src/feature/builtin/drive/wip.rs b/crates/worker/src/feature/builtin/drive/wip.rs index 086f95e79..fd11101ab 100644 --- a/crates/worker/src/feature/builtin/drive/wip.rs +++ b/crates/worker/src/feature/builtin/drive/wip.rs @@ -1,5 +1,16 @@ -//! Native non-indexable Drive entries. Publication resolves just one ID through -//! current Backend authorization; the Client owns entry/committed-request validators. +//! Native Drive projection with no indexable entry children. +//! +//! `/drive` is a discovery entrance; Tree does not enumerate file inventory. Only entries selected +//! from bounded list/search results resolve directly. Paths contain a hex-encoded Workspace ID and +//! decimal node ID, preventing substitution with another Workspace or ID. Previously acquired +//! Client entries are not added to Tree either. +//! +//! The provider resolves one entry under current Backend authorization and creates a self-scoped +//! contract for its kind. Metadata, descriptor, validator, and handler are fixed to one publication; +//! the route binds the entry/creation parent. Move destinations accept typed Entry paths. +//! Interfaces describe operations for the entry kind; the Backend rejects writes under read-only +//! grants. Native Invoke still applies original Tool permissions. [`crate::wip`] owns the shared +//! cache, validator, and cancellation contracts. use super::{ DriveFeature, SPECS, Spec, backend::{DriveError, decimal}, diff --git a/crates/worker/src/feature/builtin/flow_transition.rs b/crates/worker/src/feature/builtin/flow_transition.rs index 2a716d3a0..457062217 100644 --- a/crates/worker/src/feature/builtin/flow_transition.rs +++ b/crates/worker/src/feature/builtin/flow_transition.rs @@ -1,3 +1,17 @@ +//! Feature connecting Worker-owned Flow state to a read-only internal verifier. +//! +//! The model submits only a reason, choosing neither instance nor target state. The coordinator +//! clones the current Worker's local state and commits attempts/events to the Session before +//! replacing in-memory state. It does not use the Workspace client for transition mutations and +//! first recovers restored verifying attempts. The `flow` crate owns graph and verdict-set validation. +//! +//! The verifier receives a fixed capture of the committed parent Session, current state, reason, +//! and all outgoing conditions. Even with a Workdir, it gets only Read/Glob/Grep through a read-only +//! wrapper, not Workspace, Ticket, Memory, Worker management, write, or Bash capabilities. +//! Only complete typed `FinishFlowVerification` results are accepted; prose completion is not +//! success. State-entry instructions return as regular Tool results; this Feature owns neither +//! role-specific business loops nor side effects. + use std::collections::BTreeSet; use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, Mutex}; diff --git a/crates/worker/src/feature/builtin/memory.rs b/crates/worker/src/feature/builtin/memory.rs index febbe047b..c4973cea5 100644 --- a/crates/worker/src/feature/builtin/memory.rs +++ b/crates/worker/src/feature/builtin/memory.rs @@ -1,8 +1,14 @@ -//! Workspace-HTTP backed Memory tools. +//! Memory tools through Workspace HTTP and a Host-provided resident prompt source. //! -//! Runtime workers may have Workspace authority without direct local filesystem -//! authority. In that case model-visible Memory tools must go through the -//! workspace backend instead of resolving `.yoi/memory` from a Worker workdir. +//! Workspace authority does not imply filesystem authority, so model-visible Memory operations +//! delegate to typed Backend operations. They do not discover/import `.yoi/memory` from cwd or +//! ancestors. The legacy Workspace single document and subject-scoped Subjektiv retain separate +//! authorities/operations. Enabling the Feature neither migrates/dual-writes legacy data nor +//! replaces existing tool semantics with subject Memory. +//! +//! Resident context is supporting context, not current authority or grounds to update confirmed +//! Memory. User-managed behavior and generated Memory surfaces remain distinct; fetch/render +//! failures do not mean "unset". use crate::subjektiv::{SubjektivHostConnection, SubjektivHostError}; use std::sync::{Arc, Mutex}; @@ -729,6 +735,11 @@ impl WorkspaceSubjektivResidentSummarySource { } } +/// Detect behavior changes against the state reflected in history, not the fetched state. +/// +/// Initial load builds a prompt contribution; change refresh does not advance applied on fetch/render alone. +/// The interceptor confirms after durably appending a system item, allowing refetch if append fails. +/// Invalidate applied before rewrite so even identical bodies can be presented in the new history. #[async_trait] impl SystemPromptContributionSource for WorkspaceSubjektivResidentSummarySource { async fn load(&self) -> SystemPromptContribution { @@ -796,9 +807,14 @@ const SUBJEKTIV_RESTORE_REFRESH_HOOK: &str = "refresh-resident-surface-after-res const SUBJEKTIV_BEHAVIOR_REFRESH_HOOK: &str = "refresh-subject-behavior-before-request"; const SUBJEKTIV_BEHAVIOR_REWRITE_HOOK: &str = "invalidate-subject-behavior-before-rewrite"; -/// subjektiv-owned restore behavior. Initial prompt contribution remains a -/// shared host facility, while loading and rendering a post-restore refresh is -/// installed only with an execution-enabled, subject-attached subjektiv Feature. +/// Subjektiv-owned restore-boundary refresh. +/// +/// Initial prompt contributions are a shared Host facility; only an execution-enabled, subject-connected +/// Feature requests restore refresh. Source presence alone must not make generic Worker core infer it. +/// Saved prompt/history is unchanged; one system item representing current resident context is appended. +/// A generated surface's ready-empty state differs from ungenerated, stale, or failed; those states +/// never mean empty Memory. Pre-request hooks handle later behavior updates; the restore boundary +/// itself is not repeated. #[derive(Clone)] pub(crate) struct SubjektivResidentRestoreRefreshFeature { source: Arc, diff --git a/crates/worker/src/feature/builtin/memory_lifecycle.rs b/crates/worker/src/feature/builtin/memory_lifecycle.rs index 8ef2e9e26..68307a8a5 100644 --- a/crates/worker/src/feature/builtin/memory_lifecycle.rs +++ b/crates/worker/src/feature/builtin/memory_lifecycle.rs @@ -1,3 +1,17 @@ +//! Memory extraction lifecycle using committed runs and a restricted Internal Worker. +//! +//! Worker core owns only generic hook/task/session plumbing; this Feature owns extraction targets, +//! audit, and pointers. Workspace Memory and Subjektiv have separate pointer domains and are never +//! simultaneous automatic extraction targets. Extraction uses the parent's committed capture and +//! resolved evidence, not uncommitted child Worker output. +//! +//! Pointers advance only on success with explicit FinishMemoryExtraction, not mere Engine completion. +//! Generation fences and cancellation are checked; extensions append only when the capture's Segment +//! matches the current one, preventing pre-rewrite extraction results from marking ranges processed +//! in a new Segment. Candidate staging is an independent durable write before Finish, so later +//! failure, cancellation, or fence loss does not roll it back. Retries without pointer advancement +//! can produce semantic duplicates; consolidation owns candidate comparison. + use std::sync::Arc; use std::time::Duration; diff --git a/crates/worker/src/feature/builtin/memory_surface_lifecycle.rs b/crates/worker/src/feature/builtin/memory_surface_lifecycle.rs index 375caa9c3..324c6beeb 100644 --- a/crates/worker/src/feature/builtin/memory_surface_lifecycle.rs +++ b/crates/worker/src/feature/builtin/memory_surface_lifecycle.rs @@ -1,3 +1,15 @@ +//! Coordinates editing and publication of resident context generated from confirmed Memory. +//! +//! Budget checks cover the whole request, including the system prompt, JSON input representation, +//! and tool schemas, not just source material. Each LLM request repeats the check to account for +//! history growth in correction turns. This estimate excludes provider-specific HTTP representations; +//! it is neither a billing-token measurement nor a strict guarantee of model limits. +//! +//! Edit counts, execution time, and source refetches after publication conflicts are bounded. +//! Cancellation and background generations are rechecked before publication; the Host decides final +//! source freshness and persistence. Active Memory whose material exceeds the budget must not yield +//! successful empty generation. Generation failure does not undo confirmed Memory or applied candidates. + use std::pin::Pin; use std::sync::Arc; use std::time::Duration; diff --git a/crates/worker/src/feature/builtin/merge_request.rs b/crates/worker/src/feature/builtin/merge_request.rs index 156d86c16..a41b56852 100644 --- a/crates/worker/src/feature/builtin/merge_request.rs +++ b/crates/worker/src/feature/builtin/merge_request.rs @@ -1,3 +1,26 @@ +//! Connects enabled Merge Request Tools to the Workspace API and native WIP. +//! +//! The native projection accepts only `/merge-requests` and its immediate canonical IDs, binding +//! each item's subject to its route. It removes `merge_request_id` from descriptors and reconstructs +//! it for original schema and permission evaluation. Only enabled regular Tools are projected; +//! no Backend-wide catalog or custom query is added. [`crate::wip`] owns shared reference, +//! permission, Client-lifetime, and unknown-outcome contracts. +//! +//! # Injected review capability +//! +//! Review routes must exactly match the MR in the Host-injected Reviewer context. Capability tokens +//! never appear in descriptors or model arguments. Candidate, Ticket snapshot, approval, repository, +//! and integration evidence retain regular Tool and Backend checks. This adapter neither performs +//! Git/provider integration nor owns Ticket conclusion decisions. +//! +//! # Bounded responses and subject validation +//! +//! Read returns the existing bounded thread page, keeping linked Tickets as references. Successful +//! item-read response hashes are observed; mutations discard observations. Receipts do not fabricate +//! complete representations or local mutation generations. Open results include the new item path. +//! Item responses are checked against the bound MR; subject mismatches after mutations are not +//! treated as unexecuted operations. Optional argument omission/null stays as in original Tool input. + use crate::feature::{ FeatureDescriptor, FeatureInstallContext, FeatureInstallError, FeatureInstructionContribution, FeatureInstructionDeclaration, FeatureInstructionId, FeatureModule, ToolContribution, diff --git a/crates/worker/src/feature/builtin/objective.rs b/crates/worker/src/feature/builtin/objective.rs index d1340558f..58dd5c4f1 100644 --- a/crates/worker/src/feature/builtin/objective.rs +++ b/crates/worker/src/feature/builtin/objective.rs @@ -1,9 +1,21 @@ +//! Objective Tools and native WIP projection through the scoped Workspace API. //! -//! Objective tool registration backed by Workspace API authority. +//! Worker cannot assume Backend-local storage paths, so the Workspace API owns Objective reference +//! resolution, state, audit, and linked Ticket validation. The native projection accepts `/objectives` +//! and immediate canonical `O-*` or internal IDs, binding the subject `id` from the route. +//! Linked Tickets remain separate reference inputs, requiring neither Ticket projection nor its +//! permissions. [`crate::wip`] owns shared reference, permission, Client-lifetime, and unknown-outcome +//! contracts. //! -//! Objectives are project-level planning context. Runtime Workers may not know -//! local `.yoi/objectives` paths, so model-visible Objective tools go through -//! the scoped Workspace API. +//! # Fixed contracts and current content +//! +//! Objects/Interfaces expose operation contracts, not mutable Objective bodies. Content edits thus +//! do not invalidate fixed signatures; there is no content cache or mutation generation. Each call +//! fetches a current bounded response through the same Backend adapter; the Backend also evaluates +//! conditions such as edit's `old_string`. Host validators protect routes and Interface contracts. +//! Native operations evaluate permission under regular Tool names and invoke the same business +//! logic. Query/detail uses canonical references and bounded bodies/events, omitting internal +//! storage IDs from output. use std::collections::BTreeMap; use std::sync::Arc; diff --git a/crates/worker/src/feature/builtin/session_explore.rs b/crates/worker/src/feature/builtin/session_explore.rs index 09d5bc241..69d4ced54 100644 --- a/crates/worker/src/feature/builtin/session_explore.rs +++ b/crates/worker/src/feature/builtin/session_explore.rs @@ -1,3 +1,10 @@ +//! Feature exploring one immutable Session capture fixed by the Host. +//! +//! Worker-observation owns live Worker reauthorization and recapture; this Feature reads only its +//! retained snapshot. It has no Workspace client or Memory state, and follows shared Session capture +//! references, privacy, and exploration bounds. It exposes history to internal verifiers and similar +//! consumers without business-mutation capabilities. + use std::sync::Arc; use agen::tool::{Tool, ToolDefinition, ToolError, ToolMeta, ToolOutput}; diff --git a/crates/worker/src/feature/builtin/subjektiv_memory.rs b/crates/worker/src/feature/builtin/subjektiv_memory.rs index 3fb9435ca..cd82ca390 100644 --- a/crates/worker/src/feature/builtin/subjektiv_memory.rs +++ b/crates/worker/src/feature/builtin/subjektiv_memory.rs @@ -1,8 +1,16 @@ -//! Host-scoped model tools for change-identified subjektiv Memory. +//! Model tools for Subjektiv Memory with change IDs, bound to a Host connection. //! -//! Confirmed Memory is read-only here. Explicit remembering and change requests -//! become immutable staging candidates; T-670 remains the only authority that may -//! adopt a candidate as a confirmed change. +//! Confirmed Memory is read-only here. Remember/ProposeChange creates only immutable staging +//! candidates; adoption and confirmed changes remain under consolidation authority. The Host fixes +//! subject/session/provenance from committed captures and connections, not model input. Preferences +//! require HumanInput evidence; model-authored tool calls never become authority for human preferences. +//! +//! Non-preference proposals without entry_refs stay pending_commit and use their own tool call as +//! evidence after commit. Receipts derive from Session and tool-call identity and are reconstructed +//! from durable history for staging, enabling idempotent retries with the same receipt after hook +//! failure or restore. RunCommitted and PreLlmRequest replay them; fail-closed BeforeSessionRewrite +//! prevents compaction from dropping unstaged receipts. Uncommitted operations are never staged. +//! Staging changes neither confirmed Memory fingerprints nor surfaces. use std::collections::HashSet; use std::sync::Arc; diff --git a/crates/worker/src/feature/builtin/task/mod.rs b/crates/worker/src/feature/builtin/task/mod.rs index 9ab38be3c..e45b1b703 100644 --- a/crates/worker/src/feature/builtin/task/mod.rs +++ b/crates/worker/src/feature/builtin/task/mod.rs @@ -1,6 +1,6 @@ //! Task tools built-in feature module. //! -//! The built-in Task feature owns the session-lifetime [`TaskStore`] shared by +//! The built-in Task feature owns the session-lifetime `TaskStore` shared by //! the Task tools and reminder hooks. Worker hosts install this module through the //! feature contribution boundary and use its narrow snapshot surface for //! restore/rewind/compaction compatibility. diff --git a/crates/worker/src/feature/builtin/ticket.rs b/crates/worker/src/feature/builtin/ticket.rs index 706580a51..a653dd7bb 100644 --- a/crates/worker/src/feature/builtin/ticket.rs +++ b/crates/worker/src/feature/builtin/ticket.rs @@ -1,8 +1,32 @@ -//! Built-in Ticket feature adapter. +//! Ticket Feature and native WIP projection connected to the Workspace API. //! -//! The ticket crate owns Ticket domain logic and Tool implementations. This -//! module binds an authority-scoped Workspace client, declares the built-in -//! feature, and contributes those tools through the normal registry path. +//! The `ticket` crate owns the Ticket domain and regular Tools; the Backend owns Workspace state, +//! relationships, and audit. This module connects a scoped Workspace client to the enabled Tools. +//! Shared WIP references, permissions, and unknown outcomes follow [`crate::wip`]. +//! +//! # Route-bound subjects +//! +//! The native projection exposes `/tickets` and one immediate child segment. Items accept canonical +//! `T-*` or canonical internal IDs, never resolving Objective keys or deeper suffixes as Tickets. +//! Item descriptors omit subject `ticket`/`id`, reconstructing original Tool arguments only from the +//! resolved route. Related/dependency Tickets remain separate inputs resolved by the Backend. +//! Operations come from regular Tools actually enabled by `TicketFeatureAccess`, retaining original +//! schemas and permission identities. WIP adds neither a broader business catalog nor another workflow. +//! +//! # Bounded responses and observation invalidation +//! +//! Query returns bounded snippets and canonical item paths; read returns bounded Backend details. +//! Objectives and Merge Requests remain references, without expanding their content or native routes. +//! Successful item-read response hashes feed route validators; canonical/internal aliases share an +//! observation group. Mutation receipts are not complete representations, so updates discard +//! observation hashes. Related targets and targets in queue results are also invalidated; no mutation +//! counter is created. These local observations replace neither Backend item/state CAS nor current +//! authorization. +//! +//! Fetching canonical references after mutations uses failure classes acknowledging that writes may +//! already have succeeded. HTTP failures at that stage are not misclassified as "unexecuted". +//! Optional arguments distinguish omission from explicit null for original JSON Schema/permission +//! evaluation; defaults use the same implementation as regular Tools. use std::collections::{BTreeMap, BTreeSet, HashMap}; use std::sync::{Arc, Mutex}; diff --git a/crates/worker/src/feature/builtin/worker_observation.rs b/crates/worker/src/feature/builtin/worker_observation.rs index 65c895683..325f89615 100644 --- a/crates/worker/src/feature/builtin/worker_observation.rs +++ b/crates/worker/src/feature/builtin/worker_observation.rs @@ -1,3 +1,16 @@ +//! Feature reading committed Sessions of Host-authorized Workers. +//! +//! The Host injects [`WorkerObservationProvider`]; model input never constructs grants, URLs, +//! session handles, or provider clients. List returns only currently authorized subjects, and every +//! capture reauthorizes. Missing and unauthorized subjects return the same NotFound, preventing +//! distinction. Runtime peers and parent-owned SubWorkers are separate tagged subjects. +//! +//! Each exploration operation recaptures its target to reflect newly committed entries. Shared +//! Session capture owns exploration projections and reference semantics. This Feature does not +//! enable fixed-snapshot session-explore or Memory extraction; observation grants no Worker control, +//! Ticket mutation, or approval capability. SubWorker completion notifications contain no transcript; +//! they prompt inspection rather than substitute for reading the committed Session. + use std::sync::Arc; #[cfg(test)] diff --git a/crates/worker/src/feature/builtin/workspace_config.rs b/crates/worker/src/feature/builtin/workspace_config.rs index 1549c1792..6bfe1e8e1 100644 --- a/crates/worker/src/feature/builtin/workspace_config.rs +++ b/crates/worker/src/feature/builtin/workspace_config.rs @@ -1,5 +1,26 @@ -//! WIP-only, grant-bound Workspace configuration Feature. No local config store -//! or OS Workdir/session is created by this logical attachment. +//! WIP-only logical attachment Feature for Workspace configuration allowed by explicit grants. +//! +//! Configuration source authority resides in the Backend virtual tree; evaluated Profiles/Prompts +//! are not edited in place of source. The Feature connects to the existing ledger through a +//! Workspace-bound client. It creates no OS directory copy, WorkdirSession, Bash capability, or +//! independent attachment ledger. The Backend and configuration domain own source persistence, +//! evaluation, CAS, and policies for applying changes to running Workers. +//! +//! # Shared attach path +//! +//! Structured chat invocation and native self-attach share [`WorkspaceConfigFeature::attach`]. +//! The access argument only restricts existing grants; unavailable explicit write access is rejected, +//! not downgraded. Attach runs while preparing admitted input and records a durable result before +//! passing it to the dependent LLM. Availability alone does not prove authority: validation, including +//! replay/resume, and Backend ledger updates check current access. Opaque connection IDs identify +//! connection lifetimes; alias reuse cannot revive old connection authority. +//! +//! # Request-time projection +//! +//! [`wip`] provides the native entrance; content builds a logical subtree from current Backend +//! metadata. Source bodies stay out of observations; directories derive from source paths. This +//! tree adds no Workflow/Repository registration, credential management, or Host configuration files. +//! Shared WIP references, Client lifetimes, and unknown outcomes follow [`crate::wip`]. pub mod backend; mod content; diff --git a/crates/worker/src/feature/builtin/workspace_config/backend.rs b/crates/worker/src/feature/builtin/workspace_config/backend.rs index 97326eb72..f10c60eda 100644 --- a/crates/worker/src/feature/builtin/workspace_config/backend.rs +++ b/crates/worker/src/feature/builtin/workspace_config/backend.rs @@ -1,4 +1,11 @@ -//! Identity-bound typed Workspace transport. Never retries a dispatched request. +//! Typed Workspace configuration transport bound to Worker identity. +//! +//! Uses `workers/self/workspace-config`, never bypassing it for the owner editor API. Responses are +//! checked for Workspace, alias, content root, and connection lifetime, distinguishing typed Backend +//! `NotCommitted` from `Unknown`. Authorization, CAS, and input rejections become safe fixed +//! diagnostics; raw provider/transport bodies never reach the model. Missing, oversized, or +//! undecodable mutation responses do not prove non-commit; dispatched requests are never resent. +//! This adapter does not reimplement Backend source-persistence transactions or activation. use std::sync::Arc; use async_trait::async_trait; diff --git a/crates/worker/src/feature/builtin/workspace_config/content.rs b/crates/worker/src/feature/builtin/workspace_config/content.rs index ff1f0781e..234b97eeb 100644 --- a/crates/worker/src/feature/builtin/workspace_config/content.rs +++ b/crates/worker/src/feature/builtin/workspace_config/content.rs @@ -1,5 +1,22 @@ -//! Request-time logical subtree. Metadata is captured per resolution, never -//! cached as authority; authorization and canonical tree CAS stay in Backend. +//! Logical configuration subtree built per request from Backend metadata. +//! +//! Maps `/workspace-config` to the Backend's empty root path and children to canonical relative +//! source paths. Connection, authorization, and node information are fetched at resolution, without +//! permanent file inventory or authority caches. Read/write targets are fixed by selected routes; +//! model arguments cannot override Workspace, connection, validator, or expected digest. Root mutation +//! batches handle multiple sources and import references together, passing digest, entrypoints, and +//! node state from one tree observation as preconditions to canonical commit. +//! +//! Path and root validators have different roles. Additional observations of other paths must match +//! connection, root validator, digest, and entrypoints, never mixing snapshots in one batch. +//! Backend whole-tree CAS is the final conflict boundary with UI/other Workers; local locks cannot +//! replace it. Connection lifetimes and evaluation-contract changes can also cause staleness, so +//! identical source bytes alone do not prove freshness. Post-call validators come from returned +//! commit/read results, not subsequently read state. +//! +//! Explicit nonexistent paths provide create contracts bound to their absence observations. +//! Oversized trees/paths/text/changes or invalid metadata are not published as complete observations. +//! Only authorization denial maps to namespace absence; transport/Backend failures are not hidden as absence. use std::collections::{BTreeMap, BTreeSet}; use std::sync::Arc; diff --git a/crates/worker/src/feature/builtin/workspace_config/wip.rs b/crates/worker/src/feature/builtin/workspace_config/wip.rs index 9fd0a27d7..d3aa9d2c8 100644 --- a/crates/worker/src/feature/builtin/workspace_config/wip.rs +++ b/crates/worker/src/feature/builtin/workspace_config/wip.rs @@ -1,4 +1,10 @@ -//! Shared native self-attach entrance and production request-time config subtree. +//! Registers the native self-attach entrance and request-resolved configuration content subtree. +//! +//! Connection to the existing attachment ledger and configuration content share a root, but attach +//! is not a content mutation and creates no separate content validator. Only the production adapter +//! registers a subtree provider bound to current grants. There is no configuration-specific +//! compatibility Tool or `/tools` fallback. The parent module and content own Feature enablement, +//! connection targets, and configuration-body projection boundaries. use std::collections::BTreeMap; use std::sync::Arc; diff --git a/crates/worker/src/feature/invocation.rs b/crates/worker/src/feature/invocation.rs index a4a71a1fe..de8bc6fbe 100644 --- a/crates/worker/src/feature/invocation.rs +++ b/crates/worker/src/feature/invocation.rs @@ -1,3 +1,22 @@ +//! Registration, completion, and validation of Feature-declared structured chat invocations. +//! +//! Public names and aliases serve display/selection; source-qualified identities identify execution +//! targets. Registration must match the declared descriptor and the Feature being installed. Public +//! name/alias collisions are rejected; failed Feature installs roll back to the same checkpoint as +//! other contributions. Completion exposes only enabled handler metadata, not code or credentials. +//! +//! Worker executes only `Segment::FeatureInvoke`, never extracting syntax from ordinary text, +//! attachments, or history. Every call checks the installed descriptor, arguments, current +//! availability, and handler semantic/authority validation. Availability is rechecked after async +//! work. Completion providers come from registered metadata, not request-specified providers or +//! authority. Completion is side-effect-free discovery; oversized candidate values are omitted, +//! not shortened. +//! +//! Handlers use the Host-supplied `invocation_id` as a scoped idempotency key at the business boundary +//! and reauthorize at the commit boundary inside `invoke`. General availability alone cannot prove +//! argument-specific authority. Oversized results or ambiguous post-execution failures become +//! `OutcomeUnknown`, not successful dependent model context. Worker owns execution order and durable receipts. + use std::collections::{BTreeMap, BTreeSet}; use std::fmt; use std::sync::Arc; diff --git a/crates/worker/src/feature/mcp.rs b/crates/worker/src/feature/mcp.rs index ca13ae137..5fbeb4cf5 100644 --- a/crates/worker/src/feature/mcp.rs +++ b/crates/worker/src/feature/mcp.rs @@ -1,3 +1,12 @@ +//! Adapter projecting MCP wire discovery results into Worker Features. +//! +//! The `mcp` crate owns process/JSON-RPC lifecycle; this module owns capability-dependent bounded +//! discovery, Tool-name namespacing, schema acceptance, and model-facing result rendering. Server +//! instructions are not adopted as Worker instructions; external metadata and results remain data. +//! List-changed notifications prompt rediscovery rather than prove catalog changes; startup +//! rediscovery is also bounded. Failed server contributions are isolated with diagnostics; +//! the regular registry decides Tool-name collisions. + use std::collections::BTreeMap; use std::path::Path; use std::sync::Arc; diff --git a/crates/worker/src/fs_view.rs b/crates/worker/src/fs_view.rs index 5508718c2..42c224240 100644 --- a/crates/worker/src/fs_view.rs +++ b/crates/worker/src/fs_view.rs @@ -1,14 +1,14 @@ -//! Worker 視点のファイルシステム操作。 +//! Filesystem operations from the Worker's perspective. //! -//! `WorkdirSession` の上に「Worker が読み取りたい / 列挙したい」操作を集約する軽い wrapper。 +//! Lightweight wrapper over `WorkdirSession` collecting Worker read/list operations. //! -//! - `ReadRequirement` と `render_auto_read` — compact worker が `mark_read_required` -//! で nominate したファイルを再読し、`[Auto-read file: ...]` system message に -//! 変換する経路。`Worker::compact` から呼ばれる。 -//! - `slice_lines` — 行 offset / limit でテキストを切り出す純粋ヘルパ。 -//! compact tool 側の `mark_read_required` でも使用。 -//! - `list_file_completions` — TUI 補完用、prefix マッチでファイル候補を列挙する経路。 -//! IPC `Method::ListCompletions` 経由で呼ばれる前提（Phase 2 で接続）。 +//! - `ReadRequirement` and `render_auto_read` reread files nominated by the compact worker's +//! `mark_read_required`, rendering `[Auto-read file: ...]` system messages. +//! Called by `Worker::compact`. +//! - `slice_lines` is a pure helper slicing text by line offset / limit, also used by the compact +//! tool's `mark_read_required`. +//! - `list_file_completions` lists prefix-matching file candidates for TUI completion. +//! Intended for IPC `Method::ListCompletions` (connected in Phase 2). use std::path::{Path, PathBuf}; @@ -21,16 +21,16 @@ use workdir::{ EntryKind, ListRequest, ReadRequest, StatRequest, WorkdirPath, WorkdirSessionHandle, }; -/// 補完候補1件の最大数。`list_file_completions` がこの値を超えたら打ち切り。 +/// Maximum completion candidate count; `list_file_completions` stops at this bound. const COMPLETION_LIMIT: usize = 100; -/// submit-time directory FileRef の shallow listing で返す最大 entry 数。 -/// TUI completion と同じ浅い一覧という意味論に揃えるため、同じ上限を使う。 +/// Maximum entries in a submit-time directory FileRef's shallow listing. +/// Shares the TUI completion bound to preserve the same shallow-listing semantics. const DIR_FILE_REF_ENTRY_LIMIT: usize = COMPLETION_LIMIT; /// Provider-side bound for auto-read and submit-time referenced-file reads. const AUTO_READ_BYTE_LIMIT: usize = 4 * 1024 * 1024; -/// Compact worker が `mark_read_required` で nominate した「次セッション開始時に -/// 自動で再読すべきファイル」のエントリ。 +/// File nominated by the compact worker's `mark_read_required` for automatic rereading +/// at the start of the next session. #[derive(Debug, Clone)] pub struct ReadRequirement { pub path: PathBuf, @@ -40,23 +40,23 @@ pub struct ReadRequirement { pub limit: Option, } -/// Worker から見えるファイルシステム操作の入口。Clone は cheap（`WorkdirSession` 内 `Arc`）。 +/// Entry point for Worker-visible filesystem operations. Cheap to clone (`Arc` in `WorkdirSession`). #[derive(Debug, Clone)] pub struct WorkerFsView { session: WorkdirSessionHandle, } -/// `list_file_completions` が返す候補1件。 +/// One candidate returned by `list_file_completions`. #[derive(Debug, Clone, PartialEq, Eq)] pub struct FileCandidate { - /// 入力 prefix と整合する形のパス（prefix が absolute なら absolute、 - /// relative なら cwd 相対）。 + /// Path matching the input prefix's form: absolute for an absolute prefix, + /// cwd-relative for a relative prefix. pub path: String, pub is_dir: bool, } -/// `resolve_file_ref` の失敗理由。Worker 側で Alert に振り分けるために -/// WorkdirSession / 内部判定の両方を区別できるよう保持する。 +/// Failure reason for `resolve_file_ref`. Distinguishes WorkdirSession failures from +/// internal checks so Worker can select the appropriate Alert. #[derive(Debug)] pub enum ResolveError { /// Path resolution / scope check failed via `WorkdirSession`. @@ -251,7 +251,7 @@ impl WorkerFsView { } } -/// `text` の `offset` 行目から `limit` 行（None なら末尾まで）を、元の改行で繋いで返す。 +/// Return `limit` lines of `text` starting at line `offset` (None means to the end), joined by newlines. pub fn slice_lines(text: &str, offset: usize, limit: Option) -> String { let lines: Vec<&str> = text.lines().collect(); let start = offset.min(lines.len()); @@ -536,7 +536,7 @@ mod tests { let view = WorkerFsView::new(fs_for(&dir)); let cands = view.list_file_completions("").await; - // ディレクトリ first + // Directories first let names: Vec<&str> = cands.iter().map(|c| c.path.as_str()).collect(); assert_eq!(names, vec!["subdir", "alpha.rs", "beta.rs"]); assert!(cands[0].is_dir); diff --git a/crates/worker/src/ipc/notify_buffer.rs b/crates/worker/src/ipc/notify_buffer.rs index 6af0e75cb..c44bd2398 100644 --- a/crates/worker/src/ipc/notify_buffer.rs +++ b/crates/worker/src/ipc/notify_buffer.rs @@ -15,7 +15,7 @@ //! agent-visible WorkerEvent variants, and any future typed system reminder //! insertion all ride this queue. //! Per `tickets/notify-history-persist.md` and `AGENTS.md` (LLM -//! context の加工原則), there is **no** "transient, history-skipping" +//! context processing principles), there is **no** "transient, history-skipping" //! lane — everything injected into a request is also committed to //! history so any LLM reaction has a visible trigger across turns, //! resume, and compaction, and so the Anthropic prompt cache prefix diff --git a/crates/worker/src/job.rs b/crates/worker/src/job.rs index c32fac642..b52270d34 100644 --- a/crates/worker/src/job.rs +++ b/crates/worker/src/job.rs @@ -1,11 +1,14 @@ -//! Host-facing, result-only Jobs executed by the private Internal Worker substrate. +//! Job execution by a restricted Internal Worker; the Host owns result acceptance and persistence. //! -//! The host owns persistence, concurrency, timeout, retries and result acceptance. Normal -//! timeout/shutdown routes through the cancellation sender and awaits `run`, including Worker -//! resource cleanup. An emergency drop/abort does not confirm asynchronous cleanup; the Host -//! must retain its lease and reconcile any unknown outcome before allowing another execution. -//! The result-only Feature owns no background tasks, but an in-flight synchronous Host sink -//! cannot be preempted by dropping its tool future and must itself be bounded. +//! Selected Profile requirements are checked against explicit Host grants. Unsatisfied capabilities +//! are rejected, not silently stripped. Profile names and resolution cwd grant no filesystem or +//! Workspace authority. Success requires structured output synchronously accepted by the Host sink, +//! not a final answer or completion of execution. +//! +//! The Host manages concurrency, timeouts, and retries, and awaits `run` and resource release even +//! after cancellation. Emergency drop or abort cannot confirm asynchronous cleanup, so leases and +//! unknown-outcome state remain. The result Feature has no background work, but dropping a tool +//! future cannot interrupt a running synchronous sink. The Host must also bound the sink's work. mod result; pub use result::{JobResultFeature, JobResultFinalizer, JobResultSink, SUBMIT_JOB_RESULT_TOOL}; diff --git a/crates/worker/src/lib.rs b/crates/worker/src/lib.rs index a1cfa7c16..38ab8eba0 100644 --- a/crates/worker/src/lib.rs +++ b/crates/worker/src/lib.rs @@ -1,3 +1,5 @@ +#![doc = include_str!("../README.md")] + mod backend_job; pub use backend_job::BackendJobExecutionBinding; pub mod bootstrap; diff --git a/crates/worker/src/model_client.rs b/crates/worker/src/model_client.rs index 8da39c1aa..683fc1ff1 100644 --- a/crates/worker/src/model_client.rs +++ b/crates/worker/src/model_client.rs @@ -1,16 +1,16 @@ -//! [`ModelManifest`] を [`Box`] に落とす worker-side factory。 +//! Worker-side factory converting [`ModelManifest`] into [`Box`]. //! -//! 段階: -//! 1. `ModelManifest` を [`catalog::resolve_model_manifest`] で -//! カタログ込み [`ModelConfig`] に解決（ref → 展開 / inline → 検証） -//! 2. `AuthRef` を local secret store / ファイルから解決して [`ResolvedAuth`] に -//! 3. `scheme.required_auth()` と解決値を照合（非対応組合せは構築エラー） -//! 4. `ModelCapability` は manifest 明示 > model catalog > provider -//! default_capability > scheme 既定 の順でフォールバック（上位 3 段は -//! `catalog::resolve_model_manifest` が [`ModelConfig`] に詰め込む） +//! Steps: +//! 1. Resolve `ModelManifest` through [`catalog::resolve_model_manifest`] into catalog-aware +//! [`ModelConfig`] (expand refs; validate inline configuration). +//! 2. Resolve `AuthRef` from the local secret store / files into [`ResolvedAuth`]. +//! 3. Check the resolved value against `scheme.required_auth()`; unsupported pairs fail construction. +//! 4. `ModelCapability` falls back in order: explicit manifest > model catalog > provider +//! default_capability > scheme default. `catalog::resolve_model_manifest` puts the first three +//! levels into [`ModelConfig`]. //! -//! agen は低レベル基盤に留める方針なので、高レベル側で必要に -//! なる認証ストア解決と secret store 解決は worker 側で行う。 +//! agen remains low-level infrastructure; worker handles the higher-level auth-store and secret-store +//! resolution. use std::sync::Arc; @@ -30,7 +30,7 @@ use secrets::{SecretStore, SecretValue}; pub use manifest::model_catalog::{ModelConfig, ResolveError as CatalogResolveError}; -/// プロバイダ構築時のエラー。 +/// Errors during provider construction. #[derive(Debug, thiserror::Error)] pub enum ProviderError { #[error("model configuration error: {0}")] @@ -56,7 +56,7 @@ pub enum ProviderError { ManifestResolve(#[from] catalog::ResolveError), } -/// `AuthRef` をランタイムで使える [`ResolvedAuth`] に解決する。 +/// Resolve `AuthRef` into runtime-usable [`ResolvedAuth`]. fn resolve_auth(scheme: SchemeKind, auth: &AuthRef) -> Result { let resolver = DefaultSecretResolver; resolve_auth_with_resolver(scheme, auth, &resolver) @@ -123,9 +123,9 @@ fn resolve_auth_with_resolver( } } -/// `AuthRef::CodexOAuth` 指定時、`base_url` 未指定なら ChatGPT backend を既定とする。 -/// Codex CLI が使う `/backend-api/codex` を base に取り、scheme 側の `/responses` -/// path と結合して `https://chatgpt.com/backend-api/codex/responses` になる。 +/// `AuthRef::CodexOAuth` defaults to the ChatGPT backend when `base_url` is unset. +/// The Codex CLI base `/backend-api/codex` combines with the scheme's `/responses` path +/// to form `https://chatgpt.com/backend-api/codex/responses`. fn effective_base_url(scheme: &S, config: &ModelConfig) -> String { if let Some(b) = &config.base_url { return b.clone(); @@ -147,11 +147,11 @@ fn build_transport( scheme: config.scheme, }); } - // capability の優先順位 (上位 3 段は `ModelConfig` に既に反映済み): - // 1. manifest 明示 + // Capability precedence (the first three levels are already reflected in `ModelConfig`): + // 1. explicit manifest // 2. model catalog // 3. provider.default_capability - // 4. `Scheme::default_capability()`（scheme ごとの wire-level 安全側） + // 4. `Scheme::default_capability()` (scheme-specific conservative wire-level default) let capability: ModelCapability = config .capability .clone() @@ -191,9 +191,8 @@ fn build_from_config(config: &ModelConfig) -> Result, Provide TransportPolicy::standard(), ), SchemeKind::OpenaiResponses => { - // ChatGPT backend (codex-oauth) は `max_output_tokens` / - // `temperature` / `top_p` を 400 で弾くため、その経路では - // 送出を止める。 + // The ChatGPT backend (codex-oauth) rejects `max_output_tokens`, + // `temperature`, and `top_p` with HTTP 400; omit them on that path. let send_to_official = !matches!(config.auth, AuthRef::CodexOAuth); let scheme = OpenAIResponsesScheme::new() .with_send_max_output_tokens(send_to_official) @@ -208,17 +207,16 @@ fn build_from_config(config: &ModelConfig) -> Result, Provide } } -/// [`ModelManifest`] から [`LlmClient`] を構築する。ref / inline の -/// いずれも受け取り、カタログ解決は内部で行う。 +/// Build an [`LlmClient`] from a [`ModelManifest`], accepting refs or inline configuration +/// and resolving the catalog internally. pub fn build_client(manifest: &ModelManifest) -> Result, ProviderError> { let config = catalog::resolve_model_manifest(manifest)?; build_from_config(&config) } -/// 既に解決済みの [`ModelConfig`] から [`LlmClient`] を構築する。 -/// `ModelManifest` から既に `catalog::resolve_model_manifest` を通した -/// ケース（factory / spawn 経路でカタログ引きを 1 回だけにしたい等）で -/// 使う。 +/// Build an [`LlmClient`] from an already resolved [`ModelConfig`]. +/// Use when `ModelManifest` has already passed through `catalog::resolve_model_manifest`, +/// for example to resolve the catalog only once in factory / spawn paths. pub fn build_client_from_config(config: &ModelConfig) -> Result, ProviderError> { build_from_config(config) } @@ -360,7 +358,7 @@ mod tests { let dir = tempfile::tempdir().unwrap(); let _guard = ConfigDirGuard::new(dir.path()); - // Ollama は AuthRef::None で構築できる end-to-end path。 + // End-to-end path building Ollama with AuthRef::None. let manifest = ModelManifest { ref_: Some("ollama-local/llama3.1".into()), ..Default::default() @@ -379,7 +377,7 @@ mod tests { let dir = tempfile::tempdir().unwrap(); let _guard = ConfigDirGuard::new(dir.path()); - // Form C: 完全直書き。Ollama 相当を AuthRef::None で構築。 + // Form C: fully inline. Build an Ollama equivalent with AuthRef::None. let manifest = ModelManifest { scheme: Some(SchemeKind::Anthropic), base_url: Some("http://localhost:11434".into()), @@ -397,7 +395,7 @@ mod tests { #[test] fn ollama_config_succeeds_without_key() { - // Ollama = Anthropic scheme + base_url 差し替え + AuthRef::None + // Ollama = Anthropic scheme + overridden base_url + AuthRef::None let config = ModelConfig { scheme: SchemeKind::Anthropic, base_url: Some("http://localhost:11434".into()), diff --git a/crates/worker/src/runtime/worker_allocation/conflict.rs b/crates/worker/src/runtime/worker_allocation/conflict.rs index 686b993cb..f47bebf92 100644 --- a/crates/worker/src/runtime/worker_allocation/conflict.rs +++ b/crates/worker/src/runtime/worker_allocation/conflict.rs @@ -1,8 +1,8 @@ //! Pure functions that decide whether scope rules collide. //! //! These helpers are read-only over [`LockFile`]; they never touch the -//! file or the lock itself. The mutating operations in [`crate::mutate`] -//! call them under the [`crate::LockFileGuard`]. +//! file or the lock itself. The mutating operations in `super::mutate` +//! call them under the [`super::LockFileGuard`]. use manifest::{Permission, ScopeRule}; diff --git a/crates/worker/src/runtime/worker_allocation/lifecycle.rs b/crates/worker/src/runtime/worker_allocation/lifecycle.rs index a596c1aa9..cab09ec18 100644 --- a/crates/worker/src/runtime/worker_allocation/lifecycle.rs +++ b/crates/worker/src/runtime/worker_allocation/lifecycle.rs @@ -229,7 +229,7 @@ pub(crate) fn install_top_level_at_for_test( /// a spawning Worker. /// /// The spawning flow is two-stage: the spawner calls -/// [`crate::delegate_scope`] (with its own pid as a live placeholder, +/// [`super::delegate_scope`] (with its own pid as a live placeholder, /// `segment_id = None`), then exec's the child; the child, once /// running, calls this function to rewrite the allocation's pid + /// segment_id to its own and claim the [`ScopeAllocationGuard`] so diff --git a/crates/worker/src/runtime/worker_allocation/table.rs b/crates/worker/src/runtime/worker_allocation/table.rs index af7c6159a..b9d0a16ae 100644 --- a/crates/worker/src/runtime/worker_allocation/table.rs +++ b/crates/worker/src/runtime/worker_allocation/table.rs @@ -85,9 +85,9 @@ pub struct Allocation { /// a top-level Worker started directly by a human. pub delegated_from: Option, /// Segment ID this Worker is currently writing to. `None` means this - /// is a pre-reservation made by a spawner via [`super::super::delegate_scope`] + /// is a pre-reservation made by a spawner via [`super::delegate_scope`] /// before the child has come up; the child fills it in at - /// [`crate::adopt_allocation`] time. + /// [`super::adopt_allocation`] time. #[serde(default)] pub segment_id: Option, } diff --git a/crates/worker/src/segment_log_sink.rs b/crates/worker/src/segment_log_sink.rs index 76d9542d3..a21d9cec5 100644 --- a/crates/worker/src/segment_log_sink.rs +++ b/crates/worker/src/segment_log_sink.rs @@ -174,7 +174,7 @@ impl SegmentLogSink { /// /// Used by restore paths that load a session's complete log into /// the mirror before any subscriber is connected. Callers that need - /// to notify existing subscribers should use [`reset_with_initial`]. + /// to notify existing subscribers should use [`Self::reset_with_initial`]. pub fn replace_silent(&self, entries: Vec) { let mut mirror = self .inner diff --git a/crates/worker/src/session_capture.rs b/crates/worker/src/session_capture.rs index f67be5936..46e52c33c 100644 --- a/crates/worker/src/session_capture.rs +++ b/crates/worker/src/session_capture.rs @@ -1,8 +1,15 @@ -//! Workspace- and Memory-independent exploration of an immutable ordered session capture. +//! Exploration domain for immutable Session captures, with no Workspace/Memory mutation capability. //! -//! Hosts construct a capture from committed session items. The capture excludes reasoning, -//! assigns append-stable `SessionEntryRef` values, and provides sparse overview, bounded -//! range/search, read, and generic evidence projections without granting mutation authority. +//! A capture is an ordered projection of committed entries; exploration excludes reasoning and +//! system prompts. Overview, search, bounded reads, and evidence conversion share `SessionEntryRef`. +//! References identify entries in the selected Session and cannot be reused for another subject. +//! Appends preserve existing entry identities. Sparse real user/assistant anchors and intervening +//! entry counts index long histories. Search with an empty filter also provides bounded range lists. +//! +//! Host-fixed snapshot exploration, Memory candidate evidence validation, and current Worker +//! observation share only this domain; they do not enable or modify each other's Features. +//! The provider owns authorization to recapture a current Worker. A capture itself proves neither +//! completion nor approval and grants no business-mutation authority. use std::collections::HashSet; use std::sync::Arc; @@ -22,6 +29,8 @@ const MAX_READ_MAX_ITEMS: usize = 80; const DEFAULT_READ_MAX_BYTES: usize = 32 * 1024; const OVERVIEW_ANCHOR_STRIDE: usize = 8; +/// Append-stable exploration reference derived from an entry identity in the selected Session. +/// Shared by overview, search, read, and evidence conversion; it does not identify other Sessions' entries. #[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)] #[serde(transparent)] pub struct SessionEntryRef(String); diff --git a/crates/worker/src/wip.rs b/crates/worker/src/wip.rs index 0df009785..8c4a0849e 100644 --- a/crates/worker/src/wip.rs +++ b/crates/worker/src/wip.rs @@ -1,11 +1,65 @@ -//! Explicit-opt-in Web Interface Protocol (WIP) Worker surface. +//! Worker's Web Interface Protocol (WIP) projection and execution boundary. //! -//! The adapter intentionally keeps authority in the existing Worker host. In WIP -//! mode ordinary tools are mounted below `/tools`, removed from the model-visible -//! tool list, and invoked through three stateful tree/inspect/invoke tools. The -//! published WIP crates own protocol validation, HTTP encoding, Known Space, and -//! operation lifecycle state; this module owns only the in-process transport and -//! the compatibility projection into existing async `Tool` implementations. +//! In WIP mode, `Tree`, `Inspect`, and `Invoke` access native projections and compatibility +//! projections at `/tools/`, without duplicating individual Tools for the model. +//! The public WIP crate owns Protocol types, codecs, Known Space, and call lifecycle; this module +//! owns Host registration/resolution and connection to existing async Tools. Provider-specific +//! projections belong to their Features. +//! +//! # Object ownership and added operations +//! +//! [`WipMountRegistry`] retains root namespace owners. Object existence, identity, and resolver +//! ownership are separate from operations added by other Features. Dynamic mounts resolve only one +//! immediate collection child segment; added operations replace neither existing Objects nor resolvers. +//! Descriptors extending the same Interface must match format, documentation, and type declarations. +//! Registration rejects duplicate operation names, routes owned elsewhere, and conflicting contracts +//! for the same reference. Changing enabled operations changes the Interface validator, preventing +//! execution from old observations. +//! +//! Native projections hide compatibility Tools only for the same semantic capability or an explicit +//! claim. Feature names are not intermediate Objects; old `/features/...` routes are not aliases. +//! Old routes in persisted history are not rewritten and remain distinct from current call targets. +//! +//! # References and call-time authority +//! +//! Exact `(scope, name)` pairs identify Interfaces. The same name in different scopes means different +//! contracts; ancestry uses canonical path segments, not string prefixes. Self-scoped projections use +//! the resolved Object path as scope, fixing Object, descriptor, validator, and handler from the same +//! projection. Missing scope Object refs are never fabricated. +//! +//! The Host checks target, Object precondition, scope/membership/ref, Interface precondition, and +//! arguments before dispatching to the fixed handler. Publication, Feature enablement, and observation +//! do not grant execution authority; handlers recheck current provider capabilities and caller +//! authority. Compatibility calls validate input against original JSON Schema and evaluate permission +//! with original Tool names/arguments. `ask` is denied without an approval protocol. Workdir scope +//! and Backend authorization remain in existing implementations. +//! +//! # Client lifetime and observations +//! +//! Each [`WipRuntime`] creates a Client, distinguishing sessions by endpoint and Host-supplied opaque +//! security context. Observations stay within the Runtime, never shared or persisted across Workers. +//! Restore requires reacquisition; reference strings in history are not cache or wire authority +//! evidence. Ordinary compaction preserves the live Runtime but does not turn shortened conversation +//! into observation authority. +//! +//! Acquisition, display, and calls are separate. Tree follows only indexable edges; unobserved +//! children are not empty sets. Inspect displays only the selected Object or Interface in the +//! official Text View. Displays that cannot meet completeness, freshness, or bounds fail rather than +//! silently truncate. Displays are untrusted data; signatures are never reparsed as call targets. +//! The binding defines details. +//! +//! # Post-dispatch outcomes and cancellation +//! +//! Route/schema/permission denial, stale, provider-confirmed cancelled, interrupted, post-dispatch +//! disconnection, and unknown outcomes are distinct. Dispatched operations are never automatically +//! rerun. Unknown mutation outcomes in particular require external state or provider receipt checks. +//! Cancellation goes only to the operation and handler fixed for execution, without re-resolving the +//! Object. Sending cancellation alone does not establish a terminal outcome; completion or future +//! drop releases the execution binding. +//! +//! Native typed returns remain values; Entries and strings never trigger implicit extra acquisition. +//! Later Inspect failures do not fail completed calls. Attachments and compatibility ToolOutput use +//! existing Engine output/history paths; native attachments still require WIP return-value validation. use std::collections::{BTreeMap, BTreeSet, HashMap, VecDeque}; use std::sync::atomic::{AtomicU64, Ordering}; diff --git a/crates/worker/src/wip/binding.rs b/crates/worker/src/wip/binding.rs index 37b8017c5..8beb42454 100644 --- a/crates/worker/src/wip/binding.rs +++ b/crates/worker/src/wip/binding.rs @@ -1,5 +1,20 @@ -//! Target-specific AI binding. Retrieval, display and invocation are separate: -//! signatures are untrusted display data, never parsed back into call targets. +//! Binding separating model-facing WIP acquisition, display, and calls. +//! +//! Tree succeeds only after acquiring all indexable edges within the requested depth; it does not +//! scan Known Space by path prefix. Children at the depth boundary are unobserved, distinct from +//! observed empty sets. Acquisition completion checks freshness, data, and edge completeness, not +//! just `ensure`'s return value. Loading and insufficient capacity are reported as acquisition states, +//! not resolved by overwriting or busy loops. +//! +//! Inspect displays an Object signature with ordered Interface references, or one Interface signature. +//! It neither expands referenced descriptors automatically nor requires a prior Tree. Official Text +//! View signatures remain unchanged; paths and structured references sit in the outer envelope. +//! Rendering/envelope bound violations fail the whole display; incomplete contracts never return success. +//! +//! Invoke uses structured Interface references and named arguments. Missing, stale, or failed +//! observations get at most one pre-dispatch acquisition attempt per target; dispatched calls never +//! rerun. Explicit refresh updates only the selected Object/Interface, not referenced targets. +//! Interrupted acquisition fails only its own request, never leaving Loading permanently. use super::*; use wip_client::{InterfaceObservation, PreparedRequest}; diff --git a/crates/worker/src/wip/binding/inspect_target.rs b/crates/worker/src/wip/binding/inspect_target.rs index 6ebd6b867..4f61229b5 100644 --- a/crates/worker/src/wip/binding/inspect_target.rs +++ b/crates/worker/src/wip/binding/inspect_target.rs @@ -1,4 +1,9 @@ -//! Input-only compact Interface addresses. Never parse rendered signatures. +//! Compact Interface addresses exclusively for Inspect input. +//! +//! `scope::name` preserves the explicit absolute scope and name exactly, rejecting scope inference, +//! normalization, relative resolution, and operation suffixes. Special characters decode as JSON +//! strings. Quoting an entire Object path containing `::` distinguishes it from an Interface address. +//! Invalid Interface input never falls back to Object lookup; displayed signatures are not parsed. use super::{InterfaceReference, ToolError}; #[derive(Debug, PartialEq, Eq)] diff --git a/crates/worker/src/wip/compatibility_schema.rs b/crates/worker/src/wip/compatibility_schema.rs index 90d56649b..e37fd4f76 100644 --- a/crates/worker/src/wip/compatibility_schema.rs +++ b/crates/worker/src/wip/compatibility_schema.rs @@ -1,7 +1,13 @@ -//! Conservative JSON-shape-preserving types for the compatibility `input` value. -//! The original JSON Schema remains authoritative and visible in Inspect. -//! In particular, open records and JSON unions must not become closed records -//! or WIP's tagged unions: that would reject/change valid original tool inputs. +//! Projects compatibility Tool `input` into WIP types without changing JSON shape. +//! +//! Conservatively represents only closed records, homogeneous lists, primitives, and string enums. +//! Local JSON Pointer expansion has depth/node limits; cycles, unsupported syntax, open records, +//! and mixed unions remain `Json`. Replacing them with WIP closed records or tagged unions would +//! reject inputs valid for the original Tool or change their shape. +//! +//! Original JSON Schema also stays in Inspect-visible parameter summaries for execution validation. +//! Type projection does not replace schema constraints, rename fields, insert defaults, or add union +//! wrappers. The parent WIP Host connects display and original schemas to execution. use super::{Documentation, Json, TypeExpr}; use wip_protocol::{EnumCase, FieldDeclaration}; diff --git a/crates/worker/src/worker.rs b/crates/worker/src/worker.rs index cb46adb9f..a411b2ea3 100644 --- a/crates/worker/src/worker.rs +++ b/crates/worker/src/worker.rs @@ -89,8 +89,11 @@ const MAX_SUBMISSION_RECEIPTS: usize = 128; const MAX_INVOCATION_RECEIPTS: usize = 1024; const MAX_INVOCATION_IDENTITY_BYTES: usize = 256; -/// A durable start without a matching result is intentionally not retryable. -/// This is a conservative fence, not an exactly-once business-operation ledger. +/// Durable replay-prevention receipt retaining invocation start and a bounded result. +/// +/// Stores the typed payload digest and qualified identity, not a copy of all arguments. +/// A start without a result cannot rule out side effects, so reexecution is rejected. +/// The same ID with a different payload is also rejected. This is not a business exactly-once ledger. #[derive(Debug, Clone, serde::Serialize, serde::Deserialize)] #[serde(deny_unknown_fields)] struct InvocationReceipt { @@ -3105,7 +3108,7 @@ impl Worker { /// Note: this constructor does **not** parse `manifest.worker.system_prompt` /// as a template. `Worker::from_manifest` is the production path for /// templated prompts; callers of `Worker::new` that want a template - /// should parse it themselves and call [`set_system_prompt_template`]. + /// should parse it themselves and call [`Self::set_system_prompt_template`]. pub async fn new( manifest: WorkerManifest, worker: Engine, @@ -3714,7 +3717,12 @@ impl Worker { &self.store } - /// Rewind the current segment to just before the specified public user input. + /// Rewinds the current Segment to just before the specified public user input. + /// + /// Saves the display-history prefix and pending activation checkpoint in one Store replacement. + /// Retains invocation-only receipts and recovery-blocked state, avoiding a failure window + /// between truncation and checkpoint append. Rewinding history does not undo committed + /// business side effects. pub async fn rewind_to(&mut self, entry_id: String) -> Result { // Reject invalid targets before invoking rewrite hooks or quiescing tasks. // Resolve again under the append lock after the asynchronous barrier. @@ -3895,7 +3903,7 @@ impl Worker { /// Snapshot of the cumulative LLM Usage measurement timeline. /// /// One entry per LLM call. Restored on `restore` and appended in - /// `persist_turn`. Used by token-accounting APIs in [`token_counter`]. + /// `persist_turn`. Used by token-accounting APIs in [`agen::token_counter`]. /// Returns a clone since the underlying vector is shared with hooks /// running on the Engine. pub fn usage_history(&self) -> Vec { @@ -3905,14 +3913,6 @@ impl Worker { .clone() } - /// Snapshot of the extract (memory.extract) boundary pointer. - /// - /// `None` means no extract has run yet on the current session — the - /// next extract will start from entry 0. Updated by - /// [`try_post_run_extract`](Self::try_post_run_extract) on success - /// and reset by [`compact`](Self::compact) (the new compacted - /// session has a fresh log with no `LogEntry::Extension` entries). - /// Cheap clone via `Option`. /// Snapshot of the typed user segments tracked alongside worker /// history. The K-th entry corresponds to the K-th `Item::user_message` /// derived from `LogEntry::AnnotatedUserInput` entries (post-compaction); seed @@ -4090,7 +4090,7 @@ impl Worker { /// Attach the broadcast sender used for typed lifecycle `Event`s. /// - /// The Controller wires this alongside [`attach_alerter`] so that + /// The Controller wires this alongside [`Self::attach_alerter`] so that /// Worker-internal operations (currently: compaction) can surface /// progress to connected clients. pub fn attach_working_event_tx(&mut self, working_event_tx: broadcast::Sender) { @@ -4143,12 +4143,6 @@ impl Worker { } } - /// Push a `Method::Notify` entry onto the pending buffer. - /// - /// The notification will be appended to `worker.history` as an - /// `Item::system_message` just before the next LLM request, via - /// `WorkerInterceptor::pending_history_appends`. See [`NotifyBuffer`] - /// for overflow behaviour and the lane-of-record rationale. /// Configure the fixed idle notification coalescing window. /// /// The deadline is derived from the oldest durable notification receipt, @@ -4613,6 +4607,12 @@ impl Worker { Ok(()) } + /// Resolves Submit's Flow selector and prepares initial state and instruction projections. + /// + /// Rejects duplicate active instances, multiple Flow segments, disabled Features, and Workspace + /// mismatches. Source resolution is read-only and does not yet change the live instance. + /// The caller commits typed input, the instruction history projection, and Flow state in one + /// AnnotatedUserInput before updating in-memory state. fn prepare_flow_input( &self, input: Vec, @@ -4932,6 +4932,16 @@ impl Worker { self.handle_worker_result(result, history_before).await } + /// Processes committed Submit invocations in segment order and returns whether dependent LLM + /// execution may proceed. + /// + /// Persists start in the pending activation extension before calling the handler, then appends + /// the result and updated receipt in one AnnotatedSystemItem. If result append fails, start + /// remains, preventing reexecution of uncertain side effects within this process or after + /// restore, Pause/Resume, or compaction. Exact replay uses saved results but revalidates semantics + /// and authority before reusing successful context. Receipt exhaustion rejects untrackable + /// execution rather than dropping old evidence. The first failure/unknown result stops later + /// invocations and dependent LLM execution without rolling back earlier side effects. async fn execute_feature_invocations(&self, segments: &[Segment]) -> Result where St: Clone + 'static, @@ -6352,8 +6362,18 @@ impl Worker { Ok(()) } - /// Runs one parent-owned observable compaction service and returns the new - /// Segment ID. Lifecycle states are committed before they are broadcast. + /// Saves a new Segment and switches history references through parent-owned compaction. + /// + /// Completes pre-rewrite hooks and confirms background tasks stopped, then saves the summary + /// and retained tail in a new Segment. Creation alone does not change current history; CAS of + /// the metadata active reference is the commit point. Lock order is append barrier, allocation, + /// then metadata, excluding Submit/Notify during the switch. If allocation update fails after + /// CAS, rejects appends and retains exclusion while reporting the mismatch. Success aligns the + /// append destination, history, and cache identity before publishing the new base to subscribers. + /// + /// Confirms release of the compaction Internal Worker before activation. Cancellation or future + /// drop alone does not prove release; unconfirmed cleanup blocks the next execution. Progress + /// is a runtime observation, not a restartable durable workflow record. pub async fn compact(&mut self, retained_tokens: u64) -> Result { self.compact_with_cancel(retained_tokens, None, CompactionTrigger::RequestThreshold) .await diff --git a/crates/workspace-drive/README.md b/crates/workspace-drive/README.md new file mode 100644 index 000000000..3dc2019b0 --- /dev/null +++ b/crates/workspace-drive/README.md @@ -0,0 +1,10 @@ +# workspace-drive + +Stores the latest Workspace-owned document tree using SQLite metadata and immutable blobs. +Does not provide a POSIX filesystem, document version history, or Worker/Session-owned attachments. The Host handles HTTP authentication and grants. + +## Main entry points + +- [`Drive`]: document tree, reads, collection, and deletion +- [`Mutation`], [`Drive::mutate`]: updates based on observed mutations +- [`RequestStatus`], [`Drive::request_status`]: committed operation results diff --git a/crates/workspace-drive/src/blob.rs b/crates/workspace-drive/src/blob.rs index 766448b39..b5a38f336 100644 --- a/crates/workspace-drive/src/blob.rs +++ b/crates/workspace-drive/src/blob.rs @@ -1,10 +1,12 @@ -//! Synchronous, immutable local blobs for the SQLite drive authority. +//! Synchronous, immutable local blobs for SQLite Drive authority. //! -//! Call these methods on blocking threads, while holding the authority's SQLite -//! IMMEDIATE transaction. That transaction, not this adapter, serializes reads, -//! uploads and GC across processes. The root and its ancestors must be owned by -//! trusted workspace code: path checks reject existing symlinks, but the -//! path-based LocalFileSystem API cannot prevent hostile concurrent replacement. +//! The caller's IMMEDIATE transaction, not this adapter, provides inter-process exclusion for readers, uploads, and GC. +//! `LocalFileSystem` Create publication alone does not guarantee file/directory sync. Sync the published file, +//! then its directory, before returning success to the metadata transaction. Unreferenced blobs left by failures are eligible for GC. +//! +//! The Host protects the root and ancestors. Existing symlinks are rejected, but this path-based API is not +//! a sandbox against concurrent path replacement by a hostile local actor. Requires a local persistent filesystem +//! with working file/directory sync and SQLite WAL locks; sync failures never silently weaken durability. use std::collections::BTreeSet; use std::fs::{self, File}; diff --git a/crates/workspace-drive/src/lib.rs b/crates/workspace-drive/src/lib.rs index c1551aa1f..cfaad86f8 100644 --- a/crates/workspace-drive/src/lib.rs +++ b/crates/workspace-drive/src/lib.rs @@ -1,6 +1,5 @@ -//! Workspace-owned latest-version storage. Call these blocking operations on a -//! blocking thread, never on an async executor. SQL IMMEDIATE transactions are -//! the cross-process authority for uploads, tree changes, reads, and collection. +#![doc = include_str!("../README.md")] + mod blob; mod migrations; use migrations::MIGRATIONS; @@ -83,13 +82,16 @@ pub enum Kind { } #[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] +/// Public projection of a node with a stable Workspace-local ID. +/// IDs are never reused after deletion and use decimal strings in JSON to preserve JavaScript precision. +/// Blob keys and physical paths are excluded from the public projection. pub struct Node { pub id: NodeId, pub workspace_id: String, pub parent_id: Option, pub name: String, pub kind: Kind, - /// Request ID of the last committed mutation; empty only for the immutable root. + /// Request ID of the last committed mutation. Only the immutable root has an empty value. pub last_mutation_id: String, pub size: u64, pub content_type: Option, @@ -99,6 +101,9 @@ pub struct Node { #[derive(Clone, Debug, Serialize, Deserialize)] #[serde(tag = "operation", rename_all = "snake_case", deny_unknown_fields)] +/// Document mutations based on an observed commit request ID. +/// Update/Relocate/Delete compare the expected mutation ID with the current value in the same transaction. +/// Conflicts use neither counters nor timestamps; only one concurrent mutation based on the same observation commits. pub enum Mutation { CreateFolder { parent: NodeId, @@ -137,8 +142,8 @@ pub struct MutationResult { #[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] #[serde(tag = "state", rename_all = "snake_case")] pub enum RequestStatus { - /// No committed result in this authority snapshot. A concurrently running - /// request may still commit; retry with the SAME identity and fingerprint. + /// No committed receipt exists at this observation point. + /// Does not prove an in-flight concurrent request will not commit; retry with the same identity and payload. Uncommitted, Committed { result: MutationResult, @@ -180,6 +185,14 @@ pub struct CollectionPage { type Authorizer = dyn Fn(&Connection, bool) -> Result<()> + Send + Sync; +/// Workspace-scoped storage of the latest document tree. +/// SQLite is authoritative for the tree and latest blob references; names/parents never determine blob paths. +/// The root is immutable, names undergo no normalization or case folding, and ordinary folder deletion requires an empty folder. +/// +/// Uploads, tree mutations, reads, and GC serialize through the same IMMEDIATE transaction. +/// Processes must share the same metadata DB and blob root; distributed writes through independent DBs are unsupported. +/// Operations are synchronous, so async Hosts use blocking threads. +/// No mechanism snapshots DB and blobs together; metadata backup alone cannot restore content. #[derive(Clone)] pub struct Drive { authorizer: Option>, @@ -217,10 +230,11 @@ impl Drive { _storage: storage.clone(), }) } - /// Trusted Server attachment: SQLite IMMEDIATE also locks this authority, - /// so a cached Drive handle cannot race deletion reservation in server.db. - /// This is Host configuration, never a model-selected database path. The - /// attached authority is read-only by convention; Drive never writes it. + /// Attaches a trusted Host's Workspace authority DB to the connection. + /// IMMEDIATE transactions lock authority too, rejecting mutations through cached handles after deletion is reserved. + /// Drive does not modify attached authority or guarantee atomic writes across DBs. + /// Blob I/O also serializes with Host DB writes, so long operations delay other Host writes. + /// The DB path is Host construction input, never model input. pub fn open_with_workspace_authority( storage: &ScopedFeatureStorage, registration: &RegisteredFeature, @@ -257,10 +271,9 @@ impl Drive { })?; Ok(drive) } - /// Bind immutable caller authority to this handle and its clones. The callback - /// runs within every client operation's IMMEDIATE transaction, never at bind - /// time; attached authority stays locked through blob I/O and publication. - /// Host maintenance must use an unbound handle. + /// Binds the caller's authorization callback to the handle and its clones. + /// Runs inside every client operation transaction, not at binding, holding the authority lock through blob I/O and publication. + /// Rechecks receipt replay and result queries too. Host maintenance uses unbound handles. pub fn with_authorizer(&self, authorizer: F) -> Self where F: Fn(&Connection, bool) -> Result<()> + Send + Sync + 'static, @@ -276,8 +289,7 @@ impl Drive { None => Ok(()), } } - /// Early admission check, not a lease: every subsequent operation still - /// invokes its authorizer in that operation's own transaction. + /// Early admission check, not a lease. Subsequent operations recheck authorization in their own transactions. pub fn check_authorized(&self, write: bool) -> Result<()> { self.database.try_transaction(|tx| { self.authorize(tx, write)?; @@ -302,8 +314,8 @@ impl Drive { }) } - /// Keyset pages are ordered by stable ID, not name. Concurrent mutations may - /// appear/disappear between pages. There is no snapshot promise across calls. + /// Keyset page in stable ID order, not name order or a snapshot across calls. + /// Concurrent changes between pages may add, delete, or move entries. pub fn list(&self, parent: NodeId, after: Option, limit: usize) -> Result { validate_limit(limit, MAX_PAGE)?; self.database.try_transaction(|tx| { @@ -329,9 +341,9 @@ impl Drive { }) } - /// Recover the committed result by request identity, including pre-cutover - /// receipts whose payload cannot be verified for replay. Committed status is - /// not proof that a different actor/payload matches that request. + /// Reads the original committed result by request identity. Receipts survive later node deletion. + /// Results remain readable even if the legacy receipt's payload cannot be proven for replay, + /// but their existence never authorizes an operation with a different actor/payload. pub fn request_status(&self, request_id: &str) -> Result { validate_identity(request_id, "request_id", 128)?; self.database.try_transaction(|c| { @@ -352,12 +364,11 @@ impl Drive { }) } - /// DB commit is publication. The blob and receipt are never independently - /// treated as a successful mutation. Successful request IDs are retained for - /// the Workspace lifetime; reuse with a different actor/payload is conflict. - /// Pre-cutover replay checks the original typed-payload digest using proven - /// receipt correspondence. If that evidence is unavailable, returns Conflict - /// without executing; recover the committed result through request_status. + /// Publishes metadata updates and a receipt in one DB commit after syncing blobs. + /// Saving a blob alone is not success. New blobs left by sync/commit failures stay unreferenced and eligible for GC. + /// Request IDs persist for the Workspace lifetime; replay with the same actor/payload returns the original result. + /// The actor is trusted input from the authentication adapter; reusing an ID with a different actor/payload conflicts. + /// Legacy replay uses only the original payload digest and proven correspondence; unknown correspondence conflicts without execution. pub fn mutate( &self, request_id: &str, @@ -428,9 +439,9 @@ impl Drive { }) } - /// Each chunk requires the originally observed committed mutation. Old blobs are - /// not retained: after update/delete return conflict/not-found, never switch - /// silently to the latest blob. GC cannot run while this read holds SQL lock. + /// Pins each chunk to the observed commit request, serializing with uploads/GC via SQL locks. + /// Old blobs are not retained as history; updates/deletions return conflict/not-found rather than switch to latest content. + /// Checks referenced blob existence and total length even for empty files; missing blobs are not normal empty content. pub fn read( &self, id: NodeId, @@ -475,8 +486,8 @@ impl Drive { }) } - /// UTF-8 text over byte offsets. Callers must select character-aligned - /// ranges; invalid UTF-8 (including a split character) is an explicit error. + /// Reads a byte-offset range as UTF-8. + /// Ranges splitting character boundaries or invalid UTF-8 return explicit errors, never implicit correction. pub fn read_text( &self, id: NodeId, @@ -497,9 +508,9 @@ impl Drive { }) } - /// Search examines at most `budget` nodes after the cursor. Text inspection - /// is limited to UTF-8 text/* files <=64KiB, and matches are bounded by budget. - /// An empty matching page may still carry a continuation cursor. + /// Search with a budget limiting nodes examined after the cursor. + /// Text inspection covers only small UTF-8 text/* blobs, never unbounded reads of large blobs. + /// The cursor is the last examined ID, so continuation may exist even with no matches. pub fn search( &self, query: &str, @@ -535,9 +546,10 @@ impl Drive { }) } - /// Collection shares the exact database lock with uploads/readers. No grace - /// period or process-local mutex is used as a safety proof. Repeat full sweeps - /// from None, since new random keys can sort before a previous cursor. + /// Holds the same IMMEDIATE lock as readers/uploads and collects only unreferenced blobs and interrupted staging files. + /// Includes blobs left by rollback after a crash; relies on neither grace periods nor in-process mutexes alone. + /// Local listing scans the whole directory, but responses and retained candidates are bounded. + /// New UUIDs may sort before the cursor, so future sweeps start at the beginning. pub fn collect(&self, after: Option<&str>, limit: usize) -> Result { validate_limit(limit, MAX_PAGE)?; self.database.try_transaction(|tx| { @@ -564,14 +576,16 @@ impl Drive { }) }) } + /// Commits a persistent tombstone rejecting new operations. Receipt reads/replays never authorize new mutations. pub fn fence(&self) -> Result<()> { self.database.try_transaction(|tx| { tx.execute("UPDATE drive_state SET deleting=1 WHERE singleton=1", [])?; Ok(()) }) } - /// Retryable purge. Fence is committed first. Root and deletion tombstone - /// remain until Host FeatureStorage deletes the metadata scope. + /// Commits the fence first, then removes this Workspace's blobs, nodes, and receipts in a retryable manner. + /// Retains the root and tombstone until the Host deletes the FeatureStorage scope. + /// A Host lifecycle operation distinct from ordinary folder deletion; never processes other Workspaces' storage roots. pub fn purge(&self) -> Result<()> { self.fence()?; self.database.try_transaction(|tx| { diff --git a/crates/workspace-drive/src/migrations.rs b/crates/workspace-drive/src/migrations.rs index cf28d1c27..fd2e584fe 100644 --- a/crates/workspace-drive/src/migrations.rs +++ b/crates/workspace-drive/src/migrations.rs @@ -1,4 +1,8 @@ -//! Frozen schema upgrades. Old column/JSON names occur only to preserve existing data. +//! Frozen schema conversions preserving existing records. +//! +//! Binds current nodes to receipt-derived commit requests, preserving original fingerprints, +//! receipts for deleted nodes, and ID allocation history. Missing or ambiguous current-state correspondence rolls back the entire migration. +//! Legacy representations are used only for exact replay validation against original requests, never as authority for new mutations. use crate::{Mutation, NodeId}; use feature_storage::FeatureMigration; use rusqlite::{OptionalExtension, Transaction}; diff --git a/crates/workspace-server/README.md b/crates/workspace-server/README.md new file mode 100644 index 000000000..bbffda327 --- /dev/null +++ b/crates/workspace-server/README.md @@ -0,0 +1,11 @@ +# workspace-server + +Backend that manages persistent Workspace resources and authorization and provides an HTTP API. Coordinates Ticket, Objective, and Worker assignments, Workdir links, and Backend Jobs. Runtime and the controller own Worker execution state and input admission; this crate implements neither the product CLI nor the execution engine. + +## Design entry points + +- [`server`] — HTTP admission and resource operation coordination +- [`store`] / [`authority`] — SQLite persistence and Workspace resource projections +- [`hosts`] / [`worker_source`] — Runtime delegation and request source verification +- [`backend_job`] — Immutable input, attempt, and structured result boundaries +- [`workspace_catalog`] — Workspace creation and replay contracts diff --git a/crates/workspace-server/src/auth.rs b/crates/workspace-server/src/auth.rs index aa202f2de..286e2ced8 100644 --- a/crates/workspace-server/src/auth.rs +++ b/crates/workspace-server/src/auth.rs @@ -1,3 +1,12 @@ +//! Resolves Backend request actors from human credentials. +//! +//! User identifies the human and Account the Workspace owner; request actors retain both. +//! The store resolves valid API tokens or browser sessions, checking expiry and User existence. +//! Tokens are matched by hash; session cookies use HttpOnly/SameSite. +//! This resolution does not authorize individual Workspace resources. Server admission points +//! and domain services check ownership, Origin, and operation permissions. worker_source +//! separately verifies Runtime signing proofs. + use axum::http::{HeaderMap, header}; use chrono::{DateTime, Duration, Utc}; pub use server_api::{ActorAuthMethod, AuthPublicConfig, AuthenticatedUser, RequestActor}; diff --git a/crates/workspace-server/src/authority.rs b/crates/workspace-server/src/authority.rs index 8f8001af2..02018b96c 100644 --- a/crates/workspace-server/src/authority.rs +++ b/crates/workspace-server/src/authority.rs @@ -1,3 +1,17 @@ +//! Projects Workspace Tickets, Objectives, and Memory from domain records to API resources. +//! +//! Normal Backend reads use Workspace-scoped SQLite/domain authority, not a Ticket backend +//! chosen from Worker cwd or legacy repository file layouts. The ticket crate owns Ticket state, +//! relations, and content; the Workspace store owns assignments and Runtime Worker links. +//! Builds bounded detail, thread, relation, assignment, and evidence projections, not full +//! execution sessions as substitute records for domain decisions. +//! +//! Unobservable open MR sources are unknown; past review refs do not imply current commits. +//! Evidence filters use current projections excluding closed MRs: commit requires a settled source; +//! approved_review requires approval tied to the current Ticket content digest and all target MR +//! sources/results. Merged MRs use stored integration evidence, so provider outages alone do not +//! invalidate approval. + use std::{path::PathBuf, sync::Arc}; #[cfg(test)] diff --git a/crates/workspace-server/src/backend_job.rs b/crates/workspace-server/src/backend_job.rs index 5ec55f31b..055d3d069 100644 --- a/crates/workspace-server/src/backend_job.rs +++ b/crates/workspace-server/src/backend_job.rs @@ -1,9 +1,36 @@ -//! Durable Backend-owned jobs executed by ordinary Runtime-managed Workers. +//! Backend-owned Jobs with fixed input and execution limits. //! -//! A job is durable intent. An attempt binds that intent to one dedicated -//! Worker, and a result is accepted only through the fenced structured-result -//! API below. Runtime lifecycle and final prose are deliberately not success -//! authority. +//! Jobs retain immutable input and final results. Each attempt binds one trial to a dedicated +//! Runtime Worker, input digest, tracked submission identity, and deadline. Requesting and +//! notification Workers are sources and destinations, not parents; stopping or removing them +//! does not cancel the Job. Worker retention and execution lifecycle follow existing Runtime +//! and Workspace resource management. +//! +//! # Replay and reevaluation +//! +//! Replaying the same Job ID and fingerprint returns the current attempt; ID reuse with different +//! input is rejected. Attempts without execution slots remain durably reserved; dispatch advances +//! on completion and restart. Only explicit reevaluation adds attempts. The store checks confirmed +//! failure, remaining attempt budget, completed cleanup of prior attempt Workers, and no occupancy +//! by other Jobs. Unknown execution outcomes are not rerun as new attempts. +//! +//! Serialization-key resource conflicts return the original immutable request as resource_reused. +//! This does not mean new input was processed. Pending/unknown Jobs and unfinished cleanup +//! retain resource occupancy, distinct from replay of an identical request. +//! +//! # Structured results and notifications +//! +//! Result admission matches the authenticated Worker, Workspace, Job, current attempt, input +//! digest, and tracked run, then stores bounded JSON. Identical result replay is idempotent; +//! differing results and stale attempts are rejected. Final prose, Idle, Stopped, and successful +//! model calls are not evidence of Job success. Submission authority comes from the Job binding +//! fixed by the Host at creation/Restore, not a Profile name or purpose alone. +//! +//! Notifications have delivery records separate from results. After restart, unknown deliveries +//! are retried with the same tracked notification ID. Results survive stopped/missing destinations +//! without rerunning the model. Execution recovery matches observable Workers and tracked +//! submissions; lost evidence leaves the outcome unknown. Exactly-once execution of all external +//! side effects is not guaranteed. use serde::{Deserialize, Serialize}; use serde_json::Value; diff --git a/crates/workspace-server/src/feature_storage.rs b/crates/workspace-server/src/feature_storage.rs index b4386fd07..d1c84e974 100644 --- a/crates/workspace-server/src/feature_storage.rs +++ b/crates/workspace-server/src/feature_storage.rs @@ -1,2 +1,5 @@ -//! Backend compatibility facade for the neutral Host storage manager. +//! Compatibility boundary re-exporting shared Host feature storage management to the Backend. +//! +//! The feature-storage crate owns storage registration and migrations. This module adds neither +//! a Backend-specific storage lifecycle nor a separate schema authority. pub use feature_storage::*; diff --git a/crates/workspace-server/src/hosts.rs b/crates/workspace-server/src/hosts.rs index 23e09f96f..7b3641255 100644 --- a/crates/workspace-server/src/hosts.rs +++ b/crates/workspace-server/src/hosts.rs @@ -1,3 +1,17 @@ +//! Adapter delegating Workspace operations to embedded/remote Runtimes. +//! +//! RuntimeRegistry selects execution owners by Runtime/Worker identity and converts typed +//! requests and results. The Server owns Workspace assignments, Workdir links, and retention; +//! Runtime owns execution endpoints and lifecycle settlement. Display names, socket/session +//! paths, and catalog state do not grant operation authority. +//! +//! Remote requests match the selected Workspace binding against current configuration before +//! issuing short-lived capabilities with the current Workspace signing identity. A binding match +//! selects a destination, not proof of authentication. Stale clients with replaced/revoked bindings +//! are rejected. Method, path/query, body digest, permission, and target identity are request-bound; +//! verification follows Runtime's current Workspace trust. Past connection tests are not reused +//! as authentication state. + use crate::Error; use crate::resource_broker::BackendResourceBroker; #[cfg(test)] diff --git a/crates/workspace-server/src/lib.rs b/crates/workspace-server/src/lib.rs index fefe24d92..5d98d2fbd 100644 --- a/crates/workspace-server/src/lib.rs +++ b/crates/workspace-server/src/lib.rs @@ -1,8 +1,4 @@ -//! Local workspace web control plane backend bootstrap. -//! -//! This crate deliberately provides backend building blocks and an HTTP router; -//! it is not the product CLI facade. Tickets and Objectives are served through -//! Backend authority surfaces rather than Worker-local filesystem access. +#![doc = include_str!("../README.md")] pub mod auth; pub mod authority; diff --git a/crates/workspace-server/src/server.rs b/crates/workspace-server/src/server.rs index 29a08144c..90b6f6a7a 100644 --- a/crates/workspace-server/src/server.rs +++ b/crates/workspace-server/src/server.rs @@ -1,3 +1,25 @@ +//! Workspace HTTP admission and coordination of domain resource operations. +//! +//! The catalog dispatcher acquires a Workspace lifecycle lease and verifies a signing +//! proof or human actor before entering the inner router. Proof verification precedes +//! resource lock waits; a deleting Workspace rejects reads too. Accepted handlers are +//! not reauthenticated merely because the proof expires. Subsequent resource/provider +//! operations follow their owners' current constraints. +//! +//! Shared Worker admission lives in worker_operations; deletion draining lives in +//! workspace_admission. Generated contract routers and registered WS routes define +//! production entry points. Test helpers do not imply unauthenticated production routes. +//! +//! # Tickets and Orchestrators +//! +//! Ticket operations persist through the domain backend. Current Workspace role assignments +//! constrain Worker-origin state decisions. Queue and event notifications do not imply +//! implementation completion. Runtime Worker spawning, Workdir linking, and Ticket assignment +//! use their respective durable operations. Orchestrator backlog attention rereads current +//! Tickets and interested assignments at startup, Restore, and turn end. Subscription snapshots +//! and reconnection are not turn-end events; Running-to-Idle events are handled separately. +//! This coordinates notifications, not a general event queue or worktree layout policy. + use subjektiv::api::*; mod backend_job_profile; mod subjektiv_job; @@ -847,6 +869,13 @@ struct ExternalProviderAuditContext { attachment_alias: String, } +/// Workdir session bound to one External provider connection generation. +/// +/// Command handles map to provider handles within this session, never a new generation +/// or reconnected provider. Closing on replacement/detach requests cancellation and terminal +/// output confirmation from the old connection; unconfirmed cleanup is not reported as success. +/// Only confirmed provider shutdown can replace per-command confirmation. Reconnection does +/// not prove that missing commands have stopped. struct ExternalProviderWorkdirSession { workdir: workdir::Workdir, connection: Arc, @@ -2605,6 +2634,13 @@ fn repository_api_rejection(path: &str, status: StatusCode, message: &str) -> Re // Log here, not only in the inner Workspace router: the catalog dispatcher can // reject authentication before that router (and its API error logger) is entered. +/// Records outer-dispatcher rejections using the same safe internal diagnostics. +/// +/// Claim-derived identities are emitted as claimed hashes. Method, path without query, +/// and Workspace are request metadata with control-character and length limits. The generated +/// request ID identifies this rejection, not a caller ID or cross-service trace. HTTP responses +/// remain generic 401s without proof details. Classification describes this verification result, +/// not the cause or resolution of earlier intermittent 401s. fn runtime_request_proof_rejection( method: &str, path: &str, @@ -5245,6 +5281,14 @@ impl WorkspaceApi { Some(summary) } + /// Binds observation-based Restore intent to a Runtime-owned operation. + /// + /// First queries existing owners/receipts without preparation, handling replay and recovery + /// before new-launch eligibility. New intent rereads singleton, Workdir, and External grants + /// to build references. SSH resources are issued only after Runtime admits the same observation + /// token under its operation lock. Replay does not replace saved preparation; incomplete auth + /// preparation remains ReconciliationRequired. Runtime owns lifecycle settlement, while the + /// returned summary comes from current observation. fn restore_workspace_worker( &self, worker: &RuntimeWorkerRef, @@ -5658,6 +5702,12 @@ fn persist_external_workdir_permissions( } } +/// Converts External grants into session capabilities using READ, WRITE, COMMAND levels. +/// +/// WRITE includes READ; COMMAND includes WRITE. Arbitrary independent boolean combinations +/// are not permissions: legacy read+command without write is rejected. Provider registration +/// must match the grant's capabilities; Worker attachments intersect permissions with that ceiling. +/// Write scope alone does not create command permission. fn external_workdir_capabilities(permissions: &str) -> Result { match external_workdir_permissions(permissions)?.grant_level() { Some(server_api::ExternalWorkdirPermissionLevel::Read) => { @@ -26147,6 +26197,12 @@ async fn post_passkey_registration_options( })) } +/// Consumes a stored WebAuthn ceremony and registers only verified credentials. +/// +/// Binds the challenge to User, RP, Origin, and stored verifier state, rejecting consumed or +/// expired ceremonies. Browser assertions alone do not create keys or sessions: credentials +/// are saved and browser sessions issued only after webauthn-rs verification succeeds. +/// Login also verifies a stored ceremony. fn complete_passkey_registration( api: &ServerAuthApi, request: PasskeyRegistrationCompleteRequest, @@ -26485,6 +26541,12 @@ fn approve_device_login( }) } +/// Returns the API token generated by browser approval to the device client. +/// +/// Pending and expired flows return no credentials. Approved flows consume and return the +/// token, clearing the store's temporary token so normal repeated polling yields Consumed. +/// This does not guarantee retrieval of the same token after response loss and is separate +/// from browser-cookie authentication. async fn post_device_login_poll( State(api): State, Json(request): Json, diff --git a/crates/workspace-server/src/server/backend_job_profile.rs b/crates/workspace-server/src/server/backend_job_profile.rs index 4c8d155dc..26c6d54ed 100644 --- a/crates/workspace-server/src/server/backend_job_profile.rs +++ b/crates/workspace-server/src/server/backend_job_profile.rs @@ -1,4 +1,11 @@ -//! Profile recipes are selected by the caller. Execution grants remain Host owned. +//! Backend Job Profile selection and explicit grant checks. +//! +//! Resolves the caller's registry Profile from the Workspace virtual config bundle and Prompt +//! projection, using public catalog recipes separately from browser launch filtering. Raw sources +//! and path selectors are rejected; instructions/features are not replaced with another default +//! recipe. Recipes requiring Workspace mutation, Memory, child/peer, Workdir, Flow, or MCP authority +//! absent from the Job are rejected. subjektiv also checks explicit consolidation grants and feature +//! requirements; Profile selection alone creates no domain authority. use super::*; impl WorkspaceApi { diff --git a/crates/workspace-server/src/server/worker_operations.rs b/crates/workspace-server/src/server/worker_operations.rs index fe84fb7f2..157fd840e 100644 --- a/crates/workspace-server/src/server/worker_operations.rs +++ b/crates/workspace-server/src/server/worker_operations.rs @@ -1,4 +1,35 @@ -//! Workspace identity-bound operation reception. Runtime/controller remain execution authority. +//! Worker operation admission based on Workspace registration. +//! +//! [`WorkspaceWorker`] fixes Runtime/Worker identity; it is not a lease guaranteeing a live +//! execution or authorization. Each operation rereads registration and revalidates the request's +//! Account or Worker control grant before delegating to Runtime. The controller determines actual +//! Idle/Busy state and input admission; catalog state alone cannot determine execution eligibility. +//! +//! # Caller and input boundaries +//! +//! The Server builds [`WorkerOperationContext`] from authentication results. Worker control locks +//! per grant and rechecks the same grant ID and permissions after waiting. Account requests reread +//! credentials, Account, and active Workspace, also checking browser session Origin. Test-only +//! Backend contexts do not exist in production. +//! +//! Worker control converts User input to UserIfIdle, never queuing Submit for a running Worker. +//! Notify requires separate permission. Normal input is denied to Backend Job Workers and Workers +//! with unsettled Ticket claims. Only Runtime Accepted Stop closes the current Workdir session +//! and synchronizes attachments. WorkerRemove is a separate contract: it verifies a target-bound +//! proof, then rechecks grants and retention in the removal service. +//! +//! # Connections and results +//! +//! Dedicated and multiplexed WS use the same [`WorkspaceWorkerMethodSender`]. Connections check +//! observe; method sends check send_input and browser context. Successful sends mean transport +//! channel enqueue, not controller acknowledgment or durable Submit/Notify receipts. +//! Actual results arrive through the event stream. Connections bind to one execution and do not +//! implicitly reconnect or Restore into the next execution with the same Worker ID. +//! +//! Unary input lacks the WS Submit replay-key contract; unknown delivery outcomes remain diagnostic. +//! Restore observation tokens, controller command IDs, Submit/Notify request IDs, and cleanup plan +//! digests have separate verifying owners, not a shared success criterion or replay contract. +//! Internal SubWorkers are parent Worker sessions, outside this Workspace registry handle. use super::*; /// Caller authority supplied by authenticated Server adapters, never public JSON/identity hints. diff --git a/crates/workspace-server/src/server/workspace_admission.rs b/crates/workspace-server/src/server/workspace_admission.rs index 06c94b9a8..1769c8224 100644 --- a/crates/workspace-server/src/server/workspace_admission.rs +++ b/crates/workspace-server/src/server/workspace_admission.rs @@ -1,11 +1,19 @@ -//! Workspace lifecycle admission, not operation serialization. +//! Admission gate coordinating Workspace deletion with HTTP handler lifetimes. //! -//! Resource owners retain their own locks/transactions. Deletion first stops -//! ordinary admission, then drains admitted requests. Synchronous Runtime -//! callbacks remain admissible while a parent request is still running; once -//! the last ordinary request leaves, callback admission also closes. This -//! avoids both callback/deletion cycles and an unbounded stream of new work -//! starving deletion. All leases, including the deletion fence, are RAII. +//! Normal requests acquire leases immediately; independent handlers can run concurrently. +//! Resource owners handle Worker, Workdir, and SQLite exclusion; this gate does not serialize them. +//! Business-operation lock waits do not precede proof verification, avoiding expiry of short-lived +//! proofs while waiting for other requests. +//! +//! Deletion closes normal admission before draining existing handlers. Only Runtime callbacks +//! needed by active parent requests may acquire tracked leases; callback admission also closes +//! when the last normal lease leaves. This prevents deletion blocking callbacks awaited by parents +//! and callbacks indefinitely extending the drain. Callbacks still require authentication and +//! authorization. +//! +//! Drop releases leases and temporary deletion fences. Cancelling a deletion wait reopens normal +//! admission, but durable Workspace state blocks admission after deletion reservation succeeds. +//! Leases cover HTTP responses, not the full lifetime of upgraded WS or background work. use std::sync::{Arc, Mutex}; use tokio::sync::{Mutex as AsyncMutex, Notify, OwnedMutexGuard}; diff --git a/crates/workspace-server/src/server_drive.rs b/crates/workspace-server/src/server_drive.rs index 57c79ef7f..71291f6a8 100644 --- a/crates/workspace-server/src/server_drive.rs +++ b/crates/workspace-server/src/server_drive.rs @@ -1,5 +1,9 @@ //! Authorized Workspace document API. Storage, CAS and receipts stay in workspace-drive. //! HTTP upload consumes frames with a hard bound; DB publication occurs only after validation. +//! +//! HTTP errors and stream failures use fixed public representations, without Host storage paths +//! or raw diagnostics. Unknown mutation outcomes are OutcomeUnknown; response failure does not +//! prove the mutation was uncommitted. use super::*; use futures::StreamExt; use server_api::*; diff --git a/crates/workspace-server/src/server_drive_grants.rs b/crates/workspace-server/src/server_drive_grants.rs index 9d85eaf67..c593ad8e4 100644 --- a/crates/workspace-server/src/server_drive_grants.rs +++ b/crates/workspace-server/src/server_drive_grants.rs @@ -1,5 +1,9 @@ -//! Drive authority is rechecked inside the metadata transaction against the -//! attached Server DB. A bound handle is not an authorization cache or lease. +//! Authorizes Drive operations against current grants in the connected Server DB. +//! +//! Checks before upload body reception only admit the request; handles retain no authority. +//! Each operation's metadata transaction rechecks Worker liveness, unrevoked grants, and write +//! permission. Human requests also check current User/Account and active Workspace. Incomplete +//! or contradictory Worker/Runtime sources do not fall back to Account authority. use super::*; use rusqlite::params; use server_api::{ diff --git a/crates/workspace-server/src/server_workspace_config.rs b/crates/workspace-server/src/server_workspace_config.rs index 0e0a54959..5bec7ce6b 100644 --- a/crates/workspace-server/src/server_workspace_config.rs +++ b/crates/workspace-server/src/server_workspace_config.rs @@ -3,6 +3,12 @@ //! Files support read/edit/write and non-entrypoint deletion; directories support //! create, with atomic change sets (including non-entrypoint renames) on the root. //! The canonical main.dcdl entrypoint cannot be deleted or renamed. +//! +//! Worker operations reread unrevoked grants and current attachment connection IDs after acquiring +//! the session lock. Observation validators bind to grant, effective access, connection, config +//! content, and schema, rejecting stale observations or writes after grant replacement. +//! Write permission is the intersection of grant and attachment permissions; authorization at +//! handle acquisition is not ongoing authority. use super::*; use server_api::{ WorkspaceConfigAccess as Access, WorkspaceConfigApiError as Failure, diff --git a/crates/workspace-server/src/store.rs b/crates/workspace-server/src/store.rs index 922884608..b7c2b6dc3 100644 --- a/crates/workspace-server/src/store.rs +++ b/crates/workspace-server/src/store.rs @@ -1,3 +1,31 @@ +//! SQLite persistence and schema consistency for Workspace resources. +//! +//! # Startup and migrations +//! +//! Normal open, migration preflight, and explicit application share connection preparation. +//! Preflight copies a read-only source into memory and exercises the full migration without +//! changing the source DB. Existing DBs apply retained migrations in order; new DBs use the +//! latest canonical schema. A partially committed migration sequence is a valid history prefix +//! from which the next attempt resumes. +//! +//! Schema history must match the retained baseline's version/name sequence. Unknown, modified, +//! misordered, or unsupported histories are rejected. Workspace table rebuilds across domains +//! align with Ticket/Merge Request validation and ordering, then validate resource references +//! and foreign keys. Recovery requirements, such as unresolved legacy deletion receipts, block +//! migration before it starts. Updating only the new schema would diverge from existing DBs; +//! both creation paths must maintain the same contract. +//! +//! # Resource state and operation records +//! +//! Workspace bootstrap commits resources and creation receipts in one transaction. Jobs and +//! Workdir deletion span Runtime/providers, so immutable intent and evidence of external results +//! are stored separately. The Worker registry retains persistent identity, assignment, retention, +//! and observation projections; it does not replace controller input admission or Runtime lifecycle. +//! +//! The internal Restore journal records client intent with a request ID and observation token +//! fixed before sending. Unknown outcomes and restarts reuse that pair, settling on a confirmed +//! result. This is not a second lifecycle state machine duplicating Runtime operations or receipts. + use std::collections::{BTreeSet, HashMap, HashSet}; use std::path::Path; use std::sync::{Arc, Mutex}; diff --git a/crates/workspace-server/src/ticket_item_checker.rs b/crates/workspace-server/src/ticket_item_checker.rs index 5736d8a59..4fffb8a35 100644 --- a/crates/workspace-server/src/ticket_item_checker.rs +++ b/crates/workspace-server/src/ticket_item_checker.rs @@ -1,7 +1,20 @@ -//! Post-save Ticket item checks executed through the Backend-owned Job runner. +//! Asynchronous advisory checks of saved Ticket content using Backend Jobs. //! -//! The immutable input contains only the persisted Ticket item and the edit that -//! produced it. Conversation history is deliberately outside this boundary. +//! Authenticated Worker Create/EditItem paths freeze check input and reserve a Job after saving. +//! Failed preparation, execution, or delivery does not roll back the save or change Ticket state, +//! Queue, or assignments. Direct browser edits have no notification Worker and start no check. +//! +//! Input is limited to saved title/body, content digest, and the prior body/replacement context +//! needed for that edit, excluding conversation, user agreements, and external facts. Content +//! digests identify advisory freshness, not Job intent. Identical content with different edit +//! context or sources creates different Jobs; oversized input is not truncated and marked checked. +//! +//! Results are checked for unknown fields, duplicates, count/string limits, and quotes that are +//! exact substrings of the checked title/body. Invalid results end as invalid_result. +//! Delivery rereads the Ticket and notifies the original Worker only of findings matching the +//! saved digest. No findings and stale results end delivery with distinct suppression reasons, +//! without redelivery on recovery. Advice is neither an approval gate nor implementation guidance; +//! no findings do not prove the entire specification correct. use std::collections::BTreeSet; diff --git a/crates/workspace-server/src/workdir_create_operations.rs b/crates/workspace-server/src/workdir_create_operations.rs index a41fc8ca7..29bc8fccc 100644 --- a/crates/workspace-server/src/workdir_create_operations.rs +++ b/crates/workspace-server/src/workdir_create_operations.rs @@ -1,3 +1,12 @@ +//! Persists immutable Workdir creation intent and Repository credential candidates. +//! +//! Creation fingerprints identify Repository source, selector, requested Runtime, and display +//! name. Replay retains saved selectors and resolved resources, not current defaults. Store +//! reservations reject differences under the same operation ID; SSH candidates fix identity at +//! selection. Retained generated Workdir IDs and candidates support provider retries, not proof +//! of creation. Creation retries do not start on resources with pending removal; transitions +//! from failed to pending also check the current deletion fence within a transaction. + use rusqlite::{OptionalExtension, TransactionBehavior, params}; use sha2::{Digest, Sha256}; diff --git a/crates/workspace-server/src/workdir_removal.rs b/crates/workspace-server/src/workdir_removal.rs index df0250583..26839dfd3 100644 --- a/crates/workspace-server/src/workdir_removal.rs +++ b/crates/workspace-server/src/workdir_removal.rs @@ -1,10 +1,24 @@ -//! Durable, retryable Backend authority for persistent Workdir removal. +//! Persists durable deletion intent and attempts for Repository-backed Workdirs. //! -//! Runtime cleanup is an external side effect, so callers reserve one immutable -//! operation before invoking the provider. The final registry deletion and -//! operation completion are committed atomically. If a process stops after the -//! provider side effect but before that transaction, recovery re-observes the -//! provider and converges through the same operation. +//! Deletion intent fixes Workspace, Workdir, owning Runtime, Repository/materialization, actor, +//! and reason. Endpoints, host paths, and provider handles do not grant caller deletion authority. +//! External Workdirs follow grant revocation, not this operation. +//! +//! Only one pending operation per Workdir is allowed; attempts are claimed transactionally. +//! Process IDs and startup markers are retained. Recovery reclaims the exact owner/attempt by +//! compare-and-set only after confirming the previous owner is gone; live or unobservable owners +//! are not displaced. In-process locks alone are insufficient. +//! +//! Server execution revalidates stored materialization and current attachments, reservations, +//! assignments, retention, and creation operations, then observes occupancy and cleanliness via +//! Runtime. Dirty resources remain; unknown observations or provider failures retain the registry +//! and record attention/retryability. Only confirmed deletion or exact working_directory_not_found +//! is deletion evidence; registry removal and completed/removed commit in the same transaction. +//! +//! Normal deletion, Workspace deletion, cleanup, and startup recovery use the same reserved Server +//! operation execution. Failed DB commits after provider deletion are reobserved through that +//! operation. Recovery relies on provider identity and results, not an exactly-once guarantee +//! across all interruption points. use chrono::Utc; use rusqlite::{Connection, OptionalExtension, TransactionBehavior, params}; diff --git a/crates/workspace-server/src/worker_source.rs b/crates/workspace-server/src/worker_source.rs index 3d51476d1..f159be54f 100644 --- a/crates/workspace-server/src/worker_source.rs +++ b/crates/workspace-server/src/worker_source.rs @@ -1,3 +1,15 @@ +//! Backend boundary verifying Runtime-origin requests and WorkerRemove source proofs. +//! +//! Runtime signatures are verified against current Workspace bindings and the Backend audience. +//! Hint headers such as Worker IDs do not authenticate. General request proofs and target-bound +//! WorkerRemove proofs are separate contracts; removal services recheck control grants/retention +//! after verification. Embedded removal also consumes Runtime-generated in-process proofs, +//! without falling back to hints. +//! +//! Signature/request/time checks alone do not admit proofs: JTI consumption and current Worker +//! registration, reservations, and singleton ownership are checked too. Diagnostic verification +//! stages show progress, not an authenticated principal. + use std::sync::{Arc, OnceLock}; use std::time::{SystemTime, UNIX_EPOCH}; diff --git a/crates/workspace-server/src/worker_source/runtime_request.rs b/crates/workspace-server/src/worker_source/runtime_request.rs index 5e8b4a619..f4daf9c65 100644 --- a/crates/workspace-server/src/worker_source/runtime_request.rs +++ b/crates/workspace-server/src/worker_source/runtime_request.rs @@ -1,3 +1,19 @@ +//! Runtime request proof verification and secret-free rejection diagnostics. +//! +//! Unverified issuers only select public keys from current Workspace bindings. After matching +//! signature, Backend audience, Workspace, permission, method, path/query, body digest, and time, +//! verification consumes the JTI transactionally and checks Worker registration or creation +//! reservations and singleton ownership. One sampled clock serves time checks and JTI records; +//! the lower-level verifier owns tolerance and ordering. Diagnostics do not recompute rejection +//! with another clock. +//! +//! Rejection reasons expose only safe categories and verification stages. Claim-derived identities +//! use domain-separated hashes, omitted on decode failure. request_verified may still precede +//! replay/registration/ownership checks; it is not an authenticated principal. Diagnostics do not +//! infer key-mismatch causes or absence/revocation distinctions the store does not make. +//! Proofs, raw claims, keys, bodies, and store error text are not retained. Identity hashes are +//! not encryption and do not make guessable IDs secret. + use worker_runtime::auth::{ RuntimeAuthError, RuntimeRequestClaim, RuntimeRequestSourceClaims, RuntimeRequestSourceExpectation, decode_runtime_request_source_claims, diff --git a/crates/workspace-server/src/workspace_catalog.rs b/crates/workspace-server/src/workspace_catalog.rs index 5d9679d9f..7e4aede96 100644 --- a/crates/workspace-server/src/workspace_catalog.rs +++ b/crates/workspace-server/src/workspace_catalog.rs @@ -1,3 +1,20 @@ +//! Workspace creation and replay without requiring a Repository. +//! +//! Workspace identity is independent of Git Repositories and Workdirs. Omitting the initial +//! Repository means no asset, not Git initialization or an inferred substitute. Supplied objects +//! undergo normal validation; invalid input is not omission. The owner is the Account supplied +//! by the authentication adapter. +//! +//! Normalized requests and owners produce fingerprints. After signing material reservation and +//! preparation, the store bootstrap transaction commits Workspace, optional Repository, initial +//! settings, signing identity, and creation receipt. Different input under the same operation key +//! conflicts. Replaying material preparation retains the reserved Workspace/key identities. +//! +//! Replay returns current Workspace/settings, not creation snapshots. With an initial Repository, +//! it returns the asset currently registered under the original key, or None after deletion/rename, +//! without reregistration or initialization. Requests originally omitting Repository replay as +//! None even if assets were added later. + use std::sync::Arc; use chrono::{SecondsFormat, Utc}; diff --git a/crates/workspace-server/src/workspace_drive_host.rs b/crates/workspace-server/src/workspace_drive_host.rs index 410c0c049..84d97f3b7 100644 --- a/crates/workspace-server/src/workspace_drive_host.rs +++ b/crates/workspace-server/src/workspace_drive_host.rs @@ -1,4 +1,9 @@ //! Trusted Host placement and lifecycle for Workspace Drive (no HTTP/DTO surface). +//! +//! Workspace deletion reservations reject even existing handles through transaction serialization +//! with the Server DB. Deletion starts with a durable Drive fence and settles purge tombstones and +//! blob/metadata deletion before deleting FeatureStorage. Blob deletion failures retain +//! FeatureStorage; partial failure is not completion. use std::path::{Path, PathBuf}; use std::sync::atomic::{AtomicBool, Ordering}; diff --git a/crates/workspace-server/src/workspace_signing_identity.rs b/crates/workspace-server/src/workspace_signing_identity.rs index f724f2ca1..059399dcb 100644 --- a/crates/workspace-server/src/workspace_signing_identity.rs +++ b/crates/workspace-server/src/workspace_signing_identity.rs @@ -1,3 +1,14 @@ +//! Consistency of per-Workspace signing identities and private material. +//! +//! The store retains public identities and material references; a separate store manages private +//! material. Workspace creation first reserves immutable Workspace/key/material references and +//! prepares the same material on replay. Bootstrap activates it after checking public key/material +//! agreement. Successful material preparation alone does not commit Workspace creation. +//! +//! Capability issuance selects the current active identity, checks the claims' Workspace, key ID, +//! and fingerprint against it, then signs. Missing/corrupt material does not fall back to a shared +//! Server key. Debug redacts private material; Drop zeroizes private key strings. + use std::fmt; use std::fs::{self, OpenOptions}; use std::io::Write; diff --git a/crates/yoi/README.md b/crates/yoi/README.md index 3bb4b210b..e7888d00e 100644 --- a/crates/yoi/README.md +++ b/crates/yoi/README.md @@ -1,63 +1,10 @@ # yoi -## Role +`yoi` is the product CLI entry point. It handles argument parsing, connection and Profile selection, TUI launch, and assembly of administrative subcommands. Worker execution, transport, and terminal rendering are delegated to libraries. -`yoi` is the product CLI facade. It owns the installed binary name, top-level argument parsing, normal TUI launch, and product subcommands such as `yoi worker` and `yoi ticket`. +## Implementation entry points -## Boundaries - -Owns: - -- product command shape and user-facing CLI entry points -- profile/default selection for ordinary startup -- wiring library crates into the executable -- headless maintenance commands that are part of the product surface - -Does not own: - -- Worker runtime internals (`worker`) -- socket client mechanics (`client`) -- model turn orchestration (`agen`) -- TUI rendering/runtime implementation (`tui`) - -## Design notes - -Keeping product CLI ownership here prevents lower crates from depending on the binary name or parsing top-level arguments. Runtime launch should use typed command boundaries, not shell-command strings. - -## Workspace creation and selection - -`yoi init` remains Git-oriented and requires an explicit initial repository key: - -```sh -yoi --backend https://backend.example init --display-name Development --repository-key platform -``` - -For a Workspace with no initial repository, use the same Backend creation API explicitly: - -```sh -yoi --backend https://backend.example workspace create --display-name Research -``` - -Both commands record Workspace-to-Backend routing in the global client configuration, -not repository-local identity files. Repository-free creation prints the Workspace id; -select it explicitly from any directory, without Git discovery: - -```sh -yoi --workspace-id -yoi --workspace-id workers -yoi --workspace-id resume -yoi --workspace-id panel -``` - -`--workspace-id` selects Backend even when `default_connection` is `local`, and conflicts -with `--local`. An explicit `--backend ` overrides configured routing. Without an -explicit Workspace id, Backend commands retain Git-based Workspace discovery. - -Normal Backend startup launches the existing workdir-free Worker form. Repository and -workdir attachment selection are not prerequisites imposed by this CLI; Runtime -availability and workdir requirements remain Backend/TUI responsibilities. - -## See also - -- [`../../docs/design/overview.md`](../../docs/design/overview.md) -- [`../../docs/design/profiles-manifests-prompts.md`](../../docs/design/profiles-manifests-prompts.md) +- `src/main.rs`: product command parsing and dispatch to launch targets. +- `src/cli_connection.rs`: per-command connection requirements and target resolution. +- `src/workspace_bootstrap.rs`: Workspace creation and Backend routing records. +- [`workdir_share`] — sharing existing Workdirs. diff --git a/crates/yoi/src/cli_connection.rs b/crates/yoi/src/cli_connection.rs index 488969646..573eeafd9 100644 --- a/crates/yoi/src/cli_connection.rs +++ b/crates/yoi/src/cli_connection.rs @@ -1,3 +1,8 @@ +//! Resolves connection targets by matching command requirements with explicit selections. +//! +//! Explicit Workspace identity needs no Git discovery. Target selection is separate +//! from Workspace discovery, allowing Backend selection without a Repository. + use std::path::Path; use client::{BackendTarget, StandaloneTarget, Target, TargetKind}; diff --git a/crates/yoi/src/lib.rs b/crates/yoi/src/lib.rs index 32e2a7501..0c4f95911 100644 --- a/crates/yoi/src/lib.rs +++ b/crates/yoi/src/lib.rs @@ -1 +1,3 @@ +#![doc = include_str!("../README.md")] + pub mod workdir_share; diff --git a/crates/yoi/src/main.rs b/crates/yoi/src/main.rs index 4e4dda79a..5ec7513ea 100644 --- a/crates/yoi/src/main.rs +++ b/crates/yoi/src/main.rs @@ -1,3 +1,8 @@ +//! Parses product commands and dispatches to typed client/TUI/Runtime entry points. +//! +//! Binary-name and top-level argument interpretation stay here, separate from +//! execution in lower-level crates. + mod cli_connection; mod mcp_cli; mod objective_cli; diff --git a/crates/yoi/src/workspace_bootstrap.rs b/crates/yoi/src/workspace_bootstrap.rs index 97ce43aa2..b16091f58 100644 --- a/crates/yoi/src/workspace_bootstrap.rs +++ b/crates/yoi/src/workspace_bootstrap.rs @@ -1,3 +1,9 @@ +//! Workspace creation and Repository mapping through the Backend catalog. +//! +//! Workspaces without a Repository use the same creation API. Connection routing +//! is recorded in global client config; Workspace identity is not persisted in +//! the Repository. + use std::fs; use std::path::{Path, PathBuf}; use std::process::Command; diff --git a/docs/README.md b/docs/README.md index 41f7a97b9..3252caf0c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,66 +1,25 @@ -# Yoi documentation +# 文書の維持管理 -This directory contains maintained developer documentation for Yoi. Its job is to preserve design intent that is hard to recover from code alone. +文書には、コードから読み取れない設計理由と、開発を継続するための規則を残す。利用ガイド、設定説明、実装一覧は維持しない。同じ説明を複数の場所に置かず、変更の責任を持つコードの近くで管理する。 -It is not a dumping ground for external research, old plans, API inventories, or ticket history. Those belong in local notes, work item artifacts, code, or git history. +## 保管場所と責務 -## Reading order +- `principle/`：複数の実装に共通する設計原則。判断が競合したときの基準と、その理由を書く。個別の処理手順やデータ構造は置かない。 +- `development/`：テスト設計、検証、レビューなど、開発を続けるための規則。製品の操作手順や環境設定の説明は置かない。 +- `report/`：問題発生時の報告。試みた操作、観測した事実、影響、原因の確度、未解決事項を記録する。過去の報告は現行仕様の根拠にせず、当時の記録として蓄積できる。 +- Rustのcrate README：責務と非責務を短く示し、主要moduleへの参照を置く。各crateの`lib.rs`は`#![doc = include_str!("../README.md")]`でREADMEをcrateドキュメントとして使う。 +- Rustのmodule・itemドキュメント：所有境界、状態遷移、失敗時の保証と限界、非自明な判断理由を書く。API一覧や実装の逐語説明は繰り返さない。 -1. [`design/overview.md`](design/overview.md) — the system map. -2. [`design/durable-operations.md`](design/durable-operations.md) — cross-domain operation identity, checkpoints, retries, child operations, and disposition, including durable Workdir removal. -3. [`design/context-history.md`](design/context-history.md) — the highest-risk invariant: inputs that affect the model must be committed to history before they enter context. -4. [`design/worker-session-state.md`](design/worker-session-state.md) — Worker identity, replayable session logs, current metadata, and live process hints. -5. [`design/session-observation.md`](design/session-observation.md) — common session captures, `SessionEntryRef`, Memory evidence, and host-authorized Worker observation. -6. [`design/flow-state-graph.md`](design/flow-state-graph.md) — Workspace Flow sources, frozen content-addressed definitions, transition attempts, and bounded internal verification. -7. [`design/profiles-manifests-prompts.md`](design/profiles-manifests-prompts.md) — reusable Profiles, resolved Manifests, and prompt resources. -8. [`design/tool-permissions-scope.md`](design/tool-permissions-scope.md) — tool policy and filesystem scope. -9. [`design/plugin-packages.md`](design/plugin-packages.md) — the retained offline package format and the prohibition on ambient Plugin authority. -10. [`development/plugin-development.md`](development/plugin-development.md) — explicit-path offline Plugin package authoring and inspection. -11. [`design/memory-knowledge.md`](design/memory-knowledge.md) — deprecated single-document Workspace Memory authority and its separation from subject-scoped subjektiv Memory. -12. [`design/workspace-kanban-orchestrator-runtime.md`](design/workspace-kanban-orchestrator-runtime.md) — how Kanban operations become durable orchestration events and backend-internal routing decisions. -13. [`design/workspace-runtime-docker.md`](design/workspace-runtime-docker.md) — the WebUI / Backend / Runtime split, Docker image layout, worker launch path, and workdir materialization boundary. -14. [`development/ui-ux/design-language.md`](development/ui-ux/design-language.md) — Workspace Webの視覚・interaction原則、文法、token、汎用component。 -15. [`development/ui-ux/product-ux.md`](development/ui-ux/product-ux.md) — Workspace resource、navigation IA、resource固有pattern。 -16. [`development/ui-ux/application-architecture.md`](development/ui-ux/application-architecture.md) — application shell、Header、Sidebar、nested override architecture。 -17. [`development/ui-ux/visual-review.md`](development/ui-ux/visual-review.md) — Web UI実装者がbefore/afterを目視し、完了前に残すvisual validation evidence。 -18. [`development/server-runtime-auth.md`](development/server-runtime-auth.md) — manual Workspace Server / Runtime public-key exchange and authenticated Runtime startup checks. -19. [`development/work-items.md`](development/work-items.md) — how project work is recorded and reviewed. -20. [`development/rust-testing-strategy.md`](development/rust-testing-strategy.md) — what Yoi Rust tests should prove, where they belong, and how to name them. -21. [`development/validation.md`](development/validation.md) — how to check changes. -22. [`development/workspace-schema-migrations.md`](development/workspace-schema-migrations.md) — how to preflight, apply, verify, and roll back control-plane SQLite schema changes. -23. [`design/standalone-agent-host.md`](design/standalone-agent-host.md) — in-process standalone Worker host の依存方向、authority、lifecycle、非目標。 -24. [`design/server-feature-storage.md`](design/server-feature-storage.md) — Server管理のWorkspace/Feature専用SQLite、migration、transaction、backup/restore、削除・shutdown境界。 -25. [`design/subjektiv-store.md`](design/subjektiv-store.md) — subject-scoped Memoryのownership、authorization、Session attribution、変更履歴、candidate consolidation境界。 -26. [`design/subjektiv-surface.md`](design/subjektiv-surface.md) — confirmed Memoryから生成するresident surface、availability、append-only restore injection。 -27. [`development/subjektiv-product-cutover.md`](development/subjektiv-product-cutover.md) — legacy Workspace Memoryからsubjektivへのoperator cutover、explicit reset、backup／rollback runbook。 -28. [`design/worker-wip-mode.md`](design/worker-wip-mode.md) — explicit-opt-in Worker WIP mode, Worldspace projection, authority, cache, and retry contracts. -29. [`design/objective-wip-projection.md`](design/objective-wip-projection.md) — Host-assigned Objective collection/item routes, operation mapping, Backend authority, bounds, and compatibility suppression. -30. [`design/ticket-wip-projection.md`](design/ticket-wip-projection.md) — Host-assigned Ticket collection/item routes, role-specific operations, route identity, and typed Backend preservation. -31. [`design/merge-request-wip-projection.md`](design/merge-request-wip-projection.md) — Host-assigned Merge Request routes and preserved review/integration authority gates. -32. [`design/feature-chat-invocations.md`](design/feature-chat-invocations.md) — declarative `/` invocation identity, grammar, completion, typed draft, attachment staging, execution, and authority boundaries. -33. [`design/workspace-config-wip-projection.md`](design/workspace-config-wip-projection.md) — Backend-authored configuration as a logical WIP attachment, explicit grants, shared UI conflict domain, and activation boundaries. +WebなどRust以外の実装に固有の理由は、そのソースのmodule・型・処理に付くコメントで管理する。無関係なRust crateへ移さず、実装説明の別冊も増やさない。複数の画面・部品の設計判断に使うDesign LanguageやUX規則は文書として維持し、個別実装の説明と区別する。 -## What belongs here +`docs/`直下のファイルはこのREADMEだけとする。このファイルは文書の管理規則であり、全ファイルの索引にはしない。 -Keep documentation when it records a stable design boundary, a non-obvious rationale, or a workflow that future changes must respect. +## 追加・変更・削除の判断 -Examples that belong: +文書を追加する前に、残す必要のある判断が既存の原則、moduleドキュメント、テストのどこにも説明されていないことを確認する。コマンド出力、設定項目、型やフィールド、モデル名、schema番号、テスト件数など、実装から得られる可変情報は転記しない。APIの契約を説明する最小の例や、開発規則を実行する検証コマンドは、その目的に必要な範囲で残せる。 -- Why Worker metadata is not the session log. -- Why `Profile` and resolved `Manifest` are different layers. -- Why context-only event injection is forbidden. -- Why child Worker notifications are hints rather than completion proof. +設計変更と同じ差分で、その理由を所有するドキュメントを更新する。設計がなくなった場合は説明も削除する。移動・削除時にはリンク、`include_str!`、テスト、生成処理からの参照を確認し、古い場所に案内だけの文書を残さない。`report/`の過去のパスや引用は、当時の証拠として扱う。 -## What does not belong here +`crates/`以下のREADME・rustdoc・説明コメントは英語で統一し、それ以外の文書は日本語で書く。実行時の文言、テストデータ、引用は言語統一のために変更しない。コード識別子、API名、パスは原綴りを保つ。段落の最初に主張を置き、対象・条件・根拠を明記する。未確認の主張を断定せず、同じ内容の予告・言い換え・総括を重ねない。 -Do not keep material only because it once helped a ticket. - -Examples that should be removed or moved to `docs/.local/` / work item artifacts: - -- External project comparisons. -- Provider API snapshots, prices, or model tables. -- Old implementation plans that are no longer the design authority. -- Public type or method inventories that drift with code. -- Debug notes that are useful only for one investigation. - -`docs/.local/` is intentionally outside the maintained documentation surface. +作業計画と進捗はTicket・Objective・セッションの記録へ置く。秘密情報、認証情報、非公開プロンプト、生ログは文書へ転載しない。無視対象のローカル調査資料は、製品が参照する正本や維持対象の文書にしない。 diff --git a/docs/architecture/worker-restore-observations.md b/docs/architecture/worker-restore-observations.md deleted file mode 100644 index 4236bfdad..000000000 --- a/docs/architecture/worker-restore-observations.md +++ /dev/null @@ -1,70 +0,0 @@ -# Guarded Worker Restore and Web observations - -T-701 extends the [registry-bound operation reception](workspace-worker-operations.md), not the Worker creation, Resume or Restore state machines. - -## Different tokens protect different things - -| Operation | Observed identity / expected input | Atomic validation and execution owner | Conflict / uncertain outcome | Shared client behavior / deliberate difference | -| --- | --- | --- | --- | --- | -| Restore | Workspace + Runtime + Worker identity; Runtime-owned `restore_observation_token`; immutable `expected_observation_token` + `request_id` | Existing Runtime per-Worker lifecycle operation lock, durable admission and lifecycle commit. Preparation and execution follow admission, not a frontend GET comparison | Stale new intent is HTTP 409 (`restore_observation_conflict`); operation results remain accepted / rejected / rolled_back / reconciliation_required. A lost reply is unknown, not definite rollback | Both entrances use the same scoped POST/parser/notice classifiers and observation refresh. Never replace a rejected intent's token and automatically retry | -| Stop | Identity and reason, **not** a catalog content CAS | Runtime per-Worker operation lock and existing backend Stop settlement; Workspace session finalization only after accepted | Existing flat accepted / unsupported / rejected result with diagnostics | Same encoded target paths and bounded HTTP-error messages; refresh observations instead of writing a guessed local stopped state. Stop does not gain Restore's observation-token admission guarantee | -| Pause / Resume / Cancel / Compact | Connected execution identity and `WorkerCommandEnvelope.command_id` | Existing bound execution/controller command admission | Command acknowledgement: Accepted / StaleCommandId / Conflict / InvalidState, with authoritative state | Command IDs order commands; they are not catalog observation tokens. Resume is not Restore; connections do not silently move to a replacement execution | -| Delete / cleanup | Runtime + selected target IDs; `expected_plan_digest` of the candidate set, dependencies and blockers | Existing cleanup validation, per-target live/retention revalidation, removal service and catalog commits | Existing cleanup-plan conflicts, per-target status and diagnostics | Preserve plan protection for candidate sets and dependencies. Never substitute a Worker observation token. Page plans invalidate at the beginning of every catalog refresh and are unavailable until fresh catalog and plan observations settle | -| Pin / Unpin | Registry Worker identity and explicit boolean | Registry retention update and ordered projection publication | Existing retention result/errors | Preserve retention semantics and full catalog metadata; refresh shared observations and cleanup plan | -| Submit / Notify | Authenticated source + durable submission / notification request ID | Existing controller/session durable receipt admission | Existing accepted/rejected receipts and unknown delivery diagnostics | These are not Restore requests; Restore never manufactures input, Resume or a new Worker | - -There is no universal Worker command token or shared frontend action lock. Workspace reception resolves a stable target and rechecks caller authorization; Runtime/controller/store owners retain operation-specific atomic boundaries. - -## Browser request and result - -Normal explicit Restore is enabled only for `availability = observed`, catalog `stopped`, and an available backend observation token. Sidebar display state and foreground `worker_state` are not eligibility evidence: sidebar uses `lifecycleState`; the page uses the full catalog's `state`. Neither UI guesses snapshot, singleton, Workdir provider or authorization eligibility. - -Both entrances send the generated two-field `WorkerRestoreRequest` to: - -```text -POST /api/w/{workspace_id}/runtimes/{runtime_id}/workers/{worker_id}/restore -``` - -Each ID is independently encoded. No Ticket-assignment query, Submit, Resume or Worker-create call is used. The bounded parser validates wrapper identity, the four states, diagnostics and optional **Workspace WorkerSummary**. A Stop lifecycle result or launch summary is not a Restore response. Unknown states/fields/invalid identities cannot produce a success notice. - -Accepted means the existing Restore operation committed, not that the Worker is necessarily Running or a later run succeeded. A returned summary validates result identity; display is updated from fresh shared observations, not an old POST snapshot. Rejected and rolled_back display diagnostics without success. Reconciliation_required and transport/malformed-response uncertainty retain the original immutable request for an explicit **Retry Restore** action. That action sends the same request; it does not create an operation, exchange its observation token, or automatically loop. A conflict clears this local recovery intent and asks the user to review fresh observations before a new explicit operation. A GET alone does not settle Runtime's pending lifecycle operation. - -## Backend admission, preparation and recovery - -The Workspace `WorkspaceWorker` reception still resolves target identity and rechecks caller authority. Its Restore coordinator first dispatches the public tuple to Runtime without candidate preparation. A terminal owner returns its historical receipt; an admitted pending owner supplies its stored preparation before renewed singleton/Workdir/External eligibility is evaluated. Historical result summaries are read from the current Runtime and are not written back as catalog observations. - -For a new intent, Workspace builds read-only preparation references after the normal eligibility checks. Runtime atomically checks the observation token again under the existing per-Worker lock and persists the owner together with `pending_restore` and its existing lifecycle operation ID. Only this admitted operation can proceed to SSH resource issuance, binding/access preparation and execution. A race between the first observation check and admission therefore cannot issue credentials or launch a new Restore from the old intent. - -Preparation completion is attached to that same pending operation, not a replacement operation. Retained repository-access snapshots and exact request identity allow interrupted preparation to resume without exchanging candidate bindings. Runtime retries use the original owner's payload; a freshly generated preparation cannot rewrite the admitted target or expected observation. Stop and startup reconciliation continue to use the existing pending lifecycle ownership. When startup/legacy persistence has a pending lifecycle operation but no public receipt owner, an explicit request with the current observation may bind recovery to that exact operation ID, without applying candidate preparation. Once bound, a different request cannot take over; terminal replay does not resurrect a later stopped Worker. - -| Caller | Intent identity retained / validation boundary | -| --- | --- | -| Web sidebar and page | Immutable two-field request retained locally for explicit uncertain-result recovery; no shared in-flight lock | -| Workspace alias, scoped and Worker-control HTTP | Mandatory public request; common target/auth reception and the same Runtime admission | -| Embedded / remote Runtime providers and management HTTP | Typed Runtime request and coordination transport, including original observation and lifecycle operation owner | -| `WorkerRestore` Tool | Observed token in Tool arguments and durable Tool-call-derived request identity | -| Existing TUI Restore/resume path | Selected row's token and exact request retained through the existing flow. Unknown/pending results refresh Session observation and prompt for an explicit same-tuple retry; cancellation does not retry and reports the original tuple. Conflict exits for fresh selection, never exchanges the failed intent. No new TUI Restore entrance | -| Internal Coder, Orchestrator, subject reuse and compensation | Domain intent journal pins the request/token before dispatch and retains it through unknown result and restart. A definitive observation conflict retires only that rejected intent without automatic retry; a later deliberate invocation may pin fresh observation with a fresh request ID. It does not own lifecycle status or bypass Runtime validation | -| Runtime startup / pending reconciliation | Private recovery of the existing persisted Restore owner and operation; not an external unguarded new-intent endpoint | - -The Workspace internal intent journal stores client retry identity only. Runtime remains the lifecycle authority; it is not a second Restore state machine, Worker ledger or cross-screen lock. - -## Observation and cleanup-plan ordering - -`workspaceWorkersStore` retains the existing sidebar stream/projection and also fetches the full Workspace Worker catalog for the page. Full catalog metadata preserves retention, attachments, implementation and targets that are not sidebar entries. Stream updates overlay lifecycle/availability/foreground snapshot/token without interpreting absence as a new page exclusion. - -Every stream update or explicit refresh increments a local **observation scheduling version** and publishes `catalogRefreshing` before fetching. This local version fences asynchronous display work; it is **not** Restore admission authority and is never sent to Runtime. A newer event/GET supersedes older GET completion. Old POST summaries and page-loader data cannot overwrite the active catalog. On disposal, requests and subscription callbacks are fenced. Catalog fetch errors have their own stable alert episode rather than clearing transport/Workdir alerts. - -The page invalidates cleanup plans immediately when observations refresh. It does not fetch or install plans while `catalogRefreshing` is true. Successful refresh completion triggers plan reads even when the resulting catalog contents did not change. Workspace/lifetime and cleanup epochs reject late previous plan responses. A failed refresh leaves old candidates unavailable instead of treating old page data as fresh. - -Both components retain local busy state only. Requests pin their starting Workspace/target, and late replies after Workspace switch/disposal cannot update another Workspace's notices, list or busy state. Sidebar and page may issue competing requests: atomic Runtime admission—not a cross-component client lock—decides their outcomes. Console links and existing sidebar filtering remain unchanged. - -## Validation surfaces - -- Runtime admission tests prove stale/ABA and serialization, durable same-request replay and pending reconciliation. -- Runtime management, embedded/remote providers, Workspace aliases/scoped/control routes, Tool and existing TUI callers exercise the guarded API rather than a legacy unguarded Restore. -- `tests/runtime-workers.test.ts` and `test/sidebar/worker-actions.test.ts` prove response/request and notice contracts; the Runtime parser tests are included in the normal Web test task. -- Rendered `worker-stop.browser.spec.ts` and `workers-page.browser.spec.ts` cover menu/action eligibility, busy behavior, four outcomes and errors, explicit retry, Workspace/disposal fences, ordinary Console/Pin/Delete, sidebar-origin updates and cleanup epochs. -- `worker-restore-store.browser.spec.ts` tests the real store's full metadata, GET/event ordering, dependent-plan invalidation, no POST recovery loop, catalog-only target preservation and disposal. - -Exact executed validation counts and immutable source/review evidence belong to the repository Merge Request and final approved Ticket handoff. diff --git a/docs/architecture/workspace-worker-operations.md b/docs/architecture/workspace-worker-operations.md deleted file mode 100644 index d0f7bdd6f..000000000 --- a/docs/architecture/workspace-worker-operations.md +++ /dev/null @@ -1,140 +0,0 @@ -# Workspace Worker operations - -T-702 documents the implemented Workspace operation-reception boundary. It does not move execution authority into the Workspace registry or make all Worker-related routes use one API. The common implementation is [`worker_operations.rs`](../../crates/workspace-server/src/server/worker_operations.rs); production adapters and service orchestration are in [`server.rs`](../../crates/workspace-server/src/server.rs). - -## Identity reception is not execution authority - -`WorkspaceWorker::resolve` resolves a Workspace registry reference to a `RuntimeWorkerRef` (Runtime ID and Worker ID). Resolution first tries that exact pair, then a Workspace resource reference constrained to the requested Runtime. It returns an identity-bound handle, not a live execution snapshot, authorization lease, or promise that the Worker is running. Each operation rereads the target registry record and authorizes its own context. - -The owners have different responsibilities: - -- **Workspace registry and domain services:** durable Workspace identity, retention/pinning, Workdir links and sessions, Ticket assignments, credentials/bindings, control grants, restore coordination, and removal plans/catalog publication. Registry projections and observation streams describe Workers; they do not admit controller commands. -- **`RuntimeRegistry` and Runtime adapters:** select the Runtime owning the resolved pair and delegate execution. Remote and embedded adapters preserve their existing transport/error behavior. Runtime owns lifecycle operation settlement and checks Workspace scope at the scoped embedded execution boundary. -- **Execution backend and Worker controller/session:** own the actual execution endpoint, command admission, authoritative Idle/Busy state, in-flight run transitions, Submit FIFO and durable receipts, Notify receipts, and compaction. A catalog status is not sufficient evidence for command admission. - -The term *controller* in a `WorkerControl` context means the calling Runtime Worker holding a grant over a target. It is not the target's in-process execution controller. - -## Production ingress → common method → execution owner - -For the HTTP rows below, `R` abbreviates `/api/runtimes/{runtime_id}/workers/{worker_id}` and `W` abbreviates `/api/w/{workspace_id}/runtimes/{runtime_id}/workers/{worker_id}`. Aliases operate in the server's configured Workspace; scoped routes additionally validate the Workspace path. The declarations live in [`server-api`](../../crates/server-api/src/lib.rs), and the generated contract service implementations in `server.rs` are the production unary adapters. - -| Ingress | Common reception / coordination | Execution or persistence owner | -| --- | --- | --- | -| `POST R/restore` and `POST W/restore` | `WorkerOperationContext::from_request` → `restore_runtime_worker_with_context` → `WorkspaceWorker::restore`; scoped route first uses `scoped_restore_runtime_worker_with_context` | `WorkspaceApi::restore_workspace_worker` retains the existing singleton, Workdir, credential/binding and pending-operation reconciliation flow; Runtime/backend owns restored execution | -| `POST R/input` and `POST W/input` | Resolve → convert request → `WorkspaceWorker::input` | `RuntimeRegistry::send_input` → Runtime adapter → execution backend/controller/session | -| `POST R/stop` and `POST W/stop` | `WorkspaceWorker::stop` | Runtime owns Stop; on an **Accepted** result Workspace closes the current Workdir session and synchronizes linked Workdir state. Rejected/Unsupported results do not perform that finalization | -| `POST R/cancel` and `POST W/cancel` | `WorkspaceWorker::cancel` | `RuntimeRegistry::cancel_worker` → target execution controller; not registry removal | -| `PUT W/pin` / `DELETE W/pin` | `WorkspaceWorker::set_pinned(true/false)` | Workspace store retention update, ordered through `worker_projection`; no execution command | -| `GET /api/w/{workspace_id}/runtimes/{runtime_id}/cleanup-plan` | `build_runtime_cleanup_plan` (preview, not a Worker mutation) | Workspace cleanup planner/projections | -| `POST /api/w/{workspace_id}/runtimes/{runtime_id}/cleanup-executions` | `execute_runtime_cleanup_with_context` validates plan digest and selected candidates → each selected Worker's `delete_from_plan` | `WorkerRemovalService::execute_cleanup_removal` → existing retention planning, Runtime retention execution, and Workspace catalog commit. Workdir-only actions remain in the cleanup service | -| `POST /api/w/{workspace_id}/workers/remove` (`WorkerRemove`, not a browser removal alias) | `worker_remove_contract` verifies its source proof against the requested target → `WorkspaceWorker::remove` | `WorkerRemovalService::execute_async` checks the `remove` grant and coordinates retention/removal. It rejects self-removal and requires a stopped target for a new removal; existing durable removal operations have their own recovery path | -| Embedded `WorkerRemove` Tool | Runtime-owned in-process target-bound proof → `EmbeddedServerWorkerMutationDispatcher` verifies/consumes proof → installed `EmbeddedWorkspaceWorkerRemoveExecutor` → `WorkspaceWorker::resolve_with_services` → the same `WorkspaceWorker::remove` | Same `WorkerRemovalService::execute_async`, grant-lock/recheck, self/live/retention guards, durable plan/recovery and catalog commit as HTTP; no Runtime-to-Server dependency | -| Dedicated WS `R/protocol/ws` and `W/protocol/ws` | `worker_protocol_ws` (scoped wrapper for `W`) → `WorkspaceWorker::connect_protocol` → `workspace_worker_protocol_session` → bound sender | One remote WS or embedded execution transport; controller/session owns protocol operation results | -| Multiplexed WS `/api/w/{workspace_id}/protocol/ws`, runtime-scoped `WorkerProtocol` selector | `scoped_workspace_protocol_ws` → `serve_workspace_subscription` → resolve + `connect_protocol`; subscription stores the bound sender | Same connected execution transport as dedicated WS; the subscription layer wraps its snapshot/events, not a different execution implementation | -| Worker-control `POST /api/w/{workspace_id}/worker-control/workers/{runtime_id}/{worker_id}/{input,cancel,stop,restore}` | `send_known_worker_input`, `cancel_known_worker`, `stop_known_worker`, `restore_known_worker` authenticate the Worker source, construct `WorkerControl`, and call the same identity methods/restore adapter | Same Runtime execution owners, under the applicable active target grant; known-worker input adds its existing non-Accepted-result error translation | - -Both WS adapters call `WorkspaceWorkerMethodSender::send`, which selects the following **named methods** rather than implementing separate command semantics in each transport: - -| Public protocol method | Named common sender method | Distinct owner/receipt | -| --- | --- | --- | -| `Pause { command }` | `pause` | Controller command admission/acknowledgement; pause is not Stop | -| `Resume { command }` | `resume` | Controller resumes the paused run; not Restore | -| `Cancel { command }` | `cancel` | Controller command admission for cancelling the current run; distinct from HTTP lifecycle request/result | -| `Compact { command }` | `compact` | Controller maintenance/compaction, with command acknowledgement and subsequent lifecycle events | -| `Submit { submission_request_id, input }` | `submit` | Session durable Submit receipt/FIFO and later run execution | -| `Notify { notification_request_id, message }` | `notify` | Session durable Notify receipt and advisory delivery, not the Submit FIFO | - -All six methods funnel through `dispatch` / `send_checked`. Other browser-allowed protocol methods also use that checked dispatch; the six named methods are not an exhaustive allowlist. For example pending-submission operations, rewind, Shutdown, completions, ListWorkers, RestoreWorker and RegisterPeer are allowed by `authorize_browser_worker_method`. This forwarding does not replace their lower-layer semantics with Workspace registry operations. - -`build_inner_router` installs the three production WS routes using typed `websocket_route` registrations and merges the generated Workspace contract router for unary routes. [`workspace_subscription.rs`](../../crates/workspace-server/src/workspace_subscription.rs) handles multiplexing. Its `WorkspaceWorkers` subscription is a separate catalog stream with no method sender; only a runtime-scoped `WorkerProtocol` subscription carries an operational sender. - -## Context and authorization restrictions - -`WorkerOperationContext` is crate-private. Clients cannot submit this enum to select their authority. - -- **Request-derived Worker control:** `from_request` uses the middleware-verified Runtime Worker subject and current Workspace registry membership. The shared source-binding helper rejects Runtime-only proofs and conflicting identity hints; no hint-only request or Account fallback can impersonate a Worker. Known-worker adapters bind their source through the same verified request extension before constructing the operation context. -- **Account/browser context:** despite the `Browser` name, this also represents account-authenticated unary requests. It retains request headers and the server-derived Account input source. On each common authorization call, the server re-resolves credentials, rejects revoked credentials or an Account change, requires the configured Origin for `BrowserSession` actors, and requires an active Workspace. It does not interpret the operation's permission string as a Worker-control grant check for an Account actor; request middleware remains part of the route's authorization boundary. -- **Worker-control context:** the target and calling Worker must be registered. The server requires an active grant for this controller/target pair and the operation's permission, locks by grant ID, then rereads the target and rechecks permission and the exact grant ID after acquiring the lock. A replacement grant requires new admission under its own lock. A missing active grant is reported as UnknownWorker; a present grant lacking the permission is InvalidInput. Merely knowing an ID does not grant control. -- **Test Backend context:** `WorkerOperationContext::Backend` exists only under `#[cfg(test)]`. It bypasses context authorization after the target-record check. There is no production Backend enum variant for a client, Worker Tool, or generic internal caller to select. Production Backend services continue to own their explicit domain flows. - -Worker-control grants are Workspace-owned records, not inferred parentage or universal access to every Runtime Worker. The production spawn flow persists a controller/subject grant with `send_input`, `notify`, `cancel`, `stop`, `restore`, `remove`, and `observe`; that default does not include `pin`. Known-worker discovery follows active grants, while Internal children use their separate parent-owned registry. Other grant origins/permission sets must be read from their own authority rather than assumed from this spawn default. - -Permissions are `restore`, `stop`, `cancel`, `pin`, and `remove` for those identity operations. HTTP input uses `notify` for Notify and `send_input` for all other accepted kinds. `connect_protocol` checks `observe`; every connected method dispatch checks `send_input`, requires an interactive target, then explicitly requires a **Browser** context. Thus an observe grant is not a mutation grant, and a Worker-control grant does not enable browser WS method forwarding. WS Notify also uses this browser-only `send_input` check, not HTTP input's `notify` grant mapping. - -`input` and all connected protocol method sends reject Workers with a Backend Job projection: their immutable input is Backend-owned and their Console is read-only. Connecting for observation is not itself rejected by that interactive check. - -For Worker-control user input, `input` converts `User` to `UserIfIdle`; admission belongs to the target controller and never queues behind an active run. Browser Submit may durably queue while Running/Paused. Notify is advisory input, not a queued Submit; it can be incorporated at a between-turn boundary or drive the notification flow while Idle. The known-worker input adapter turns definite non-Accepted input results into errors, but keeps uncertain outcomes as the existing typed result with a `worker_input_outcome_unknown` diagnostic, and supplies the Idle-only guidance for user input. - -Scoped Restore may coordinate Ticket assignment only when both `ticket_id` and `assignment_operation_id` are supplied. A Worker-control context cannot use either assignment parameter. Aggregate cleanup rejects all Worker-control requests: Worker callers must retain the stronger target-bound WorkerRemove proof/retention path. A control grant is not authority over Ticket assignments or Workdir cleanup. The cleanup coordinator revalidates Account credentials, Origin, and active Workspace even for a Workdir-only request. - -The HTTP and embedded `WorkerRemove` entry points share the registry-bound removal method. `WorkerRemove` intentionally has a distinct authorization contract from the other operations. It accepts a verified mutation source rather than `WorkerOperationContext`, verifies its target-bound proof before calling the identity method, and keeps source-specific and grant revalidation in the removal service. It is not permission-free just because it does not call `authorize_operation`. - -The embedded executor retains only the removal services with a **weak** RuntimeRegistry reference. Its removal-only `WorkspaceWorker<()>` specialization shares resolution, current-record checks and the removal method with the full `WorkspaceWorker` handle. This service projection avoids a strong `WorkspaceApi → Runtime → dispatcher → WorkspaceApi` ownership cycle; it is not another Worker ledger or an authorization fallback. It cannot call the full handle's restore/input/protocol methods. The existing independent Runtime-owned proof forwarder and verification/replay seam stay unchanged. - -## Tokens and outcomes are operation-specific - -There is no universal Worker command token or universal success result. - -- `RuntimeWorkerRef` identifies the registry target; a Workspace resource reference is an ingress lookup form. Neither identifies a particular run or proves authority. -- `WorkerCommandEnvelope.command_id` orders controller lifecycle commands. `WorkerCommandAcknowledgement` contains command kind, disposition (`Accepted`, `StaleCommandId`, `Conflict`, `InvalidState`) and authoritative state. These IDs belong to controller command admission, not durable Submit identity or Workspace removal planning. -- `submission_request_id` is the caller's Submit retry key in an authenticated-source namespace. A durable `SubmissionAccepted` receipt also contains the Worker-allocated `submission_id` and `Started`/`Queued` disposition. `SubmissionRejected` is correlated by request ID. Acceptance is not a successful completed run; `last_finished_submission_request_id` is a teardown fence, not a result-success claim. -- `notification_request_id` correlates `NotificationAccepted` / `NotificationRejected`. Notify has its own durable receipt and exact-payload retry behavior; it does not allocate a Submit FIFO entry. Browser Submit/Notify are rewritten to `SubmitTracked`/`NotifyTracked` with the server's Account source. Browser-supplied tracked forms, `SubmitIfIdle`, and `WorkerEvent` are rejected. -- Runtime `WorkerLifecycleOperationId` is a durable lifecycle/reconciliation identity, separate from a controller command ID. Ticket `assignment_operation_id`, cleanup plan digest, and retention plan/operation IDs and fingerprints likewise belong to their own domain checks. A multiplexed `subscription_id` only routes frames to an already-bound sender. - -The common sender returns `Result<()>` after authorization and enqueueing on its transport channel. This is **not** a controller acknowledgement, durable receipt, or proof of upstream delivery. Actual outcomes arrive on the event stream. The dedicated WS reports dispatch errors as protocol error events; the multiplexed adapter currently terminates the connection when sending a method fails. It also requires an initial session Snapshot before accepting the WorkerProtocol subscription. - -Unary input/lifecycle responses preserve their existing API projections: `WorkerOperationState` is `Accepted`, `Unsupported`, or `Rejected`, with Runtime/Worker IDs and diagnostics. Restore instead has `Accepted`, `Rejected`, `RolledBack`, and `ReconciliationRequired`. Pin returns retention state; cleanup returns per-target actions/statuses; WorkerRemove has its own response/error body. These are not interchangeable acknowledgements. - -A transport or store failure can leave an **unknown outcome** after effects have started. Lower-layer input results have `WorkerInputDisposition::{Accepted,Rejected,Unknown}` and an optional `runtime_run_id`, but `worker_input_result_to_api` projects only state, target IDs and diagnostics: the public unary input result does not expose those two fields. Unknown delivery is explicitly preserved as a `worker_input_outcome_unknown` diagnostic; neither HTTP adapter changes it into definite rejection or retries it. Runtime execution results also distinguish Busy/Rejected/Errored/Unsupported; error handling and durable restore/removal recovery remain lower-layer responsibilities. Do not infer that a rejected-looking projected response or a closed WS proves that nothing was accepted. The new reception layer does not add automatic replay or an exactly-once guarantee across every ingress. In particular, `worker_input_request_from_api` sets `submission_request_id` to `None`; the unary Workspace input contract does not provide the browser WS Submit retry-key contract. - -## Connected execution lifetime - -`WorkspaceWorkerMethodSender` stores both the resolved Worker handle and the sending channel privately. A WS caller cannot pair a different Worker handle with a subscription's raw sending capability. Registry identity can outlive an execution, but an operational connection does not follow it to a replacement execution. - -`connect_workspace_worker_protocol` obtains an observation source from the Workspace proxy (falling back to the Runtime only for WorkerNotFound). It then connects to one remote execution WS or attaches one embedded `WorkerProtocolTransport`. The embedded path captures the real endpoint and its initial snapshot, not the WorkerRef-wide catalog observation bus. - -Embedded dispatch uses `send_connected_protocol_method_scoped`. Runtime verifies that the transport target equals the authorized WorkerRef, checks scope and pending-operation settlement, and validates the captured endpoint before any durable mutation. Even a delayed terminal-close observation cannot make the old endpoint Stop or send to a new execution under the same identity. - -Shutdown and a closed execution relay terminate the operational stream. Embedded lifecycle/store/transport uncertainty is terminal; a settled Busy/Rejected/Unsupported rejection may leave it usable only if the endpoint still validates. Remote bridge failure closes its channels. `send_checked` reports `worker_protocol_closed` when its channel is closed and explicitly states that no implicit Restore or retry occurred. No bridge in this path retargets, restores, or reconnects to the next execution. The caller must explicitly Restore when authorized and establish a new connection; this is different from catalog observation continuing to describe the same registry identity. - -## Internal Workers are parent-owned - -Internal SubWorkers and service Internal Workers are not Workspace registry targets or independently addressable Runtime protocol subjects. They remain separate, parent-owned sessions. Their visible identity is `InternalWorkerRef.session_id` (`name` is display-only), and snapshots/events are presented through the parent's protocol stream; service-private Internal Workers are excluded from the public child inventory. - -Direct-child control/observation stays in the parent-owned Worker machinery. A Workspace Runtime Worker grant is not a grant to an arbitrary Internal Worker, and this registry handle does not create an Internal Worker Runtime record, independent transport, or implicit Restore path. The Workspace observation capture adapter explicitly rejects a non-RuntimeWorker subject; it does not resolve a SubWorker as a registry Worker. - -## Regression helpers are not production routes - -Lower-level test functions in `server.rs` must not be mistaken for installed unauthenticated endpoints: - -- `set_worker_retention`, `send_runtime_worker_input` / `dispatch_runtime_worker_input`, and `execute_runtime_cleanup` / `scoped_execute_runtime_cleanup` are `#[cfg(test)]` wrappers using the test-only Backend context. -- `worker_protocol_ws_session`, `remote_worker_protocol_ws_session`, and `embedded_worker_protocol_ws_session` are `#[cfg(test)]` transport regression helpers. Production dedicated WS uses `workspace_worker_protocol_session` with the common bound sender; production multiplexed WS uses `serve_workspace_subscription` with that same sender type. -- Production `connect_workspace_worker_protocol`, `connect_remote_worker_protocol`, and `connect_embedded_worker_protocol` are real shared connection helpers, not those test-only sessions. The grant-checked observation capture service also calls the connection helper to read a committed Snapshot; that read-only capture is outside the six operational WS methods. - -Source presence alone is not route wiring. Follow `build_inner_router`, generated contract implementations, and the typed route declarations when auditing production ingress. - -## Minimum scope, limitations, and future guard - -The boundary above covers the T-702 minimum operation set. It is not a claim that every session/file/completion API has been migrated: - -- Session snapshot/history and retained session attachment reads remain their existing observation/retention paths. They are not controller command admission. -- Attachment upload grants and upload cancellation keep their Account/target-bound grant checks; file upload and uploaded-file deletion still resolve the target and delegate through `api.runtime`, rather than calling a new `WorkspaceWorker` file method. -- Unary completions still resolve the registry reference and call `api.runtime.worker_completions`. Protocol `ListCompletions` is also browser-allowed through checked WS dispatch. Those are different ingress paths. -- Worker creation, Backend Job execution, rollback compensation, Workspace deletion, and other domain-owned orchestration are not universally rewritten to this handle. Legitimate lower-level Runtime calls still exist. - -Known limitations worth follow-up, rather than undocumented guarantees: - -1. Browser credentials/Account are rechecked for each common operation or WS method send, but not continuously for every outgoing observation event. This is not a promise of immediate revocation-driven stream teardown. -2. Cleanup is sequential rather than an all-target atomic transaction. The existing cleanup plan digest and each removal service's live revalidation remain distinct protections. -3. Unary input has its existing narrower wire contract, rather than the WS authenticated receipt envelope. Identity unification does not create a new retry contract or manufacture caller request IDs across transports. - -T-701's [stale-observation Restore guard](worker-restore-observations.md) connects the caller's `expected_observation_token` and `request_id` to Runtime's existing operation lock, distinguishes a new Restore intent from pending reconciliation, and handles UI conflicts. This boundary leaves those Restore inputs/results and the pending-operation state machine intact rather than creating a duplicate guard. - -### Implementation references - -- [`server/worker_operations.rs`](../../crates/workspace-server/src/server/worker_operations.rs): identity handle, contexts, authorization, named methods and private bound sender. -- [`server.rs`](../../crates/workspace-server/src/server.rs): contract adapters, `build_inner_router`, scoped Restore/cleanup, WorkerRemovalService, dedicated WS and shared transport bridges. -- [`workspace_subscription.rs`](../../crates/workspace-server/src/workspace_subscription.rs): multiplexed WorkerProtocol binding and catalog subscriptions. -- [`hosts.rs`](../../crates/workspace-server/src/hosts.rs): Runtime registry/adapters and distinct input/lifecycle result projections. -- [`protocol`](../../crates/protocol/src/lib.rs): command envelopes/acknowledgements, Submit/Notify receipts and Internal Worker presentation. -- [`worker-runtime/runtime.rs`](../../crates/worker-runtime/src/runtime.rs) and [`execution.rs`](../../crates/worker-runtime/src/execution.rs): scoped connected endpoint validation, lifecycle identities and execution results. diff --git a/docs/backend-auth-account-model.md b/docs/backend-auth-account-model.md deleted file mode 100644 index e4d6bf443..000000000 --- a/docs/backend-auth-account-model.md +++ /dev/null @@ -1,54 +0,0 @@ -# Backend account model and login flows - -This document records the first Backend identity/auth layer introduced for the multi-workspace Backend direction. - -## Account and owner model - -The Backend store now separates a human user from the owner namespace used by workspace ownership: - -- `accounts`: canonical namespace records. `kind = "user"` is implemented now; `kind = "organization"` is reserved in the schema for a later organization model. -- `users`: human user records. Each user owns exactly one `accounts` row. -- `workspaces.owner_account_id`: optional account owner reference. Existing/local workspaces may keep it unset until account bootstrap/ownership migration code assigns it. - -This avoids baking `owner_user_id` into workspace authority while keeping the current implementation small. - -## Browser login - -Human browser auth is modeled as Passkey/WebAuthn ceremony endpoints: - -- `POST /api/auth/bootstrap-user` -- `POST /api/auth/passkeys/registration/options` -- `POST /api/auth/passkeys/registration/complete` -- `POST /api/auth/passkeys/login/options` -- `POST /api/auth/passkeys/login/complete` -- `GET /api/auth/whoami` - -The store persists registration/login challenges, serialized WebAuthn ceremony state, verified passkey credentials, browser sessions, and challenge consumption. Successful login mints an HttpOnly `SameSite=Lax` cookie session. - -Completion endpoints require browser `PublicKeyCredential` responses and verify them with `webauthn-rs` before a passkey is stored or a browser session is issued. Fake or stale credential responses are rejected by the verifier, and consumed challenges cannot be reused. - -## CLI/TUI device login - -CLI/TUI clients use a device-login flow instead of browser cookies: - -- `POST /api/auth/device-login/start` creates a `device_code` and human `user_code`. -- The CLI prints `verification_uri_complete` and polls `POST /api/auth/device-login/poll`. -- A browser-authenticated user approves with `POST /api/auth/device-login/approve`. -- Approval mints a Backend API token and returns it exactly once to the polling client. - -`yoi login [--backend ] [--no-wait]` starts this flow and stores the received token in `$XDG_CONFIG_HOME/yoi/backend-tokens.json` or `$HOME/.config/yoi/backend-tokens.json`. - -## Request actor extraction - -`workspace-server::auth::resolve_request_actor` resolves a typed request actor from either: - -- `Authorization: Bearer ` for CLI/TUI API tokens, or -- the configured browser session cookie. - -The resolved actor includes `user_id`, `account_id`, `handle`, `display_name`, and `auth_method`. This helper is intended to be used by mutating Backend APIs as authorization gets added route-by-route. - -## Non-goals for this step - -- Organization membership and RBAC. -- Password login, OAuth login, or other human auth methods. -- Token scoping beyond Backend API bearer authentication. diff --git a/docs/design/checkout-wip-projection.md b/docs/design/checkout-wip-projection.md deleted file mode 100644 index 51e28bfda..000000000 --- a/docs/design/checkout-wip-projection.md +++ /dev/null @@ -1,304 +0,0 @@ -# Attached checkout WIP projection - -T-699 publishes the contents of **current Worker attachments**, independently of -Repository/Workdir inventory and management. This is a Yoi implementation of -request-time projection, **not a dependency on `wip-fs-server`**. The upstream -read-only source/specification informed the design; its byte `read` and -truncate/write `replace` contracts are not substituted for Yoi's text/edit/save -contracts. No file-tree store, synchronization, or per-file resident registration -is created. - -```text -/ -├── repositories inventory/reference -├── workdirs identity/lifecycle -├── workdir-attachments Backend connection lifetimes -└── checkouts current accessible content entrances; list - └── directory; list, glob, grep, create_file - -# Not indexable; resolve known paths directly or discover via Operations: -/checkouts//README.md file; read, edit, write -/checkouts//src directory; list, glob, grep, create_file -``` - -Operations are Interface declarations, not children or `/features/...` routes. -File tools are native when the normal Worker WIP surface is installed. Bash stays -in compatibility mode. Ordinary Tools mode retains Read/Edit/Write/Glob/Grep. -Feature activation never grants READ, WRITE, EDIT, search, COMMAND, or delegation. - -## Placement and ownership - -`worker::checkout` owns Worldspace binding, contextual Interface declarations, -permission input reconstruction and safe output. It never opens a host pathname. -It resolves the live `WorkdirSessionRouter` on each request. The existing -`WorkdirSession` provider executes `checkout_observe` and `checkout_execute`; -Runtime HTTP and External transports carry typed operation/result envelopes at -their existing authorization, connection-generation and cancellation boundaries. -The Workspace-attached session forwards through the existing Backend broker. -Endpoint paths, credentials and materialized roots do not enter WIP descriptors, -Objects, arguments or results. - -The common Host registry has an **explicit asynchronous subtree provider** -contract, separate from the existing one-segment dynamic item families. It -validates every resolved projection's path/owner/descriptor and every enumerated -child's immediate-parent relationship. Static and dynamic/subtree overlaps are -rejected. Observe resolves only the requested depth; successful observations are -complete through that depth, not partial trees masked as success. Directory -listing Operations and native search use the shared List/Glob/Grep engines through typed -`CheckoutSearchRequest` forwarding. Scoped sessions append intersecting rule -layers, translate their cwd, and require provider-side visibility checks before -entry descent/content reads. Results retain session-logical coordinates, so a -SubWorker cwd does not duplicate a path prefix in its Object links. Unsupported -providers fail closed rather than falling back to a broader source search. -Direct resolution also uses current provider authority, not possession of a -cached Interface. - -## Indexable boundary (T-714) - -Only `/checkouts` and accessible immediate attachment entrances are indexable. -No directory content List runs during tree/observe, even at deep depth or when a -known deep directory is the observation root. Empty, small and large directories -all have the same boundary. Positive-depth content observations return -`children: []` (no **indexable** children), not an omitted field and not -an assertion that the directory is empty. At depth zero children are omitted; -`observe("/checkouts", 1)` omits entrance children, depth 2 returns [] there. -The Client retains discovered/inspected paths without adding indexable edges. - -Known authorized paths remain directly resolvable and operable. Use List/Glob/Grep -for discovery, then Inspect only selected entry coordinates; no successful search -is undone when an entry disappears before Inspect. Bash/COMMAND is unnecessary. - -## Names and connection isolation - -Aliases use T-697's canonical encoding: nonempty ASCII letters/digits/`-`/`_` are -unchanged; every other UTF-8 alias is `~` plus lowercase UTF-8 hex, with `~` itself -encoded. This is injective, not display-name slugification, Unicode normalization, -URL decoding or case folding. Only canonical spellings are accepted. - -`checkouts.list` returns alias, slug, Workdir ID, Worker-local attachment generation -and `path`. Attachment inventory retains its Backend connection lifetime ID and -can return `checkout_path`/`checkout_slug` for the matching live router attachment. -Attach results point to the attachment catalog and checkout entrance. Workdir -existence alone does not expose its content. Following a reference still requires -current scope/capability and an accessible filesystem projection. - -Objects have no shared inode `ref`: hard-linked names and different attachments -must not be mistaken for one path-independent representation. Interface -references are structured `(scope, name)`. Each entrance is its own -scope; content Interfaces use that entrance as ancestor scope, irrespective of -indexability. Local names bind target coordinates, projection incarnation and -live attachment generation. The provider's target-qualified-name routing hint -only selects a target; fetch re-resolves one coherent current publication and -checks the exact Interface, Descriptor, validator and scope under the pinned -router connection. It never turns a cached registration or name into authority. -Validators combine that namespace with the provider's -opaque identity/state validator. Restoring a Worker creates a fresh incarnation; -detach/alias reuse changes the router generation. An old Interface, validator or -read-history entry cannot retarget a new connection. No separate connection ledger -is added. The existing router's active-operation exclusion and begin/finish/cancel -detach paths remain authoritative; detach does not remove the Workdir/files. -Collection enumeration rejects connection-set changes between sampling and -publication instead of pairing old items with a new validator. Result publication -checks the captured attachment generation, not merely a -reused alias. It maps typed provider paths without observing/statting every -result. Provider awaits are covered by the operation deadline. Objects publish -no scope_ref here; legitimate ref-less binding still checks membership, -validators, current provider authority and attachment generation. - -## AI Operation mapping - -All operations bind their attachment and target from the WIP route. There is no -`target_workdir`, Workdir ID, host path or replacement `file_path` argument. -Arguments not declared by the descriptor are rejected before dispatch. Paths -below are relative to the **bound directory**, not the Worldspace or endpoint. - -| Object | Operation | Arguments | Result | -| --- | --- | --- | --- | -| `/checkouts` | `list` | none | bounded `items` containing alias/slug/Workdir/generation/path | -| directory | `list` | optional `limit` (default 100, 1..1000), `after` (returned `{kind, entry}`) | direct-content records `{kind, size, entry}`, `truncated`, optional `after` | -| directory | `glob` | `pattern`, optional `path` | normal bounded Glob text/summary plus Descriptor-typed same-checkout `entry` coordinates | -| directory | `grep` | `pattern`; optional `path`, `glob`, `type`, `case_insensitive`, `-A`, `-B`, `-C`, `multiline`, `output_mode`, `head_limit`, `offset` | normal Grep grouped/context/count text and typed provider search targets mapped to Descriptor-typed `entry` coordinates | -| file | `read` | optional line `offset`, `limit` | line-numbered text and normal Read summary | -| file | `edit` | `old_string`, `new_string`, optional `replace_all` | replacement count/summary and preview | -| file | `write` | `content` | existing-file save summary; never creation | -| directory | `create_file` | `path`, `content` | create-new summary and created-file link; permitted missing parents are created | - -Native operation results declare a typed record with `summary`, optional `content` -and `items`. Each item contains `entry: entry` in its Descriptor (an absolute -Worldspace coordinate, encoded as a string in WIP Value), never a guessed ordinary -string or an OS path. List additionally declares `kind`, byte `size`, `truncated` -and optional `after: {kind: string, entry: entry}`. File operations have empty -items except Create, which returns the new coordinate. -Search links are constructed from **typed provider paths**, not by parsing Grep's -rendered filenames/line delimiters. Provider scope remains authoritative for -search output as well as links. Glob/Grep share normal provider ignore/pattern, -context/output mode, offset and bounded result behavior. Descriptor-safe ignore -loading preserves root/nested rules and prunes ignored directories, checking -scope before opening each rule source. It cannot read global/ambient ignore -configuration outside provider authority or chase linked-worktree administration -pointers; source-class precedence, per-provider Git-marker/error behavior and -text/aggregate limits are detailed in `crates/workdir/CHECKOUT_PROVIDER.md`. -Search does not claim an -atomic snapshot of every descendant: it fences the bound directory before/after -search and each provider read/search retains its own consistency guarantees. - -Read offsets are zero-based **lines**, limit defaults to 2000, displayed numbers -are one-based. The shared Read retains at most 256 KiB of source text and bounds -its numbered presentation to 256 KiB, with an explicit truncation marker when -numbering/Unicode expansion exceeds that budget. No byte-range -operation is published by this native surface. Partial Read records the provider's -full bounded-source hash and total line count, not the hash of only its displayed -slice. Native and ordinary Tool calls use the same original alias/generation -tracker, mutation coordinator, rendering and parameter validation. - -Edit requires a nonempty, different `old_string`/`new_string`; `old_string` must -occur exactly once unless `replace_all` is true. Existing Write/Edit require a -prior successful Read through the same attachment. The read-history hash check -and the provider identity validator are **both** required: a same-content inode -replacement cannot be blessed merely by matching a hash. Failed/conflicting Read -must not record history; successful mutations update hash/change statistics only -after provider success. Create is strictly beneath the bound parent, creates only -an absent file, and must not overwrite an existing destination or symlink. - -Manifest permission identities are Read/Edit/Write/Glob/Grep plus **List** for -native directory listing; create_file uses Write. List publication uses provider -Read capability and potential List allow policy; execution reconstructs the bound -directory's logical path for the exact List policy. Allowing Glob or COMMAND does -not override a List deny/ask rule, and Interface scope validity grants no file -access. Normal Tools mode is unchanged (no new top-level List tool). The Host reconstructs the same route-bound input for policy evaluation; -provider read/write scope, capabilities and delegated leases are checked again. -`ask` fails closed. Native replacement claims remove `/tools/Read`, `/tools/Edit`, -`/tools/Write`, `/tools/Glob`, `/tools/Grep`, not unrelated Tools. - -## Directory List pagination - -List uses the existing provider List engine, **not** glob("*") or shell ls. It -includes direct visible empty directories, hidden and ignored names: unlike -Glob/Grep, provider List does not apply search ignore files. Existing Read scope, -readability and can_enumerate_directory checks still gate names; no ancestor or -nonrecursive enumeration permission is inferred from a known descendant. - -Order is directories first, then lexical canonical provider path, with every -other kind in the second group. after is the last returned exclusive ordering -key, rebased to a canonical Worldspace entry by the Host. Pass the returned record -unchanged to continue on the same directory; a foreign checkout, non-direct child, -absolute provider path or unknown field is rejected. No cursor stat is required: -the key may have disappeared. This is **live pagination, not a snapshot**. New -keys before the cursor are not revisited; later keys may appear, removals vanish, -and renames/kind changes may omit or repeat an entry. Directory validators fence -each call, not all pages; refresh Inspect after a stale rejection before issuing -a new explicit call. Detach/reuse expires the old Interface and validator. - -Each page scans the live directory under provider cancellation/deadline and the -existing traversal-entry ceiling, retaining at most limit+1 ordering candidates -and the aggregate path-byte budget. It never retains the whole unbounded listing. -The next ordering key is explicit whenever truncated; no next key means complete. -Provider resource failures stay explicit failures, not silent truncation or retries. - -UTF-8 canonical names are retained verbatim; unrepresentable names are skipped by -the existing provider. Native List may reject visible symlinks under the checked -provider's no-follow policy. If a provider returns symlink/other kinds they remain -classified path coordinates, not a promise of inspectable/operable Objects: -direct checkout resolution only publishes authorized regular files/directories. - -## Consistency, safety and resource policy - -The provider resolves/pins the entry/parent, compares its opaque validator and -executes shared `fs-operation` processing within the same checked boundary. The -validator represents identity and protected metadata, independently of the Read -content hash. Read checks state across its stream and returns a validator for -that same state. The WIP response uses the **execution result's validator**, not a -later independent path stat that could bless a replacement after execution. - -The native checked filesystem projection deliberately rejects symlink traversal -and unsupported/special entries. It does not silently broaden existing resolved -or logical scope. Ordinary Tools retain their existing symlink-policy behavior. -External sessions retain the root descriptor pinned **before approval/grant**; -renaming/replacing its ambient pathname must not change the approved root. -Absolute paths, traversal, alternate alias arguments and special-file mutation -are not native capabilities. UTF-8 names are byte-for-byte; non-UTF-8 names must -not be lossy-normalized into another Object name. Root/directory permissions are -not inferred from catalog visibility or from a descendant pathname supplied by -the model. - -Saves retain Yoi's temporary-file publication rather than in-place truncation. -Thus saving one hard-link pathname replaces that name rather than changing all -aliases of the old inode. Checked preconditions detect known changes before -commit. Advisory locks cannot exclude arbitrary noncooperating external editors; -read-time/commit-time identity checks do not create a filesystem-wide transaction. -The supported race guarantees and remaining external namespace race window are -specified by the provider implementation and tests, not by a claim that a pinned -fd makes any rename-based update universally atomic. - -A guaranteed pre-effect rejection is distinct from **OutcomeUnknown** after -mutation/parent creation may have started, commit succeeds but state cannot be -returned, response validation/encoding fails, or a transport/deadline loses the -result. Never automatically retry uncertain mutations. Inspect current state and -rediscover/reread. Do not report partially created parents as unchanged. - -Host observations are bounded to depth 32 and 1024 nodes with a 30-second total -cooperative deadline. The checkout adapter permits at most 16 concurrent requests -and bounds admission waiting and the complete operation (including bounded result -publication) to 30 seconds each; provider path/depth, source/response, -mutation/search and transport bounds apply too. No initial context expands an -entire tree. Resource-limit errors do not turn incomplete observation into a -successful truncated tree; bounded search results may explicitly report -truncation under the existing search contract. - -## Typical sequence - -1. Use Tree on `/checkouts` with depth 1; Inspect `/checkouts` directly and Invoke `list`. -2. Inspect an entrance or known deep directory directly; Invoke `list`/`glob`/`grep`. - For List, pass returned `after` with the next limit to obtain another live page. -3. Follow a returned file `entry`, Inspect that path and Invoke line `read`. -4. Call `edit` or `write`; Client-managed validators protect the exact observed - object while the shared tracker enforces the prior Read. -5. Rediscover after external changes; a stale rejection is not permission to skip - Read. ReRead after successful save to verify text. -6. Inspect a directory and call `create_file` with a descendant relative path, - then discover/Read the returned new file. -7. Use `/workdir-attachments/` for detach. Old checkout - observations and history cannot operate a newly attached alias. - -Workspace configuration (T-695) remains a separate logical provider; it is not -replaced by raw Linux filesystem writes. Bash native conversion, broad POSIX -operations, lifecycle duplication, upstream changes and deployment/restarts are -outside this feature. - -## Reproducible validation matrix - -Run from the main repository, on Linux with `openat2` and `/proc/self/fd`: - -```sh -cargo test -p worker --lib checkout_wip -cargo test -p workdir --test list_pagination -cargo check -cargo test -p fs-operation -p workdir -p tools -p worker -# Workdir External/HTTP features and dependent server fixture: -cargo test -p workdir --features http-client -cargo test -p yoi-workspace-server --lib backend_workdir_session_proxy_executes_typed_operations -node scripts/verify-wip-dependencies.mjs -cargo fmt --all -- --check -git diff --check -``` - -| Boundary | Focused regression locations | -| --- | --- | -| Deep mounts, multiple aliases, route-only arguments, exact permission identities, no duplicate compatibility file Tools | `worker/src/checkout_wip_tests.rs` and existing `wip.rs` registry/controller tests | -| Native search → numbered Read → unique/all Edit → Write → Create; returned validator cached chains | `checkout_wip_tests.rs`, `tools/src/checkout_tests.rs` | -| SubWorker write leases, sparse/nonrecursive/nested scopes, denied basename/content pruning, result-coordinate rebasing, revocation | `workdir/src/checkout_search_tests.rs`, `checkout.rs`, native scoped-child test | -| Approved External root replacement; same-content inode/name/parent substitution; hard-link save isolation; symlinks/special files; Unicode/non-UTF8 paths | `workdir/src/checkout.rs`, `fs-operation/src/checked.rs` | -| Mid-read updates; independent hash and validator fences; parent partial creation and post-save state failure | `checked.rs`, `checkout.rs`, Tools tests | -| Detach/alias reuse/restore references and read history; collection/search response generation fences | `checkout_wip_tests.rs`, `checkout_race_tests.rs`, existing router/scope/catalog lifecycle tests | -| Traversal depth/entry/cancellation bounds; stalled observation/link publication; numbered output expansion; post-dispatch encoding loss | `fs-operation/src/traversal_tests.rs`, `checkout_race_tests.rs`, `tools/src/read.rs`, `wip.rs` | -| Real prefixed HTTP provider → Remote session → router → native WIP; auth, read-only, lost/malformed post-commit response without retry | `worker/src/checkout_http_tests.rs` | -| Workspace proxy/External typed pairing and unknown-outcome propagation; Backend/Runtime Workspace, grant and generation isolation | checkout additions in `manage_workdir.rs` and Backend `server.rs`, plus full existing host/provider suites | -| Root/nested ignore parity, source precedence/negation/anchoring, directory pruning, denied/symlink rule confinement and bounded loading | `fs-operation/src/ignore_tests.rs`, `workdir/src/checkout_search_tests.rs`, native WIP ignore/link regression | -| Ordinary Tools text/hash/edit/write/search and provider symlink policy | full Tools, fs-operation and Workdir suites | - -The HTTP fixture tests actual network/codec boundaries but is not a production -Backend grant-network end-to-end deployment. Production routing/authorization is -covered separately by the existing host suites. Fault tests are deterministic -boundary simulations, not power-loss tests. Metadata/pinned-handle tests do not -close the external-editor final-check/rename window documented above. No live -Worker or deployed environment needs updating/restarting to execute this matrix. diff --git a/docs/design/compaction.md b/docs/design/compaction.md deleted file mode 100644 index bcd66f0c9..000000000 --- a/docs/design/compaction.md +++ /dev/null @@ -1,124 +0,0 @@ -# Compaction - -Compaction exists because long-running Workers need durable continuity without sending the entire transcript forever. - -## Pruning vs compaction - -Pruning is request-local packing of existing history. It chooses what to include for one model request while preserving the underlying committed log. - -Compaction is a durable transition. It creates a new summarized state and should be recorded so later turns can explain why older details are no longer directly present. - -## Token-budget protection - -Safety checks should use effective backend/context limits, including in-flight usage. Cached input tokens still occupy context even if upload displays subtract them. - -When a provider returns an exact `input_total_tokens` measurement for the whole prompt shape, Yoi can treat it as authoritative and estimate only incremental growth after that measurement. It should not fake a system/history split or tune thresholds just to mask estimator bugs. - -## After compaction - -Compaction output is not automatically safe. The post-compact context must be revalidated before the next request. - -A `just_compacted` flag must not bypass safety checks. It is easy for a compact summary, retained tail, or prompt resource change to still exceed a context limit. - -`result_context_max_tokens` is an advisory warning threshold for the assembled replacement -history, not a hard activation cap or a provider context limit. Its estimate includes the summary, -auto-read files, references, retained tail, and TaskStore context. Exceeding it emits a Compactor -warning and continues without discarding the generated summary or stripping auto-read/retained -content. `0` disables this warning. Normal request-time safety checks still apply after activation. - -## Large sessions - -Large-session compaction should not send an entire prefix transcript as the summary input. Prefer bounded overview/index inputs plus exploration, then keep the retained tail small and explicit. - -This keeps compaction cost predictable and avoids turning a context-recovery mechanism into the largest prompt in the session. - -## History integrity - -Compaction should preserve persisted reasoning history and avoid serializing unverified hidden reasoning context. Trace and metrics can count request shape and reasoning items without smuggling hidden provider state into model input. - -The important property is explainability: after compaction, records should still show what summary replaced which older context and why future turns can rely on it. - -## Binary tool details - -Durable binary attachments are not text and must not be charged by applying the text -`bytes / 4` fallback to their base64 persistence encoding. Provider usage measurements remain -authoritative for requests that actually include an attachment. - -An image returned by `ViewImage` is sent for the immediate follow-up model request. Later requests -project the consumed image body out while retaining the tool summary, so the Worker can call -`ViewImage` again when it needs the current file. Compaction likewise never carries image bodies -into the replacement Segment: retained images become related-file references with their Workdir -alias and path where available. - -The compactor may auto-read only UTF-8 text with `mark_read_required`. Images and other binary -files can be nominated only with `add_reference`; the next Worker fetches them on demand with -`ViewImage` or the appropriate read tool. - -## Activation boundary - -A compacted Segment is staged completely before it can become authoritative. The -activation boundary uses this lock order everywhere a live Worker changes Segment: - -1. stop the compaction service and remove its registry record; -2. acquire the Worker's append barrier; -3. write the final pending-submission checkpoint and stage the replacement Segment; -4. acquire and preflight the machine-wide Worker allocation lock; -5. compare-and-swap `metadata.json` from the source Segment to the replacement; -6. atomically replace the allocation table while still holding its separate lock; -7. publish the append destination, session projection, sink, and in-memory history; -8. release the append barrier. - -Submit and Notify acceptance take the same append barrier. The controller persists -requests received while manual compaction is running immediately rather than holding -volatile deferred methods, so an overlapping Shutdown cannot discard a SubWorker -completion. Consequently an accepted item is either durably included in the -replacement checkpoint, or waits until every live authority points at the replacement -and is then appended there. Restore admission takes the allocation lock, so it cannot -observe the interval between the metadata CAS and allocation hand-off. - -The metadata CAS is the commit point. Failures before it leave the source Segment -active. A failure updating allocation after it marks the old in-memory writer -unusable and retains the machine-wide allocation lock: further acceptance fails -closed and restore admission cannot register a competing writer until process -teardown, after which restore converges from the replacement named by metadata. -The Worker must likewise remain non-idle while terminal compaction-service cleanup -is pending. Cleanup authority is stored on the Worker (not only on one compaction -future). A failed stop returns a typed cleanup-pending outcome without clearing that -authority; the next compaction boundary retries it. The authority and lifecycle -reference are cleared exactly once after the registry record is gone, before -activation or a terminal idle-capable result. While either activation repair or -cleanup attention is outstanding, the controller fences Submit/Notify dispatch and -does not start queued work. Shutdown is itself a cleanup barrier: it retries the -Worker-owned stop authority with backoff and does not emit its terminal event until -the registry record is gone. - -## Metrics and comparison procedure - -Compaction measurements stay out of the ordinary transcript. They are appended as -`metrics` extensions and are read only through the explicit -`session-metrics` reader/export path. `read_session_metrics` attaches the durable -`segment_id` and `SegmentStart.compacted_from` lineage to each record. - -1. Compare runs with the same workload and model settings. Record manual versus - automatic mode, `threshold_policy`, and `retained_token_budget`. -2. Start with the `compact.start` correlation ID. Join it to `compact.finish` and - the metric breakdown on the result Segment, then to the next normal request's - `compact.post_request`. Use `compacted_from`, rather than timestamps, to prove - the source-to-result Segment relationship. -3. Compare `compact.retained_tokens`, `compact.overview_tokens`, - `compact.summary_tokens`, `compact.auto_read_tokens`, and - `compact.result_context_tokens` to explain the context-size change. -4. Compare the Compactor's input/output/cache-read/cache-creation tokens, request, - turn, tool-call, and duration metrics. The current provider `UsageEvent` has no - pricing authority, so `compact.cost_usd` is valueless with - `status=unavailable` and `reason=provider_usage_unpriced`; do not fabricate a - zero cost. Record a numeric value only after a price authority exists. -5. Aggregate failure and cancellation using the fixed `failure_category` values. - Metrics must never contain provider error text, prompt/session content, host - paths, or secrets. -6. Report at least success rate, next-request token/cache changes, Compactor token - use, and duration per workload before changing retention or thresholds. - -Export ordering is deterministic through timestamp, phase, Segment, and -`log_index`. Causality still comes from `correlation_id` and `compacted_from`, not -from timestamp order alone. diff --git a/docs/design/context-history.md b/docs/design/context-history.md deleted file mode 100644 index 1c63af0b0..000000000 --- a/docs/design/context-history.md +++ /dev/null @@ -1,49 +0,0 @@ -# Context and history - -Any input that can influence the model across a turn must be committed to history before it is placed in model context. - -This rule protects both explainability and prompt-cache behavior. If the model reacts to a context-only insertion that is not in history, later turns cannot explain why the assistant made that decision, and cache anchors become harder to reason about. - -## Allowed context transformations - -A context transformation is acceptable when it is reproducible from durable Worker state and does not introduce new volatile facts. - -Examples: - -- Pruning old history according to persisted/session-derived state. -- Truncating oversized tool result content while preserving the committed result boundary. -- Adding prompt-cache anchors derived from stable context structure. -- Rendering the same materialized system prompt string after compaction. - -These transformations change how context is packed, not what happened. - -## Forbidden context injection - -Do not insert turn-crossing information directly into context without first appending it to `worker.history`. - -Forbidden examples: - -- Delivering a `Notify` or `WorkerEvent` only as a temporary context note. -- Adding a system reminder that explains behavior but is not persisted. -- Rewriting old messages to include new facts. -- Letting UI/controller-only state become model-visible without a committed record. - -If new information should affect the model, append it to history and commit it. `history.json` / session persistence follows from the Engine history path. - -## Prompt cache implications - -Yoi optimizes for predictable prompt shape, not only small requests. Hidden context insertions make cache behavior hard to reproduce because the model-visible input can change without a corresponding history record. - -Ordinary new inputs are not the problem: user messages, committed events, compacted summaries, and tool results are expected additions to history. The avoidable problem is changing prior or side-channel context in a way that is neither durable nor explainable from records. - -## Pruning and compaction - -Pruning is a request-time packing operation that chooses which existing history to include under a token budget. Compaction is a durable history operation that creates a new summarized state and must be committed. - -After compaction, context safety must be revalidated. A flag such as `just_compacted` must not suppress safety checks, because compact output can itself be too large or malformed for the next request. - -## UI and orchestration consequences - -The TUI should display committed events rather than inventing transcript blocks. Orchestration should treat child notifications as prompts to inspect state, not as state themselves. - -This keeps future turns able to answer: "what did the agent know, when did it know it, and where is that recorded?" diff --git a/docs/design/durable-operations.md b/docs/design/durable-operations.md deleted file mode 100644 index f1c878c2f..000000000 --- a/docs/design/durable-operations.md +++ /dev/null @@ -1,418 +0,0 @@ -# Durable operation design - -Durable operation records make retries converge on the same authorized intent -and preserve the facts needed to explain an externally visible result. They are -not execution traces and must not mirror every Rust function or implementation -step as persisted state. - -This document defines the cross-domain rules for operation identity, state, -checkpoints, child operations, failure evidence, and terminal disposition. -Domain code may use different records where its atomicity boundary differs, but -it should classify the operation before choosing a schema. - -## Core rule - -Persist authority and non-reconstructable facts, not control flow. - -A value belongs in durable operation state only when at least one of the -following is true: - -- it identifies the caller's stable intent and detects conflicting reuse; -- it freezes authority or configuration that an exact retry must continue to - use; -- it binds a preallocated or created resource to the operation; -- it records an externally visible side effect that cannot be safely derived or - repeated; -- it records the final domain result or disposition; -- it provides bounded evidence needed for retry, reconciliation, or audit. - -A local step does not become durable merely because it occurs before or after -another function call. If current authority can be reread or the step can be -repeated safely, derive or repeat it instead of adding a stage. - -## Resilience boundary - -These rules cover normal product retry and recovery boundaries: duplicate -requests, returned errors, timeouts, known partial-completion outcomes, process -restart from the last committed facts, and retries of provider operations with -an explicit idempotency or observation contract. - -They do not require the system to survive an unexpected stop at every point -while Rust code is executing. A panic, abort, process kill, machine loss, or -power failure may occur between an external side effect and its next durable -checkpoint. Yoi does not attempt to close every such instruction-level crash -window by writing a stage before and after every `await`, function, or provider -call. - -Consequently: - -- do not claim general exactly-once execution; -- do not introduce write-ahead stages solely to model arbitrary Rust - control-flow interruption; -- rely on SQLite transaction atomicity for work inside one database transaction; -- prefer provider idempotency, compare-and-swap, stable resource identity, and - authoritative observation for external effects; -- when an uncovered crash window cannot be reconciled automatically, retain the - last committed facts and surface an `unknown` or attention-required - disposition rather than guessing that the side effect did or did not happen. - -A domain may require a stronger crash-consistency contract for a specific -destructive or security-sensitive effect. That requirement must be explicit and -must define the provider protocol, checkpoint ordering, replay behavior, and -reconciliation evidence. It is not implied by calling a record a durable -operation. - -## Classify the record before adding state - -Not every record containing an `operation_id` is a state machine. Use one of the -following shapes. - -### Atomic idempotency ledger - -Use an idempotency ledger when all authoritative mutations and result recording -commit in one database transaction. - -The record normally contains: - -- Workspace and operation identity; -- a fingerprint of stable caller intent; -- the created resource or result identity; -- the committed content digest or immutable change ID and timestamp where relevant. - -It does not need `pending`, `executing`, or intermediate stages. An exact retry -returns the recorded result. Reusing the same operation identity with a -different fingerprint fails. - -Repository secret mutation results, Workspace resource creation results, and -transactionally appended domain events are examples of this shape. - -### Reservation - -Use a reservation when an identity or exclusive right must exist before a later -binding can complete. - -Persist factual transitions such as: - -- the reserved resource identity; -- the immutable request fingerprint and authority snapshot; -- the concrete resource or assignment bound to the reservation; -- reservation expiry or release evidence when the contract requires it. - -Do not model internal dispatch, validation, construction, or callback steps as -reservation states. A nullable result binding or a small `reserved | created` -state can be sufficient when those values correspond to real authority facts. - -### Durable side-effect operation - -Use a durable side-effect operation when work crosses a database/provider -boundary and a retry needs durable intent or result evidence. - -The default lifecycle is deliberately small: - -```text -pending -> completed -pending -> failed -failed -> pending # only when the domain explicitly permits retry -``` - -Existing code may use `succeeded` for the successful terminal value; new naming -should prefer `completed`. Do not rewrite applied migrations or historical audit -text only to normalize that word. - -The operation should contain: - -- stable operation identity and request fingerprint; -- immutable resolved authority needed by an exact retry; -- preallocated resource identity where it prevents duplicate creation; -- only the necessary irreversible checkpoints; -- bounded failure evidence; -- the final result and domain disposition. - -`pending` means that the intent remains open and current authority must be -reread before progress. It does not identify which Rust function should execute -next. `failed` records the latest terminal attempt outcome; retryability is an -explicit domain rule, not something inferred from the word. `completed` means -the operation's required result and evidence are durably committed. - -### Parent workflow - -A parent workflow coordinates domain operations but does not duplicate their -lifecycle. - -Persist: - -- the parent intent and fencing authority; -- stable child operation identities; -- the final workflow result or disposition; -- bounded attention or decision evidence. - -Read child state from the child authority. Do not copy child states, provider -stages, Worker status, attachment status, or Workdir status into a second parent -state machine. A parent cleanup workflow will often need only -`pending | completed`; child failure remains on the child operation and appears -in the parent as current attention metadata. - -Creating or binding a child must itself be idempotent. Prefer a deterministic -child operation identity or persist the child reference atomically with the -parent decision so a retry cannot create siblings for one intent. - -## Checkpoint rules - -A checkpoint records a fact that changes retry semantics. It is not a progress -notification. - -Add a checkpoint only when all of the following hold: - -1. A side effect may already have occurred outside the current transaction. -2. Current authority cannot derive the fact reliably enough for safe retry, or - repeating the effect is not safe under the provider contract. -3. The retry algorithm changes after the fact is committed. -4. Tests can exercise behavior before and after the checkpoint. - -Prefer factual fields over stage names: - -- `provider_deleted_at` is evidence that provider deletion succeeded; -- `child_operation_id` binds delegated work; -- `result_digest` identifies the committed result content; -- `target_ref_after` records verified merge evidence. - -Avoid fields such as `validating`, `closing_session`, `detaching`, -`deleting_registry`, or `finalizing`. Those names describe code location, not -durable authority. If those steps are safe to rerun or their result can be read -from Worker, attachment, Workdir, repository, or provider authority, they are -not checkpoints. - -A checkpoint must never claim more than the authority that produced it. For -example, sending a provider request is not proof that provider deletion -completed, and receiving a Worker notification is not proof that a Ticket or -cleanup workflow completed. - -## State, failure, blockers, and disposition are separate - -Do not overload one enum with unrelated dimensions. - -- **Operation state** says whether the intent is open, completed, or has a - recorded failed attempt. -- **Failure evidence** records a bounded category, timestamp, and safe - diagnostic detail for the latest failure. -- **Blockers and eligibility** are normally derived by rereading current - authority. Persist them only as audit or attention evidence, not as a - substitute for live validation. -- **Disposition** records what the domain decided to retain, delete, release, - tombstone, abandon, or leave unknown. -- **Attention metadata** explains why automated progress currently cannot - continue and what authority must change. - -Values such as `blocked`, `executing`, `stale`, `dirty`, `retained`, and -`deleted` therefore do not all belong in one operation-state enum. Some are -derived conditions, some describe transient execution, and some are domain -results. - -Before every retry or side effect, reread live authority and revalidate its -fence. A previously recorded blocker does not prove that the operation remains -blocked, and a previously unblocked operation does not retain permission after -assignment, ownership, content, or attachment authority changes. - -## Identity and fingerprinting - -Every externally retryable operation has a stable identity in its owning -Workspace or authority scope. The operation fingerprint represents stable caller -intent, not generated results or mutable observations. - -Include inputs whose change would mean a different requested operation. Exclude: - -- generated resource IDs when the Server allocates and persists them as the - result; -- timestamps assigned by the Server; -- retry counters and diagnostics; -- current provider observations that are expected to change; -- secret bytes and credential material. - -Resolved authority snapshots may be stored separately from the caller -fingerprint. An exact retry uses the persisted snapshot where replay convergence -requires it; a new operation resolves current authority. Unknown, foreign, or -conflicting operation identity fails closed. - -## Transactions and external providers - -Keep database work in one transaction whenever the owning authority and result -live in the same database. Do not create a durable operation merely to split a -transaction that can remain atomic. - -When an external provider is involved: - -1. reserve stable intent and identity if retry needs them; -2. invoke the provider with the strongest available idempotency, expected-old - content digest, or stable resource key; -3. verify the provider result through authoritative response or observation; -4. commit only the checkpoint or result evidence that changes retry behavior; -5. on retry, reread both the operation and current domain/provider authority - before acting. - -Compensation is a domain operation, not an invisible `finally` block. If -compensation has its own external side effects or retry lifecycle, give it a -stable child operation identity rather than expanding the parent into a list of -cleanup stages. - -## Workdir removal application - -Workdir removal is one durable side-effect operation in the Workspace Server -DB. It binds the Workspace, Workdir, owning Runtime, -Repository/materialization identity, source actor, stable intent fingerprint, -lifecycle, retry metadata, and bounded result. Runtime URL, provider handle, -host path, credentials, and caller-selected Runtime are not operation inputs. - -A durable one-pending-operation constraint plus an atomic attempt claim prevents -concurrent callers from entering the provider side effect for the same Workdir; -the in-process resource lock is an additional serialization layer, not the sole -authority. Each active attempt persists the Server process ID and process-start -marker. Recovery reclaims only an owner proven missing or replaced; a live or -unobservable owner is never stolen. The reclaim transaction compare-and-set -checks the exact proved owner snapshot and attempt count so stale orphan proof -cannot overwrite a newer live claim. - -Each attempt: - -1. resolves or revalidates the persisted same-Workspace Workdir, Runtime, - Repository, and materialization identity; -2. checks current attachments, attachment reservations, current assignment - occupancy, retention/cleanup holds, and pending materialization authority; a - failed Workdir-create retry must atomically return to `pending` before - provider work and is rejected while removal is pending; -3. retains dirty, occupied, blocked, or otherwise unknown Workdirs without - detaching a Worker or forcing deletion; -4. observes the owning Runtime/provider and calls its existing Workdir cleanup - only for an eligible clean Workdir; -5. treats only successful provider cleanup or exact - `working_directory_not_found` as removal evidence; -6. deletes the Backend Workdir registry row and commits the operation's - `completed`/`removed` result in one SQLite transaction. - -A provider error leaves the registry intact and records a bounded -`attention_required` result with explicit retryability. Startup recovery lists -`pending` and retryable `failed` operations, then executes this same path after -rereading live authority. `WorkdirDelete`, Workspace REST removal, Runtime -cleanup execution, and recovery must not maintain separate inline -provider-delete paths. - -The public request contains only `working_directory_id` plus a bounded reason. -The public result contains only the Workdir ID, -`removed | retained | attention_required`, retryability, and an optional bounded -failure category. Internal operation identifiers, checkpoints, provider paths, -and credentials are not public DTO fields. - -## Backend-owned bounded Jobs - -A Backend Job is durable side-effect intent whose model execution is delegated to -one ordinary Runtime-managed Worker per attempt. The Workspace Server owns the -Job; `source_worker` and `notification_target` are provenance and delivery -addresses, not parent ownership. Stopping or losing either Worker does not revoke -Job intent. Attempt records retain the bound Worker identity as historical -evidence without foreign-keying Worker retention to Job retention; normal Worker -stop, archive, pin/hold, and removal policy remains authoritative. - -The durable split is deliberate: - -- `backend_jobs` stores immutable bounded input, its fingerprint, the current - attempt number, and the final structured result or terminal failure; -- `backend_job_attempts` binds one attempt and input digest to one dedicated - `(Workspace, Runtime, Worker)` identity, a precommitted tracked Runtime - submission request identity, a deadline, and its bounded outcome; -- `backend_job_deliveries` records best-effort notification separately from the - result. Its fixed-size collision-resistant durable delivery ID is also the - Runtime's tracked notification request ID. Runtime reports acceptance only - after the Worker has persisted the correlated notification receipt, so restart - recovery can replay an ambiguous `sending` claim without duplicating Worker - context. A missing or stopped recipient never discards a result or reruns - model work. - -Exact dispatch retry reuses the deterministic attempt and Worker allocation key. -Only explicit re-evaluation of a definitively failed attempt increments the -bounded attempt number; unknown outcomes are not replayable. The runner -uses the existing embedded Runtime spawn, Worker registry, Session input, -observation, usage, and lifecycle paths with no Workdir attachment. Each client -explicitly selects an existing registry Profile. The runner resolves the complete -Builtin/Workspace project catalog through the normal virtual config bundle and -prompt projection path; browser launch candidates remain a narrower UI policy. -Raw Profile source and filesystem/path selectors are not accepted. A selected -recipe's instruction and features are not replaced with `backend-job` defaults. -Profiles requiring ungranted Workspace mutation, ordinary Memory, child/peer, -Flow, Workdir or MCP authority fail explicitly; no caller scope is inherited. -The Ticket checker continues to select `builtin:backend-job` itself. - -Job identity/result capability comes from an immutable Host-authored create and -restore binding, not the Profile name, tag, instruction, Worker display name or -Runtime placement. Backend checks the exact live Job/attempt/input digest and -authenticated Worker before accepting results or domain operations. Profile -policy alone never confers a subject grant. - -A common optional resource key serializes Jobs using their existing durable -intent/attempt records (schema 81), not a second domain workflow table. Exact -request replay is distinct from a resource collision: the latter returns the -original active immutable request with `resource_reused`, and never claims that -new input was processed. Pending/unknown Jobs and **all** unfinished attempt -cleanups retain the fence. Retry requires definitive failure, remaining attempt -budget, completed prior cleanups and no competing resource owner. Unknown intent -remains attention-required even after its Worker is removed. - -Success is accepted only by the structured result boundary. Before Runtime -input can execute, the attempt is fenced by Workspace, Job ID, attempt ID, input -digest, bound Runtime Worker identity, and a fixed-size collision-resistant -tracked submission request identity derived from the attempt. Ordinary Worker -input and Console mutation are rejected for -Job-owned Workers, so no later run in the same Session can exercise the result -capability. Result JSON bytes are bounded before persistence. Worker final prose, -Idle, Stopped, or a successful model call are not success evidence. Identical -structured submission is idempotent; conflicting replay, stale or late attempts, -wrong input digests, and other Workers fail closed. - -On restart, an unbound reservation is redispatched with its stable allocation. -A dispatched attempt continues only when its bound Worker and precommitted -tracked submission identity remain observable; otherwise it becomes `unknown`. -Ambiguous dispatch outcomes are also `unknown` and cannot be retried as a new -attempt. Durable deadlines fence acceptance before the runner requests Worker -stop. Notification recovery is independent: `sending` or `unknown` delivery -claims return to `pending` and replay the same tracked notification request ID. -The design does not claim general exactly-once model execution: a crash between -Runtime creation and durable binding converges through stable allocation where -available, and unrecoverable external evidence is surfaced as `unknown` rather -than guessed. Historical attempts preserve the reason for retry without -duplicating the Runtime Worker lifecycle state machine. - -Workspace Worker subscriptions expose an explicit Job binding (`job_id`, -`attempt_id`, and `purpose`) in snapshot and live projections. Sidebar hiding is -a Web View policy over that canonical metadata; the Worker remains registered, -observable, and eligible for its canonical read-only Console route. - -## Diagnostics and audit - -Persist bounded error categories and identifiers needed to investigate or retry. -Do not persist credentials, provider handles, raw command output, raw prompts, -full session transcripts, or host paths in ordinary operation diagnostics. - -Attempt counts, last-attempt timestamps, and safe provider categories may help -operations, but they are telemetry and evidence rather than lifecycle authority. -Logs may describe detailed execution stages; the durable record should remain -centered on intent, checkpoints, result, and disposition. - -## Applying this rule - -For a new or materially changed operation: - -1. identify the owning authority and transaction boundary; -2. classify it as an atomic ledger, reservation, durable side-effect operation, - or parent workflow; -3. define stable identity, fingerprint, and exact-retry behavior; -4. list external side effects and decide which are idempotent or authoritatively - observable; -5. add only checkpoints that change retry behavior; -6. keep child operation state in the child authority; -7. separate failure, blocker, attention, and disposition from lifecycle state; -8. state the unsupported crash windows honestly; -9. test fingerprint conflict, exact retry, authority revalidation, checkpoint - replay, and result/disposition projection as applicable. - -Existing operation schemas need not be rewritten solely for vocabulary -consistency. When an operation is changed for functional reasons, use this -classification to remove derived or control-flow stages rather than adding -another special-case lifecycle. diff --git a/docs/design/feature-chat-invocations.md b/docs/design/feature-chat-invocations.md deleted file mode 100644 index f1948f00e..000000000 --- a/docs/design/feature-chat-invocations.md +++ /dev/null @@ -1,92 +0,0 @@ -# Declarative Feature chat invocations - -Feature chat invocations are explicit structured user input. They let an enabled Feature contribute discoverable chat actions without teaching each client Feature-specific syntax or asking the LLM to infer an operation from text. - -## Namespaces and identity - -Composer sigils have separate meanings: - -- `:` is a client control command for the whole draft. -- `@` is a Worker-readable file reference. -- `/` selects a Feature invocation and may occur among ordinary prose. -- `#` has no invocation meaning. - -A public name such as `attach` is presentation, not authority. The payload carries the installed source-qualified `FeatureInvocationIdentity` (for example `builtin:attachments/attach`). Installation rejects duplicate public names and aliases, so clients never choose an arbitrary winner. Disabled, failed-to-install, or unavailable Features are absent from completion and cannot be resolved at execution. - -## Declarative grammar - -The common spelling is: - -```text -/name(positional, named=value) following natural language -``` - -The closing `)` is the explicit argument/prose boundary. String values use JSON double quoting and escaping (including Unicode escapes and escaped control characters). Bare values end at whitespace, comma, or `)`; comma separates arguments and a trailing comma is accepted. Integers are decimal signed 32-bit values (`-?[0-9]+`); booleans are `true` or `false`. Positional indices are contiguous; every argument can also be represented by its stable name in restored chip text. Mixed named/positional input advances the positional cursor only for positional values, and assigning the same argument twice is an error. - -A descriptor is declarative data, for example: - -```json -{ - "identity": "builtin:prepare/prepare", - "name": "prepare", - "aliases": [], - "display_name": "Prepare context", - "description": "Prepare scoped context before processing the request", - "syntax": "parenthesized", - "arguments": [ - {"name": "path", "position": 0, "required": true, "value_type": {"kind": "worker_file"}, "completion": {"kind": "worker_file"}}, - {"name": "mode", "required": false, "value_type": {"kind": "enum", "values": ["safe", "fast"]}, "completion": {"kind": "static", "values": ["safe", "fast"]}} - ] -} -``` - -`/prepare("資料/a b", mode=safe) explain the result` produces a typed invocation and a separate following Text segment only when explicitly confirmed. Missing required arguments, unknown or duplicate arguments, invalid enum/primitive values, and incomplete quotations remain validation errors, not partial business execution. - -Descriptors declare requiredness, value type, positional index, completion strategy, and optional description. Completion strategies are static values, Worker files, client files, or a host-owned provider. Metadata and completion are side-effect free. Arbitrary parser or provider code never crosses into a client. - -Clients may parse this spelling only for input assistance. A call becomes executable only after the user selects a descriptor and the client emits `Segment::FeatureInvoke`. Free text, URLs, paths, pasted content, attachments, assistant output, and tool output are not scanned or promoted by the Host. - -## Composer and completion lifecycle - -TUI and Web composers retain selected invocations as atomic typed chips containing identity and structured arguments. Backspace/Delete and range deletion remove an unsent chip; removal is not an undo for an already executed operation. Draft history, transport retry, queueing, Session history, snapshot/live projections, and rewind restoration preserve the typed segment rather than flattening and reparsing it. - -Completion requests carry a kind, fresh `request_id`, and source-qualified argument context. Host and Runtime echo the request ID unchanged. Clients correlate replies with that nonce plus their current draft identity/content and edit invalidation state, cursor, target Worker, authority snapshot, and connection generation; even identical-prefix/context ABA replies are discarded. Old uncorrelated replies cannot establish a selected invocation. Client-file arguments use a declared `InvocationClientAdapter`; they do not ask the Worker to resolve a client-local path. - -## Attachment adapter - -`/attach` is the built-in real use of the generic contract. Its descriptor declares a required client-file argument and the attachment client adapter. - -- TUI may select a path visible to the TUI process, stage an upload, and replace the invocation intent with an `UploadedFile` chip. -- Web opens the existing browser picker; it does not enumerate the browser host filesystem. -- The local path is never sent to or resolved by the Worker. -- Uploading is cancellable draft staging, not invocation execution. Removing an in-progress upload chip by Backspace/Delete, range replacement, or draft abandonment cancels its reservation and adapter upload; the removed upload must no longer block a text-only Submit. Late callbacks and Undo cannot restore that cancelled reservation as sendable input. A completed unsent uploaded-file chip may retain its staged resource while editor Undo can restore it, releasing abandoned resources at the draft boundary; a late completion must never turn a deleted pending reservation into such an Undo-owned resource. -- Authoritative `SubmissionAccepted` (not socket-send success) transfers uploaded references to the durable input lifecycle. Draft removal cannot delete an in-flight reference whose admission may already have succeeded, or an accepted attachment. Rejection restores the exact typed intent for retry; retry retains request/invocation IDs. Upload completion alone is not submission acceptance. -- TUI standalone currently has no client-local upload staging transport. It reports the unsupported adapter and preserves the draft; it does not reinterpret a local path as a Worker host path. - -There is no `/clear-attachments` command. Normal composer editing is the removal mechanism. - -## Installation and execution - -A Feature declares invocation descriptors using `FeatureDescriptor::with_chat_invocation` and registers matching handlers with `context.chat_invocations().register` (or `register_with_completion`) during `FeatureModule::install`. Registration must match the declared descriptor and the installing Feature's source-qualified identity. The invocation registry is checkpointed and rolled back with other Feature contributions. Only successfully installed, currently available descriptors are exposed through completion. - -Authority-sensitive handlers override the side-effect-free `is_available()` discovery hook and `validate()` semantic hook. They must still revalidate authorization and use scoped idempotency at their business commit boundary inside `invoke()`; defaults grant no extra domain authority. Availability is checked again after asynchronous provider/handler work. Reusing a successful preparation result or resuming its dependent model run also rechecks the side-effect-free semantic/argument-specific authority validation; general Feature availability alone is insufficient. Completion strategies come from installed metadata, never from a client-selected provider identifier. Enum and boolean candidates can derive directly from value types when no explicit completion strategy is supplied. - -Completion output is bounded to 64 entries / 64 KiB encoded data, with per-field byte limits. Oversized candidate values are discarded instead of altered. Handler results allow 4 KiB messages and 16 KiB context; oversized results/errors after execution become `OutcomeUnknown`, withholding dependent model context rather than assuming the operation was unexecuted. - -The Worker validates every structured payload again against the installed descriptor. It executes invocations in segment order only when the durable submission is activated, after committing the user input and before the dependent LLM request. A completion choice, draft edit, queued-but-cancelled submission, reconnect, history view, or restored snapshot never executes a handler. - -A submission may contain at most eight invocations with distinct invocation IDs. The first failed or outcome-unknown result fences later invocations **and prevents the dependent LLM run**; no cross-invocation rollback is promised. Every result is committed as typed Session history and rendered as bounded model-visible context. `OutcomeUnknown` is distinct from failure and is never automatically retried. - -Before calling a handler, the Worker appends a start receipt through the existing `worker.pending_activations.v1` extension checkpoint. The result and updated receipt are appended together with the annotated system item. An unresolved start (including handler completion followed by a failed result append) remains a fence across same-process retry, restart, Pause/Resume, compaction, and rewind. Reusing an ID with changed arguments is rejected; exact replay can reuse its recorded result but does not invoke again. Receipt capacity is bounded and refuses untracked execution rather than evicting recovery evidence. Restoring or displaying history never reexecutes a handler. - -Rewind preserves invocation-only receipts and the recovery-blocked fence independently of submission/notification queues. The retained history prefix and recovery checkpoint replace the segment together through the existing atomic Store replacement contract. There is no truncate-then-checkpoint repair window in which a crash can erase a completed or ambiguous business operation's recovery evidence. - -The recovery receipt retains the invocation ID, qualified identity, SHA256 typed-payload digest, and bounded result—not another copy of every argument. Invocation IDs are nonempty and at most 128 UTF-8 bytes; clients generate fresh UUIDs on explicit selection and preserve them through transport retry. The session retains at most 1,024 recovery receipts without eviction. Corrupt, duplicate, inconsistent, or oversized recovery evidence fails closed. - -`submission_request_id` protects submission admission and retry, while `invocation_id` is the stable idempotency key passed to a side-effecting handler. A handler must scope that identity at its domain boundary. This design does not claim a new generic exactly-once substrate. - -## Authority boundaries - -Possession of syntax, metadata, identity, or a typed payload grants no permission. Runtime target, enabled Feature, argument schema, and captured domain authority are checked at execution. Completion providers cannot perform business mutations or expose credentials. Handlers may share an underlying domain operation with Worker tools, but chat discovery does not expose every tool automatically. - -Text-only clients submit text. The Backend and Host never reinterpret text as an invocation. Unknown typed input is visibly unresolved in restoration and **rejected on submission**, not silently flattened or processed by the LLM. Notify remains text-only: extra typed wire fields or typed Runtime input cannot be discarded into an advisory message or executed as a side-effecting call. diff --git a/docs/design/flow-state-graph.md b/docs/design/flow-state-graph.md deleted file mode 100644 index ffada9be8..000000000 --- a/docs/design/flow-state-graph.md +++ /dev/null @@ -1,111 +0,0 @@ -# Flow state graph and verifier - -Flow is a Workspace-scoped declarative state graph. A Worker operating under an active Flow can request evaluation of its current state's outgoing conditions, but cannot select or write a target state directly. - -## Source authority - -A Workspace-authored Flow source is one DCDL document stored under a virtual path such as `flows/coder-review.dcdl`. Built-in Flow sources are compiled resources under `resources/flows/*.dcdl` and use read-only virtual paths such as `builtin/flows/coder-review.dcdl`. The source document is the graph authority; states and transitions are not normalized into independently editable relational records. - -Every invocation uses a source-qualified typed selector: - -```text -builtin: -workspace: -``` - -Unqualified selectors and implicit override precedence are rejected. `builtin:coder-review` resolves from the embedded resource catalog. `workspace:coder-review` resolves from the current Workspace DB source. Built-in and Workspace sources with the same slug coexist as distinct logical records. - -The compiler pipeline is: - -```text -DCDL source - -> decodal evaluation - -> private Serde-compatible value - -> typed Flow source schema - -> graph validation - -> CompiledFlowDefinition -``` - -The Serde-compatible intermediate is private compiler infrastructure. Public APIs and persisted Flow runtime records use typed Flow domain values. - -`flow_sources` stores one current logical Workspace-authored source per Workspace/slug. Every changed source retains the original content, content digest, and compiled definition in immutable source history. Built-ins are read-only embedded resources: resolving one compiles and returns its exact source snapshot without writing it into Workspace DB or maintaining a manual change counter. Runtime pins source identity, content digest, and compiled definition in Worker state, so editing a Workspace source or updating a built-in resource never changes an existing instance. - -The compiler rejects unknown fields, unsupported schema versions, invalid/reserved identifiers, unknown transition targets, authored `$cancelled` state/targets, terminal states with outgoing transitions, non-terminal states without transitions, unreachable states, and reachable closed paths that cannot reach a user-declared terminal state. It injects the synthetic exceptional-cancellation transition and `$cancelled` terminal state after validation. - -## Runtime-owned instance and event authority - -Flow source authority and Flow execution authority are split at the immutable source snapshot boundary. - -The Workspace Backend stores only current Flow sources and immutable source history. Resolving a source-qualified selector returns the Workspace id, Flow id, content digest, and compiled definition. Resolution is read-only with respect to Flow execution: it never creates an instance, attempt, or event. - -One Runtime Worker durably owns: - -- the pinned source snapshot and compiled definition; -- its active Flow instance, current state, and lifecycle status; -- its active transition attempt; -- its ordered Flow events. - -The complete `FlowRuntimeState` is persisted as a typed `flow.runtime.v1` Worker session extension. Initial Flow state is committed in the same `UserInput` log record as the entered-state instructions and remaining Submit segments. Backend therefore cannot contain an active instance that Worker history has never observed. - -Transition mutations clone the current Runtime state, append ordered events, persist the new session extension, and only then replace the in-memory projection. A persisted verifying attempt survives Runtime restart and same-Worker restore; the next transition request recovers it instead of creating a competing attempt. - -The executable checks live in `FlowRuntimeState::begin_or_recover_transition` and `resolve_active_transition`, with private `begin_transition` / `resolve_transition` helpers: - -- Before starting, require an active instance, no other active attempt, and the same compiled content digest as the pinned source. Reject an attempt ID already present in `TransitionRequested` events, including cancelled/failed attempts. -- Before consuming a result, require its exact instance and currently reserved attempt ID. Then compare the compiled and captured definition digests against the pinned digest. -- Before applying a successful verdict, require the instance to remain active and its current state to equal the captured source state. Validate the complete outgoing-condition result set and accept exactly one met condition. -- Completion or cancellation releases the active attempt. A delayed result cannot consume a newer attempt even if the graph returns to the same state. The persisted event log prevents reusing the old attempt ID after restore. - -There is no independent state counter or manually incremented built-in source counter. Event `sequence` remains only the append order of the actual event log, not a substitute for any of these conditions. - -Worker stop retains the Flow with the Worker session. Restoring the same Worker reconstructs the latest state from session extensions and the saved Profile still determines `feature.flow` eligibility. Worker deletion removes the owning Worker/session; Flow state is not implicitly handed to another Worker. - -Workspace Server schema migration v26 removes the legacy `flow_instances`, `flow_transition_attempts`, and `flow_events` tables. Backend/Web visibility, when needed, is a bounded Runtime Worker projection rather than a second instance authority. - -## Worker boundary - -Flow invocation uses the normal Submit segment vector rather than a Worker-create field: - -```json -{ - "method": "submit", - "submission_request_id": "018f4f15-5c41-7d3a-8a72-2e755bc71681", - "input": [ - { "kind": "flow", "selector": "builtin:coder-review" }, - { "kind": "text", "content": "Ticket 00001... implementation" } - ] -} -``` - -Runtime accepts exactly one Flow segment only when the resolved Profile enables `feature.flow` and a Workspace client is available. The Worker asks Workspace authority only for an immutable source snapshot, creates the instance locally, replaces the Flow segment with the entered state's instructions, and commits that runtime state atomically with the remaining Submit segments before LLM execution. A Worker with an active Flow rejects the duplicate input without changing its local state or events. - -The generic model-facing `WorkerSpawn` accepts `initial_submit: Vec` and routes them unchanged through the shared Workspace spawn request into Runtime `CreateWorkerRequest.initial_input`. It does not have a parallel `initial_text` or a role-specific `SpawnCoder` wrapper. Backend derives the flat content projection from the canonical segment vector, validates Flow shape before spawn, and includes the segment vector in lifecycle idempotency fingerprints. Runtime does not commit Worker creation or report spawn success merely because the initial Submit request entered the Worker's in-memory channel: Runtime assigns the Submit an opaque id, the Worker commits that id as an extension on the same `UserInput` entry as any initial `FlowRuntimeState`, and the execution backend must return a matching typed input-commit acknowledgement. Restoring the same Worker never replays spawn initial segments. - -When an Orchestrator supplies `ticket_id` to generic `WorkerSpawn`, the Worker tool derives the assignment operation id from the durable tool-call id rather than accepting lifecycle authority from model input. The shared Workspace worker-create route projects that request into a Coder Ticket-role intent and atomically applies the existing queued-Ticket assignment operation only after Runtime has returned the input-commit acknowledgement. A spawn or pre-commit input failure therefore leaves the Ticket queued and unassigned. - -`RequestFlowTransition` accepts only: - -```json -{ "reason": "bounded explanation" } -``` - -It does not accept Workspace, Flow, instance, Runtime, Worker, state, transition, or target identifiers. `RuntimeFlowCoordinatorClient` reads and persists only the current Worker's local `FlowRuntimeState`; no Workspace mutation client participates in transitions. Profile enablement makes the capability eligible but does not create or select an instance; without an active instance the transition request fails closed. - -The transition tool begins or recovers a locally persisted attempt, runs one internal verifier, and resolves the typed verifier outcome into the Worker-owned state. The resulting state and next-state instructions are committed in the normal tool result history. - -## Internal verifier authority - -The internal verifier receives: - -- one immutable snapshot of committed parent-session entries through `session-explore`; -- the captured current state, request reason, and complete outgoing condition list; -- `FinishFlowVerification`; -- when the parent has a Workdir session, only `Read`, `Glob`, and `Grep` backed by a `ReadOnlyWorkdirSession` capability-reducing wrapper. - -It does not receive Workspace, Ticket, Memory, Worker-management, write/edit, Bash, or command authority. The read-only wrapper reports only read capabilities, rejects mutation/command operations, and closing it does not close the parent's source session. - -`FinishFlowVerification` accepts exactly one `met | not_met | indeterminate` result and a bounded rationale for every transition id in the attempt. Missing, unknown, or duplicate ids are rejected before a result is recorded. A prose-only internal Worker completion is a failed verifier outcome, not a successful transition. - -## Separation from the role-owned loop - -This state graph does not implement Coder/Reviewer lifecycle, state-entry side effects, deterministic condition providers, or arbitrary Flow state data. The downstream role-owned loop submits the built-in Coder Flow segment, follows entered-state instructions, and uses Flow events plus typed review/repository evidence for its higher-level completion decisions. diff --git a/docs/design/memory-knowledge.md b/docs/design/memory-knowledge.md deleted file mode 100644 index 88d6e7842..000000000 --- a/docs/design/memory-knowledge.md +++ /dev/null @@ -1,23 +0,0 @@ -# Legacy Workspace Memory authority - -The single-document Workspace Memory product is a **legacy/deprecated** compatibility surface. Its document, staging candidates, and staging resolutions are durable Workspace context owned by the Workspace Server control-plane database. It is separate from subject-scoped subjektiv Memory, whose persistence authority is the Workspace's Server-managed `subjektiv` Feature database. - -Legacy Runtime Workers query and mutate the old document through the established typed `/api/w/{workspace_id}/memory...` operations and `MemoryQuery`, `MemoryReadDocument`, `MemoryUpdateDocument`, and legacy `MemoryStaging*` tools. While those surfaces remain available, their existing inputs keep their existing Workspace-document meaning. They must not be silently redirected to subjektiv, assigned an inferred subject, or reused with a new interpretation. - -Worker-side `memory` code owns shared schemas and transport types, not persistence. Legacy `AppendAudit` is acknowledgement-only and has no dedicated durable Memory-audit rows. General Session logs and diagnostics remain independently durable under their own retention policies. - -Repository or ancestor `.yoi/memory`, `.yoi/knowledge`, and malformed marker trees are ignored. There is no cwd/ancestor discovery, dual read, startup import, or automatic migration. The removed `yoi memory lint` command is intentionally not a compatibility surface; invoking `yoi memory ...` is an unknown command. - -Memory remains supporting context rather than implementation authority. Use Tickets, Objectives, repository files, Git history, and append-only Session records for exact current facts. - -## subjektiv cutover and reset - -Enabling `feature.subjektiv` does not migrate or erase legacy rows. The only supported destructive boundary is the explicit, authorized legacy reset API described by the [subjektiv product cutover runbook](../development/subjektiv-product-cutover.md). It atomically deletes the old document, staging candidates, and staging resolutions while retaining Workspace Memory settings, Sessions and diagnostics, Tickets, Knowledge, Skills, the subjektiv Feature database, and unrelated Workspace data. The reset is operator-triggered and idempotent; it is never a startup migration or Profile side effect. - -## Upgrade guidance - -Old repository-local Memory data is not imported automatically. Before upgrading from a version that used `.yoi/memory`, explicitly export any data that must be retained with that old version and import it into the intended Workspace-backed product surface. Do not assume the legacy control-plane reset imports repository-local or legacy Workspace data into subjektiv. Stale files may be archived or deleted only after verification. - -## Preserved filesystem boundaries - -Repository-local Memory removal and the control-plane legacy reset do not affect repository checkouts, Git metadata, Runtime-owned Workdirs, Session logs or artifacts, Server data directories, XDG client/server configuration, or the subjektiv Feature database. They remove only their explicitly named legacy authority. diff --git a/docs/design/merge-request-wip-projection.md b/docs/design/merge-request-wip-projection.md deleted file mode 100644 index 14ddee2d8..000000000 --- a/docs/design/merge-request-wip-projection.md +++ /dev/null @@ -1,72 +0,0 @@ -# Merge Request native WIP projection - -## Purpose and authority - -When a Worker explicitly runs in `wip` mode and its Merge Request Feature has at least one enabled operation, the Worker Host projects that exact operation set as native Worldspace objects. The projection is an adapter over the existing scoped Workspace Merge Request tools and Backend routes. It does not create a second Merge Request store, provider integration path, review grant, approval rule, or completion authority. - -The Host assigns the Merge Request provider the root `merge-requests` namespace through `WipMountRegistry` and mounts: - -```text -/merge-requests -/merge-requests/ -``` - -There is no Feature Object or `/features/merge-request` compatibility alias. Legacy paths remain historical display data only and fail as current input with rediscovery guidance. - -An item identity is a single canonical path segment. Nested paths and percent-encoded target substitution do not resolve. Item operations remove `merge_request_id` from their descriptor and reconstruct it only from the selected object route. Discovery does not mint authority: every call still uses the Worker's scoped `WorkspaceClient`, the enabled Feature flags, ordinary Tool permission rules, and the Backend's Workspace, repository, assignment, provider-ref, review, and integration checks. - -## Current Feature inventory - -The native surface is assembled from `MergeRequestFeatureConfig`, not from a broader Backend catalog. Only an ordinary Merge Request tool that would be registered for the Worker is projected and claimed from `/tools`. The current Feature has no list/query Tool, so the native adapter does not invent one: callers can resolve an existing known MR ID as a direct item route, while a successful `open` result includes the newly created item path. - -Collection interface `yoi.merge-request/collection/v1`: - -| Native operation | Existing Tool | Preserved authority inputs | -| --- | --- | --- | -| `open` | `OpenMergeRequest` | Ticket, repository key, source selector, target selector, summary | - -Item interface `yoi.merge-request/item/v1`: - -| Native operation | Existing Tool | Preserved non-identity inputs or authority | -| --- | --- | --- | -| `read` | `ShowMergeRequest` | bounded thread `after` and `limit`; Backend resolves current source and target refs | -| `check_readiness` | `CheckMergeRequestReadiness` | Backend resolves the current source subject and derives current approval/blockers | -| `review` | `ReviewMergeRequest` | decision/body/findings plus the injected Reviewer capability, captured request event, Ticket content digest, linked result snapshot, and exact candidate | -| `complete` | `CompleteMergeRequest` | operation ID, approval event, target-before/after refs, merge strategy and conflict resolution; Backend requires Orchestrator authority | - -Typical role surfaces are therefore: - -- Coder: collection `open`; item `read`. -- Reviewer: item `read` and `review`. -- Orchestrator: item `read`, `check_readiness`, and `complete`. - -`CompleteTicket` belongs to the Ticket Feature and route-bound Ticket item `complete`, alongside state and close decisions. It takes current item/state CAS, a replay key, reason, and optional references; it requires neither an MR result set nor a Reviewer approval. The former MR collection `complete_ticket` operation is not published. The review descriptor never contains the injected capability token, and a review item route must exactly match the Merge Request captured by that capability before the ordinary review Tool can run. - -## Preserved review and integration gates - -The native projection delegates to the same ordinary tools and REST endpoints. In particular, it does not replace or weaken: - -- Workspace and configured-repository isolation; -- current Coder assignment and assigned-Workdir source-selector proof for opening/review requests; -- provider resolution of the exact current source candidate; -- ReviewRequested binding to the Ticket item `content_digest` and complete linked-MR/source snapshot; -- injected Reviewer capability consumption and exact MR/candidate matching; -- source movement invalidating source approval while target-only movement preserves unchanged-source review; -- unresolved request-change and revoked/stale approval rejection; -- current target observation, target-before/after proof, strategy, and conflict-resolution checks; -- Orchestrator-only Merge Request integration; -- immutable integration/result evidence and exact linked-result-set requirement attestation. - -The Worldspace adapter performs no Git or provider integration. A visible `complete` operation remains unusable without the ordinary Backend's authenticated online Orchestrator source and integration evidence. - -## Bounded reads, staleness, and outcomes - -`read` exposes the existing bounded thread page with optional `after` and a `1..=200` limit. Responses keep linked Tickets as references/data, so Ticket or Objective native projections are not prerequisites. Descriptors never expand repository diffs, unbounded thread history, credentials, or capability material. - -A successful item read records a response fingerprint. A successful item mutation advances a local mutation generation. Opening records the returned MR and adds its native path to the result. Ticket decisions advance the Ticket provider's object generation, not the MR integration state or proofs. Ending Ticket work can affect later live source observation, so clients reread MR authority before MR actions; Backend checks remain authoritative. The common WIP Host rejects calls made with stale object/interface validators; callers must rediscover before deciding whether to call again. - -Optional fields preserve ordinary JSON-input semantics: omission remains omission and explicit WIP unit/null remains JSON `null` for schema and permission matching. The reconstructed object is validated against the original Tool JSON Schema before the Tool executes. - -Backend 401/403 responses map to permission denial, 404 to missing objects, and deterministic 400/409/422 authority or workflow failures to invalid arguments. Cancellation and interruption remain distinct. Transport/server failures after dispatch, malformed successful mutation responses, and identity-mismatched mutation responses are reported as unknown outcomes and must not be retried automatically. - -Normal Tool mode is unchanged. In WIP mode, only the enabled Merge Request compatibility tools represented by these native objects are suppressed, preventing duplicate native and `/tools` publication. diff --git a/docs/design/objective-wip-projection.md b/docs/design/objective-wip-projection.md deleted file mode 100644 index b3933df71..000000000 --- a/docs/design/objective-wip-projection.md +++ /dev/null @@ -1,57 +0,0 @@ -# Objective native WIP projection - -## Purpose and authority - -When a Worker explicitly runs in `wip` mode and the Objective Feature is enabled, the Worker Host projects Objectives as native Worldspace objects. The projection is an adapter over the existing scoped Workspace API. It does not introduce another Objective store, state machine, reference resolver, or audit path. - -The Host assigns the Objective provider the root `objectives` namespace through `WipMountRegistry` and mounts: - -```text -/objectives -/objectives/ -``` - -There is no Feature Object or `/features/objective` compatibility alias. Legacy paths are retained only as historical text and fail as current inputs with rediscovery guidance. - -`` is either a canonical `O-` key or an existing canonical internal alphanumeric ID. Only one direct child segment is accepted. Nested suffixes, Ticket keys, and unrelated or hidden prefixes do not resolve as Objective objects. - -The Backend remains authoritative for existence, `O-*`/internal-ID resolution, Workspace scope, validation, audit events, pagination, body/event bounds, and linked-Ticket validation. A discovered route is not independent authority; every operation still uses the Worker's current `WorkspaceClient`. - -## Current operation inventory - -The collection interface is `yoi.objective/collection/v1`: - -| Native operation | Existing Tool permission identity | Parameters | -| --- | --- | --- | -| `query` | `QueryObjective` | `query`, `states`, `linked_ticket_id`, `updated_after`, `updated_before`, `sort`, `limit`, `cursor` | -| `create` | `ObjectiveCreate` | `title`, `body_md`, `state`, `linked_tickets` | - -The item interface is `yoi.objective/item/v1`: - -| Native operation | Existing Tool permission identity | Parameters | -| --- | --- | --- | -| `read` | `ShowObjective` | `event_limit`, `event_cursor` | -| `edit` | `ObjectiveEdit` | `title`, `old_string`, `new_string`, `replace_all` | -| `set_state` | `ObjectiveSetState` | `state` | -| `link_ticket` | `ObjectiveLinkTicket` | `ticket_id` | -| `unlink_ticket` | `ObjectiveUnlinkTicket` | `ticket_id` | - -The item operations intentionally do not accept an Objective `id`. The handler is created from the resolved item route and reconstructs the equivalent ordinary Tool permission arguments with that route-bound identity. Descriptor validation rejects a caller attempting to supply a second Objective target. - -`deny` and `ask` permission results fail closed before Backend dispatch. Backend 401/403 map to permission denial, 404 to missing target, and deterministic client validation failures to invalid arguments. Transport, server, or response-projection failures after dispatch retain the common unknown-outcome contract; mutating operations must not be retried automatically when the outcome is unknown. - -## Bounded reads and references - -`query` retains the ordinary Worker limit of 1–100 and returns stable snippets, canonical `O-*` references, linked `T-*` references, `next_cursor`, and `has_more`. Each result also includes its canonical item path. It does not expand Objective bodies, complete events, or Ticket records. - -`read` retains the ordinary Worker event limit of 1–50 and projects the Backend-bounded body, `body_truncated`, linked-Ticket summaries, bounded events, and event pagination. Internal Objective, Ticket, and event-storage IDs are removed from model-visible results. - -`create` returns the authoritative Objective detail plus its canonical item path. Mutations return the authoritative post-operation detail. The Object publishes fixed operation signatures, not the Objective's mutable content: editing content does not invalidate an unchanged Object/Interface contract. The adapter therefore keeps no content-freshness cache or synthetic change identifier. Each invocation checks current permission and dispatches to current Backend authority; `edit` evaluates its explicit `old_string`/uniqueness conditions there. Reads always fetch current Backend data rather than replaying cached content. Host validators still protect route and Interface signatures before dispatch. - -Linked Ticket references remain strings passed to the existing Objective Backend operation. Their presence does not publish a Ticket object, grant Ticket read or mutation authority, or require a Ticket-native WIP projection. - -## Compatibility and Tool mode - -The native collection/item pair claims all seven Objective compatibility capabilities in WIP mode. Therefore `/tools/QueryObjective`, `/tools/ShowObjective`, and `/tools/Objective*` are not published alongside the native routes. Unrelated enabled tools keep their compatibility projections. - -Normal `tools` mode is unchanged: it still exposes the same seven Objective tools with their existing names and input schemas. Both surfaces share the same `WorkspaceHttpObjectiveBackend` and scoped Backend endpoints. diff --git a/docs/design/orchestrator-worktree-layout.md b/docs/design/orchestrator-worktree-layout.md deleted file mode 100644 index ce6f533a4..000000000 --- a/docs/design/orchestrator-worktree-layout.md +++ /dev/null @@ -1,32 +0,0 @@ -# Orchestrator worktree and implementation worktree layout - -Yoi can run an Orchestrator from a dedicated orchestration worktree while implementation worktrees still belong under the original workspace root. The runtime must therefore distinguish role execution cwd/workspace from the target repository root used for implementation branches and merge-completion. - -This decision records the layout policy for Ticket `00001KTTB479X`. - -## Roots - -- `role_workspace_root`: the launched role Runtime's explicit execution/workdir root. Ticket authority remains the Workspace Server control-plane database reached through the Workspace API. -- `original_workspace_root`: the repository root from which implementation worktrees should be created. -- `implementation_worktree_root`: `/.worktree`. -- `merge_target_workspace_root`: the workspace/branch where merge-completion validates and merges. - -When these roots differ, implementation worktrees must be created under `implementation_worktree_root`, not under the Orchestrator process cwd. Merge-completion must run against `merge_target_workspace_root`, not whichever worktree happens to host the Orchestrator. - -## Policy - -- Workspace Server Ticket records remain authoritative; none of these filesystem roots selects or hosts a Ticket backend. -- Main-workspace draft Tickets are not an implicit Orchestrator queue. Queue acceptance still requires explicit Ticket state and routing checks. -- Implementation branches should not include unrelated Orchestrator bookkeeping. Orchestration progress stays in the Workspace Server Ticket/event record; child implementation worktrees contain implementation diffs and branch-local artifacts only when needed. -- Prompt and workflow guidance must treat merge-ready dossier as a checkpoint. If reviewer approval, clean/safe target workspace, and standing/user merge authority are present, Orchestrator should continue through merge, validation, Ticket close, and cleanup instead of stopping solely because a dossier exists. - -## Implemented in this branch - -- Ticket role launch prompts now emit a `Workspace routing context` section for Orchestrator launches, and for any launch that explicitly provides original/target roots. -- `TicketRoleLaunchPlan` carries `original_workspace_root`, `implementation_worktree_root`, and `target_workspace_root` as explicit plan fields for callers/tests. -- Orchestrator role prompt resources now say to create implementation worktrees under the original workspace root and to run merge-completion against the recorded target workspace. - -## Follow-up boundaries - -- Panel/workspace orchestration can later populate `original_workspace_root` and `target_workspace_root` when launching Orchestrator from a dedicated worktree. -- A future implementation can add a first-class orchestration-worktree launcher and cleanup policy, but it must keep the four-root distinction above visible in launch diagnostics and tests. diff --git a/docs/design/overview.md b/docs/design/overview.md deleted file mode 100644 index 5fdb66d45..000000000 --- a/docs/design/overview.md +++ /dev/null @@ -1,35 +0,0 @@ -# Design overview - -Yoi is organized around one rule: durable authority must live in explicit records, while runtime conveniences remain reconstructable hints. - -That rule shapes the crate split. The runtime can restart, attach, compact, or delegate work without relying on hidden controller state, and developer tooling can inspect the records that explain why an agent acted. - -## Core layers - -- `yoi` owns the product CLI and top-level command shape. It is the façade that wires connection selection, product commands, and normal TUI launch. -- `worker` turns a `Engine` into a named runtime entity with scope, session persistence, protocol handling, tools, and Worker metadata integration. -- `agen` owns model-facing turns: history append, retries, continuation, pruning/compaction mechanics, tool loops, and provider-independent callbacks. -- `session-store` owns replayable append-only conversation/session logs. -- `pod-store` owns current Worker metadata keyed by Worker name. -- `protocol` defines the socket message boundary between clients and Workers. -- `standalone` owns the client-side, one-process Standalone host and its dedicated session store. It does not create Runtime Workers, PID/socket catalogs, or Workspace product authority. -- `client` contains reusable Backend Workspace/Runtime clients plus the shared `Target` boundary. Its normal targets are exactly Standalone and Backend; it is not a subprocess launcher or repository-local product backend. -- `manifest` resolves Profiles, Manifests, model/provider references, scopes, prompts, and tool permission policy into a runtime contract. -- `tools` implements built-in tools with bounded output and policy-aware execution. -- `memory` owns shared legacy Memory transport plus extraction/evidence schemas, not persistence. Legacy single-document Memory lives in the Workspace Server control-plane database; subject-scoped Memory lives in the Workspace's Server-managed `subjektiv` Feature database. -- `workspace-server` is the local Workspace control-plane seam. It can project Tickets, Workers, lifecycle, usage, and orchestration events, but browser/API operations must stay on opaque backend identities instead of raw local paths, sockets, Worker names, or session files. -- `tui` is a UI over either one in-process Standalone session or Backend Workspace/Runtime Worker authority; it should not invent a local Worker catalog or durable product state. - -## Why these boundaries exist - -The Engine should not know process identity, Worker names, live sockets, spawned children, or UI state. It should know how to run an LLM turn over committed history and tools. - -The Worker should not make provider-specific wire decisions. It coordinates runtime identity, persistence, scope, and protocol delivery around a Engine. - -The TUI should not be an alternate source of truth. It may queue local input, show optimistic affordances, and render snapshots, but durable state comes from Worker/session records. - -The CLI should own product command shape. Other crates should expose library APIs and typed runtime commands rather than re-parsing product arguments. - -## Documentation boundary - -Design docs explain the constraints that code must preserve. They should not duplicate every public type, exact schema field, or command output; those are owned by code, tests, and work items. diff --git a/docs/design/plugin-component-model.md b/docs/design/plugin-component-model.md deleted file mode 100644 index 8e8bf4fbc..000000000 --- a/docs/design/plugin-component-model.md +++ /dev/null @@ -1,29 +0,0 @@ -# Plugin Component Model boundary - -Dynamic Worker Plugin execution is not part of the current product. The `.yoi-plugin` component metadata retained in `manifest` is an offline package-format contract only. - -## Current behavior - -- Worker creation and restore install no dynamic Plugin modules. -- Manifest/Profile input rejects `plugins` and `feature.plugins`. -- Runtime and Server startup perform no repository, ancestor, cwd, or user-data Plugin discovery. -- No persisted local `package_path` is execution authority. -- Only statically compiled trusted built-in Features contribute Worker capabilities. -- `yoi plugin check` parses an explicitly named directory or package without instantiating a component. - -The package validator may reject legacy core-Wasm artifacts and require Component Model metadata, but passing validation does not make an artifact installable or executable. - -## Future platform constraints - -A future Server Plugin platform may execute Wasmtime Component Model packages only after the architecture is implemented as a coherent authority boundary: - -1. an operator installs an immutable package into Server-owned artifact authority; -2. a Workspace owner selects an installed package through an immutable Addon configuration snapshot bound to exact package/content digests; -3. Backend authors a per-Worker execution plan containing exact identities, digests, configuration, and bounded grants; -4. Runtime fetches only Server-authorized digests and verifies package bytes and execution-plan identity; -5. Runtime instantiates a fresh bounded Wasmtime Store with no ambient WASI authority; and -6. restore uses the persisted execution plan and exact artifact rather than current Workspace settings or a filesystem path. - -Default components receive no filesystem, sockets, environment, clocks, randomness, subprocess, Workdir, broad Workspace client, credential, or network authority. Any host import must be narrow, typed, explicitly granted, live-revalidated where necessary, bounded, and audited. - -This future platform must not restore `.yoi/plugins`, user-data catalogs, cwd/ancestor discovery, native dynamic libraries, downloaded Cargo manifests, or local paths as compatibility authority. diff --git a/docs/design/plugin-packages.md b/docs/design/plugin-packages.md deleted file mode 100644 index 2256541bc..000000000 --- a/docs/design/plugin-packages.md +++ /dev/null @@ -1,31 +0,0 @@ -# Plugin packages - -Yoi retains `.yoi-plugin` as an offline authoring and inspection format. The format is not a Worker capability source. - -## Current product boundary - -Normal Worker creation, restore, Profile/Manifest resolution, Server/Runtime startup, and CLI execution do not discover Plugin catalogs from: - -- repository or ancestor `.yoi/plugins` directories; -- user-data Plugin directories; -- the current working directory; or -- persisted local `package_path` values. - -`plugins` and `feature.plugins` are rejected in Worker Manifest/Profile input. Dynamic Plugin Tools, Services, Ingress handlers, and WASM components are not installed or executed. Worker capabilities come only from statically compiled trusted built-in Features. - -## Offline format operations - -The CLI keeps only operations whose input or destination is explicit: - -```sh -yoi plugin new rust-component-tool ./example-plugin -yoi plugin check ./example-plugin -yoi plugin pack ./example-plugin --output ./example-plugin.yoi-plugin -yoi plugin check ./example-plugin.yoi-plugin -``` - -These commands parse, validate, or write the named local artifact. They do not scan a Workspace, mutate Profile/Manifest configuration, install a package, register Worker Tools, or execute Plugin code. `plugin list` and `plugin show` were removed because their catalog semantics depended on ambient repository and user-data stores. - -## Future installation authority - -Server-installed Plugin packages and Addons are future work. That platform must provide explicit immutable package identity and digest, Server-owned artifact delivery and Workspace selection, a Backend-authored per-Worker execution plan, Runtime verification, and sandboxed execution. It must not restore repository-local or user-data discovery as a compatibility fallback. diff --git a/docs/design/profiles-manifests-prompts.md b/docs/design/profiles-manifests-prompts.md deleted file mode 100644 index e9b1624f2..000000000 --- a/docs/design/profiles-manifests-prompts.md +++ /dev/null @@ -1,133 +0,0 @@ -# Profiles, Manifests, and prompts - -Profiles are reusable recipes. Resolved Manifests are runtime contracts. Prompt resources are managed assets. Keeping those layers separate prevents runtime state from leaking into reusable configuration. - -## Profiles - -A Profile describes how a Worker should normally be built: worker language, model/provider selectors, prompt choices, tool policy defaults, and other reusable preferences. - -A Profile should not contain runtime-bound fields: - -- `worker.name` -- concrete delegated `scope.allow` -- sockets or process identifiers -- session pointers -- restored spawned-child state -- raw secret values - -Those fields depend on one run, one parent, or one machine. Putting them in a reusable Profile makes reuse unsafe. - -Yoi Profiles are data artifacts resolved from builtin profile definitions, `profiles.toml`, Workspace Config values, or explicit JSON/TOML artifacts. Decodal is the Workspace authoring syntax; JSON/TOML artifacts are the low-level resolver interchange format. Runtime launch does not depend on executable profile scripts. - -### Workspace registration and migration - -Register the Profile **value**, not a source pathname: - -```dcdl -{ - profile = { - default_profile = "project:my-companion"; - entries = [{ - selector = "project:my-companion"; - label = "My Companion"; - description = "My reusable Worker recipe"; - profile = import "./profiles/my-companion.dcdl"; - }]; - }; -} as WorkspaceConfigSchema -``` - -The imported file can contain `{ worker = { mode = "wip"; }; model = { ref = "codex-oauth/gpt-5.6-sol"; }; feature = { task = { enabled = true; }; }; }`. That record can also be written directly at `entries[0].profile`. Composition uses ordinary Decodal expressions, for example `(import "./profiles/base.dcdl") // { worker = { mode = "wip"; }; }` for an explicit patch, or `&` for compatible refinement. No dedicated Profile file or `profiles/` directory is required. Imports are relative to the **containing virtual config file**, not a host filesystem path; missing imports and virtual-tree escapes are rejected. - -`selector` must be a nonempty `project:*` selector, unique in the Workspace. `label` and `description` are optional display metadata. An omitted `default_profile` selects `builtin:companion`; an explicitly unknown default is an error, not a fallback. A project Profile is a partial recipe resolved against `WorkerManifestConfig::resolution_defaults()`, **not** implicitly inherited from a builtin role. Explicit values remain explicit and omitted Profile fields remain absent in the transported value. To inherit a builtin recipe, import its public value explicitly; there is no implicit builtin inheritance. - -### Public builtin imports - -Workspace Decodal can import embedded Profile values through the read-only `$builtin/` namespace: - -```dcdl -(import "$builtin/profiles/companion.dcdl") // { - worker = { mode = "wip"; }; - feature = { task = { enabled = true; }; ticket = { enabled = false; }; }; -} -``` - -Use this expression anywhere a Profile value is accepted, including `profile.entries[*].profile` or an imported Workspace recipe. `$builtin/profiles/companion.dcdl` is a source path, not the launch selector `builtin:companion`. Public paths under `$builtin/profiles/` are `base.dcdl`, `default.dcdl`, `standalone.dcdl`, `standalone-subjektiv.dcdl`, `standalone-subjektiv-consolidation.dcdl`, `coder.dcdl`, `companion.dcdl`, `intake.dcdl`, `reviewer.dcdl`, `orchestrator.dcdl`, `job.dcdl`, `backend-job.dcdl`, `memory-consolidation.dcdl`, and `subjektiv-memory-consolidation.dcdl`. This is an explicit embedded catalog, not arbitrary resource or filesystem access. Unknown paths, namespace escapes, host paths and network URLs are errors. - -Relative imports **inside a builtin source** resolve against its builtin virtual directory: companion's `./base.dcdl` means `$builtin/profiles/base.dcdl`, never the Workspace's `profiles/base.dcdl`. Relative imports in Workspace files continue to use their containing Workspace virtual directory. Builtin values obey ordinary Decodal composition: `//` applies an explicit patch and `&` refines compatible values (conflicting explicit values remain errors). Importing a builtin does not add a special merge or fallback mechanism. - -Builtin dependencies share the Workspace snapshot budgets: at most 256 entries across Workspace entries and distinct observed builtin sources, 256 KiB per source, and 4 MiB of total source bytes. Unused Workspace entries still count; repeated imports of the same builtin count once. The observed import graph must be acyclic and at most 32 edges deep, **including the synthetic evaluation wrapper's import of the entrypoint**. These guards apply before the loader returns sources to Decodal, including imports in lazy object fields; builtins do not bypass them. Launch archives retain their separate transport limits (64 sources, 256 KiB per source, 1 MiB total source bytes). - -The former `entries = [{ selector = "project:name"; source = "profiles/name.dcdl"; }]` registration is no longer accepted. Migrate it explicitly to `profile = import "./profiles/name.dcdl";`, preserving the selector, label, description and default. For registrations in a nested config file, adjust the relative import to the original virtual-tree entry (for example `../profiles/name.dcdl`). Do not rename or recreate the Profile file merely for registration. Previously saved source-based saved content snapshots remain readable as config data and are not rewritten, deleted, or silently substituted with a builtin. Their Profile projection/launch reports `profile_source_registration_removed` with the replacement form; saves under the new schema reject the old field. Existing Workers continue using their saved Manifest while the author migrates and saves the config. - -The schema contribution separates its materialized registration schema from an optional, fingerprint-bound `authoring_source`. The latter supplies editor field suggestions inside array elements and partial Profile records, plus a separate constraint-only analysis pass for supplied nested values (including imported recipes); that pass never materializes the shape, makes omitted fields required, or supplies defaults to the saved projection. Editor diagnostics point to the supplied value's virtual file/span. The authoring shape is never substituted for the persisted evaluation schema. Decodal structural diagnostics cover registration and nested Profile field types; the shared Backend save boundary additionally checks full Profile/Manifest semantics (including mode values, forbidden runtime fields, duplicate selectors and default references). UI and WIP saves use that same boundary; invalid candidates leave the active config content untouched. - -`scope` and `delegation_scope` retain both supported forms: a string such as `"workspace_read"` / `"workspace_write"`, or a table such as `{ intent = "workspace_read"; deny_write = ["private"]; symlink_policy = "resolved"; }`. Decodal 0.4 has no union range, so these fields are opaque to the editor constraint pass; a non-materialized table hint supplies nested completion without rejecting the string form or rewriting it. The shared Backend save validator checks scope forms, intent values and table fields (including their types and unknown keys). Other structurally typed fields such as `worker.mode` and `feature.task.enabled` retain editor type diagnostics. Completion hints never become Profile defaults or saved values. - -Extraction `reasoning` in both `feature.memory` and `feature.subjektiv`, like `engine.reasoning`, accepts either a string effort or a signed integer budget. These union values are also opaque to editor constraints and semantically checked on save. Compaction authoring covers the complete token settings and their supported `compact_*` aliases, plus the ratio helper's `request` / `worker` aliases; values are not renamed or defaulted during Workspace evaluation. The shared test corpus `resources/config-schema/profile-authoring-values.json` proves exact values in native/WASM analysis and effective settings through save/projection/Runtime consumption. Its canonical compaction case must cover every serialized `CompactionConfigPartial` field, making future contract drift visible in tests. - -## Manifests - -A resolved Manifest is the concrete contract used to create or restore a Worker. It carries defaults, resolved paths, permissions, scope, prompt references, provider/model decisions, and runtime identity. - -Source/partial layers may omit fields. Resolved manifests should be explicit enough that Worker creation does not depend on ambient configuration later changing under it. - -`--manifest ` exists as an explicit low-level escape hatch. Normal fresh startup selects a `builtin:*` or `project:*` Profile from the Backend-managed Workspace Config content snapshot rather than applying an ambient manifest cascade. - -Project Profiles are evaluated as part of the content-addressed Virtual Config, including its imports and value composition. The Backend packages the **evaluated selected value** into a digest-bound JSON entry in the existing Profile archive transport, and binds the launch bundle to the Workspace identity, tree/projection digests and schema/toolchain fingerprints. Runtime verifies the archive and resolves that exact value without re-evaluating Decodal, scanning imports or reading Workspace files. When Workspace evaluation imports builtins, the archive additionally seals the exact observed transitive builtin closure for the Workspace contract: canonical `$builtin/profiles/...` source keys, original Decodal bytes, content types, sizes and source digests, all bound by the outer archive and launch bundle digests. These are provenance dependencies, not runtime entrypoints or runtime import instructions. Missing or corrupt provenance fails verification even though the entrypoint is JSON; no local file or current embedded builtin can rescue it. Builtin Profiles selected directly retain their embedded Decodal archive path. Changing an embedded builtin so that it changes the evaluated Workspace value requires an **explicit config save/re-evaluation before a fresh launch**. The Backend compares current evaluation with the saved projection digest and rejects a mismatch rather than silently adopting the new value. T-711 retains the existing version-2 toolchain contract/fingerprint tuple so existing stored contracts remain readable; builtin provenance is bound by the archive and bundle digests, not a new catalog fingerprint field. Neither a builtin update nor a config save rewrites a sealed archive or silently rebases an existing Worker. Saving config changes only future fresh launches; the Worker persists the resulting resolved Manifest and restore uses that saved snapshot rather than selecting the current recipe again. Files below the Workdir are not implicit Profile override layers. - -## Local stdio MCP server declarations - -Profiles and manifest layers may declare named local stdio MCP servers under `mcp.stdio_server`. This is a typed configuration surface only. Declaring a server does not start a subprocess, discover packages, negotiate MCP capabilities, or register tools/resources/prompts. - -Example TOML artifact fragment: - -```toml -[[mcp.stdio_server]] -name = "filesystem" -command = "node" -args = ["server.js", "--root", "."] - -[mcp.stdio_server.cwd] -kind = "path" -path = "./mcp" - -[mcp.stdio_server.env] -inherit = ["PATH"] - -[mcp.stdio_server.env.set.SAFE_MODE] -kind = "literal" -value = "1" - -[mcp.stdio_server.env.set.API_TOKEN] -kind = "secret_ref" -ref = "providers/mcp-token" - -[mcp.stdio_server.env.set.UPSTREAM_TOKEN] -kind = "env_ref" -name = "MCP_UPSTREAM_TOKEN" -``` - -`command` is a direct executable name/path, not a shell string. `args` are passed as argv entries by future lifecycle code. `cwd.kind = "path"` is resolved relative to the Profile or manifest layer; omit `cwd` or use `{ kind = "inherit" }` when the lifecycle caller should choose. Environment handling is explicit: future spawn code should inherit only names listed in `env.inherit` and set only variables in `env.set`. `literal` values are for non-secret data; credentials should use `secret_ref` or explicit `env_ref`. Diagnostics and Debug output must redact env literal values and must not print secret plaintext. - -Local stdio MCP servers are ordinary local executables running with the user's OS permissions. Yoi's feature flags, Plugin permissions, and MCP config validation are not an operating-system sandbox and cannot prevent filesystem/network/process side effects once a later lifecycle implementation chooses to spawn a configured server. - -## SubWorkers - -`SubWorkerSpawn.profile` is optional and resolves through defaults when omitted. The only concrete capability delegation in the tool call is `SubWorkerSpawn.scope`, and it must be a subset of the parent Worker's effective scope. - -`inherit` derives reusable settings from the parent's resolved Manifest while replacing child identity and delegated scope. It should not blindly reuse the parent's original Profile source or runtime state. - -## Prompt resources - -Prompts live under `resources/prompts` so builtins, project overrides, and user overrides have one asset boundary. - -The prompt layer should explain policy and behavior, but it should not smuggle volatile state into model context. Runtime facts that affect later turns must still go through history. - -Builtin resources should be embedded at compile time. Project Profiles and prompt overlays belong to the Backend-managed content-addressed Workspace Config and travel as digest-bound source archives. User Profile registries, provider/model catalog overrides, and explicit low-level Manifests remain filesystem-based where those local resolution paths are used. - -## Why this separation matters - -Without this split, configuration becomes unreproducible: a Profile might accidentally depend on a parent Worker's socket, a prompt override might act like hidden state, or a restored Worker might observe different defaults than the run that created it. - -The boundaries make it clear which information is reusable authoring, which is resolved runtime contract, and which is durable run history. diff --git a/docs/design/provider-model-boundary.md b/docs/design/provider-model-boundary.md deleted file mode 100644 index 99edd2157..000000000 --- a/docs/design/provider-model-boundary.md +++ /dev/null @@ -1,45 +0,0 @@ -# Provider and model boundary - -The Engine should be provider-independent. Provider-specific wire formats, auth mechanisms, model catalogs, reasoning knobs, and terminal error shapes belong below or beside it, not inside ordinary turn orchestration. - -## Engine responsibility - -`agen` owns turn lifecycle: - -- committed history append -- tool loops -- retry before stream start -- continuation after safe partial output -- pruning/compaction decisions -- provider-neutral callbacks and events - -It should not know where a Codex OAuth token lives, how a provider encodes reasoning effort, or which environment variables a provider once supported. - -## Provider responsibility - -Provider/model layers own: - -- resolving model aliases and builtin model metadata -- provider-specific request/response conversion -- auth integration -- context/window metadata -- terminal error classification -- trace points around HTTP/auth/body/SSE lifecycle - -External API facts change quickly. Repository docs should record Yoi's boundary decisions, not snapshots of vendor documentation. - -## Secrets - -Local credentials use explicit secret references. Secret values must not appear in diagnostics, Debug output, CLI/TUI output, work items, docs, session logs, model context, or persisted plaintext store files. - -Codex OAuth remains a separate integration because its existing `auth.json` / `CODEX_HOME` shape is not the same as normal provider secret refs. - -## Error and trace semantics - -Provider terminal errors that are displayed live, such as context-length failures, must persist as errored runs rather than successful empty turns. - -Event trace sidecars are optional parsed lifecycle traces. They are not complete raw SSE logs and should not become a hidden source of model context. - -## Why this boundary exists - -Provider APIs drift. If provider details leak into Engine logic, every new model behavior risks changing core orchestration. Keeping the boundary explicit lets Yoi adapt provider integrations while preserving stable turn semantics. diff --git a/docs/design/server-feature-storage.md b/docs/design/server-feature-storage.md deleted file mode 100644 index cc97a74ee..000000000 --- a/docs/design/server-feature-storage.md +++ /dev/null @@ -1,138 +0,0 @@ -# Server-managed Feature storage - -`yoi-workspace-server::feature_storage` provides the SQLite lifecycle boundary for trusted Server-side Features. It is not a Worker feature API and it is not a model-visible SQL tool. Subject-scoped Memory is persisted in the Workspace's `subjektiv` Feature database through this boundary; legacy single-document Workspace Memory remains in the control-plane database and is not owned, imported, or reset by `FeatureStorage`. - -## Ownership boundary - -The Server owns: - -- the physical layout under `feature-storage//features/.sqlite`; -- validated Workspace and Feature identifiers (no caller-provided database path); -- connection reuse, `foreign_keys`, WAL, `busy_timeout`, and write serialization; -- migration timing/history, shutdown admission, online backup/restore, and Workspace deletion fencing. - -A Feature owns: - -- its tables, migration bodies, queries, and domain transactions; -- validation and typed repository/API methods exposed to its Server handler; -- authorization before the repository is called. - -A Feature database is an API/ownership isolation boundary inside the trusted Server process. It does not sandbox arbitrary native code and does not replace Workspace authorization. A `FeatureDatabase`, SQLite path, or general SQL endpoint must never be sent to a Worker/Runtime or added to a model tool contract. - -## Registration and repository example - -Registration is trusted startup code. Merely constructing or registering storage is lazy and creates no directory or database; the first `open` applies migrations. The simplified example below demonstrates storage ownership and transaction atomicity, not the production subjektiv schema or Memory-change API; those contracts are described in [subjektiv-store.md](subjektiv-store.md). - -```rust -use rusqlite::{Transaction, params}; -use yoi_workspace_server::{ - FeatureDatabase, FeatureMigration, FeatureRegistration, FeatureStorageError, - WorkspaceApi, -}; - -type StorageResult = - std::result::Result; - -fn create_subject_memory(tx: &Transaction<'_>) -> StorageResult<()> { - tx.execute_batch( - "CREATE TABLE subjects ( - subject_id TEXT PRIMARY KEY, - state TEXT NOT NULL CHECK (state IN ('active', 'retired')) - ); - CREATE TABLE memories ( - subject_id TEXT NOT NULL, - memory_id TEXT NOT NULL, - body TEXT NOT NULL, - PRIMARY KEY (subject_id, memory_id), - FOREIGN KEY (subject_id) REFERENCES subjects(subject_id) - ); - CREATE TABLE interventions ( - intervention_id TEXT PRIMARY KEY, - subject_id TEXT NOT NULL, - detail TEXT NOT NULL, - FOREIGN KEY (subject_id) REFERENCES subjects(subject_id) - );" - )?; - Ok(()) -} - -static MIGRATIONS: &[FeatureMigration] = &[ - FeatureMigration::new(1, "subject memory baseline", create_subject_memory), -]; - -const REGISTRATION: FeatureRegistration = - FeatureRegistration::new("subjektiv", MIGRATIONS); - -struct SubjektivRepository { - database: FeatureDatabase, -} - -impl SubjektivRepository { - fn apply_candidate( - &self, - subject_id: &str, - memory_id: &str, - body: &str, - intervention_id: &str, - ) -> StorageResult<()> { - // Candidate content and intervention recording are - // one atomic transaction in the subjektiv Feature database. - self.database.transaction(|tx| { - tx.execute( - "INSERT INTO memories (subject_id, memory_id, body) - VALUES (?1, ?2, ?3)", - params![subject_id, memory_id, body], - )?; - tx.execute( - "INSERT INTO interventions (intervention_id, subject_id, detail) - VALUES (?1, ?2, 'candidate applied')", - params![intervention_id, subject_id], - )?; - Ok(()) - }) - } -} - -fn initialize_subjektiv_repository(api: &WorkspaceApi) -> StorageResult { - let registered = api.feature_storage().register(REGISTRATION)?; - let database = api.feature_storage().open(®istered)?; - Ok(SubjektivRepository { database }) -} -``` - -A production Feature should register once while constructing its Server service and retain the returned `RegisteredFeature`/typed repository. Duplicate registration is rejected. `FeatureDatabase` methods are synchronous because `rusqlite` is synchronous; async HTTP/background handlers must run nontrivial repository work on a blocking executor rather than blocking a Tokio core thread. - -## Migration and transaction contract - -- Versions are contiguous from 1. Migration schema changes and the corresponding history rows are committed in one `BEGIN IMMEDIATE` transaction. -- Concurrent opens in one manager are single-flight. Independent Server managers/processes serialize migration writers through SQLite and reread history after acquiring the transaction. -- A migration callback error rolls back the complete migration batch. A later open retries it; failure is never recorded as success. -- Stored migration names are verified. A history mismatch is rejected, and a schema newer than the registered plan is never overwritten or downgraded. -- `FeatureDatabase::transaction` covers multiple tables in one Feature database. No atomic transaction is promised with the Server control-plane database or another Feature database. -- A domain rejection can return `FeatureStorageError::operation(...)`; the transaction then rolls back. - -## Lifecycle and operations - -### Disable and restart - -Not registering/opening a disabled Feature creates nothing. Disabling a previously used Feature does not delete its database. On restart, registering and opening the same Feature/Workspace reuses the persisted file and verifies/applies its migrations. - -### Backup and restore - -`WorkspaceFeatureStorage::backup` uses SQLite's online backup API for each discovered Feature database, so a live WAL is not ignored or copied as a bare database file. Each image passes `integrity_check` and `foreign_key_check`, and a manifest binds the snapshot to its Workspace and Feature identifiers. The snapshot directory is published with an atomic rename only after all images succeed. - -Each Feature image is internally consistent. There is deliberately no cross-database snapshot instant or distributed transaction between Feature databases or the Server DB. A caller that needs a higher-level coordinated backup must first stop relevant Feature writers. In particular, the [subjektiv product cutover](../development/subjektiv-product-cutover.md) stops legacy and subjektiv activity before pairing a control-plane backup with the subjektiv Feature backup; the explicit legacy reset never deletes or recreates the Feature database. - -`restore` rejects the wrong Workspace manifest, corrupt/foreign-key-invalid images, an existing destination, or any live connection. It builds validated databases in a temporary directory and atomically publishes the Workspace storage directory. Restore is an operator/Server lifecycle operation, not a model API. - -### Workspace deletion - -Workspace deletion calls `WorkspaceFeatureStorage::delete` after Worker/Workdir cleanup and before finalizing removal from the Server DB. It fences new opens, waits for connection-locked operations, closes handles, and removes only the validated Workspace subtree. The admission fence remains until the cached Workspace API is discarded, preventing a background task from recreating storage between filesystem deletion and control-plane finalization. Retrying deletion is idempotent. - -### Shutdown - -After HTTP graceful drain, Server serve paths explicitly call `shutdown`. Shutdown rejects new registration/open, waits for in-progress connection operations to release their locks, closes all managed connections, and makes retained handles fail with `DatabaseClosed`. Operations must not treat a shutdown error as success. - -## Failure handling - -Identifier, registration, migration-history, unsupported-schema, maintenance, shutdown, snapshot, I/O, and SQLite failures are typed as `FeatureStorageError`. Server handlers should map them to a bounded domain/API error without exposing paths, SQL, or unrestricted database access. Migration/restore/delete failures must stop the corresponding lifecycle transition; they are not warnings to ignore. diff --git a/docs/design/session-observation.md b/docs/design/session-observation.md deleted file mode 100644 index 24ce5dc03..000000000 --- a/docs/design/session-observation.md +++ /dev/null @@ -1,47 +0,0 @@ -# Session capture and Worker observation - -Yoi uses one session-entry exploration domain for host-provided snapshots, Memory extraction evidence, and authorized observation of active Workers. The domain is implemented in `worker::session_capture` and has no Workspace or Memory mutation authority. - -## Common capture contract - -A host constructs an immutable ordered `SessionCapture` from committed session items. The capture: - -- excludes reasoning before overview, search, read, or evidence projection; -- does not include the session system-prompt field; -- assigns an append-stable `SessionEntryRef` (`E...`) from the committed source position; -- uses the same reference in sparse overview anchors, search results, bounded reads, and Memory evidence conversion; -- pages sparse real user/assistant anchors and reports the number of non-reasoning entries between anchors; -- supports bounded range search and compact range listing when no filter is supplied; -- bounds read item count and bytes. - -`SessionEntryRef` is local to the selected session subject. Runtime peers use the canonical `{ kind: "runtime_worker", runtime_id, worker_id }` reference; parent-owned children use `{ kind: "sub_worker", name }`. A model must first select a host-projected session and must reuse both the structured subject and entry references returned for that session. - -## Independent features - -The three feature modules share only the capture domain: - -- `session-explore` installs `ShowOverview`, `SearchEntries`, and `ReadEntry` for one immutable host snapshot. It has no Workspace client or Memory state. -- `memory-extract` installs `StageMemoryCandidate` and `FinishMemoryExtraction`. It validates every staged `entry_ref` against its co-installed capture before converting it to typed Memory evidence. -- `worker-observation` installs `ListWorkerSessions`, `ViewSessionOverview`, `SearchSessionEntries`, and `ReadSessionEntry`. It captures the selected Worker again on every operation, so newly committed entries become visible while existing append-only references remain stable. - -The features do not enable or mutate each other. Feature-registry collision checks remain authoritative for tool names. - -## Observation authority - -`WorkerObservationProvider` is a host-injected authority boundary. The Worker receives only an `Arc`; model input never supplies grants, Workspace credentials, Runtime URLs, session handles, repository paths, or provider clients. - -The provider must: - -1. list only active subjects already granted to the current Worker; -2. reauthorize every capture instead of trusting a previous list result; -3. return the same not-found result for missing and unauthorized subjects; -4. return only committed session items; -5. keep subject identifiers opaque and bounded. - -Runtime/Backend integrations enable the feature through the Backend-only `WorkerSpawnRequest.resolved_worker_observation_enabled` field, forwarded as `CreateWorkerRequest.worker_observation_enabled`. Canonical same-Runtime peers may also be supplied through `resolved_worker_observation_grants`; Runtime revalidates those against live weak handles and Workspace scope. For cross-Runtime peers and dynamically added Workers, `WorkspaceClientWorkerObservationProvider` calls the Workspace-scoped Server projection on every list/capture. Server authorizes that route against the current Workspace Orchestrator identity, recomputes the active Workspace Worker set, and reads the selected Worker’s committed protocol snapshot. Runtime binds these providers through `Worker::bind_worker_observation_provider` on spawn and restore. Parent-owned SubWorkers use the same provider contract through `SpawnedSubWorkerObservationProvider`; their subjects use the tagged `sub_worker` variant. - -Observation is read-only evidence access. It does not authorize Ticket, Memory, Worker, or Workdir mutations and is not completion or approval authority. - -## SubWorker output - -SubWorkers no longer expose a separate output cursor or lifecycle tool family. `WorkerList`, `WorkerSendInput`, and `WorkerStop` retain parent-owned lifecycle control through a `{ kind: "sub_worker", name }` subject, while committed child output is read through `worker-observation`. Turn-completion notifications carry no transcript and only tell the parent to inspect the authoritative committed session at a natural boundary. diff --git a/docs/design/standalone-agent-host.md b/docs/design/standalone-agent-host.md deleted file mode 100644 index 984014456..000000000 --- a/docs/design/standalone-agent-host.md +++ /dev/null @@ -1,48 +0,0 @@ -# Standalone Agent Host - -`crates/standalone` は、既存の Yoi Worker を同一プロセス内で起動する最小の host 境界である。 -新しい Agent 実行系ではなく、`manifest`、`worker`、`session-store`、`workdir` の既存契約を固定構成で組み立てる。 - -## 依存方向 - -```text -standalone - ├─ manifest - ├─ worker - ├─ session-store - └─ protocol - -worker - └─ WorkerBootstrap / start_worker_controller -``` - -`standalone` は `tui`、`worker-runtime`、`yoi-workspace-server` に依存しない。 -`worker` の direct entrypoint、`worker-runtime` の fresh/restore factory、`standalone` は同じ controller lifecycle を使う。fresh runtime/standalone construction は `WorkerBootstrap` を共有し、Runtime は `prepare()` 後かつ Feature install/controller exposure 前に Workdir、observation、Flow の live binding を追加する。restore は replay 済み Worker を `PreparedWorker` へ渡して同じ pre-exposure lifecycle を通す。 - -## authority と lifecycle - -- launch は `ProfileExecutionTarget::Standalone` で built-in/XDG Profile を resolve する。repository-local Profile と path selector は authority にしない。 -- canonical cwd を `WorkerFilesystemAuthority::local` の root/cwd とし、host ごとに top-level Worker と process-owned `WorkdirSession` を一つ作る。 -- Controller transport は `InProcess` に固定する。通常起動で Worker subprocess、HTTP/WS server、Unix socket を作らない。 -- model provider は通常の resolved Manifest から構築する。埋め込み host と deterministic test は `start_with_model_client` で同じ bootstrap に process-owned client を注入できる。 -- feature plan/install は既存 `WorkerController` が行う。Task や optional direct SubWorker を standalone 側で再実装しない。 -- `shutdown()` は Host-owned Job service の scheduling を閉じ、Internal Worker の取消・cleanup とローカル SQLite connection の close を確認してから、既存 `Method::Shutdown` を送る。controller が active run、SubWorker registry、Workdir session、MachineScope allocation を順に片付けた後の confirmation を待つ。cleanup を確認できない drop/error では lease を保持する。 -- `jobs()` は Feature へ明示的に渡す汎用 Job 実行 capability。対話 Session・モデル作成 SubWorker とは別で、Profile/result authority と永続化の契約は [Standalone Jobs](standalone-jobs.md) に記録する。 -- startup error は category のみを公開し、credential、prompt 本文、session metadata、内部 path を error text に含めない。 - -## CLI / TUI routing - -- `yoi` の connection-aware command は `TargetKind::Standalone | Backend` の二択で dispatch する。`--local` は Standalone を、明示的な CLI selector、または `$XDG_CONFIG_HOME/yoi/client.toml` の `default_connection = "backend"` と `default_backend` は Backend を選ぶ。repository-local `.yoi/workspace.toml` / `.yoi/client.config.toml` は connection や Workspace selection の authority ではない。 -- `yoi init --display-name --repository-key ` は現在の Git repository root を Server DB に登録し、`$XDG_CONFIG_HOME/yoi/client.toml` に Backend routing のみを保存する。repository-local `.yoi/workspace.toml` は作成しない。 -- Backend workflow で Workspace selector が省略された場合、client は現在の Git repository の canonical local source または `origin` URI を Server DB の Repository catalog と照合する。exactly one の Workspace だけを選択し、zero/multiple match や Backend offline は明示的に失敗して repository-local file へ fallback しない。 -- Standalone の通常起動は `StandaloneHost`、restore は専用 `StandaloneStore` の session picker を使う。Workspace Worker list、PID、Unix socket、subprocess は探索しない。 -- `workers`、Backend Worker restore、Workspace panel、Ticket、Objective は Backend authority を要求する。Standalone から repository-local filesystem backend へ fallback しない。 -- `yoi worker` は Runtime や明示的な process-owned integration が使う direct Worker entrypoint として残るが、通常の `yoi` / TUI 起動経路からは呼び出さない。 - -## 非目標 - -- TUI/CLI routing や画面実装 -- Runtime / Workspace Server / Orchestrator / Ticket authority の内包 -- standalone 独自の HTTP/WS API -- Worker/Tool/Task/SubWorker protocol の fork -- subprocess Worker launcher や runtime-owned Worker catalog diff --git a/docs/design/standalone-jobs.md b/docs/design/standalone-jobs.md deleted file mode 100644 index 1ab7d7ab6..000000000 --- a/docs/design/standalone-jobs.md +++ /dev/null @@ -1,152 +0,0 @@ -# Host-owned standalone Jobs (T-708) - -## Implementation design recorded before code changes - -Base: `3c424fd17d1a5e8bb5542de2d0ad88f55338e0ea` (current origin/develop; T-704 integrated). T-709's subjektiv consumer is not part of this change. - -The reusable `job` crate owns the transport-neutral immutable request (registry Profile selector, input snapshot/reference and SHA-256 digest, caller Job ID, instruction, serialization key and bounded limits), result envelope/digest and validation. Backend retains its existing SQLite state machine, Runtime binding, domain grants, authorization, unknown-outcome and cleanup implementation; its adapter uses the shared request validation and result contract, not a second runner. - -Standalone owns a separate adapter, not a WorkspaceClient impersonation. `StandaloneHost` owns a Job service and its connection lifecycle. The concrete local placement is `//jobs.sqlite3` (using the existing Worker directory layout; alongside, not inside, interactive sessions). The `standalone` crate owns the versioned local schema: durable intents, immutable attempts, execution state, accepted result/digest, cancellation and consumer acknowledgement. A single Host connection is protected by the existing Worker lease. SQLite transactions serialize reserve, dispatch, result acceptance and terminal transitions. No subjektiv tables or fake Workspace/Runtime registry are created. - -Local adapter states follow the existing intent/attempt/result distinctions. Exact immutable request replay returns the same Job, never starts another attempt. Explicit retry creates a bounded new attempt only for known failure; unknown outcomes require caller reconciliation and never auto-replay. Unstarted pending attempts remain pending on shutdown/reopen. Interrupted dispatch/running attempts without an accepted result become unknown; accepted results survive restart and remain available until consumer acknowledgement. Cancellation is durable and never auto-resumed. Consumer acknowledgement/delivery failure is separate from model success and never reruns the model. - -Execution uses the existing private Internal Worker substrate with a new Host-facing bounded Job entry point. It gets a distinct ephemeral session, explicitly resolved selected Profile/system prompt/model, no ambient parent tools, filesystem, Workspace or Feature connections, and only a Job/attempt/input-digest-bound structured result submission capability. Profiles requiring unavailable tools/features are explicitly rejected rather than silently stripped. Result authority is independent of Profile name; final prose, Idle and Worker termination are never success evidence. This first generic adapter grants no domain operations; future consumers must provide explicit domain capabilities rather than gaining them through a profile tag. - -The Host limits concurrent active resources independently of caller limits, enforces timeout/turn/output limits, routes cancellation into the Internal Worker lifecycle and waits for resource cleanup. Accepted results are committed before resource release; required domain postprocessing must finish before submission (or be independently durable in its consumer). No daemon, pool, subprocess, HTTP/WS listener or Server dependency is introduced. - -Validation will distinguish scripted in-process Internal Worker executions and SQLite reopen/recovery fixtures from any real-provider/process tests. It will cover alternative Profiles, immutable retry, invalid/stale results, resource limits, failure/cancellation/timeout, shutdown and restart. Backend focused regression tests must still prove its existing fences and cleanup. - -## API and ownership - -`job::{JobRequest, JobLimits, JobResultSubmission, JobOutcome}` are the common -Feature-facing data contract. `JobOutcome` distinguishes intent state from attempt -state and omits Workspace/Runtime identities. These enums are read vocabulary, -not another transition engine. Backend converts neutral requests with -`BackendJobRequest::from_job_request` and projects reservation/acceptance with -`.outcome()`; Backend domain grants, provenance, delivery and cleanup remain on -its own adapter and retain their existing durable/wire format and fingerprints. - -Standalone consumers receive `StandaloneHost::jobs()` explicitly, not a fake -Workspace client and not a tool on the interactive Worker. They can persist -`request`, read `get`/`pending`, explicitly `start` or `retry`, `cancel`, and -`acknowledge`. `.wait()` observes execution cleanup; `.outcome()` on the returned -snapshot is the common read view. `Completed` with its accepted structured result -is success authority; merely returning from `wait` is not. The injected-client -entry point resolves/validates the same selected Profile and model configuration; -it replaces only the model transport for embedders/tests. - -A typical generic consumer supplies its own stable Job ID/reference and immutable input, -immutable JSON input, instruction and registry Profile: - -```rust,ignore -let jobs = host.jobs(); -let snapshot = jobs.request(request)?; -if snapshot.state == standalone::jobs::JobState::Pending { - jobs.start(&snapshot.request.job_id).await?; -} -let outcome = jobs.wait(&snapshot.request.job_id).await?.outcome(); -// Validate/apply the consumer's domain result here. A failed notification or -// delivery leaves the successful result unacknowledged; do not retry the model. -if outcome.state == job::JobState::Completed { - deliver_idempotently(&outcome)?; - jobs.acknowledge(&outcome.job_id)?; -} -``` - -The Host retains tasks/cancellation even while a consumer waits; dropping a wait -future does not detach cancellation ownership or release a concurrency slot. -Active resources are bounded to eight by the Host and by the minimum of all live -requests' ceilings, including after result acceptance and before cleanup. -Serialization keys likewise fence live resources, not just pending result state. -Busy `start` returns a typed capacity error and leaves the reserved attempt -unchanged; callers decide when to start/resume pending work. There is no pool, -background daemon, implicit retry or independent Feature-owned Internal Worker. - -## Profile and capability policy - -The caller must select an existing built-in or user registry Profile. Path/raw -selectors and `inherit` are not accepted. Registry existence, selected instruction -and model configuration are checked before dispatch. The ordinary user settings -registry and Prompt catalog are used; repository-local recipes are not discovered -by the standalone Host, and default never silently maps to a different Profile. - -`builtin:job` is an **optional**, compatible result-only policy. It is not a -runner-selected default or authority token. User registry Profiles with different -instructions/models/bounds execute the same way, as demonstrated by the scripted -Worker tests. This adapter deliberately rejects active unsupported Feature, -filesystem/delegation scope, MCP, skills, compaction and durable event-trace -requirements. Existing `builtin:backend-job` has such requirements and is rejected -by this standalone adapter, while its Backend behavior is unchanged. A future -consumer requiring domain operations must arrange explicit Host capabilities; -choosing `builtin:subjektiv-memory-consolidation` alone grants nothing. - -Every run uses the existing private Internal Worker substrate, a distinct identity -and ephemeral Session. Only `SubmitJobResult` is installed by a FeatureModule. -Its model input is `{result: ...}`; Job/attempt/input-digest bindings come from -the Host capability, not model arguments. The sink checks those bindings, the -current persisted attempt, deadline, byte limit and result digest. Exact result -replay is idempotent; changed/foreign/stale results are rejected. Submission is -sealed before acceptance, so uncertain responses permit only identical retries. -An ambiguous sink failure is reconciled against SQLite; absent accepted evidence -it is `Unknown`, not a known retryable model failure. No domain logic is embedded -in this sink. Required consumer/domain postprocessing must complete before -submission or have its own durable ownership outside the ephemeral Worker. - -Selected instructions are retained. The Host execution contract is rendered from -`resources/prompts/internal/job_system.md` before the run's initial history is -created, not injected into a previous/transient turn. Turn/output policy is -attenuated to at most 32 turns and 8192 output tokens per model request, preserving -tighter selected Profile bounds. Structured results retain the shared 16 KiB -default / 64 KiB absolute byte limits, not a larger standalone exception. - -## Persistence and termination - -Local schema version **1** is owned only by `standalone::job_store`: `job_intents` -contains immutable serialized intent/fingerprint, serialization key, current -attempt and acknowledgement; `job_attempts` contains immutable identity/input digest, -execution state, failure and accepted result/digest. Foreign keys, immutable -identity/result triggers, transactions, `WAL` and `synchronous=FULL` protect the -boundary. No Server schema or subjektiv/Memory schema is copied or opened. -The Host opens and recovers the store only under its existing top-level Worker -lease; clones cannot open or recover it themselves. - -- **Exact request replay:** same immutable intent returns its current snapshot. - Changing any intent field under the same ID is a conflict, not a new attempt. -- **Known failure:** only explicit `retry` creates a new immutable attempt, up to - the intent's bounded maximum. Prior attempts/results remain in SQLite. -- **Timeout/shutdown before acceptance:** fence `Unknown` before requesting Engine - cancellation; late result submissions cannot create success. Await normal - cleanup; retain the Host lease and report an error if confirmation is lost. -- **Normal Host shutdown:** seal scheduling/mutations, cancel active runs, await - resource cleanup, close SQLite, then stop the interactive controller and release - its lease. Reserved/unstarted requests remain resumable. Old cloned handles - return `Closed`, rather than mutating a store reopened by another Host. -- **Unexpected Host drop:** request cancellation and retain the lease because - synchronous drop cannot attest cleanup. No daemon continues after process exit. -- **Process loss/reopen:** reserved attempts stay pending; unaccepted dispatched - attempts become unknown; accepted results/acknowledgements remain intact. - SQLite result evidence, never final prose or an ephemeral Session, decides success. -- **Cancellation:** durable and never automatically resumed. Cancelling already - unknown intent does not erase its attempt's uncertain execution evidence or - release its serialization fence. `retry` still refuses it. There is no generic - operation that declares uncertain domain side effects unexecuted; callers must - reconcile domain evidence rather than bypassing the fence with an implicit replay. -- **After acceptance:** model error, cancellation or consumer delivery failure - cannot erase success or rerun the model. Acknowledgement is independently durable - and idempotent. Execution resources are reclaimed independently of result lifetime. - -Emergency cancellation has a bounded grace (one second for cancellation setup, -two seconds for normal cleanup). Expiry is **not** cleanup confirmation: it returns -an error, keeps unknown evidence/accepted results, and the Host retains its lease. -The result-only Feature owns no async/background tasks or domain connections; -the dropped engine stream/tool pumps request abort. Synchronous SQLite submission -is bounded by its five-second busy timeout and cannot be preempted mid-transaction. -No resource or side effect is claimed rolled back merely because a future dropped. - -## Validation evidence - -See [T-708 validation ledger](../report/2026-10-06-t708-validation.md). It separates -scripted in-process Worker/Host runs, direct SQLite/reopen/rollback fixtures and -an actual test-owned child-process loss/Host restore test from live-provider or -live dogfood validation. T-709's subjektiv consumer is intentionally not migrated -here; Backend T-704 Profile selection and T-690 cleanup remain regression-tested. diff --git a/docs/design/standalone-subjektiv.md b/docs/design/standalone-subjektiv.md deleted file mode 100644 index a98dec422..000000000 --- a/docs/design/standalone-subjektiv.md +++ /dev/null @@ -1,43 +0,0 @@ -# Standalone subjektiv (T-709) - -## Implementation design, recorded before implementation - -Base: `13055d7ee18f9d29edc641d0bd0e57c3d034dfa0`, including T-704 and T-708. No Backend database, fake Workspace, local fallback, new daemon, or subjektiv-specific Internal Worker runner is introduced. - -The operator-managed standalone state directory (existing CLI/XDG standalone state setting, not cwd or a repository file) is the local sharing boundary. `/subjektiv/` is a private directory. A persisted random storage-scope identity is issued here; Subject IDs use the common domain UUID issuer, never cwd/Profile/Worker name. The common Feature SQLite manager opens the subjektiv database under this root; it owns connection options, migration execution, close/reopen and transaction locking. The common subjektiv crate owns the existing schema/migrations/domain/CAS/receipt/surface rules. Existing Backend layout and physical schema remain unchanged. Job domain state is not embedded into Memory tables or Session text. - -Subjects are explicitly created/listed in that state scope and selected at normal launch; enabled policy alone does not create or attach one. Each selected Subject has a separate OS advisory lock file under `/subjektiv/leases/`. The Host holds the exclusive lock for body execution and all delegated Jobs until confirmed shutdown. A second process cannot acquire the same lock. Process death releases the kernel lock, not a PID/Profile guess; uncertain in-process cleanup retains ownership. A separate Subject can run concurrently, with SQLite serializing common database transactions. Symlinks and non-private state nodes must be rejected at this boundary. Knowledge of an ID/path is not a model capability. - -The selected Subject and local storage-scope identity are persisted in the standalone Worker record. Restore verifies them and reacquires the original Subject lock; resume never silently changes Subjects. Subject-free and disabled-policy launches retain the old path with no subjektiv persistence side effects. Ordinary SubWorkers/Reviewers receive no independent Subject/extraction connection. - -The Feature receives an explicit Host-scoped subjektiv capability, separate from WorkspaceClient and Profile policy. Backend adapts its existing authenticated API operations; standalone binds one local store, Subject, Worker and committed Session authority. Session attribution is established by the Host, including Sessions without candidates. Reads use the existing committed public Session projection and standalone Session store, not a copy in the Memory database. No system prompt, hidden reasoning, uncommitted history or arbitrary filesystem/foreign-Subject access is granted. - -Consolidation uses the existing generic standalone Jobs service and common request/result contracts. For a selected Subject the Host owns `/subjektiv/jobs//jobs.sqlite3` under that Subject lease, allowing durable intents/unknown outcomes/results to follow it across Workers. Subject-free Hosts retain the T-708 Worker-local location. Subjektiv selects the consolidation Profile. Job capabilities are explicitly attenuated to the immutable candidate batch, recall/decision/surface operations and public Session references; they never confer the Subject body's execution lease or ambient parent filesystem/Workspace tools. Common receipt/CAS rules handle partial apply and response loss; unknown execution is never automatically replayed. Surface failure leaves confirmed Memory intact. The generic runner owns cancellation, timeout, result binding, resource cleanup and shutdown; processing stops with the standalone process and pending durable work is recovered only on a subsequent explicit Host connection. - -Validation must distinguish scripted model/Host and real test-owned process/lock runs from live-provider model quality. Root `cargo check`, changed-crate tests and narrow Backend/Feature/Job regression tests are required, plus format/diff checks. Concrete CLI/UI usage and recovery are documented in [standalone.md](../standalone.md). - -## Implemented boundaries and validation - -The validation counts and former API terminology below record T-709 at the cited source, not new executions. Current Memory changes and Job input-digest bindings follow [the store contract](subjektiv-store.md) and [the Job contract](standalone-jobs.md). - -- `feature-storage` owns the shared SQLite connection policy/Feature manager; `subjektiv` owns the unchanged domain schema/migrations, Host-neutral API dispatch, and T-704 consolidation request/result/surface validation. Server modules remain thin adapters; Worker/standalone do not depend on Workspace Server. -- `worker::subjektiv` exposes a nonserializable live Host capability plus a separate attenuated consolidation capability. Backend creation still requires the actual authenticated client and bound snapshot. Local creation is explicit Host injection; unavailable Backend never falls back locally. -- The local catalog keeps private scope/language settings and common Feature storage under `subjektiv/`, separately from generic Subject Job ledgers. The common attribution table's existing `runtime_id` lookup key is the local storage namespace for local rows, not a running Runtime identity; public local evidence leaves `runtime_id`, Workspace, and Account fields absent. Worker IDs, Subject binding, and operator-managed scope must all match before Session content is read. -- The shared strict FsStore public index is also used for committed evidence admission. Stable references are intersected with visible committed entries; numeric hints accept the two existing capture producers. Open-Run evidence is deferred only before a request, never exposed by Session tools or recorded before commit. Persisted local human provenance is distinct from Backend Account provenance and maps to the existing public `HumanInput` class. -- Standalone integration tests use real Hosts/SQLite, scripted extraction/consolidation/surface models, and test-owned second processes. Nine cases cover inert/disabled policy, reopen, saved-binding restore, cross-Worker recall/source discovery, exact explicit commit staging, normal extract/pointer → generic Job → CAS apply → resident surface, same-Subject cross-process exclusion/abnormal-exit recovery, and transport loss after apply followed by surface-only repair without duplicate immutable Memory change. -- Generic Job tests additionally cover immutable grant persistence, missing grants, foreign grants, mismatched input bindings and wrong-consumer grants, per-attempt/deadline revocation, cancellation, timeout, cleanup failure, and stale capabilities. Session adapter tests cover no-candidate discovery, foreign Subject/scope/Worker denial, bounded UTF-8 and JSON output, filter/generation-bound cursors, missing/corrupt storage, and hidden/uncommitted/partial-tail exclusion without observation writes. - -Verified commands: - -```sh -cargo test -p standalone --test subjektiv -- --test-threads=1 -cargo test -p protocol -p feature-storage -p subjektiv -p manifest -p session-store -p standalone -p worker -p tui -p yoi -p yoi-workspace-server -cargo test -p worker-runtime --lib protocol_ -cargo check -cargo check -p protocol --features typescript,json-schema -cargo check -p tui --features e2e-test -cargo fmt --all -- --check -git diff --check HEAD -``` - -The changed-crate run passed, including 836 Worker unit tests, 603 Workspace Server unit tests (authorization/singleton/T-704 regression), 48 standalone unit tests, and nine standalone Subject integration tests. Existing ignored tests remain ignored. TypeScript generation check reports existing unsupported serde-attribute warnings but succeeds. These commands prove mechanism/authorization/storage contracts, not live-model quality or full interactive terminal E2E behavior. diff --git a/docs/design/subjektiv-consolidation-jobs.md b/docs/design/subjektiv-consolidation-jobs.md deleted file mode 100644 index b18ce191d..000000000 --- a/docs/design/subjektiv-consolidation-jobs.md +++ /dev/null @@ -1,103 +0,0 @@ -# Backend subjektiv consolidation Jobs (T-704) - -subjektiv requests the common Backend Job runner with an explicit -`builtin:subjektiv-memory-consolidation` Profile. The runner has no purpose-based -Profile switch. Subject-body Workers and the independent Workspace Memory feature -are not Jobs and are not migrated or reset here. The shared/standalone adapters -planned in T-708/T-709 are outside this Backend change. - -## Input, grants and concurrency - -The existing backlog/byte thresholds and missing/stale/failed surface policy -remain the launch decision. A Job captures an immutable bounded batch (up to 100 -candidate IDs) plus the Subject `memory_fingerprint`. Selection also reserves the Job's -16 KiB result budget, accounting for JSON-escaped IDs appearing in both the batch -claim and compact verified outcomes. Excess candidates remain staged for a later -batch. An empty batch is valid for a -surface-only rebuild. The Backend explicitly binds the Subject and candidate -batch grant in durable Job intent before Worker execution. No subject-body -singleton ownership or Session attribution is transferred to the Job. - -Authenticated `(Workspace, Runtime, Worker)` registry binding and the exact -current dispatched attempt authorize each operation. Names, Profile tags and -model-provided `subject_id` are not authority. Candidate listing exposes only -unresolved IDs in the captured batch; reads/decisions outside it are denied. -Surface prepare requires all batch candidates to have durable dispositions. -Surface generation records preserve immutable Job/attempt provenance (subjektiv -schema migration 6); publication, failure and completion reject another attempt's -generation, including pre-Job/legacy generations. - -The common resource key `subjektiv-consolidation:` prevents parallel -Jobs for a Subject. Concurrent requests return the original active intent and -report `skipped_existing_job`; they do not expand its grants. New candidates and -items beyond the bounded batch stay in staging. A subsequent consolidation -request snapshots remaining work; it is not counted as processed by the first -Job. There is no domain-specific status ledger, pool or persistent idle Worker. -Definitive failures use the common bounded retry API with the original intent; -unknown execution stays fenced for attention. A later explicitly requested Job -may snapshot remaining candidates once all previous Workers are safely cleaned. - -## Memory and surface completion - -`MemoryApplyCandidate` still uses the existing `expected_change_id` checks, atomic -apply/close and immutable decision receipts. Partial application survives model -failure, timeout, lost response, retry, Worker deletion and Backend recovery. -Job replay is not a second memory apply: resolved candidates retain their -existing dispositions, and exact candidate decisions converge on their receipts. - -`SubmitBackendJobResult` for a bound consolidation Job receives only the Subject -and captured candidate IDs. Its Host capability awaits the existing bounded -clean-context surface editor and publish-or-recorded-failure, then adds the actual -structured surface outcome. It does **not** schedule a Worker-owned post-Run task. -The result call serializes and caches the exact generated outcome for ambiguous -submission replay. The Backend verifies every batch disposition and the current -Memory-input fingerprint, generation-run ownership and persisted ready/failed surface fields; -it enriches the durable result with actual immutable candidate disposition -summaries (`candidate_id` and `action`). Full reasons, affected change references and -source edges remain in the existing immutable receipt store; they are not copied -into a bounded Job result. Exact replay uses the same verified compact projection. -Thus confirmed-memory consolidation and surface readiness are separate outcomes. -A recorded surface failure does not undo Memory. A stale/unfinished/fabricated -surface cannot be reported completed. Final prose or Idle is never success. - -Only after durable structured result acceptance does the existing terminal Job -cleanup run. Its tracked Run teardown fence lets the result tool finish before -stop/remove. Timeout/unknown paths cancel unfinished Worker execution through -that same cleanup; later operations/results are rejected. The next request can -regenerate failed/stale surfaces without touching confirmed memory. - -## Legacy cutover and recovery - -The subject-specific direct spawn and Idle resubmission paths are removed. -On a consolidation request the Backend checks the **persisted singleton owner** -for the old `subjektiv-consolidation:` key. It revalidates Workspace, -key and old dedicated recipe provenance, not the display name. A running/paused -legacy foreground operation is left alone and returns `skipped_legacy_inflight`; -no Job is launched alongside it. Legacy owned operations can finish during this -bounded cutover. - -After foreground Idle (or Stopped) is confirmed, Backend stop is confirmed before -canonical WorkerRemovalService removal. Any legacy surface background task is -cancelled at stop; the subsequent Job rebuilds from confirmed Memory and does not -infer successful surface completion from Idle. Busy/unobservable/mismatched -owners fail closed. Empty/ready Subjects also retire their idle legacy owner. -Removal failures remain diagnosable through existing durable removal recovery; -a subsequent request rechecks ownership instead of deleting storage directly. -Subject-body/ordinary/Workspace-Memory Workers are never selected by this key. - -The Job runner uses its existing durable dispatch allocation, tracked input, -T-701 registry-bound restore/recovery, deadlines, unknown outcomes, result replay, -delivery and T-690 cleanup. Job results, candidate receipts, Memory changes, -provenance and surfaces remain in their existing Backend stores after Worker -removal. No live dogfood Worker or environment is stopped/updated by development -validation. - -## Validation boundary - -Server recording-backend fixtures prove reservation/binding/auth/result/cleanup -contracts, and existing store tests prove candidate CAS/atomic receipts and -surface freshness. Worker scripted-client tests prove the clean-context editor -completion-before-result boundary and result replay. They are not real-provider -LLM or live dogfood process evidence. See the T-704 validation report for exact -commands/results and inherited intermittent failures; later passes do not erase -those failures. diff --git a/docs/design/subjektiv-store.md b/docs/design/subjektiv-store.md deleted file mode 100644 index 0cca8638c..000000000 --- a/docs/design/subjektiv-store.md +++ /dev/null @@ -1,324 +0,0 @@ -# subjektiv subject Memory store - -The trusted Server-side `subjektiv` repository in -`yoi_workspace_server::subjektiv` is the persistence authority for subject-scoped -Memory. It is a domain store, not a Worker tool. Legacy single-document Workspace -Memory remains a separate, deprecated control-plane authority until an operator -performs the explicit reset defined by the -[subjektiv product cutover runbook](../development/subjektiv-product-cutover.md); -legacy rows are never imported, dual-written, or silently reinterpreted as subject -Memory. - -## Ownership and scope - -- The Server-managed `FeatureStorage` owns the SQLite path, connection, common - pragmas, migration lifecycle, backup/restore, shutdown, and Workspace deletion. -- subjektiv owns its migration contents, tables, issued subject/Memory/snapshot - identifiers, validation, queries, and domain transactions. -- Each repository is bound to the Workspace of its `FeatureDatabase`. A - `store_scope` row verifies that binding on every open. Subject IDs select data - inside that boundary; knowing an ID grants no authorization. -- Authorization must happen before calling the repository. Neither a subject ID - nor Memory text can confer Workspace permissions or assignments. -- No SQLite path, connection, or SQL surface is sent to a Worker or Runtime. - Async callers must run these synchronous repository calls on a blocking - executor. -- The store contains no `current_worker_id` or equivalent current link. Yoi's - keyed singleton remains the sole authority for the current Worker. T-668 - connects Workers using the opaque `subjektiv:` key and records - immutable historical Worker/Session attribution as provenance; that history is - never consulted as a current Worker link. - -Registration is deliberately separate from open. Trusted Server construction -registers `REGISTRATION` once per `FeatureStorage` manager, then reuses its -`RegisteredFeature` for each Workspace. Merely adding this module does not open -or create a subjektiv database. T-668 opens it lazily for explicit subject -operations or an authenticated subject extraction write; leaving the Feature -disabled has no subject-store side effect. - -## Immutable changes and schema contracts - -All new persisted JSON records use `schema_version: 1`. Subject staging retains -existing extract `StagingRecord` schema version 2 and its six kinds: -`preference`, `working_assumption`, `constraint`, `decision`, `open_question`, -and `lesson`. The host adds only `subject_id` and `created_at`; the model does -not provide subject scope or provenance. - -A subject record has a store-issued random ID, an explicit role, lifecycle state, -and lifecycle timestamps. It starts `active` with no confirmed Memory. Retirement is terminal -in this baseline and is distinct from stopping a Worker. Retirement prevents new -writes but retains all reads and history. - -User-managed `behavior_md` is separate from generated Memory context. Updating it compares the exact previously read text through `expected_behavior_md`, not a behavior counter. Conflicts require rereading the actual current text; changing behavior neither changes confirmed Memory nor rewrites Worker history. - -An immutable Memory change contains: - -- a stable store-issued Memory ID, opaque `change_id: String`, and optional `previous_change_id`; -- subject ID, existing six-way kind, state, claim, optional Markdown body, - `why_useful`, and optional staleness condition; -- exact source candidate IDs and exact `{memory_id, change_id}` derivations; -- a required change reason and host-issued creation/update times. - -The first change must start `active`. For every kind, `active` may remain active or -become `resolved`/`retracted`; `resolved` may remain resolved, reopen to active, -or be retracted. `retracted` is terminal. A corrected experience after -retraction receives a separate Memory ID and may cite the old fixed change. -`active` means eligible for ordinary recall, not proven true and not authorized. - -Refinement/correction of the same experience appends a new immutable change under the -same Memory ID. A distinct experience or lesson gets a new ID. Every write uses -`expected_change_id` equal to the current change ID; stale callers receive a typed `MemoryChangeConflict`. -Old rows, evidence edges, and derivations are immutable and retained. - -`memory_fingerprint` is SHA-256 over the sorted, length-framed `(memory_id, current_change_id)` set, including all Memory states. Each committed content/state/provenance change gets a fresh change ID, preventing ABA reuse; staging alone does not change this fingerprint. - -A surface snapshot stores only its body, exact Memory change references, the -subject's `built_from_memory_fingerprint`, and host-issued ID/time. Creation requires -the captured fingerprint to equal the current `memory_fingerprint` and permits at -most one fixed change of each Memory. The snapshot may deliberately select a -bounded subset; selection, generation, bounds, and injection policy are later -work. - -## Relational invariants and retention - -The feature database contains `subjects`, `staging_records`, -`staging_resolutions`, `memory_records`, immutable `memory_changes`, normalized -candidate/derivation edges, and immutable `surface_snapshots` with normalized -references. Composite foreign keys always include subject scope. As the physical -database is already Workspace-scoped, Workspace identity is stored once in -`store_scope` rather than copied into every row. - -`memory_records` is the `current_change_id` pointer and current JSON projection; -`memory_changes` is the immutable historical authority. Candidate JSON is -immutable. A resolution stores both the resolution JSON and the exact staged JSON -bytes that were resolved. Aggregate seal rows prevent evidence, derivations, -resolution targets, or snapshot references from being appended after their -parent JSON is finalized. Foreign keys plus retention triggers prevent dangling -candidate, derivation, resolution-target, and snapshot references. Ordinary -subject operations expose no record deletion API; the legacy reset contract does -not touch this Feature database. - -Search columns and JSON projections are written together by the typed repository. -Raw SQL remains private trusted implementation detail. Existing Workspace Memory -rows are neither imported nor dual-written. - -## Transactions - -`FeatureDatabase::try_transaction` preserves typed Feature-domain errors while -retaining the same `BEGIN IMMEDIATE`, rollback, lifecycle fencing, and connection -ownership as `transaction`. - -`SubjektivStore::apply_candidate` and `apply_candidates` atomically: - -1. verify the active subject and every unresolved candidate in repository scope; -2. check the expected Memory change ID and every exact candidate/derivation ref; -3. derive the change's candidate provenance from that same candidate set; -4. insert and seal the immutable Memory change and evidence edges; -5. update the current Memory projection and recompute the fingerprint of current Memory identities; -6. insert and seal each immutable candidate resolution and exact target change. - -Generic candidate resolution cannot emit `applied`, and direct Memory writes -cannot attach candidate provenance; those relationships must go through this -atomic path. - -Any error rolls the complete operation back. The transaction is confined to the -subjektiv Feature database; it does not promise atomicity with the Server -control-plane database or another Feature database. - -## Operational contract - -Reopening the same Server-managed feature database verifies migrations and the -stored Workspace scope before serving reads/writes. `WorkspaceFeatureStorage` -online backup/restore includes the entire subjektiv database, including immutable -history and provenance. Workspace deletion and shutdown use the common lifecycle -fences and close retained repository handles. Disabling subjektiv does not erase -its database. - -## Extraction connection (T-668) - -`feature.subjektiv` is an explicit Worker profile boundary. It reuses the -committed-run threshold, restricted Internal extraction Worker, -`SessionEntryRef` evidence resolution, explicit finish, generation fence, and -success-only pointer progression from the established Memory lifecycle. Its -pointer domain is separate from legacy Workspace Memory, and a Worker is rejected -at installation when both automatic extraction destinations are enabled. Ordinary -SubWorkers and Reviewer SubWorkers have lifecycle Features disabled by their host; -only evidence actually committed into the parent Session can enter the parent's -capture. - -The staging request contains the existing model contract (`kind`, `claim`, -`why_useful`, optional `staleness`, and entry references after host resolution) -plus the committed parent `session_id`. It contains no subject id. The Server -derives subject scope from authenticated Runtime/Worker source proof and the -current keyed-singleton lease, then attaches the session id to every source -reference and records immutable historical `{subject, runtime, worker, session}` -provenance atomically with staging. A Session already attributed to another -subject is rejected. Preference candidates still require exclusively HumanInput -evidence. - -Subject retirement does not stop a Worker, and stopping/replacing a Worker does -not retire the subject. A replacement uses the same `subjektiv:` key; -retries therefore recover through Yoi's singleton authority without a second -current-worker mapping in this database. - -Candidates are immutable writes that occur before `FinishMemoryExtraction`. If a -run stages one or more candidates and then fails, is cancelled, or loses its -generation fence, those candidates remain recorded and the extraction pointer -does not advance. A retry may therefore produce semantically duplicate candidates; -subsequent consolidation must compare them using the existing candidate -identity/content boundary. The system must not claim that the partial writes were -rolled back or silently delete them. Subject recall, consolidation, and legacy -Memory migration remain separate work. - -## Recall and explicit proposal tools (T-669) - -An enabled `feature.subjektiv` installs five names that do not overlap the legacy -single-Markdown Memory tools: `SubjektivMemoryQuery`, `SubjektivMemoryRead`, -`SubjektivMemoryListChanges`, `SubjektivMemoryRemember`, and -`SubjektivMemoryProposeChange`. Runtime-signed Worker source plus current -`subjektiv:` singleton ownership is re-evaluated for every operation; -none of the model-visible inputs contains a subject, Runtime, Worker, Session, -origin, or raw provenance object. - -Query searches only current projections. Omitted `states` means `active`; explicit -empty `states` or `kinds` is invalid. Output defaults to 20 and is capped at 100, -ordered by `updated_at DESC, memory_id ASC`. Its opaque cursor binds the subject, -canonical filters, offset, and subject `memory_fingerprint`; any confirmed-Memory -change makes it a typed stale-cursor conflict rather than silently mixing -snapshots. Change-ID conflicts and stale cursors retain their diagnostic code as a -structured `{status: "error", error: {code, message}}` tool result instead of -requiring models to parse prose. Change history follows the immutable `previous_change_id` chain, newest first. Its first page fixes `head_change_id`, so later appended changes neither duplicate nor displace old page members. IDs are not counters and are not ordered numerically. - -Read accepts an exact nonempty `change_id` or resolves the current change once. A -missing historical change never falls back to current. Markdown pagination is -line-based (default 200, maximum 1000) with a preliminary 16 KiB UTF-8 body -cap and a global 56 KiB budget over the exact pretty-serialized model-visible -response. JSON escaping, provenance, and metadata therefore reduce the current -body page before the Worker content ceiling is reached. It reports -`body_truncated`, `body_next_offset`, and `body_next_byte_offset`; an oversized -or escape-heavy single line resumes from the returned UTF-8 byte boundary within -that same line. Every continuation offset must also supply the exact `change_id` -returned by the first page, so a current change cannot mix body contents. -Provenance uses a separate immutable-change-bound cursor and returns one -candidate/derivation reference per page. New staged candidates contain at most 10 -evidence records and 10 source references; read pages project at most two of each, -with displayed anchor text capped at 64 UTF-8 bytes and nested offsets carried by -the ordinary evidence cursor. Compatible candidates persisted before the admission -cap remain readable across as many evidence pages as needed, while every current -staging path rejects new larger anchor sets. Candidate evidence is the bounded -host-resolved anchor saved in staging; raw Session bodies are not copied -into Memory responses. Resolved, retracted, and historical changes remain -addressable by ID. - -Remember and ProposeChange only stage candidates. Entry references are resolved -against the host's committed Session capture, and preference candidates continue -to require exclusively `HumanInput` evidence. When no entry is supplied, a -non-preference request returns `pending_commit`; after the run commits, the host -reconstructs pending receipts from the durable tool-call/tool-result history and -uses the committed tool-call entry itself as model-origin evidence. Receipt lookup -reports `pending_commit`, `staged`, or `missing`; the post-commit hook retries -immediately, the pre-request hook replays durable pending receipts after Worker -restore, and the fail-closed pre-rewrite hook prevents compaction from dropping an -unstaged receipt. Hook failure and process restart therefore retain an idempotent -retry path, while an operation that never commits is not staged. Receipt identity is derived from Session and tool-call identity, -and exact backend retries return the first candidate rather than creating another -one. Neither path changes `memory_records`, `memory_changes`, the confirmed-Memory fingerprint, -or a surface snapshot. - - -## Historical Session discovery (T-673) - -An enabled `feature.subjektiv` also installs exactly three read-only tools: -`SubjektivSessionList`, `SubjektivSessionSearch`, and `SubjektivSessionRead`. -The Server binds subject, Workspace, Runtime, Worker, retention catalogue, and -archive authority from the authenticated caller. Model input contains only -Session/segment/entry selectors, search filters, limits, and opaque cursors; -knowing any identifier does not grant access. Reads use Yoi's retained or -committed-archive Session storage and a shared public projection. They do not -copy Session bodies into the subjektiv database, restore Workers, migrate -Session storage, create candidates or Memories, revise a surface, or promote a -re-read tool result to HumanInput. - -The public projection covers committed records in every persisted segment, -including non-active branches and pre-compaction entries, while excluding -system prompts, hidden reasoning, traces/diagnostics, unfinished run tails, and -attachment or pasted-artifact bodies. Results preserve exact Session, segment, -entry, provenance, and lineage identity so inherited entries are not presented -as a current decision. Full entry reads are UTF-8-boundary paged and remain -subject to the same projection as snippets and search. - -The subject's first committed primary-Worker Session is attributed independently -of extraction thresholds or candidate creation. Recording is idempotent and is -retried on later committed-run and pre-request lifecycle points after transient -failure; no attribution is attempted before a committed capture exists. This is -not a distributed transaction with Session persistence. In particular, the -system does **not** backfill older unattributed Sessions from their text, display -name, current singleton ownership, Memory references, or other inference. -Historical Sessions without trustworthy Host-recorded attribution remain outside -subject discovery until a separately authorized provenance mechanism exists. - -Pagination cursors bind scope, operation, filters, stable ordering, and the -public storage generation and are never authorization credentials. Every page -rechecks current subject and observation authority. Deleted-Worker archives are -eligible only when both the committed Server catalogue and Runtime manifest -agree and the archived observation grant remains valid; incomplete, expired, -corrupt, unavailable, or changed sources are reported explicitly rather than as -an empty successful search. - -Change proposals add optional `change_proposal` metadata to the existing v2 -`SubjectStagingRecord` envelope; automatic extraction remains proposal-free and -its model schema is unchanged. The metadata is serialized atomically with the -candidate and therefore remains present in candidate reads and immutable -resolution copies: - -```json -{ - "intent": "revise", - "memory_id": "memory-…", - "expected_change_id": "change-…", - "change_reason": "The committed evidence corrects the prior condition" -} -``` - -This is the T-670 handoff fixture. Consolidation must re-read this typed metadata, -re-check subject ownership, `expected_change_id`, and the requested state -transition in the same transaction that applies the candidate, and leave a stale -proposal unresolved/conflicted. It must not parse target information from the -claim, retarget to the latest change, or revive a retracted Memory. Valid -transitions at staging are revise while active/resolved, active→resolved, -active/resolved→retracted, and resolved→active. Retraction remains terminal. - -## Candidate consolidation and corrections (T-670) - -subjektiv requests a bounded Backend-owned Job with the explicit -`builtin:subjektiv-memory-consolidation` Profile and an immutable Subject/candidate -batch grant. The common runner resolves the recipe; Profile names do not grant -permissions. The Host derives authority from the authenticated current -Job/attempt/Worker binding; model-visible inputs cannot select a Workspace, -subject, Runtime, Worker, or Session. Its tools list/read only pending candidates -in that batch, query/read confirmed changes, and make one candidate decision. -Candidate text is evidence rather than authority. No subject-specific idle Worker -is resubmitted; see [Job execution and cutover](subjektiv-consolidation-jobs.md). - -Every decision carries a stable request ID. An exact retry returns the committed -receipt, while reuse with different input is a conflict. Applying a candidate -atomically writes one new confirmed change, its exact candidate and derivation -edges, the immutable resolution and affected change references, and updates -the subject Memory fingerprint. A non-applied decision explicitly records -`discarded`, `invalid`, `duplicate`, or `already_covered`; an empty model response, -transport failure, or aborted consolidation records no disposition and leaves the -candidate pending. A successful applied response is the only evidence that -confirmed Memory changed. - -A typed change proposal must be applied to its exact `memory_id` and -`expected_change_id`, preserving its intent and change reason. The transaction -rechecks the current change ID and state transition. It never rebases or retargets -a stale proposal, revives retracted Memory, or extracts a target from claim prose. -Correction/refinement of the same experience appends under the same Memory ID; -a corrected experience after terminal retraction receives a new Memory ID and may -cite the retracted fixed change as a derivation. Surface generation is a -separate post-consolidation lifecycle: its failure cannot roll back a committed -candidate decision. - -For the product activation, legacy reset, and operator recovery boundary, follow -the [subjektiv product cutover runbook](../development/subjektiv-product-cutover.md). diff --git a/docs/design/subjektiv-surface.md b/docs/design/subjektiv-surface.md deleted file mode 100644 index dee59f61f..000000000 --- a/docs/design/subjektiv-surface.md +++ /dev/null @@ -1,85 +0,0 @@ -# Subject Memory surface generation - -The subject Memory surface is a bounded start-time context derived from confirmed Memory with immutable change history. It is not another Memory authority and is never used as extraction or consolidation input. - -## Fixed policy - -The Backend captures one bounded input snapshot from current `active` records for exactly one subject. Staging records, Session history, historical changes, and prior surfaces are not queried. - -Selection is deterministic: - -1. categories use the fixed order `preference`, `constraint`, `decision`, `working_assumption`, `open_question`, `lesson`; -2. each category is ordered by `updated_at DESC, memory_id ASC`; -3. at most 8 records are retained per category; -4. records are allocated round-robin by category, up to 24 records total; -5. the 10,000-token material budget is initially divided equally across categories as a fairness reservation; unused shares are then reclaimed in the same canonical round-robin order, so one large category cannot exclude another non-empty category or strand usable capacity; -6. both category and aggregate material bounds use the provider-independent estimate `ceil(UTF-8 bytes / 4)`; -7. before invoking the editor, the Worker measures the actual normalized Agen request fields containing the rendered system prompt, conversation items (including the fixed question and material JSON), and the typed `SubmitMemorySurface` tool definition/schema against the 12,000-token complete-input budget. It reapplies equal per-category fairness reservations under that final estimator, reclaims unused shares in canonical order, fails rather than publishing a false empty surface if no grounded material fits, and guards every subsequent correction request before it reaches the provider. Provider-specific HTTP envelope bytes are outside this normalized request budget. - -The clean-context editor uses the effective `builtin:subjektiv-memory-consolidation` profile model (`codex-oauth/gpt-5.6-luna`, medium reasoning). A generation permits three model turns (the initial attempt plus at most two correction turns). A Memory-input fingerprint conflict causes at most one complete re-read and regeneration. The published Markdown is capped at 1,024 tokens by the same estimate. - -Every editor point carries one or more exact `{memory_id, change_id}` references. The Host checks the output shape, non-empty grounding, budget, subject scope, membership in the captured materials, and current `memory_fingerprint`. These checks prove reference existence and freshness only; they do not claim semantic correctness. - -Publication is idempotent for identical output for one exact Memory-input fingerprint. A different concurrent output cannot replace the first published surface. Confirmed Memory writes atomically mark the prior surface stale. Failed surface work never rolls back confirmed Memory or candidate disposition. - -Resident injection uses five distinct product states: - -- `ready`: inject the non-empty snapshot only when its `built_from_memory_fingerprint` - equals the subject's exact current `memory_fingerprint`; -- `ready-empty`: a successful current generation selected no Markdown; inject no - prose, but preserve the explicit successful-empty state; -- `ungenerated`: no successful current generation exists; inject nothing; -- `stale`: confirmed Memory changed after publication (or the snapshot predates - this generation policy); inject nothing until regeneration succeeds; -- `failed`: the generation attempt failed; inject nothing and preserve the typed - failure for diagnostics/retry rather than representing it as empty Memory. - -The resident Backend transports availability separately from optional Markdown. -In storage, `ready-empty` is a current `ready` snapshot with empty Markdown; it is -never conflated with `ungenerated`, `stale`, `failed`, request failure, or -omission. Publication is append-only: generation inputs/runs, failures, snapshots, -and exact references remain immutable history, while current state points at the -applicable result. - -Runtime-owned new Workers install Features before durably materializing the -initial Session head. Initial resident prompt construction uses the host's shared -prompt-contribution source contract, so an enabled Feature can contribute its -current surface without owning Session persistence. Restore refresh is different: -it is a subjektiv Feature lifecycle hook installed only for an execution-enabled, -subject-attached Worker. The generic Worker host announces the restore boundary -and durably commits typed system items requested by installed Features; it does -not infer refresh behavior from the presence of a resident source. Consequently, -legacy Memory-only Workers, policy-only unattached Workers, and lifecycle-suppressed -Internal Workers/Reviewers do not read a subjektiv surface, render the restore -template, or append a refresh. - -The subjektiv restore hook preserves the persisted prompt and all prior history -verbatim. It requests one durable restore-boundary system item that supersedes -earlier resident summaries with the latest `ready` surface, an explicit -`ready-empty` state, or a no-current-surface tombstone for `ungenerated`, `stale`, -or `failed`; subsequent turns in that restored process do not append the boundary -again. No regeneration, correction, failure, or cutover rewrites a committed -Session entry. - -## Semantic fixture and verification - -Use a test subject with these active materials: - -- constraint: “Never send credentials to tools.” (`staleness`: “Until credential brokering is redesigned.”) -- constraint: “A trusted broker may deliver short-lived credentials to the Git provider.” -- decision: “Publish only the Ticket work branch; never force-push.” -- working assumption: “The provider resolves selectors after a normal push.” -- open question: “Does the provider preserve approval after target-only movement?” -- lesson: “Checking only the local branch previously produced non-reviewable evidence.” -- duplicate lesson: “Local HEAD alone is not immutable review evidence.” - -A valid review fixture must: - -- retain “never” and the credential-redesign expiry condition; -- present the apparently conflicting broker constraint without silently declaring one side invalid; -- preserve the target-only-movement item as an open question rather than a decision; -- combine the two review-evidence lessons only if both meanings remain; -- cite only exact refs supplied with those materials; -- distinguish mechanical citation validation from a human/model semantic review of the wording. - -The repository tests exercise deterministic bounds and fair budget reclamation, subject isolation, active-current-only selection, complete normalized editor-request accounting, output/reference rejection, empty/ungenerated/failed/stale states, legacy regeneration, generation races, bounded conflict retry, terminal failure recording, parallel publication, correction/retraction staleness, and publication idempotency. The prompt fixture above is the manual semantic check for condition, negation, reservation, conflict, expiry, and duplicate editing behavior. diff --git a/docs/design/ticket-evidence.md b/docs/design/ticket-evidence.md deleted file mode 100644 index 122399440..000000000 --- a/docs/design/ticket-evidence.md +++ /dev/null @@ -1,25 +0,0 @@ -# Ticket evidence and immutable Merge Request results - -Ticket lifecycle, repository integration, and current requirement attestation are separate facts. An incomplete read-side evidence summary does **not** undo a recorded `done` state or a `CompleteTicket` event. - -## Evidence subjects - -- An open MR uses the Repository/Runtime authority's provider-resolved current source. No Server-local checkout is assumed. Failure is a tagged `source_ref_observation: { status: "unavailable", code }`, not proof that review is stale. Ticket review status is `unknown`; `missing` includes `source_ref_unavailable`, not `review_after_rescope` merely because observation failed. -- A merged MR uses its stored MergeResult's `approved_source_ref` and `approval_event_id`. Branch deletion/movement, checkout absence, and provider outage do not trigger source observation. The summary's `current_subject_ref` is the immutable merged source and observation is `not_required`. -- Integration approval must exist, approve that exact source, predate the merge, have matching ReviewRequested evidence, and not have been revoked. `integration_evidence_error` is a bounded machine code for invalid stored evidence. `merged` alone never creates approval. - -A merged MR's review status/timestamps describe **integration approval**, not a later requirement approval. Paged MR detail reads evaluate the full stored result before paging; UI must not infer approval from a merge event appearing in one page. - -## Current requirements - -`merge_request::requirement_approval` evaluates Ticket read-side MR evidence; it is not a Ticket completion gate. It validates an effective Reviewer approval of the current item content digest and the **exact complete linked MR/source snapshot**, including the hosting MR's own source and corresponding ReviewRequested event. There is no primary-MR fallback, comment-derived approval, or timestamp-only freshness rule. A post-integration review may attest new item content/a linked-result snapshot without replacing MergeResult or its integration approval. - -`approved_current_subject` describes valid source/integration approvals. `review_after_rescope` retains its wire name but now describes exact current requirement attestation, not ordering of wall-clock timestamps. `approved_review` query filtering requires that current attestation. `stale_after_rescope` requires substantive item-edit history and a known source-approved snapshot lacking current attestation; unknown observations are only `missing_evidence`. - -`complete_for_integration` means a nonempty linked MR set has valid source approvals and current requirement evidence, including for partially or fully integrated sets. It is not a claim that open repositories have been integrated, that target integration readiness has been checked, or that Ticket completion is authorized. An MRless Ticket has no applicable MR evidence: `missing` is empty, `review_status` is null, and approval/integration flags remain false. It does not match `missing_evidence` or `approved_review` merely because its state is done. - -CompleteTicket belongs to the Ticket domain. It records the authenticated actor's reason/result and optional references using current item `content_digest`/state CAS and an operation receipt. It needs no MR or Reviewer approval, and does not create or modify those proofs. State setting, completion, close, and reopen share the same transactional update boundary. Exact replay returns current Ticket authority without reapplying a historical decision or appending an event. Historical MR-based `ticket_completion` records remain readable and distinct from new judgment receipts. - -Writable targets authorize deliverables, not mandatory MRs for every target. Every linked MR remains part of the exact requirement snapshot; no arbitrary latest-review or single-repository shortcut is used. - -Query candidate SQL narrows Ticket metadata, not live evidence. Evidence predicates are evaluated from the same full MR projection as ShowTicket, after source observation. Bounded scans can return an empty page with `has_more` and a cursor after the last scanned candidate; callers must follow that cursor rather than treat an empty page as exhaustion. This prevents stale SQL/report heuristics or early nonmatching candidates from hiding later evidence matches. diff --git a/docs/design/ticket-item-checker.md b/docs/design/ticket-item-checker.md deleted file mode 100644 index e38f3b436..000000000 --- a/docs/design/ticket-item-checker.md +++ /dev/null @@ -1,39 +0,0 @@ -# Asynchronous Ticket item checker - -Worker-originated `TicketCreate` and `TicketEditItem` mutations are checked after the authoritative SQLite save. The checker is advisory: its scheduling, model execution, result delivery, or failure never rolls back a Ticket mutation and never changes Ticket state, Queue eligibility, assignments, or implementation authority. - -## Applied paths - -The trigger is in the Workspace Server's authenticated Worker Ticket REST path. A complete, verified Runtime/Worker source header pair is required before a check is scheduled. Every successful Worker `Create` and `EditItem` operation schedules the persisted item content digest; failed saves and thread/state events do not schedule work. - -The browser's direct item-edit handler currently does **not** schedule this check because it has no originating Runtime Worker advisory destination. Browser edits must not be presented as checked. Adding that path later requires an explicit notification/audit destination rather than borrowing an unrelated Worker identity. - -## Immutable input and content identity - -The Job input contains only: - -- canonical and user-facing Ticket IDs; -- the exact saved item `content_digest`; -- exact saved title and body; -- whether the operation created or edited the item; -- changed-field flags and the necessary previous title/body for edited fields. - -The digest is SHA-256 of editable Ticket item content, matching `TicketDetail.content_digest`; it is not a `create`/`item_edit` event ID or an edit counter. Conversation history, user requests, agreements, Worker transcripts, and external facts are not captured. The generic Backend Job input limit is 128 KiB; an oversized exact snapshot is not truncated or misrepresented as checked and produces a server diagnostic after the Ticket remains saved. - -## Execution and bounds - -The trigger reserves durable intent synchronously after save, then dispatches without waiting for model completion. Concurrency pressure never rejects that intent: attempts above the four-Job cap remain durably `reserved`, and terminal attempt transitions plus restart recovery drain queued attempts through an atomic `reserved` → `dispatching` slot claim. It uses T-674's Backend-owned Job runner and an ordinary registered embedded Runtime Worker with `builtin:backend-job`. That profile selects the existing lightweight `codex-oauth/gpt-5.6-luna` model through normal model configuration and authentication. The checker has only `SubmitBackendJobResult`; it has no Ticket mutation, Queue, Worker management, conversation discovery, Web, Memory, or Workdir tools. - -Each check is bounded to 4 concurrent Backend Jobs, 45 seconds, one attempt, an 8 KiB result, at most 8 findings, bounded finding fields, and the Backend Job runner's fixed input/instruction/delivery limits. The deterministic Job ID hashes the complete checker intent, including the saved content, edit context, instruction and source/notification identity; exact retries reuse a Job, while a different edit or source producing the same Ticket content does not collide with it. Generic Job and attempt records retain completed results and categorized dispatch, timeout, invalid-output, and other failures. - -## Result and advisory behavior - -`internal.ticket_item_checker` defines the text-only categories and exact structured output. Server validation denies unknown fields, rejects duplicate or excessive findings, bounds every string, and requires each quote to be an exact substring of the inspected saved title or body. Invalid output terminalizes the attempt with `invalid_result` rather than treating it as a clean check. - -Before advisory delivery, the server reloads the Ticket and compares the current item content digest with the immutable checked digest. Findings for the current content are sent only to the authenticated source Worker over the existing tracked `Notify` path. A notification identifies itself as a post-save checker advisory, includes Ticket/content digest, exact quote, reason, and correction/confirmation direction, and explicitly states that it is not a user request, approval gate, workflow blocker, or implementation instruction. - -Clean results and stale results send no Worker input. Their durable delivery records are terminalized with distinct suppression categories so restart recovery cannot later emit or repeatedly reclaim them. Notification failures remain separate from the completed checker result. A stopped or missing destination is never recreated or started. - -## Authoring responsibility - -`common.tickets` separately instructs Ticket authors not to turn their own current-turn scope into durable requirements and not to invent user constraints. Those rules use the author's request context. The independent checker deliberately does not infer user intent or constraint provenance and does not certify overall specification correctness when it returns no findings. diff --git a/docs/design/ticket-wip-projection.md b/docs/design/ticket-wip-projection.md deleted file mode 100644 index 146e1e193..000000000 --- a/docs/design/ticket-wip-projection.md +++ /dev/null @@ -1,82 +0,0 @@ -# Ticket native WIP projection - -## Purpose and authority - -When a Worker explicitly runs in `wip` mode and its Ticket Feature is enabled, the Worker Host projects the enabled Ticket surface as native Worldspace objects. The projection is an adapter over the existing scoped Workspace Ticket API and typed Ticket tools. It does not add a Ticket store, resolver, audit path, state machine, readiness rule, queue path, or relation implementation. - -The Host assigns the Ticket provider the root `tickets` namespace through `WipMountRegistry` and mounts: - -```text -/tickets -/tickets/ -``` - -There is no Feature Object or `/features/ticket` compatibility alias. A legacy path supplied as current input fails with rediscovery guidance; saved history containing that text is left unchanged. - -`` is either a canonical `T-` key or an existing canonical internal alphanumeric ID. Only a direct child is accepted. Nested suffixes, Objective keys, unrelated prefixes, and hidden routes do not resolve. - -Every call retains the Worker's scoped `WorkspaceClient`, current Ticket Feature access, and ordinary Tool permission policy. Discovery is not mutation authority. A direct call to an unpublished operation fails even when the caller knows its name. `deny` and interactive `ask` permission outcomes fail closed before Backend dispatch. - -## Current operation inventory - -The native surface is assembled from the tools actually enabled by `TicketFeatureAccess`; it is not a second broad Ticket catalog. The native operation delegates to the named ordinary Tool and validates the reconstructed input against that Tool's exact JSON Schema. - -Collection interface `yoi.ticket/collection/v1`: - -| Native operation | Existing Tool identity | Subject identity | -| --- | --- | --- | -| `query` | `QueryTicket` | Collection; preserves bounded filters, limit and cursor | -| `create` | `TicketCreate` | Collection; Backend assigns identity | - -Item interface `yoi.ticket/item/v1`: - -| Native operation | Existing Tool identity | Non-subject inputs | -| --- | --- | --- | -| `read` | `ShowTicket` | `event_limit`, `event_cursor` | -| `edit` | `TicketEditItem` | title/body replacement and targets edit | -| `comment` | `TicketComment` | comment body | -| `mark_ready` | `TicketMarkReady` | optional reason | -| `intake_ready` | `TicketIntakeReady` | intake summary and optional reason | -| `queue` | `TicketQueue` | none | -| `transition` | `TicketWorkflowState` | expected/target state, reason and body | -| `close` | `TicketClose` | resolution | -| `dependency_check` | `TicketDependencyCheck` | none | -| `record_relation` | `TicketRelationRecord` | relation kind, **target Ticket**, optional note | -| `remove_relation` | `TicketRelationRemove` | relation kind and **target Ticket** | -| `record_orchestration_plan` | `TicketOrchestrationPlanRecord` | kind, related Ticket, note and accepted-plan data | -| `query_orchestration_plans` | `TicketOrchestrationPlanQuery` | relation kind and bounded limit | - -The subject `ticket`/`id` parameter is removed from every item descriptor and reconstructed exclusively from the resolved route. A caller cannot replace it through arguments. A relation target or related Ticket remains an explicit, distinct input and is resolved by the existing typed Backend. - -Access profiles therefore expose different native interfaces: - -- read-only/review: collection `query`; item `read`; -- workspace authoring: query/create plus read/edit/comment/ready/queue/close/relation operations; -- intake: the authoring surface plus `intake_ready`; -- workflow/orchestration: query/read/comment, workflow transition/close/dependency checks, relation operations and orchestration-plan operations; -- work-report: query/read/comment only. - -No native operation is inferred for a Tool that the Feature did not register. Enabled Ticket tools claimed by the native projection are removed from `/tools`; normal Tool mode is unchanged. - -## Bounded reads and references - -`query` uses the authoritative Workspace bounded query and cursor contract. Results include canonical `T-*` references and native item paths without embedding unbounded thread or evidence history. `read` uses the authoritative bounded detail projection for thread events, relations, Objective references, assignments, Merge Requests and evidence. Objective and Merge Request values remain references/data; this projection neither assumes their native routes nor grants their authority. - -Backend 401/403 failures map to permission denial, 404 to missing objects, and deterministic 400/409/422 validation or workflow failures to invalid arguments when the typed operation fails before any successful mutation. Cancellation and interruption remain distinct. Post-mutation canonical-reference lookups are explicitly marked as unknown-outcome stages, so even a later deterministic HTTP status cannot misreport an already-committed write as a rejection. Transport, server, malformed response, and other post-dispatch failures retain unknown outcome, and mutating operations are not retried automatically. - -Successful reads observe the authoritative item `content_digest`. Successful mutations advance a local observation generation even when the first call uses an internal ID whose canonical alias has not yet been learned. Once a read or mutation reveals the canonical key, the canonical/internal alias groups are merged and invalidated together. Relation mutations also invalidate the reported target Ticket, while queue results invalidate every reported queued Ticket, because those objects' incoming relations or workflow states changed. A stale validator is rejected by the common WIP Host before another Backend dispatch; the caller must rediscover/reinspect and explicitly decide whether to call again. - -## Preserved Ticket contracts - -The projection invokes the existing Tool and scoped Backend routes, so the following remain authoritative and are not reimplemented by WIP: - -- item content digest and audited item/thread/state events; -- title/body/targets input validation; -- planning-to-ready target validation and write-target requirement; -- queue dependency traversal, planning-dependency rejection and cycle detection; -- expected-state workflow transitions and close semantics; -- canonical `T-*`/internal-ID resolution and Workspace isolation; -- exact forward relation ownership, inverse views and relation target validation; -- orchestration-plan validation and bounded query behavior. - -The ordinary JSON Schema is retained as a second validation boundary behind the WIP descriptor. Optional fields omitted by the caller stay omitted, while an explicitly supplied `null` stays present as JSON `null`, so permission matching sees the same input shape as normal Tool mode. Defaults are applied only by the same ordinary typed Tool that applies them in Tool mode. diff --git a/docs/design/tool-permissions-scope.md b/docs/design/tool-permissions-scope.md deleted file mode 100644 index 0d2d88b4c..000000000 --- a/docs/design/tool-permissions-scope.md +++ /dev/null @@ -1,37 +0,0 @@ -# Tool permissions and scope - -Yoi treats tools as explicit capabilities. Model-visible tool names are not permission by themselves; the resolved Manifest and Worker scope decide whether a call is allowed. - -## Permission policy - -Tool permissions are built into PreToolCall policy. - -- `allow` permits the call to proceed. -- `deny` rejects only that call with a synthetic error result. -- `ask` fails closed until a real approval/resume protocol exists. - -Failing closed matters because an unresolved approval state is not the same as permission. A future approval flow must be able to pause and resume the same unresolved call, not merely ask the model to retry later. - -## Filesystem scope - -Filesystem scope is separate from tool allow/deny. A file tool may be registered and permitted, but the concrete path still must be inside readable or writable scope. - -Symlinks do not grant extra authority. Access decisions should be made on the canonical target when it exists, and broken or out-of-scope links should produce diagnostics rather than escaping scope. - -Directory traversal tools should not follow symlink directories as a way around scope. If an external checkout is needed, add its real path to read scope. - -## Delegation - -Child Workers receive an explicit subset of the parent's scope. Delegation is a capability loan, not a copy of all parent authority. - -When a child stops, shuts down, or is pruned as unreachable, delegated write permissions must be reclaimed. Explicit base denies remain in force. - -## Tool output - -Tool output should be bounded before it enters history/model context. The system may truncate or summarize mechanical output boundaries, but it should not hide the fact that a tool call happened or fabricate successful results. - -Network tools such as WebSearch/WebFetch are disabled/unconfigured by default, fail closed, and need explicit manifest/profile configuration. Their outputs are untrusted content and must remain bounded. - -## Why this design exists - -LLM tool calls are suggestions from an untrusted planner. Yoi can let the model propose operations while keeping final authority in manifest policy, scope checks, and durable records. diff --git a/docs/design/workdir-wip-projection.md b/docs/design/workdir-wip-projection.md deleted file mode 100644 index 4c6db4ded..000000000 --- a/docs/design/workdir-wip-projection.md +++ /dev/null @@ -1,186 +0,0 @@ -# Repository, Workdir and attachment WIP projection - -## Objects and operations - -The Host owns these root namespaces, using the same `WipMountRegistry` -namespace/resolver/contribution contracts as Tickets, Objectives and Merge Requests: - -```text -/ -├── repositories list -│ └── read -├── workdirs list, create -│ └── read, attach, delete (when supported) -└── workdir-attachments list - └── read, detach -``` - -Operations are Interface declarations, **not child files**. There is no -`/features/manage-workdir`, no writable Repository catalog and no Repository -content tree here. `/checkouts` content projection is a separate provider (T-699), -as is Workspace configuration (T-695). Management Objects do not grant access to -unattached contents. A future content provider should return its entrance from -an authorized, live attachment; it must not infer content/command capability -from catalog or lifecycle access. - -`path` in inventory results is an absolute WIP Object path. It is a reference, -not an OS path, a capability grant, or an instruction to reinterpret another -string argument as a route. Source Repository references use `repository_path`; -attachment references use `workdir_path`. Attach results include -`attachments_path` to discover the current connection and its lifetime ID. - -### Identity and route encoding - -Repository identity is the formal Backend `repository_key`, never its display -name or a guessed `main`. Workdir identity is `working_directory_id`, never a -Worker-local alias. A segment containing only ASCII letters, digits, `-`, `_` -is unchanged. All other UTF-8 identities are encoded as `~` followed by lowercase -hexadecimal UTF-8 bytes. The `~` marker is itself always encoded. Only the -canonical encoding is accepted; no alternate spellings, recursive URL decoding, -case folding, slash traversal or namespace fallback are accepted. For example, -`repo/name` is `/repositories/~7265706f2f6e616d65`. This is injective and accepts -existing valid keys, including Unicode and slash-containing identities. - -Attachment identity is a fresh opaque Backend connection ID for each lifetime, -including detach/reattach to the **same** alias and Workdir. Read exposes alias, -Workdir ID and effective attachment capabilities as attributes. Workdir existence -and Worker connection are separate Objects. Detaching a connection does not -remove the Workdir or delete a logical settings space. - -## Feature and authority separation - -`feature.workdir_catalog.enabled` independently supplies read-only Repository, -Workdir and attachment reference operations. It defaults enabled so a WIP Worker -with Workspace authority and appropriate permissions can find registered keys -without Ticket or conversation hints. It does not add ordinary Tools or require -a filesystem/Bash location. Standalone Workers without Workspace authority have -no Workspace catalog mounted. - -`feature.manage_workdir.enabled` supplies management operation implementations -on the **same Objects** using `contribute_operations` and -`contribute_dynamic_operations`. It remains default disabled. It neither owns -another resolver nor confers permission. The object provider is unique; duplicate -namespace, route, operation name or inconsistent Interface contributions fail at -registration through the common registry. - -Native permission identities are: - -| Operation | Manifest permission identity | Bound identity in permission input | -| --- | --- | --- | -| repositories.list | RepositoryList | list arguments | -| repository.read | RepositoryRead | repository_key | -| workdirs.list | WorkdirList | list arguments | -| workdir.read | WorkdirRead | working_directory_id | -| workdirs.create | WorkdirCreate | original create arguments | -| workdir.attach | WorkdirAttach | working_directory_id + alias | -| workdir.delete | WorkdirDelete | working_directory_id + reason | -| attachments.list | WorkdirAttachmentList | list arguments | -| attachment.read | WorkdirAttachmentRead | connection_id | -| attachment.detach | WorkdirDetach | current bound alias | - -Read permissions gate direct item resolution as well as list item visibility. -The manifest's existing default policy is retained; `ask` is denied fail-closed. -An argument-dependent policy may advertise an operation that could be allowed, -but every call checks its exact reconstructed input before execution. Mutation -arguments cannot override path-bound Workdir or connection identity. Having -RepositoryRead is not WorkdirCreate; inventory/management permissions are not -file access, delegation or command execution grants. Backend caller/Workspace -identity, source proof, attachment capability ceilings and lifecycle policy are -checked by the existing paths on every request. - -## Contextual Interfaces and stale observations - -Providers opt into common Host contextual Interfaces. Operation publication is -the intersection of registered implementation, current target support and current -subject policy. The collection reference feature can remain available while the -management feature is disabled. External Workdirs without cleanup authority do -not expose delete. Only currently supported lifecycle operations are advertised; -no delete/command is added to logical attachments solely because they appear as -workspaces. - -A contextual Interface reference is opaque and path-qualified as -`/@/`. Clients must use the returned reference, not -construct it. Fetch and call freshly resolve the same Object and filtered -Interface; base references cannot fetch an unfiltered descriptor. Descriptor and -Object validators invalidate old observations after target/connection/permission -changes. Validators are managed by the WIP Client, not manually by the LLM. -Cached Interface possession and known direct paths never bypass current checks. -Historic session text remains append-only and is not rewritten or replayed. - -## Backend lifecycle and safe output - -Catalogs are request-time projections of existing Backend authority, not another -store or synchronization job. WIP list pages default to 50 and accept 1–100 -items. Repository pages sort the existing complete Repository catalog by stable -key. Workdir pages use the signed caller-bound Backend -`workers/self/workdir-catalog` operation, with SQL keyset paging over the existing -Workdir registry in stable ID order. This is not the legacy bounded -`working-directories` browser/Tool snapshot: Workdirs older than its 200-record -cap remain reachable. `next_cursor` advances over scanned rows even when the -current read policy or unsupported legacy source makes a page sparse or empty; -`empty` describes that page, and `has_more` remains authoritative for progress. -Attachment pages use the Backend's SQL-bounded, alias-ordered limit/offset query -and opaque `offset:` cursors. Both Workdir and attachment responses supply an -opaque complete-set `digest` derived from existing authoritative records in -the same database snapshot as the page, not by hashing the first page. The Host -uses this complete-set digest for the collection Object validator. A lifetime -or capability change beyond row 50 therefore invalidates old collection -observations without adding a ledger or edit counter. A cursor is not -a snapshot or authority grant: restart listing after concurrent collection -changes. Filtering by exact `connection_id` is applied in the caller-scoped -Backend ledger before paging, so direct item lookup also works beyond the first -page. Unknown/expired IDs yield no item, never a current alias substitute. -Repository projection uses an explicit field -allowlist: formal key, kind/provider, any existing display metadata and explicit -nullable default selector. Source URI/path, credentials, remotes, Runtime -connection information and internal fingerprints are not model-visible. -Workdir projection includes stable ID, optional display name, public source kind -and Repository key, existing selector/ref/tree, status, cleanliness and occupancy -when available. Cleanup handles and External grant internals are not forwarded. -No missing attributes or implicit selectors are fabricated. Backend errors are -sanitized rather than forwarding raw diagnostic bodies with possible host paths. - -Create and attach reuse the ordinary Workdir backend, request identity, -Worker-session router, mutation lock and Internal SubWorker lifecycle hooks. -Creation does **not** attach. Attach installs the Backend-attenuated session -capabilities, never a fabricated `ALL` capability for External Workdirs. Detach -uses the same begin/finish/cancel-detach path and command/operation exclusion as -Tools; the Backend checks expected connection identity under its existing -current-Worker lifecycle lock before closing sessions, then conditionally -updates that exact row in the existing connection ledger. Alias reuse cannot -retarget an old connection operation. Compensation retains the original -connection identity. Existing removal dispositions `removed`, `retained`, and -`attention_required`, plus retryability, remain intact. - -Stale identity/validator rejection is distinct from unknown operation outcome. -Transport failure, invalid post-dispatch response or uncertain lifecycle failure -is **not** proof that no effect occurred; it must not be automatically retried. -Inspect current state or request operator guidance. No new retry or attachment -ledger is introduced by WIP. - -When native management is mounted, `WorkdirList/Create/Attach/Detach/Delete` are -claimed as represented native capabilities, so `/tools/Workdir*` is not also -published. Ordinary Tools mode keeps its names and inputs. Existing native -Ticket/Objective/Merge Request routes retain their T-698 contracts. - -## First-Workspace example - -1. `Tree(path="/", depth=1)` finds `repositories`, `workdirs` and - `workdir-attachments` when permitted. -2. Inspect the Repository collection path and Invoke `list` with a - bounded limit. An empty registered/visible catalog explicitly returns an empty - list; do not invent a key. Follow `next_cursor` when supplied. -3. Inspect the returned item `path` directly and Invoke `read`. - `default_selector: null` means no configured default, not `main`. -4. Inspect the Workdir collection and `create` using the discovered formal key, - with optional selector, display name and Runtime under the existing contract. -5. Follow created `item.path`, Inspect directly and Invoke `attach` with a chosen - Worker-local alias. This is separate from creation. -6. Follow `attachments_path`, `list`, then discover/read the returned connection - path to confirm alias, Workdir and effective capabilities. -7. Call that connection's `detach` with no identity override. The observed - lifetime ID guards alias reuse; the Workdir remains in its inventory. - -Each step uses current WIP observations and no Ticket-derived key, host path, -Runtime endpoint or manual validator. Explicitly refresh/reset discovery after -external changes and before reconstructing a rejected stale call. diff --git a/docs/design/worker-session-state.md b/docs/design/worker-session-state.md deleted file mode 100644 index 57b6aa1d5..000000000 --- a/docs/design/worker-session-state.md +++ /dev/null @@ -1,95 +0,0 @@ -# Worker aggregate, Session, and run authority - -A Runtime-managed Worker is the canonical durable aggregate. One Worker owns exactly one Session for its lifetime; compaction and forks create Segments inside that Session rather than replacing the Session identity. - -This identity rule separates durable conversation history from execution attempts: - -- **Worker ID** identifies the Runtime catalog aggregate and remains stable across stop/restore. -- **Session ID** identifies that Worker's sole replayable history and remains stable across compaction. -- **Segment ID** identifies a branch or compacted history projection inside the Session. -- **run identity** identifies one process/controller execution attempt. Fresh spawn uses a unique run ID; Restore uses the durable lifecycle operation ID so reconciliation addresses the same attempt rather than allocating another controller. - -## Canonical Runtime layout - -Filesystem Runtime stores materialize one aggregate under `workers//`: - -```text -workers// - worker.json - metadata.json - session/ - session.json - segments/ - .jsonl - .trace.jsonl - runs/ - / - worker.sock - worker.out.log - worker.err.log - artifacts/ - spawned/ - restore-/ - worker.sock - worker.out.log - worker.err.log - artifacts/ - spawned/ -``` - -`worker.json` is the atomic Runtime catalog authority: Worker identity and Workspace attribution, the settled lifecycle status, the create/restore request and execution binding, plus any pending Restore or Stop reconciliation operation. `metadata.json` is the current Worker projection used to restore active/pending Segment pointers, resolved manifest state, delegation metadata, and child/peer visibility. It is not a second transcript. - -`session/session.json` fixes the single Session ID for the aggregate. The Worker-specific Session store rejects attempts to address another Session ID. Session JSONL is the append-oriented replay authority for committed user inputs, assistant items, tool results, system/runtime events, Segment lineage, and effective snapshots required to explain later behavior. - -The normal execution path resolves all three stores from the trusted `WorkerRef`. It does not fall back to process-global Session or Worker-metadata roots. Those roots are legacy migration inputs only. - -## Segment lifecycle - -A new Worker materializes its Session when the initial Segment is created. The Session ID then remains stable. - -Compaction writes a new Segment in the same Session with `compacted_from` lineage. Forking likewise writes a sibling Segment in the same Session with `forked_from` lineage. Allocation, UI, and event surfaces therefore report the new **Segment ID**, never a "new Session". A different Session requires a different Worker aggregate. - -This keeps existing Worker IDs, Session IDs, Segment references, observation entry references, and UI routes meaningful across compaction and restore. - -## Run lifecycle - -A lifecycle transition has a stable operation ID distinct from settled Worker status. Runtime commits a pending operation into `worker.json` before invoking a side-effecting Backend call: - -- fresh spawn uses a unique run ID and commits the connected aggregate only after Backend startup succeeds; -- Restore records `ReconciliationRequired::Restore(operation_id, mode, last_settled_status, restore request)` after preflight and before controller startup; -- Stop records `ReconciliationRequired::Stop(operation_id, last_settled_status, execution evidence)` before requesting controller shutdown. - -Restore run directories are named `restore-` and use `create_new` semantics. Retry and startup reconciliation reuse that operation ID. An in-process Backend returns an already registered controller for the same operation; after Backend restart, the existing operation-scoped directory prevents the same operation from allocating a second run generation. A different Restore operation receives a different operation ID and directory only after the previous operation has reached a proved terminal outcome. - -Startup reconciles pending Stop operations before considering automatic Restore. A proved Stop commits `Stopped` atomically and is therefore never selected as an active restore candidate. Pending or uncertain operations remain orthogonal to the last settled status and block conflicting input, lifecycle, deletion, and retention operations. - -Restore candidates keep protocol events and observation registration isolated until the Active aggregate commit succeeds. After that commit, Runtime activates the Backend observation registration and flushes buffered protocol events in order. A rollback discards the candidate context, so external observers cannot see a live candidate beside a settled `Stopped` catalog record. - -Stopping a Worker waits for controller shutdown completion and removes `worker.sock`. Run directories and diagnostic files remain evidence. How old runs are retained, archived, or purged is a separate policy; aggregate creation and migration do not invent that disposition. - -Live sockets and provider sessions are execution hints, not durable identity. Restore reconstructs Workspace client attribution, observation registration, and Workdir/provider bindings from Runtime/Backend authority while keeping the Worker and Session identities unchanged. - -## Legacy migration - -Startup migration recognizes the versioned `worker-aggregate-v1` format and treats legacy process-global metadata/Session directories as read-only sources. - -The migration is serialized by an OS file lock and writes a versioned manifest with per-Worker checkpoints and diagnostics. For each unambiguous catalog Worker it: - -1. validates Worker metadata identity and the referenced Session; -2. parses Segment and trace JSONL, tolerating only the existing crash-truncated final-line rule; -3. stages `session.json` plus all Segment files under the target Worker aggregate; -4. fsyncs files and directories, atomically renames the staged Session directory, and fsyncs its parent; -5. atomically writes and fsyncs `metadata.json`; -6. atomically updates the migration checkpoint. - -Reruns validate exact Session identity, the complete source/target filename set, and file bytes before accepting an existing target. Mixed old/new stores and a crash between Session rename, metadata copy, and checkpoint update therefore converge without replacing divergent data. - -Migration fails closed on target collisions, corrupt complete JSONL records, or ambiguous ownership. If multiple legacy metadata sources reference one Session—including metadata with no catalog Worker—the Session is not assigned to either aggregate. The manifest records the shared reference. Legacy metadata or Session directories with no catalog-backed owner are also recorded as orphans and left untouched; they do not become normal authority and are not silently deleted. - -Legacy sources remain available for audit and recovery after a successful copy. Archive/retention disposition and Orchestrator-driven Worker removal are intentionally outside this migration contract. - -## Child, peer, and notification state - -Parent-visible children and peer registrations are current Worker metadata, distinct from Session history and run liveness. Restoring a parent reconstructs reachable children where possible and retains stopped-but-restorable visibility when metadata supports it. Delegated write scope is a capability loan; stopping or pruning a child must reclaim the parent's effective permissions. - -Peer registration does not grant filesystem authority, imply parent ownership, or make notifications completion proof. Notifications remain UX hints committed through the normal Worker history path. Completion decisions must reread durable Ticket, repository, review, and test evidence rather than relying on a socket event or final assistant message. diff --git a/docs/design/worker-wip-mode.md b/docs/design/worker-wip-mode.md deleted file mode 100644 index e1d957568..000000000 --- a/docs/design/worker-wip-mode.md +++ /dev/null @@ -1,169 +0,0 @@ -# Worker WIP mode - -## Status and dependencies - -Workers support two explicit model-facing modes: - -```toml -[worker] -name = "example" -mode = "tools" # default -``` - -```toml -[worker] -name = "example" -mode = "wip" -``` - -Profiles select the same policy without owning the runtime Worker name: - -```toml -worker = { mode = "wip" } -``` - -Omission resolves to `tools`, so existing manifests, profiles, persisted snapshots, and restored Workers retain the ordinary individual-tool surface. A direct Internal SubWorker selected with `profile = "inherit"` inherits the parent mode as reusable behavior. `default` and named profile selections use that profile's own mode and do not implicitly inherit WIP. - -The implementation uses the published **WIP 0.2.0** Client, HTTP, Protocol and official Text View packages from **crates.io**. Exact workspace version constraints and Cargo.lock registry checksums reproduce one Protocol type source; no local attachment, path override, Git snapshot or old registry 0.1.0 substitutes for the release. Canonical scope/Text View contracts were checked against wip-reference **6086f3c5ef10aa1464ce9c824750d7f667217bfd**. See [validation and resolution checks](../development/wip-integration-validation.md). CI/Nix dependency acquisition requires no upstream Git credentials. Yoi retains its async provider/transport and existing execution boundaries; upstream owns Client cache/lifecycle, codecs and signatures. - -## LLM surface and route contract - -A WIP Worker exposes exactly these generic LLM tools instead of exposing every enabled ordinary tool twice: - -- `Tree(path, depth)`: retrieve the full indexable range within depth 0..8, or report incomplete/loading/failed/limited coverage as an error. Children at the requested boundary remain unobserved (null), not empty. Cached nonindexable paths are never included by prefix scanning Known Space. -- `Inspect(path)`: acquire and display exactly the selected entity. An Object path returns only its official Object signature and ordered Interface references (each with an explicit inspection path); it does not fetch Interface descriptors. An absolute `scope::name` Interface address returns only that Interface's complete official signature. Neither form requires a prior Tree or Inspect. The path is explicit JSON envelope context outside the path-free signature. Summaries are complete; runtime metadata and documentation details are not appended to signatures. -- `Invoke(path, interface, operation, arguments)`: use a structured `{scope, name}` reference and named argument record. The Client owns validators and observed scope_ref. Missing/stale/failed observations are recovered with at most one pre-dispatch retrieval per subject. A dispatched operation is never automatically retried. - -Tree and Inspect accept explicit `refresh: true` for blocked observations. Inspect refreshes only the selected Object or Interface, not referenced entities. An ensure API returning None is never itself readiness evidence: freshness, complete edge coverage and retained data are checked. Loading and capacity pressure are reported rather than superseded or busy-looped. - -The Yoi binding uses the Text View compact reference notation as an **input address**, not a parser for rendered signatures. For example: - -```json -{"path":"/tickets"} -{"path":"/::\"yoi.tool/Read/v1\""} -``` - -The first inspects an Object; the second inspects one root-scoped Interface. Object results contain `object_signature` and `interfaces: [{reference: {scope, name}, path}]`, without Interface signatures. Pass an entry's `path` directly to Inspect to obtain `{path, reference, interface_signature}`. This explicit per-entity display selection is a Yoi binding policy; it replaces the earlier automatic expansion of every Interface. Invoke continues to acquire missing observations automatically and accepts structured references, never a compact string. - -Reference components follow Text View quoting: scope allows unquoted `[A-Za-z0-9_./-]+`, name allows `[A-Za-z0-9_.-]+`; other components use JSON string quoting/escapes. There is no scope inference, normalization, relative resolution, or operation suffix. A literal Object path containing `::` must be JSON-quoted inside the `path` value (for example `{"path":"\"/literal::name\""}`) to distinguish it from an Interface address. Ordinary Object paths retain Protocol path syntax, including Unicode. Invalid or unavailable Interface addresses fail rather than falling back to Object lookup. - -### Model usage guidance - -The three tool descriptions are the recurring model-facing usage instructions, including after compaction. Use a shallow Tree only when needed to locate entrances, inspect only relevant Objects and unknown Interface contracts, and then work through Invoke. Reuse known paths and understood contracts for the **same exact scope/name reference**; do not inspect every node or repeat Inspect as a prerequisite for every operation. Same local names under different scopes are not evidence of the same contract. Reinspect only when a reference/contract is new, changed, or no longer available in the conversation context. Cache maintenance is the Client's responsibility, not a reason for extra model calls. Use domain read/list/search/query Operations for content and large/nonindexable collections. Current authorization checks and the prohibition on blindly retrying dispatched operations remain unchanged. - -The compatibility projection mounts enabled ordinary tools at: - -```text -/tools/ -``` - -Each compatibility Object publishes an explicit root-scoped reference: - -```text -{"scope":"/", "name":"yoi.tool//v1"} -``` - -Its single operation is `call`. For this compatibility route, `Invoke.arguments` is `{ "input": }`, matching the descriptor's named `input` parameter. Native projections instead receive a JSON object keyed by the descriptor's declared parameter names. Native input is decoded with the published `wip-http` descriptor-bound codec, so named and composite types are resolved, an integral JSON number remains a WIP `Number` when declared as such, and WIP `Bytes` use the codec's canonical RFC 4648 base64 JSON representation. Tool argument IDs remain JSON values; the adapter never infers a domain object or Worldspace route from them. - -Compatibility input is projected conservatively into JSON-shape-preserving WIP types: primitives, closed records with required/optional fields, homogeneous lists, and string enums. Local JSON Pointer references are expanded within bounded depth/node budgets; cycles and unsupported constructs stay `Json`. Open records, nullable/mixed unions, composition, and tuple elements are not falsely modeled as closed records or tagged WIP unions. The original input JSON Schema, including constraints, descriptions, defaults, and local definitions, is included in the **visible parameter summary** as well as retained documentation details. Thus a single Interface Inspect reveals the usable input contract even where WIP's type system cannot express it or an expansion budget is reached; it is never just an opaque `input: json` with hidden instructions. Full Inspect output still obeys response limits and is not silently truncated. - -The call envelope remains `{ "input": }`; no tagged union wrapper, field renaming, or default insertion is introduced. Before execution, the Host validates the reconstructed JSON value with the exact original schema and then delegates to the original async `Tool`. Structural projection never replaces original-schema constraints or permission checks. A schema that cannot be compiled prevents WIP startup instead of creating a weaker projection. - -## Native projection extension and collisions - -`worker::wip::WipMountRegistry` is the Host-owned namespace, Object, and operation registry. A provider requests a root namespace with `allocate_namespace(owner, namespace)`; the owner is retained as Host metadata and is not inserted into the public path. It then mounts the collection Object directly at `/` and may install a bounded direct-child resolver. Features do not appear as intermediate Objects. - -Object resolution and operations are separate contracts. One mounted `WipProjection` owns a static Object route and its initial operations. Another Feature may add operations with `contribute_operations` without replacing that Object. Dynamic families likewise keep one `WipDynamicItemResolver`, while `contribute_dynamic_operations` supplies disjoint handlers for every resolved item without a second resolver. The Host merges only contributions with the same interface format, documentation, and type declarations; duplicate operation names, inconsistent interfaces/validators, missing targets, duplicate namespace owners, and unrelated route claims fail deterministically at registration. The merged interface validator changes with the effective operation set, so stale Known Space cannot authorize a newly contributed operation. - -- Routes must be canonical absolute WIP paths, and allocated namespaces are one lowercase root segment. -- Object names must match the final route segment. -- Object existence and resolution have one owner; operation contributors cannot remount or substitute the Object. -- Multiple Features may contribute disjoint operations to the same Object interface. -- Unrelated duplicate routes, duplicate operation names, and conflicting interface shapes fail startup. -- A bounded dynamic mount may resolve exactly one item segment below an already mounted native collection; deeper paths never fall through to the resolver. -- A native projection replaces a compatibility projection either by the original same-route semantic capability or by an explicit registry capability claim owned by a mounted native route. -- Conflicting native claims fail startup. When native is selected, the claimed compatibility tool is hidden rather than exposed through a second entry. Normal Tool mode is unaffected because it does not install the WIP registry. - -`Inspect` delegates to official `wip-text-view::render_object` and `render_interface`; it renders the complete effective descriptor, including contributed operations, descriptor-local named declarations, and recursive record, list, enum, and union shapes with required flags and declaration documentation. The tests `host_namespace_object_resolution_and_operation_contributions_are_independent`, `operation_contributions_reject_interface_and_target_conflicts`, and `native_projection_preserves_descriptor_and_decodes_typed_arguments_end_to_end` demonstrate root allocation, unique resolution, contribution merging, collision rejection, current call-time authorization, complete descriptor inspection, and descriptor-typed invocation. - -The built-in native route migration is: - -| Historical route | Canonical route | -| --- | --- | -| `/features/ticket/tickets[/]` | `/tickets[/]` | -| `/features/objective/objectives[/]` | `/objectives[/]` | -| `/features/merge-request/merge-requests[/]` | `/merge-requests[/]` | - -The built-in Repository/Workdir reference provider allocates `/repositories`, `/workdirs`, and `/workdir-attachments` using this contract; `manage-workdir` contributes operations rather than owning another Object. See [Repository, Workdir and attachment WIP projection](workdir-wip-projection.md) for identity encoding, reference/management permissions, attachment lifetime safety, and first-Workspace discovery. `/checkouts` content projection remains a separate provider. - -### Registering a subsequent provider - -1. Allocate `allocate_namespace("repository", "repositories")`. The returned `WipNamespaceRoute::root()` is `/repositories`; the owner is independent of any Feature registration name. -2. Mount a native collection `WipProjection` at that root with capability `repository:collection`, matching Object name and one declared interface. Static Objects underneath the root use the same capability owner prefix. A mounted collection retains its own identity, interfaces, and validator even when it has static children. -3. For dynamic items, register one `WipDynamicMount` on the collection with capability `repository:item`. A foreign capability owner is rejected. Its `WipDynamicItemResolver` owns reference acceptance, Object identity, and current Object validator. It returns only a bounded direct child; it must not reinterpret a reference as another namespace or grant permission from existence. -4. The reference provider supplies its read operations and handlers independently of management Feature enablement. An Object-only provider may use an empty operation descriptor. Enabled management Features register `WipOperationContribution` for static Objects or `WipDynamicOperationContribution` for a dynamic family; they do not allocate the namespace again or replace its resolver. -5. Contributions use the existing interface reference and identical format, documentation, and type declarations, supplying only their new operations. Interface references are exact (scope, name) keys: extending a descriptor that is also mounted elsewhere must not create different descriptors/validators under the same reference (registration fails). Use separate references for collection and item contracts. Failed contributions leave the registry unchanged. -6. Handlers are responsible for checking current target capability and current subject authorization before dispatch. Registration/discovery is metadata, not a grant. A resolver/handler must retain the original permission identity and domain audit/failure contracts. Revocation after discovery must fail before provider dispatch. Compatibility mounts are restricted to `/tools/` and cannot bypass namespace ownership or republish legacy routes. - -Tests `management_enablement_is_independent_of_read_and_operation_permissions` and `contributed_interface_invalidates_observations_and_restore_requires_discovery` cover all eight enablement/read/manage combinations, direct-call rejection, effective descriptor changes, fresh runtime isolation under the same Worker identity, reset, and post-observation revocation. - -## Authority and execution - -Discovery is not authority. Object publication, operation implementation, target capability, and subject permission are independent checks. An Object may remain discoverable through an enabled read provider when a management Feature contributes no operations; conversely, publishing an operation never bypasses the target's current capability or the subject's current permission. Every handler re-evaluates permission at call time, so revocation after discovery is rejected before provider dispatch. Feature enablement selects implementations and does not itself grant read or mutation authority. - -The Worldspace contains only the operations registered from enabled Features and the Worker's current execution context. Calling a compatibility operation: - -1. resolves the exact mounted route; -2. verifies object and interface membership and validators; -3. validates the WIP descriptor call; -4. validates the original JSON Schema; -5. evaluates manifest tool permission rules against the **original tool name and original arguments**; -6. invokes the original async `Tool` without blocking the Tokio runtime. - -Filesystem scope, provider Workdir capability checks, Backend Workspace authority, Feature enablement, and input-specific restrictions stay in the original tool implementation and are checked again at invocation. `ask` permissions remain denied fail-closed because the runtime has no approval protocol. The gateway does not mint authority or expose authentication material. - -Pre/post Engine history still records one bounded `Invoke` result, while the compatibility result retains the original `ToolOutput`, including attachments and normal Engine output pruning. WIP call audit records retain the route, operation, request identity, and terminal classification. - -## Client lifetime, restoration, and compaction - -The stateful `wip-client::Client` is owned by one Worker WIP runtime. Sessions are keyed by the canonical endpoint and an opaque security-context identity derived by the Host from Workspace, Worker, and session identity. Authentication credentials are not included in that identity and are never emitted to descriptors, tool output, or history. - -Known Space and interface observations are therefore not shared across Workers, endpoints, or security contexts. A restored Worker creates a new runtime/client and must explore again. `refresh = true` explicitly supersedes requested retrievals. Old string references in saved history remain historical text, not cache/wire authority; observations are runtime-only and are not persisted or restored. Resume by inspecting the target path, not by migrating an opaque display string. A normal compaction keeps the live runtime but the three gateway schemas are supplied again on every LLM request, including direct Inspect and indexable Tree semantics; the compacted transcript is not treated as cache authority. - -Saved Session/history entries remain append-only evidence and are never rewritten or replayed as calls. In particular, historical `/features//...` text stays displayable but is not a current route alias. Passing such a legacy path to discovery or call returns `NotFound` with guidance to rediscover from `/`; the Host does not translate it, resolve it under another namespace, or expose old and new routes in parallel. - -## Errors, cancellation, and retries - -The adapter keeps these outcomes distinct: - -- **rejected/not dispatched**: route, membership, validator, descriptor, schema, or permission failure; the original operation does not run; -- **stale**: object/interface validator mismatch; refresh before reconstructing a call; -- **cancelled**: the original provider confirms cooperative cancellation and its bounded cancelled output is retained; -- **interrupted**: provider interruption is retained separately from cancellation; -- **disconnected/timeout after dispatch**: `wip-client` records an unknown outcome; -- **operation outcome unknown**: an executing provider fails without a rollback/non-execution guarantee, a response cannot be completed after dispatch, or the call future is force-closed. - -Cooperative cancellation is scoped to the active execution's selected operation and exact resolved handler instance. Static/dynamic dispatchers forward `cancel_operation` only to that operation's provider, once per cancellation request, never to unrelated contributors or every alias backed by a shared handler. The runtime pins the dispatcher used for execution rather than resolving the Object again on cancellation, since dynamic resolvers may return stateful instances. Providers may implement `cancel_operation` for operation-specific control or the shared `cancel` default. Terminal completion or a dropped execution future releases the active binding; unknown/finished execution IDs do not invoke any provider. A cancellation request is not itself terminal confirmation. - -An unknown mutating outcome must never be retried automatically. The model must inspect external state or ask for operator guidance. Retrieval failures may be explicitly refreshed. - -## Measurement - -The runtime tracks bounded counters for: - -- ordinary individual-tool schema bytes hidden by WIP mode; -- the three WIP gateway schema bytes; -- discover, descriptor-inspection, and operation round trips; -- successful operations. - -These values are returned with discovery/inspection output and are intended for comparing prompt input volume, exploration round trips, and representative operation success with ordinary Tool mode. No fixed reduction percentage is promised. - -## Scoped publication and display boundaries - -References are structured everywhere; wire requests reject legacy/display strings. Root and self scopes are valid; ancestry uses canonical path segments, not string prefixes. Contextual descriptors use the resolved Object path as their explicit self scope, not an encoded suffix on the local name. Fetching a self-scoped Interface freezes descriptor/validator and optional scope Object ref from the same live projection. A scope without a ref is valid; the adapter does not invent one. Calls check target existence, Object preconditions, scope/membership/ref, then Interface preconditions and arguments, with the selected descriptor and handler pinned for execution/result validation. Existing providers repeat current authorization and their captured Workdir/Workspace metadata checks at their execution/commit boundary. The registered artifacts alone are not call authority. - -Separate Interface namespaces on one static Object can be registered with `mount_interface`; the owner/identity cannot be replaced and duplicate Interface keys fail. Inspect preserves each Interface reference, even with identical operation names, and Invoke selects the exact namespace. Contribution merging retains one Interface and disjoint operation names. - -Official signature quoting, escapes and summary layout are preserved verbatim. Host documentation is untrusted data in Tool output, not instructions. RenderError categories and the failed path/reference are reported as display failures. Library input limits and Yoi envelope limits reject the entire display rather than truncating it or reporting no operations. No signature parser, shorthand-input parser, independent lexer, or custom escape engine is installed. - -Typed Entry results remain values; no eager result acquisition, guessed string-path conversion, or inferred indexable edges occur. A later Inspect failure never reclassifies a completed successful operation. FS indexable/list policy is a separate provider concern (T-714), not Client-side search/ranking. diff --git a/docs/design/workspace-config-wip-projection.md b/docs/design/workspace-config-wip-projection.md deleted file mode 100644 index febeeefd4..000000000 --- a/docs/design/workspace-config-wip-projection.md +++ /dev/null @@ -1,252 +0,0 @@ -# Workspace configuration as a WIP logical filesystem - -## Authority and source mapping - -Workspace configuration is not the Backend host's configuration directory. The -canonical authored source is already a SQLite-backed, content-addressed virtual tree: -`workspace_config_trees`, `workspace_config_entries`, and append-only -`workspace_config_tree_history`. `WorkspaceConfigState` combines its -`ConfigTreeSnapshot`, toolchain/schema contract and projection digest. The -`main.dcdl` entrypoint and its imports are authored files; evaluated profiles, -prompts and other projections are not replacement source files. - -The builtin Workspace schema composes profile, prompts, runtime, -repository-access and Skills settings. Profile sources include model settings. -Do not assume that all imported Profile sources live under `profiles/`: -`VirtualPath` identifies authored entries, including shared imports. Skill -Markdown and text resources are entries in the same virtual tree. - -The WIP provider must resolve this authority at request time, not copy it to an -OS directory, maintain a synchronized config cache/store, or register every -entry as a permanent Object. A logical attachment selects a Workspace-bound -configuration subject; it is not a Bash cwd or a host filesystem grant. - -## Existing save and activation contract - -The browser source-tree editor uses `ConfigCommitRequest`: a `base_digest` of the complete source tree, changes with expected entry digests, and the `main.dcdl` -entrypoint. Candidate evaluation applies changes, formats/normalizes Decodal, -and evaluates the composed schema. Commit rechecks both the tree digest and the -base toolchain/projection identity in an SQLite Immediate transaction before -replacing active entries. Source content remains immutable and digest-addressed; -evaluation provenance is retained by source, toolchain and projection identity, -so reevaluating unchanged source does not erase earlier schema/compiler evidence. -A candidate prepared before a schema-only refresh cannot reactivate its old -contract. Worker operations must share that conflict domain with -UI and other Worker edits; a per-Worker mutex alone would not prevent lost -updates. - -The Backend commit orchestration validates the prompt projection and -repository-access references, then commits and best-effort publishes the prompt -projection to the embedded Runtime. Saving establishes new authoritative -Workspace configuration, **not universal live application to existing Workers**. -New Worker configuration bundles are digest-bound. In-flight consumers can -retain immutable projections. Skill availability and activation are separate -contracts. A transport failure after dispatch is not proof of rollback and is -not authorization to retry a mutation automatically. - -## Separate authorities and excluded objects - -The following are not files of the authored Workspace source tree: - -- Backend host `server.toml`, OS configuration paths and materializer state. -- Workspace identity, signing material, runtime trust/registration, - credential stores, authentication/session/passkey/token data and private keys. -- Repository registrations and Flow records, which use separate typed APIs. -- Workspace display name and Memory language, which have independent CAS - domains; they must not be presented as one atomic source-tree transaction. -- Worker retention/lifecycle policy and destructive administrative operations. - -Repository-access **references** in authored config are not credential private -material. Granting config access does not grant secret-management operations, -filesystem access, command execution, or arbitrary Workspace access. Authored -text can itself contain sensitive information; configuration access must be an -explicit operator grant, not a consequence of possessing Read/Write Tools. -Existing secret stores must never be serialized into observation, operation -results, diagnostic bodies or history through this provider. - -## Enablement and operator grants - -Choose Worker WIP mode and the Feature implementation independently of authority: - -```toml -[worker] -mode = "wip" - -[feature.workspace_config] -enabled = true -``` - -The Feature defaults disabled; enabling it does not add a settings grant. An -operator authenticated as the Workspace owner grants access to a specific -registered Runtime/Worker through `workspace_config_grant_create`: - -```text -POST /api/w//workspace-config-grants -{"runtime_id":"","worker_id":"","access":"read_write"} -``` - -Use `read_only` for observation/read-only access. Grant creation is a separate -owner administration action, not an approval prompt for each edit. To revoke, -use the owner operation `workspace_config_grant_revoke` on the returned grant -ID. Worker requests carry authenticated source identity, never these IDs as a -substitute for authorization. A grant from another Workspace or another Worker -cannot be adopted by knowing a route or possessing a previous Interface. - -Explicit grants do not replace the UI editor's user authentication. Authored -source-tree read/entry/history/commit routes and the UI Profile settings catalog -reject Runtime/Worker source proofs in both single-Workspace and server-scoped -middleware, including workers with a read-write grant. `workspace.request` is -not a configuration-editor capability: granted Workers use the WIP adapter, -never the UI editor as an alternate access path. User/session/API-token editing -continues to share the canonical validation and CAS boundary with WIP saves. - -The audit found no separate Profile/Prompt/Runtime/Skill source-save endpoint; -authored writes converge on source-tree commit. Existing runtime consumption of -Prompt projections, Skill catalog/activation and resolved config bundles remains -a separate read-only/activation contract needed for Worker startup and normal -operation. Those evaluated projections are not authored source-tree bodies, -content history or a second editor. Do not fence all `/config` routes as one -capability: that would break the existing Prompt reader's runtime contract. - -The existing connection ledger remains authoritative. Each granted Worker has -an independently identified logical registry target over the **same** authored -configuration tree. This preserves the ordinary Workdir exclusive-owner -invariant without making separately granted Workers maintain separate config -stores. Every attachment has an opaque connection lifetime ID; alias reuse is -not identity reuse. No Runtime filesystem session is created for this target. -Detaching ends that connection's use but deletes neither configuration nor its -registry target. Configuration is not a Workdir-cleanup or command target. - -## User invocation and Worker access - -Select `/workspace-config` using the common Feature completion menu. Complete -its declared parenthesized argument form (close `)` to use the default access), -then send the typed chip with the request, for example: - -```text -/workspace-config() このWorkspaceのモデル設定を変更して -``` - -The composer preserves the selection as a typed invocation chip. Ordinary text -that happens to contain this spelling is not a command. Feature preparation -calls Backend attach before dependent LLM input and displays the selected -logical subject and effective access. Failure/unknown preparation must not be -reported as a successful attachment. Optional declarative `access` values are -`effective` (default), `read_only`, and `read_write`. They restrict existing -rights; explicit `read_write` fails rather than silently downgrading. A Worker -can invoke the native attach operation at `/workspace-config` through the same -Feature/Backend handler. Neither path creates a grant or a second ledger. - -The logical content root is `/workspace-config`, not `/checkouts` or a host -path. Observe exposes requested structure and available Interfaces; contents -are retrieved through Operations. Implicit directories reflect current -`VirtualPath` entries and are not independently stored directories. - -## Backend transport and consistency - -Worker operations use identity-bound routes under -`/api/w//workers/self/workspace-config`: - -| Interaction | Route | Contract | -| --- | --- | --- | -| Resolve connection | GET root | Current grant checked even if authorized but unattached | -| Attach | POST root | Idempotent reuse of the active canonical `workspace-config` alias | -| Observe | POST `observe` | Requested paths/depth; metadata only, no source bodies | -| Read | POST `read` | Connection + path validator; canonical body/type/digest | -| Save | POST `commit` | Connection + root validator + canonical config changes/CAS | - -Observe returns root digest and path validators from the same snapshot. -Opaque validators bind Workspace, Worker, grant, connection lifetime, logical -path, whole-tree digest and active/applied evaluation contract. Identical source -paths, content types and bytes retain the same digest; saving them again under -the same contract does not create an edit counter or invalidate a content -observation by itself. A schema/toolchain refresh invalidates observations even -when source bytes are unchanged. Whole-tree CAS is -intentionally conservative: an unrelated UI edit can require fresh observation. -The Client holds validators; the LLM must not manage or fabricate them. - -Authorization, active connection lookup and operation use share the per-Worker -session lock with detach/revoke. The canonical Immediate-transaction CAS remains -the final cross-Worker/UI commit boundary. Failed validation, stale state and -permission rejection do not overwrite configuration. Potential auxiliary -repository-secret effects and uncertain persistence are not described as a -full rollback. Sanitized failures distinguish `not_committed` and `unknown`; -a timeout/disconnect after dispatch never causes automatic mutation retry. - -Structural observation is bounded to depth 0–8 and 256 returned nodes, paths to -512 bytes and each text entry to 256 KiB. Oversized observations fail rather -than presenting a truncated result as complete. Source-tree creation/deletion -is expressed through canonical changes, including atomic multi-file changes -needed to update a source together with its import/registry references. The -`main.dcdl` entrypoint cannot be deleted/renamed. These are configuration -operations, not a generic POSIX filesystem or an OS mount. - -## Native operation mapping - -| WIP target | Operations and model arguments | Canonical action | -| --- | --- | --- | -| Root | `attach(access?)` | Same Backend connection operation as the selected slash invocation | -| Root | `apply_changes(changes)` | One atomic create/update/delete/rename batch; source and import/registry edits can be combined | -| Directory | `create(path, content, content_type?)` | Create an entry below this directory; ancestor directories are implicit | -| File | `read()` | Return canonical text and content type, not a byte-stream or OS descriptor | -| File | `write(content)` | Update this exact entry, preserving its content type | -| File | `edit(old_string, new_string, replace_all?)` | Unique string replacement unless `replace_all` is selected | -| File | `delete()` | Canonical deletion subject to entrypoint/import validation | -| Explicit missing path | `create(content, content_type?)` | Create that exact absence-observed entry, not a substituted path | - -Content type is `decodal` (default for creation) or `text`. File operation -targets come from the resolved WIP route. The root change batch uses canonical -relative source paths; it does **not** accept `expected_digest`, tree digest, -validator, Workspace, grant or connection overrides as model arguments. The -adapter captures metadata and fills those preconditions automatically, requiring -all paths in a batch to match one observed tree state. Unsupported/invalid -configuration mutations fail instead of performing a partial multi-file save. - -Example `apply_changes` argument: - -```json -{"changes":[{"kind":"create","path":"notes/readme.md","content_type":"text","content":"Configuration notes"}]} -``` - -These are native Interfaces only. There is no configuration-specific -compatibility Tool or fallback under `/tools`, no Bash location, and no -`wip-fs-server`/Linux real-filesystem provider substituted for these operations. - -## Validation scope - -Start with the changed contracts: - -```sh -cargo test -p manifest --lib workspace_config -cargo test -p worker --lib workspace_config -cargo test -p worker --lib manage_workdir::wip -cargo test -p yoi-workspace-server --lib workspace_config_integration -cargo test -p yoi-workspace-server --lib config_source::tests -cargo test -p server-api --lib workspace_config -cargo test -p tui --lib backend_worker_picker -``` - -The Backend integration tests use production Feature completion selection, -structured `Worker::run` input, request preparation and the real Backend router -and SQLite configuration store. At the recording LLM-client boundary they -assert that attach and its durable context happened first, then use the actual -WIP Client/Host to read/edit/write/create/delete and atomically change a source -with its import. The existing Prompt reader confirms the saved projection. -Signed-route, owner-grant, readonly/revoke, detach/lifetime, multi-Worker/UI CAS, -same-content replacement, invalid configuration, storage-failure and dropped -post-save-response cases complement that sequence. - -Web tests exercise the production generic composer component's declaration -completion, selection and typed submission, plus removal/literal-text behavior. -Logical-source parsers and launch/sidebar/TUI tests ensure that a settings -attachment is not treated as a repository commit or process cwd. The two -layers are complementary; the Rust fixture uses a recording LLM client and -in-process transport, not a live browser-to-provider deployment. No live -Backend/Runtime/LLM browser or physical power-loss result is claimed. Bounds -are finite input/tree and lock/request deadlines, not a promise that synchronous -configuration evaluation is forcibly preemptible at a hard wall-clock limit. - -Completion validation also runs the current-source root `cargo check`, complete -changed-crate suites, the server-api TypeScript configuration and generated -contract checks, formatting and diff checks. Updating this code does not update -or restart the running dogfooding environment. diff --git a/docs/design/workspace-drive-web.md b/docs/design/workspace-drive-web.md deleted file mode 100644 index 76eede51c..000000000 --- a/docs/design/workspace-drive-web.md +++ /dev/null @@ -1,37 +0,0 @@ -# Workspace Web Drive - -The Web Drive is a latest-only shared Workspace resource, independent of Git, repositories, Worker sessions and Workdir occupancy. The Server's DB node tree, last committed request, receipt and current authorization are authoritative. The UI never uses physical blob keys/paths or stores a second persistent hierarchy. - -## References and authorization - -`/w//drive/` is a human-facing stable latest URL. Its node ID remains a canonical positive decimal **string** (including values above JavaScript's safe-integer range). Rename/move retain the URL; deleting and recreating a name does not revive the deleted ID. Copy latest URL emits this Web route, not the API's safe binary download URL. Ticket/Objective/comment Markdown can link to this route. Possessing any URL/ID grants no access: metadata, content, mutations, download and receipt queries all use the current authenticated API. - -Ordinary Drive usage retains the existing member-facing API policy, not owner-only settings policy. Worker grants are owner-controlled on the existing Workers page, using its current authoritative Worker catalog. Read-only/read-write are explicit Backend grants, not consequences of enabling a Profile tool. The UI does not invent a Web read-only ACL: Backend denial is displayed and suppresses mutations. Worker read-only denial and revoke are separately exercised against the real API. - -## Drafts, observed changes and delayed operations - -The optional node route reuses one controller across Workspace/entry switches. Drafts and mutation receipts are keyed by Workspace/node identity in this page session. Reads/writes/status/list/search/ancestor/image completion are generation fenced, including A→B→A. No delayed response can select a new target or change its draft/save destination. Leaving the Drive page prompts before destroying a dirty draft or unresolved receipt; they are not claimed to survive a browser reload. - -Text edits retain the `last_mutation_id` returned with the read and send it as `expected_mutation_id`. A conflict preserves the draft and that original observation, displays an explicit conflict and never silently rebases, automerges or resubmits. Refresh can show the latest saved text alongside the draft. Discard/load-latest is an explicit destructive user decision. Partial previews cannot be saved. Save acknowledgments retain edits made while saving. - -## Transfer and media policy - -- Markdown/plain text: 64 KiB UTF-8 read/edit limit. Markdown uses the existing raw HTML-rejecting safe document renderer. Other active/binary media are download-only. -- Raster image preview: PNG/JPEG/GIF/WebP, authenticated committed-request-bound fetch, 256 KiB bounded bytes/Blob; decode/load failure is explicit. Object URLs are released on switching/destruction. Larger images remain downloadable. -- Download: native same-origin authenticated binary endpoint, bound to the observed committed request; Server's attachment/nosniff/sandbox/no-store headers remain authoritative. -- Markdown inline images never automatically fetch external/cross-Workspace/API resources. They display alternative text and require explicit navigation to an authorized Drive URL. Same-Workspace resource links navigate normally; external HTTP(S) links have no userinfo, open a new tab with `noreferrer`; script/data/file, protocol-relative, arbitrary API/relative paths are not links. -- Upload/replacement: at most 16 MiB raw Blob request, not JSON/base64 or DOM bytes. SHA-256 needs one bounded ArrayBuffer; there is no constant-memory browser hashing claim. Transfer/pending is distinct from the DB-published acknowledgment. -- A request ID is allocated before a mutation. Transport/response loss or abort after transmission is conservatively unknown, not success/failure. Its current authorized receipt endpoint is queried; no new request or automatic reupload is generated. `uncommitted` means no commit observed yet, not proof that an in-flight publication cannot still finish. Unknown requests block new writes to that entry. - -Name conflict, changed-content conflict, limit, denied, missing node, storage/I/O failure and outcome unknown are not translated into empty contents/empty Drive/success. Initial folder deletion is empty-only with explicit confirmation; no recursive operation or blob-copy rename is introduced. - -## Validation - -Browser API fixtures and real authority evidence are intentionally separate: - -- `web/workspace/src/lib/workspace/drive/*test.ts`: malformed payload/ID and transport/state races at the client/controller boundary. -- `drive-ui`/`drive-grants` component tests: actual safe rendering/interaction. -- `web/workspace/test/drive-browser`: production static shell + synthetic API fixtures, pinned Chromium interactions and visual evidence. -- `crates/workspace-server/src/server/tests/drive_web_roundtrip_tests.rs` and `web/workspace/test/drive-real-api`: hermetic real Server HTTP, SQLite, FeatureStorage and LocalFileSystem, repositoryless bootstrap and production Web adapter. Only the Runtime process boundary is fake; no live dogfood data is used. - -See the test READMEs and T-724 validation record for exact commands/results, artifact paths, source commits and any deliberate failing fixture state. diff --git a/docs/design/workspace-drive.md b/docs/design/workspace-drive.md deleted file mode 100644 index c0edb4bc6..000000000 --- a/docs/design/workspace-drive.md +++ /dev/null @@ -1,391 +0,0 @@ -# Workspace Drive storage - -## Scope and ownership - -Drive is a Workspace-owned **latest-version document tree**, not a mounted or -POSIX-compatible filesystem. `workspace-drive` owns storage, hierarchy, conflicting -changes, mutation receipts, bounded reads/search, and garbage collection. -`WorkspaceApi::drive` is the trusted Server adapter. The authorized HTTP API and -Backend-managed grants are documented below. Worker Tools/WIP, Web UI, and full -end-to-end routing belong to T-723–725. -Session attachments remain Session-owned and are not Drive authority. Worker, -Session, and Workdir cleanup do not delete Drive. - -Only a single Server Host with persistent local disks is supported initially. -Multiple processes must use **the same metadata DB and blob root**. Separate -Hosts/independent DB copies may not write the same Drive. Workers on other Hosts -use the authorized API, not direct storage access. S3 deployment/migration, -POSIX/inode semantics, histories, mounts, and distributed quotas are not included. - -## Configuration and persistent volumes - -Server startup uses the existing resolved Yoi data directory. `ServerConfig` keeps -that resolved path in `data_root`; Drive code does not read environment variables -or introduce another default. At the default layout: - -``` -$YOI_DATA/drives//blobs/ -$YOI_DATA/server/server.db -$YOI_DATA/server/feature-storage//features/workspace-drive.sqlite -``` - -The FeatureStorage root is derived from the authority database path using the -existing Host convention; it is **not** placed below `drives`. Custom database -placements remain possible: configure the trusted `ServerConfig.data_root` -explicitly and persist both locations. Bind `$YOI_DATA` (and any custom database -root) to persistent volumes, not container writable layers, Worker checkouts, -or temporary directories. The storage identifier is the immutable internal -Workspace ID, never its display name or a user-supplied path. No blob key, -physical path, credentials, or SQL surface is included in node/read DTOs. Internal -storage errors contain diagnostics for Host logs; the authorized API adapter must -map them to sanitized failure responses, not expose raw error strings. - -The directories and every ancestor are Host-owned and must not be writable by -untrusted actors. Existing symlink components, symlink blobs, non-directory -ancestors, nested directories, and path traversal are rejected. This is not a -sandbox against a hostile local administrator replacing paths concurrently: -`LocalFileSystem` uses path-based operations, not descriptor-relative no-follow -operations. External edits of the store are outside the Drive operation contract. - -## Metadata and tree contract - -Migration 1 registers `workspace-drive` through FeatureStorage. Each scoped DB -contains its Workspace binding, nodes, deletion tombstone, and mutation receipts. -SQLite `AUTOINCREMENT` node IDs are stable, scoped to a Workspace and never -reused after deletion. They are opaque application IDs, not inodes. Node IDs serialize as canonical positive **decimal strings**, preserving signed -64-bit precision across JavaScript DTO boundaries. `last_mutation_id` is the -existing request ID of the last committed mutation, not a counter or timestamp. -The immutable root has an empty value. Restore preserves `sqlite_sequence`, -node IDs, request IDs and receipts. - -The DB is the only tree/latest-reference authority. Each node has one parent; -root is the sole parentless directory, has empty name, and cannot be renamed, -moved, deleted, or written. All other names are 1–255 UTF-8 bytes, compared -case-sensitively with SQLite binary equality. No Unicode normalization or case -folding is performed: canonically equivalent strings are distinct. Japanese is -supported. Control characters, `.`, `..`, slash and backslash are rejected. Names -never become blob paths. Same-parent names are protected by a unique DB index. -Parents must be directories in this scoped Workspace. Self/descendant moves are -rejected. Ordinary folder deletion is **empty-only**, never implicitly recursive. -Workspace destruction is the separate fenced purge operation. - -Every update/move/delete requires `expected_mutation_id`. The same IMMEDIATE -transaction compares it with the node’s last successful request ID before writing. -A different committed request or -same-name/request-identity competitor returns `Conflict`; it never falls back to -unconditional writes. Root mutations and malformed requests are `Invalid`; -missing/deleted IDs are `NotFound`. No update counter is allocated or incremented. -Actor identity is trusted caller input (the authorization adapter supplies it), -and successful mutations record actor and UTC epoch milliseconds. - -`BEGIN IMMEDIATE` is held for validation, blob operations, metadata mutation and -receipt insertion. It serializes the whole Workspace hierarchy and all blob -read/upload/GC activity across independent processes. A→B/B→A, parent deletion -versus child creation, and updates based on the same observation therefore have one valid winner. -FeatureStorage configures foreign keys, WAL, `synchronous=FULL`, and a 5-second -busy timeout. Busy/SQL/I/O failures remain errors; they do not mean an empty Drive. -Before opening SQLite or admitting a Feature handle, FeatureStorage creates and -syncs **every metadata directory ancestor link**, including configured path and -resolved target chains for trusted Host symlink paths. It re-syncs existing -directories on retry after any prior flush failure. Directory sync failure aborts -admission before DB initialization/migration; no usable connection is cached. -SQLite's DB/WAL and immediate-directory synchronization alone would not protect -new FeatureStorage/Workspace ancestors. Both this metadata link durability and the -separate blob link durability below are required for acknowledged Drive success. - -The Server uses `Drive::open_with_workspace_authority` to attach its trusted -`server.db` to the Feature DB connection. An IMMEDIATE transaction also locks that -attached authority, and checks that the Workspace remains `active`. Consequently -a cached Drive handle cannot mutate after the deletion reservation commits, even -before the separate Drive tombstone is persisted. Drive never writes the -attached authority; only its metadata DB is changed. This conservative initial -implementation also serializes against other Server DB writes during blob I/O. -There is no cross-DB atomic write promise or distributed transaction: only the -Drive DB is written. Long uploads/searches can increase Server write contention; -all calls are blocking and async callers must use `spawn_blocking`. - -## Blob publication and durability - -Content creates/updates choose a fresh canonical UUID blob key unrelated to names -or parents. `object_store` **0.13.2**, with the `fs` feature and without cloud -features, supplies `local::LocalFileSystem`. `PutMode::Create` forbids overwrites. -Rename/move change metadata only; no blob move/copy/stat is needed. List/metadata -queries are SQL-only and still work if a referenced blob is missing, whereas -content reads diagnose the missing reference. - -The pinned local implementation writes a `create_new` staging file, publishes -Create with `hard_link`, and best-effort removes staging. Inspection of 0.13.2's -local implementation found **no `with_fsync` option and no fsync** ([versioned -source](https://docs.rs/object_store/0.13.2/src/object_store/local.rs.html)). -The dependency is pinned to 0.13.2 because staging cleanup also depends on its -local filename protocol; reinspect publication/staging/durability before upgrading. -Atomic -publication is not power-loss durability. The small adapter compensates: - -1. Create/validate Host directories; `sync_all` their entries and parents. -2. Use LocalFileSystem to atomically create the immutable blob. -3. `sync_all` the published file, then its containing directory. -4. Only after both succeed, update the DB reference/last committed request and insert the - receipt in the **same SQL transaction**. -5. SQLite commit is the sole success/publication point. - -Delete also syncs the directory, including on a missing-file retry. Sync, disk -full, permissions, and other errors propagate; metadata is not committed if blob -persistence fails. A failed sync may leave an unreferenced blob for GC. If DB -commit fails, the old reference remains and the new blob is likewise reclaimable. -There is no external-store/DB two-phase commit or bespoke object-store engine. - -Supported durability target: **Linux, local persistent filesystem with working -file and directory fsync and SQLite WAL locking**. Filesystems/drivers that do not -support these calls fail rather than silently downgrade. Windows, network -filesystems, and other OS/storage combinations have not been qualified. Disk -controllers and devices must honor flushes. Unit tests verify ordering/failure -semantics with injected errors and actual isolated files/SQLite; they are **not a -power-cut experiment** and do not prove hardware power-loss behavior. - -## Request result reconciliation - -Every mutation includes a bounded request identity (1–128 bytes) and trusted -actor (1–256 bytes). A SHA-256 fingerprint covers the actor and typed mutation, -including content. Its committed result is stored with metadata in the same -transaction. Replaying the exact same request returns the recorded result; -reusing the ID with different input/actor is `Conflict`. Results are retained for -the Workspace lifetime, including results referring to later deleted nodes. -There is no clock-based deduplication window or partial FS-recovery engine. - -After response loss or uncertain SQL/I/O errors, query `request_status`: -`Committed` includes the original result; `Uncommitted` means no committed row in -that authority snapshot. It is **not proof that an in-flight concurrent request -can never commit**. Retry with the same identity/payload, never manufacture a new -ID to resolve an unknown outcome. Failed/uncommitted operations have no success -receipt. A committed result can still be queried/replayed after the Drive is -fenced; it does not authorize new mutation. - -## Bounds, pagination, reads and search - -Hard limits are exported constants, not additional environment variables: - -| Operation | Limit | -|---|---:| -| File create/update | 16 MiB (empty files supported) | -| Binary/text read chunk | 1 MiB, positive requested length | -| List page / GC page | 200 nodes / 200 blob keys + up to 200 staging files | -| Search work per page | 128 nodes | -| Text inspected per candidate | 64 KiB | -| Search query | 256 UTF-8 bytes | -| Name | 255 UTF-8 bytes | -| Content type | 128 ASCII bytes, media-type separator required | - -No offset writes or shared write handles exist; whole content is replaced. Size, -invalid bounds and invalid UTF-8 errors are explicit `Invalid`; storage/capacity -errors remain failures for the API to present. There is no quota reservation. -Request and node/receipt volume remain bounded by available Host disk capacity; -operators must monitor free space for both blobs and DB/WAL. Files/receipts can -consume space until GC/Workspace deletion; there is no distributed quota system. - -List uses ascending stable ID keyset pagination. Metadata is not a multi-call -snapshot: concurrent nodes may appear/disappear and relocation can change page -membership. Continuation is the last examined ID, not an offset or name. Search -uses the same non-snapshot ID semantics, examining at most its work budget per -page; an empty matching page can still have continuation. Name search is exact -substring matching. Optional bounded text search inspects only UTF-8 `text/*` -files no larger than 64 KiB, never images/large files. No index/embedding exists. - -Each read supplies node ID **and the observed committed mutation**, and is locked to that -blob for the entire chunk. The SQLite lock excludes upload/GC while reading. -After an update the previous observation expires: the next chunk returns `Conflict` (or -`NotFound` after deletion), not the new generation. No long-lived read lease or -full-history retention is promised. Binary ranges are byte-based and exact; -text reads require UTF-8 character-aligned offsets/ranges and reject split or -invalid characters. Missing referenced blobs and length inconsistencies are -errors, including missing zero-byte files, never successful empty content. - -## Garbage collection and restart - -GC takes the **same SQL IMMEDIATE lock** as readers/uploads. Thus there is no -in-flight upload while sweeping; every currently referenced key is checked in -metadata before deletion. A grace period or process-local mutex is not a safety -proof. After a process crash SQLite releases/rolls back the transaction; an -unfinished staging file or fully saved but uncommitted blob can be reclaimed on -a later sweep. Recognized 0.13.2 staging names are cleaned under that lock; foreign -files are not interpreted as blobs. Deleted/replaced blobs are reclaimed too. - -`collect` is explicit Host maintenance, not a background task tied to a Worker. -Repeat its cursor pages, then start future sweeps at `None` because new UUID keys -can sort before the preceding cursor. Local object listing necessarily traverses -the flat blob directory; response/deletion batches and candidate memory are -bounded, unlike SQL list which does no blob traversal. Holding the write lock for -local listing is a conservative safety/throughput tradeoff. A corrupt or redirected -storage layout fails closed. GC never deletes a key referenced by the DB. - -## Workspace deletion - -Deletion reservation in Server authority changes the Workspace to `deleting`. -The attached authority fence immediately rejects new mutations from cached Drive -handles. Deletion start also persists the Drive tombstone before responding or -scheduling cleanup; API restoration reapplies it for blocked/failed deletions. -Purge commits that fence first, removes only this Workspace's blobs/staging, -clears its nodes/receipts, then Host removes the blob directory and calls -FeatureStorage's scoped delete. The feature scope's admission fence prevents -cached tasks recreating DB files during finalization. Failures retain a retryable -Server deletion operation; repeating purge/fence/delete is safe. Other Workspace -roots are never traversed or removed. Root/tombstone remain in the domain DB until -the Host removes the scope; there is no normal client recursive folder deletion. - -## Consistent backup and restore (offline initial procedure) - -A FeatureStorage DB snapshot **alone is not a Drive backup**. No online -DB-plus-object snapshot is implemented. Use this administrative procedure: - -1. Stop request admission, **all** Server processes sharing the authority, - uploads/readers, maintenance/GC and deletion. Do not merely stop Workers. - Complete or reconcile in-flight requests before backup. -2. Checkpoint/close the metadata DB using SQLite/FeatureStorage shutdown or use - its SQLite backup API while all Drive writers and GC remain stopped. Never - copy an active DB and omit its WAL. Capture the existing FeatureStorage scoped - metadata tree (or its snapshot manifest + databases), Server authority and - other deployment-required secrets/config using their respective procedures. -3. Copy the **same stopped generation's blob tree** with the metadata. Include - all referenced immutable objects; copying unreferenced blobs/staging is safe - and later GC can reclaim them. Preserve the internal Workspace ID. Record - both roots, object_store/schema version and checksums in an operator manifest; - restrict backup permissions because it contains user documents. -4. Verify each file node's blob exists and length matches `size`; missing content - is a corrupt backup, not an empty file. Flush/sync the backup using the backup - destination's own durability facilities before treating it as complete. -5. Restore only with admission/writers/GC still stopped, into absent FeatureStorage - scope and Host blob roots. Use `ScopedFeatureStorage::restore` for its snapshot - format, or restore the entire stopped deployment consistently. Never restore - just `server.db` or create a new empty Drive DB over surviving blobs. -6. Restore blobs from the matching generation; restore internal IDs, committed request IDs, - receipts and `sqlite_sequence` unchanged. Validate references/lengths and - sanitized metadata/text/binary reads before reopening admission. Resume GC - only after validation. A deletion tombstone must not be cleared on restore. - -The hermetic backup test stops writers, snapshots actual FeatureStorage SQLite, -copies actual LocalFileSystem blobs, restores into absent roots, and checks stable -IDs/latest content/receipts and non-reused ID allocation. It does not validate a -production backup scheduler, volume snapshot, or physical power cut. - -## Authorized typed HTTP API - -The Rust `server-api` contract owns `/api/w/{workspace_id}/drive/` operations: - -- `GET root`, `metadata`, `list`, `search`, `read-text`, `read-chunk`, `download`; -- `POST mutate` (create folder/bounded UTF-8 text, replace text, relocate, delete); -- `PUT upload` (binary body with flat query metadata); -- `GET requests/{request_id}` for the committed receipt snapshot; -- owner-controlled `POST/GET grants` and `DELETE grants/{grant_id}`. - -Ordinary usage follows the existing member-facing authenticated Server-user -policy. Non-ownership alone does not deny use. Grant management follows the -existing Workspace-owner account boundary. No new general membership ACL is -introduced. Browser/API-token identity is supplied by authentication middleware; -user/account existence and active Workspace state are checked again in the -operation transaction. Cookie mutations require the configured application -Origin through the existing middleware. No ordinary DTO accepts an actor. - -Runtime-forwarded requests require the existing signed source proof and live -Runtime/Worker catalog membership. Workers receive an explicit whole-Drive -`read_only` or `read_write` grant, independently of Profile, Subject, Ticket, -Workdir or commands. Incomplete/mismatched Runtime or Worker identity cannot fall -back to a browser actor. Grant creation rechecks owner, active Workspace, and -existing live Worker in the Server DB transaction; active access changes require -revoking the previous grant first (no silent downgrade). Migration 84 stores -creation/revocation actor and time. Removal fences and catalog deletion revoke -access and retain audit; grant rows cascade with Workspace deletion. No alternate -Worker registry or Drive Workdir attachment is created. - -A clone-bound Drive authorizer queries the attached Server authority under every -client-operation `BEGIN IMMEDIATE`, including receipt replay/result queries. -The lock spans validation, blob persistence and DB publication. Grant revocation -and Worker removal use the same Server DB write authority: whichever transaction -wins first is ordered first. A mutation committing before revocation is valid; -a revoked cached handle cannot commit later. Upload admission also checks access -before consuming the body, but that admission is not a lease: publication always -rechecks current access. Download checks authorization separately for every -chunk, and stops on revocation, deletion, a changed committed request or storage failure. - -References contain **both** Workspace ID and stable decimal node ID. JSON uses -`DriveEntryRef`; flat GET/binary queries require `entry_workspace_id` matching the -route Workspace before resolving `id`/`parent_id`. Node IDs are canonical -positive decimal strings within signed 64-bit range. Mutation IDs are request -strings (1–128 UTF-8 bytes, no control characters); file sizes/offsets are -bounded `u32`. List/search cursors carry Workspace identity and the last node ID; -they are continuation hints, not capabilities or snapshots. Returned `latest_url` -is a relative authenticated URL bound to Workspace and ID, independent of logical -name/parent. It confers neither public sharing nor approval. File URLs download; -folder URLs return metadata. Deletion/recreation never reuses the old reference. - -Web upload consumes binary HTTP frames with an observed **16 MiB** ceiling and -checks declared size and lowercase SHA-256 before publication. Oversized announced -or received data, mismatches, aborted bodies and I/O errors never return success. -The storage boundary currently requires a whole-file Vec, so uploads retain at -most one bounded binary file in memory before calling T-721; they are not -constant-memory object-store multipart uploads and are never expanded to -JSON/base64. Runtime source-proof verification separately buffers a bounded body -before granting trusted ingress, as in other signed Server operations. Web -downloads stream **64 KiB** committed-request-bound chunks; they do not materialize the -whole file. Generated Rust clients retain their explicit bounded-response policy -and return exact `BinaryBody` bytes. Worker JSON-facing adapters can request -bounded text or binary chunks through this same contract, but Tools/WIP and -model-context/image presentation belong to T-723; this API does not emit large -base64 JSON blobs. - -Downloads validate bare MIME types and always use `Content-Disposition: -attachment` with percent-encoded UTF-8 names, `nosniff`, a sandbox/default-none -CSP, `private, no-store`, and a SHA-256 ETag over Workspace, node ID and committed request ID. Arbitrary HTML/SVG -bytes are storable, not safe app-origin inline previews. Content length is exact; -if a later chunk loses authority or expires, the response stream fails rather -than returning another generation or a successful truncated file. Even an empty -file validates its referenced blob before HTTP success. Conditional 304/range -HTTP semantics are not added; observation-bound chunk reads are explicit. - -Errors use fixed, path-free typed codes: `denied`, `not_found`, `conflict`, -`invalid`, `limit`, `storage_unavailable`, `outcome_unknown`. SQL/commit/task -uncertainty at a mutation boundary is `outcome_unknown`/`unknown`; known rejection -or blob persistence failure is `not_committed`. A successful DB commit alone -returns a mutation response. After a lost response, query the same request ID; -`uncommitted` only means no committed receipt in that snapshot and must not be -interpreted as proof that an in-flight request can never commit. Exact replay uses -the same identity/payload/transport-bound actor and current permissions. Never -blindly create a fresh request ID to resolve an unknown outcome. Physical paths, -blob keys and raw provider errors are not public. - -Contract regeneration/freshness: - -```sh -cargo run -q -p server-api --example export_openapi -- openapi/server-api.json -cargo run -q -p server-api --example export_openapi -- --check openapi/server-api.json -cargo run -q -p server-api --features typescript --example generate_drive_api_types > web/workspace/src/lib/generated/drive-api.ts -deno fmt web/workspace/src/lib/generated/drive-api.ts -cargo test -p server-api --features typescript generated_drive_api_contract_is_current -# Existing OpenAPI-derived projections embed the whole-contract digest: -for example in generate_repository_openapi_types generate_runtime_api_types generate_ticket_api_types generate_companion_api_types generate_worker_launch_api_types; do - cargo run -q -p server-api --example "$example" - cargo run -q -p server-api --example "$example" -- --check -done -``` - -The existing BinaryBody contract also supports binary **success responses**, -including declared response headers, without changing binary request bytes or -JSON/error behavior. It exports inline OpenAPI binary bodies, not JSON byte arrays. - -## Upgrade to receipt-bound mutation checks - -Migration 2 obtains each non-root node’s latest request ID from its existing -committed receipt, transforms historical receipt payloads, and drops the old -update counter. The current-state receipt must be unique; missing or ambiguous -evidence aborts and rolls back the entire migration. Equal timestamps do not -change the result. No content, node identity, request fingerprint or blob is -regenerated. Deleted nodes’ receipts and AUTOINCREMENT history remain intact. - -The frozen migration module retains old SQL/JSON names and proven predecessor -correspondence only to verify existing receipts against their original request -fingerprints. This evidence never authorizes a new mutation. Active node CAS and -HTTP contracts do not accept the old numeric precondition. Exact pre-upgrade -Update/Relocate/Delete replays are accepted only when the complete actor/payload -and mapped predecessor reproduce the original fingerprint. If that evidence -cannot be established, recover the committed result through `request_status`; -changing the actor, payload or precondition and reusing its ID is a conflict, -not permission to execute it again. diff --git a/docs/design/workspace-kanban-orchestrator-runtime.md b/docs/design/workspace-kanban-orchestrator-runtime.md deleted file mode 100644 index 80ee7a4fc..000000000 --- a/docs/design/workspace-kanban-orchestrator-runtime.md +++ /dev/null @@ -1,285 +0,0 @@ -# Workspace Kanban to backend Orchestrator runtime - -Workspace Kanban operations are control-plane requests. They may change Ticket state and request orchestration, but they must not directly execute shell, git, filesystem work, or send authority-bearing messages to raw local Worker sockets. The durable boundary is an orchestration event consumed by a backend-internal Orchestrator Worker. - -This document records the design boundary for connecting Kanban operations, Tickets, the Workspace backend, `WorkerRuntimeRegistry`, and filesystem-capable Workers. It is intentionally a planning artifact: it does not require the Workspace backend to implement every table, API, remote protocol, or spawn adapter immediately. - -## Core rule - -A browser click or API request can create durable intent; it cannot be the authority to perform implementation side effects. - -The minimum chain for implementation work is: - -1. A user or authorized caller changes a Ticket through the Workspace/Ticket API. -2. The state change and its orchestration event are committed durably in the same logical operation. -3. A backend-internal Orchestrator Worker reads the event with domain-specific tools. -4. The Orchestrator records a routing decision. -5. If the Ticket is queued, unblocked, and accepted for implementation, the Orchestrator records `queued -> inprogress` plus the acceptance decision before any implementation side effect. -6. The Orchestrator creates typed spawn intents for filesystem-capable Coder, Reviewer, or helper Workers. -7. A local or remote runtime adapter resolves those intents into concrete worker launch/configuration using its own authority and capability policy. - -`ready -> queued` is therefore a human gate for Orchestrator routing, not an unattended scheduler and not a lease that automatically starts code execution. - -## Durable orchestration events - -Orchestration events are immutable control-plane records derived from Ticket operations. They are not raw LLM messages, Worker notifications, or socket writes. - -Initial event kinds: - -- `ticket_queued`: emitted for `ready -> queued`; requests Orchestrator routing/start-if-unblocked checks. -- `ticket_state_changed`: emitted for other lifecycle transitions that may affect orchestration state. -- `ticket_returned_to_planning`: emitted when a Ticket is moved back to `planning` because concrete requirements, decisions, dependencies, or acceptance evidence are missing. -- `ticket_done`: emitted when implementation/review flow records `done`; used for completion projection and close-readiness, not implicit close. - -A stable event shape should include: - -```text -orchestration_event { - event_id: opaque id, # durable event identity - workspace_id: opaque workspace id, - ticket_id: canonical Ticket id, - kind: ticket_queued | ticket_state_changed | ticket_returned_to_planning | ticket_done, - before_state: optional Ticket state, - after_state: optional Ticket state, - actor: { kind, key, display, source? }, # human, worker, api client, system - source: { kind, surface, operation }, # kanban_api, ticket_api, orchestrator, import, ... - request_id: opaque idempotency/correlation id, - caused_by_event_id: optional event id, - occurred_at: timestamp from the authoritative mutation, - recorded_at: timestamp when stored, - body: bounded structured reason/summary, # no raw transcript or raw local path authority -} -``` - -`request_id` is a correlation and idempotency key. For API-originated state changes, retrying the same request must return or reference the already-created Ticket transition and orchestration event instead of creating a second routing command. The event store should reject duplicate `(source, request_id)` pairs for mutation-producing requests, while allowing new events for distinct state changes. - -The Ticket state mutation and event append must be atomic from the backend's perspective. If the Ticket changes but the event cannot be recorded, routing must be considered failed and visible; if an event exists, the Orchestrator must be able to recover it after backend restart. - -## Event processing, retry, ack, defer, fail - -Event delivery state is separate from the immutable event. A future implementation can store per-consumer delivery rows such as `(event_id, consumer_id, status, attempts, visible_after, last_error, updated_at)`. That delivery mechanism is only for reliable control-plane processing; it must not redefine `queued` as a scheduler state. - -The backend-internal Orchestrator handles each event idempotently: - -- Reload the current Ticket, relations, orchestration plan, worker links, and relevant runtime summaries before deciding. -- Treat stale events as evidence, not commands. For example, a `ticket_queued` event for a Ticket that is now `planning` should be acknowledged with a stale/no-op decision rather than spawning work. -- Record a decision, waiting reason, or failure summary before acknowledging a routing event. -- Create spawn intents with stable `intent_id`s so dispatch retry can detect already-created intents. - -Delivery outcomes: - -- `ack`: the event has been interpreted and its durable outcome is recorded. The outcome may be `no_op_stale`, `routed_to_worker`, `blocked_waiting`, `returned_to_planning`, or `completion_projected`. -- `defer`: the event is valid but cannot progress now. The Orchestrator records a waiting reason and optional retry condition/backoff. Defer is appropriate for dependency not done, target conflict, dirty workspace reported by a capable runtime, missing worker capacity, or runtime unavailable. -- `fail`: processing cannot safely continue without operator or developer intervention. Fail is appropriate for invariant violations, malformed event payloads, missing Ticket authority, identity ambiguity, repeated dispatch inconsistency, or a request that would require forbidden authority. - -Retries must be bounded and idempotent. A retry may re-read the Ticket and registry and either dispatch a previously recorded intent, update a waiting reason, or fail with escalation. It must not execute shell/git/filesystem work in the backend process to "check" whether progress is possible. - -## Backend-internal Orchestrator Worker - -The backend-internal Orchestrator is a routing/control-plane Worker. It can be hosted inside the Workspace backend runtime because its tools operate on domain records and runtime registry abstractions rather than the workspace filesystem. - -Responsibilities: - -- Read durable orchestration events and delivery state. -- Inspect Tickets, Ticket relations, accepted orchestration plans, worker links, and bounded project-record projections. -- Decide whether a queued Ticket is ready for implementation, should wait, should return to planning, or should request review/closure follow-up. -- Record decisions, audit summaries, blocker/waiting reasons, and orchestration plan artifacts. -- Select a runtime by required capabilities. -- Create and dispatch typed spawn intents through `WorkerRuntimeRegistry`. -- Append/read worker run overviews, lifecycle summaries, and usage aggregates. -- Escalate when authority, requirements, or runtime capabilities are insufficient. - -Non-responsibilities: - -- No `Bash` authority. -- No raw workspace `Read`/`Write`/`Edit` authority. -- No direct git/worktree/build execution. -- No raw local Worker socket or session path authority. -- No use of browser-supplied local paths, executable paths, runtime registry paths, `display_ref`, `worker_name`, or runtime display names as operation authority. -- No raw session transcript full ingest into the Workspace database. -- No permission/auth, remote runtime protocol, Ticket DB migration, Kanban UI completion, or Coder/Reviewer spawn implementation completion in this design step. - -If routing needs evidence that only filesystem access can provide, the Orchestrator records a helper spawn intent for a filesystem-capable runtime or records a waiting/escalation reason. It does not temporarily grant itself filesystem tools. - -## Domain-specific tool surface - -The internal Orchestrator should receive backend tools, not generic Worker tools. The tools should be narrow enough to enforce lifecycle and authority rules and broad enough to let future Orchestrator prompts reason without hidden context injection. - -Required operation groups: - -- Ticket operations: - - list Tickets by state/risk/assignment bounds; - - show Ticket details, thread summaries, and relevant artifacts; - - append Ticket comments/implementation reports/review notes; - - perform validated state transitions, including `queued -> inprogress`, return-to-planning, and `done` recording. -- Relation and plan operations: - - read typed Ticket relations and derived blockers; - - read/write bounded orchestration plan artifacts; - - record conflict/capacity/waiting/accepted-plan notes. -- Event delivery operations: - - read pending orchestration events; - - ack, defer, or fail events with durable reason codes; - - query retry/defer state by event, Ticket, or consumer. -- Runtime registry operations: - - list runtimes and capability summaries; - - look up Workers by canonical `runtime_id` + `worker_id`; - - query capability support such as backend-internal tools, filesystem, shell, git, worktrees, bounded transcript read, stream availability, and worker spawn support. -- Spawn intent operations: - - create spawn intents; - - dispatch intents to a selected runtime; - - read intent state and acceptance evidence; - - associate intent/Worker summaries with Tickets. -- Worker overview and usage operations: - - append/read run overview entries; - - append/read lifecycle summaries; - - read usage aggregates for dashboard/control-plane display. -- Audit and decision operations: - - append routing decisions with actor/source/request id; - - append authority-boundary failures and escalation requests; - - query bounded decision history for a Ticket. - -Forbidden operation groups for the internal Orchestrator: - -- shell execution; -- raw filesystem read/write/edit over repository paths; -- raw Unix socket connects or socket path notification; -- raw session full transcript ingest; -- local Worker metadata path or session path access as authority; -- browser-provided display labels, paths, or executable strings as authority. - -## WorkerRuntime registry and spawn intents - -`WorkerRuntimeRegistry` is the boundary between backend control-plane decisions and runtime-specific worker launch. The Orchestrator asks for capabilities and submits typed intents; it does not construct low-level process commands. - -A spawn intent should describe policy and purpose rather than launch mechanics: - -```text -worker_spawn_intent { - intent_id: opaque id, - parent_event_id: orchestration event id, - request_id: idempotency/correlation id, - ticket_id: canonical Ticket id, - role: intake | orchestrator | coder | reviewer | helper, - purpose: route | implement | review | inspect | validate | summarize, - required_capabilities: [backend_internal_tools | workspace_fs | shell | git | worktrees | build | bounded_transcript], - workspace: { workspace_id, repository_targets? }, - cwd_semantics: role_default | ticket_worktree | target_repository | runtime_resolved, - profile_intent: builtin role/profile selector intent, - workflow_intent: optional workflow slug/phase intent, - input_packet_ref: durable bounded context reference, - acceptance_requirement: socket_ready | run_accepted(expected_segments) | decision_recorded, -} -``` - -The browser must not provide raw workspace roots, child cwd, executable paths, raw profile files, socket paths, local Worker names, or runtime display names in this intent. API callers can request high-level operations such as "queue this Ticket" or "open this canonical Worker"; the backend and runtime adapters resolve launch details from trusted workspace records, runtime configuration, and capability policy. - -Runtime adapters are responsible for translating an accepted intent: - -- A backend-internal runtime may create routing-only/intake/dashboard-assistant Workers with backend tools and no filesystem scope. -- A local Worker runtime may resolve a Coder/Reviewer intent into Worker launch arguments, scope, delegated filesystem paths, branch/worktree policy, prompt/profile/workflow selection, and acceptance evidence. -- A remote runtime may perform the same adaptation on a different machine without exposing local paths to the browser or storing them as API authority. - -Dispatch success means the runtime accepted the typed intent and returned durable acceptance evidence. It does not by itself prove the Ticket is done. Worker progress is projected through lifecycle, overview, review, and Ticket state records. - -## Runtime selection by capability - -Runtime choice is capability-driven: - -- Backend-internal runtime is suitable for routing-only Orchestrator work, intake refinement, dashboard assistant behavior, event processing, decision recording, and registry lookups. -- Filesystem-capable local or remote runtimes are required for Coder, Reviewer, worktree creation, git operations, builds, tests, repository inspection, and helper checks that need repository files. -- Bounded transcript read is a debug/support capability. It is not a substitute for overview/decision/lifecycle projections. - -If no runtime satisfies the required capabilities, the Orchestrator records `runtime_unavailable` as a waiting reason and defers or escalates. It must not silently downgrade to the backend-internal runtime for work that requires filesystem, shell, git, or worktree authority. - -## Worker identity, API, and database projection - -External API identity is runtime-scoped and opaque: - -- Worker detail: `GET /api/runtimes/{runtime_id}/workers/{worker_id}`. -- Cross-runtime list: `GET /api/workers`, with each item carrying `runtime_id`, `worker_id`, and display fields. - -`worker-name@runtime-name` is a display label (`display_ref`) only. It is not unique enough for authority and must not be accepted as the target of mutating operations. Similarly, local Worker `worker_name`, runtime display names, raw runtime registry paths, and socket/session paths are implementation diagnostics, not API authority. - -A browser-safe Worker summary can expose: - -```text -worker_summary { - runtime_id, - worker_id, - display_name, - runtime_display_name, - display_ref, - role, - state, - capabilities, - implementation: { - kind, - display_hint, - worker_name? # local Worker runtime only; diagnostic/display hint, not authority - } -} -``` - -For database projection, prefer a surrogate worker record id plus a uniqueness constraint on runtime-scoped identity: - -```text -workers ( - id INTEGER PRIMARY KEY, - runtime_id TEXT NOT NULL, - worker_id TEXT NOT NULL, - display_name TEXT NOT NULL, - runtime_display_name TEXT NOT NULL, - display_ref TEXT NOT NULL, - implementation_kind TEXT NOT NULL, - implementation_display_hint TEXT, - observed_at TEXT NOT NULL, - UNIQUE(runtime_id, worker_id) -) -``` - -Run overviews, lifecycle events, Ticket worker links, and usage aggregates may reference the surrogate `workers.id` internally for stable joins. External APIs should continue to expose and accept only `runtime_id` + `worker_id` for Worker identity. - -## Session, overview, lifecycle, and usage boundary - -The Workspace backend durable projection should center on: - -- orchestration events and delivery outcomes; -- routing decisions and audit records; -- worker lifecycle summaries; -- worker run overviews; -- Ticket state/relation/plan projections; -- usage aggregates. - -Raw session JSONL, provider traces, verbose event streams, local sockets, and local Worker metadata files remain runtime-local source/debug logs. The backend may expose bounded debug reads later, but that surface must be explicit, purpose-limited, permissioned, size-limited, and never treated as the normal Kanban/Orchestration UI data model. - -This keeps dashboard views stable across local/remote runtimes and prevents raw transcript contents from becoming hidden durable authority for why a control-plane decision happened. If a decision matters, it must be written as a decision/audit/overview record. - -## Failure, blocker, and waiting reason semantics - -The Orchestrator records why it did not dispatch work as carefully as why it did dispatch work. Initial reason categories: - -- `dependency_blocked`: required upstream Tickets or relations are unresolved. -- `conflict_blocked`: target paths, repositories, branches, or worker assignments conflict with active work. -- `dirty_workspace`: a filesystem-capable runtime reports that the relevant checkout/worktree is dirty or unsafe. The backend-internal Orchestrator does not inspect the filesystem itself. -- `missing_requirement`: the Ticket lacks a concrete decision, acceptance criterion, permission, or scope needed to start; the Orchestrator may return it to `planning` with a reason. -- `runtime_unavailable`: no registered runtime satisfies required capabilities or capacity. -- `identity_ambiguous`: the requested Worker/runtime cannot be resolved by canonical `runtime_id` + `worker_id`. -- `forbidden_authority`: completing the request would require raw shell/filesystem/socket/session/path authority in the backend process. -- `dispatch_inconsistent`: a spawn intent retry observed inconsistent runtime acceptance evidence. - -Waiting records should include ticket id, event id, reason code, human-readable summary, observed evidence, retry/unblock condition if any, and timestamp. A waiting reason may be cleared by a new Ticket event, relation change, runtime capability change, worker completion, or explicit operator action. - -Returning a Ticket to `planning` requires a concrete missing-decision or missing-information reason. Risk flags, unknown implementation details, or a need for reviewer focus are not sufficient by themselves. - -## Implementation sequence for future Tickets - -This design suggests the following order without making any of it part of this Ticket: - -1. Persist orchestration events for Kanban/Ticket state mutations, including idempotency by request id. -2. Add event delivery tools and decision/audit append tools for a backend-internal Orchestrator Worker. -3. Add runtime-scoped Worker detail APIs and backend worker projection records with surrogate ids and `UNIQUE(runtime_id, worker_id)`. -4. Add spawn intent persistence and registry dispatch stubs that preserve authority boundaries. -5. Implement local Worker runtime adaptation for Coder/Reviewer/helper intents. -6. Add remote runtime protocol only after the local typed-intent boundary is stable. - -At every step, keep the invariant that durable control-plane records explain why the system acted, while runtime-specific sockets, sessions, paths, and process launch details remain adapter-local implementation details. \ No newline at end of file diff --git a/docs/design/workspace-optional-repository.md b/docs/design/workspace-optional-repository.md deleted file mode 100644 index ea8a9e982..000000000 --- a/docs/design/workspace-optional-repository.md +++ /dev/null @@ -1,56 +0,0 @@ -# Workspace creation with optional Repository intent - -A Workspace is usable without a Repository. Its initial Git Repository is an -optional asset, not an identity or a prerequisite for connection. Creation does -not initialize a Git repository or infer a replacement asset. - -## Creation and replay contract - -`POST /api/workspaces` accepts `operation_key`, `display_name`, and an optional -`repository` intent. Omitted and explicit `null` both mean no initial Repository. -An object is validated normally: a partial object, invalid URI, or invalid key is -not treated as omission. Ownership comes from the authenticated request actor, -not a caller-supplied owner field. - -The bootstrap transaction persists the Workspace, optional Repository, initial -config, memory settings, Ticket/Objective/Worker resource-key counters, active -signing identity, and create receipt together. Signing-material provisioning -retains its existing durable reservation/recovery protocol; a failed bootstrap -leaves no partially created Workspace or Repository. - -The fingerprint includes normalized creation intent and owner identity. No -Repository differs from any Repository intent; repository-present fingerprints -keep the existing encoding so saved create receipts remain replayable. The same -operation key and input converge, including concurrent calls; changed input is a -conflict. No schema migration is needed for the existing fingerprint/Workspace -receipts. - -A replay is not a historical snapshot. It returns the current Workspace and -config content snapshot. If creation supplied a Repository, it returns the current -record **at the originally requested key**. Removal or renaming away from that -key produces `repository: null`; registration at that key later returns that -current record. Replay never re-registers or resets an asset. If creation omitted -a Repository, replay always returns `null`, even after other assets are added. - -## Entry points and zero-Repository use - -- Web creation requires only a display name. Optional Repository fields are - disclosed explicitly; Settings retains the later-registration entry point. -- `yoi [--backend URL] workspace create --display-name NAME` creates without Git - discovery. Select the returned identity with `yoi --workspace-id ID`. -- `yoi init` remains Git-oriented and still requires an explicit Repository key. -- The TUI creation picker allows Enter at the optional URI prompt to skip Git - registration. Workspace selection and ordinary embedded Worker startup do not - require a Repository or Workdir. Operations that use Git/Workdirs retain their - own prerequisites. -- Empty Repository lists are valid, without a missing-configuration warning. - Reopening and deletion use the usual Workspace authority and lifecycle guards; - in particular, the last-accessible-Workspace deletion guard is unchanged. - -Optional creation preserves the existing owner/member responsibility split: -settings are owner-controlled, while ordinary viewing and use are member-facing. -Non-ownership alone is not a reason to deny ordinary viewing. Existing -member-facing non-owner reads, owner-gated settings mutations, owner-scoped -catalog discovery, authentication and operation-permission checks are unchanged. -This creation change does not introduce a membership authority or an owner-only -viewing filter. diff --git a/docs/design/workspace-runtime-docker.md b/docs/design/workspace-runtime-docker.md deleted file mode 100644 index c5ad50030..000000000 --- a/docs/design/workspace-runtime-docker.md +++ /dev/null @@ -1,164 +0,0 @@ -# Workspace / Runtime / Docker launch boundary - -Yoi's browser-facing deployment is a three-process boundary: WebUI, Workspace Backend, and Worker Runtime. Docker Compose currently exercises that boundary with locally loaded Nix-built images; it is not a separate authority model. - -This document records the current design state for the split launch path, Docker image layout, and working-directory ownership rules. It is intentionally about boundaries and invariants rather than a complete operations manual. - -## Process boundary - -The browser launch path is: - -```text -Browser WebUI - -> Workspace Backend Server - -> Worker Runtime - -> Worker process -``` - -Responsibilities are split as follows: - -- WebUI renders the workspace and submits browser-safe requests. It does not hold filesystem authority, raw runtime paths, raw socket paths, or local Worker names as operation authority. -- Workspace Backend Server is the workspace control plane. It owns workspace-scoped API identity, Runtime registry projection, Worker registry projection, Ticket/workspace records, memory backend access, and browser-safe launch options. -- Worker Runtime owns runtime-specific process spawning, Worker controller state, workdir materialization, runtime-local workdir cleanup, and Worker protocol connectivity. -- Worker process owns model turns, tool execution inside its granted scope, session history append, and Worker protocol handling. - -The Backend can project Runtime and Worker state, but it should not become a hidden filesystem/runtime implementation. Runtime observations should be reconstructable from Runtime APIs and committed Backend records. - -Remote Runtime authentication follows the same Workspace boundary. The Server signs each Runtime request with the target Workspace signing identity, and the Runtime verifies it against the installed Workspace issuer bundle. Runtime-to-Server source proof is signed by the Runtime identity and uses the bundle's Backend URL as audience. Server-global signing identities, Runtime-side global Server trust, and static bearer fallback are not Remote Workspace authority. Provisioning and rotation are described in [Workspace ↔ Runtime authentication](../development/server-runtime-auth.md). - -## Docker image layout - -Docker images are built through Nix `dockerTools.buildImage`, not through a root Dockerfile. - -Current image roles: - -```text -yoi-runtime:latest - yoi-runtime - -yoi-server:latest - yoi-server - -yoi-webui:latest - nginx serving built WebUI assets and proxying API requests -``` - -The Compose setup assumes those images have already been loaded into the local Docker daemon. Compose is a local wiring layer, not the builder. - -Typical development flow: - -```text -nix build .#docker-runtime -nix build .#docker-server -nix build .#docker-webui -docker load < result -# repeat load for each built image result -docker compose up -``` - -The Compose files live at: - -```text -compose.yaml -``` - -The WebUI container serves static assets and proxies `/api` to the Backend Server. The Backend Server registers the Runtime container as a remote Runtime such as `docker-runtime`. The Runtime container runs `yoi-runtime` and owns Worker spawning/materialization for that runtime. - -The operator-owned `/etc/yoi/server.toml` is mounted read-only at `/server-config/server.toml`. Its `browser.public_url` is the single browser-facing setting used for WebAuthn, device-login URLs, cookie policy, and cookie-authenticated mutation origin checks: - -```toml -[browser] -public_url = "https://yoi.example.com" -``` - -Create this host file before starting Compose and set it to the exact Nginx-facing origin. It is not an API/backend URL and must not include a path, query, or fragment. It is deployment topology outside the source and Workspace repositories, not Workspace DB state or repository-local configuration. - -Container user and writable data directories matter: runtime/server images must be able to write their configured data directories and named volumes. The current local-image Compose setup avoids an image-level `User` override and sets data-directory permissions accordingly. - -## Worker launch path - -A browser-created Worker follows this control flow: - -```text -WebUI new-worker form - runtime_id / profile / initial input / optional workdir selection - -Workspace Backend create-worker API - validate workspace scope - resolve Runtime by canonical runtime_id - resolve profile/manifest using Backend authority - convert browser workdir selection into a Runtime working-directory claim - forward a typed Worker creation request to the selected Runtime - -Worker Runtime create_worker - reject a requested primary workdir if an existing live/registered Worker already owns it - spawn the Worker through the runtime execution backend - return Worker detail, run state, and workdir binding - -Workspace Backend post-spawn projection - sync Worker registry - sync workdir registry from Runtime summary - sync Worker/workdir link - return browser-safe Worker detail -``` - -The Runtime guard against assigning the same primary workdir to two active Workers is required even when the Backend has a registry projection. It is the last authority that sees runtime-local Worker requests and execution bindings. - -Worker interaction uses the unified Worker protocol WebSocket path. The older event-only stream is not an operation authority. Normal interaction should go through protocol transport and runtime/backend forwarding rather than raw local sockets. - -## Working-directory model - -A working directory is a materialized repository for a workspace on a concrete Runtime. - -The normal invariant is: - -```text -workdir { - workspace_id # Backend workspace scope - runtime_id # concrete Runtime that owns the materialization - repository_id # repository being materialized - working_directory_id - selector / resolved commit - materialization status - cleanliness -} -``` - -`repository_id` is part of the normal model. A workdir without a repository is not a normal reusable workspace workdir; it should be treated as diagnostic/orphan/corrupted state rather than as a first-class launch candidate. - -The removed `management_kind` state must not be reintroduced. Whether a row was first authored by Backend creation or later observed from Runtime is provenance, not current workdir authority. Reuse eligibility must be derived from current state and ownership: - -- Runtime reports the workdir as active/present; -- cleanliness is clean; -- no active Runtime Worker owns it as a primary workdir; -- Backend worker/workdir links do not show an existing owner; -- `primary_worker_id` is absent in the browser-safe summary. - -Runtime-observed workdirs that satisfy those conditions are valid reusable candidates for the selected Runtime. Hiding them because Backend did not originally author the registry row creates a stale provenance bug. - -## Workspace and Runtime sharing assumption - -The current Docker and browser launch path treats a configured Runtime as serving the Workspace that registered it. Runtime workdir summaries do not carry independent workspace authority sufficient for arbitrary multi-workspace sharing. - -If one Runtime is later shared by multiple Workspace Backends, the Runtime workdir materialization record/API must carry enough workspace identity to let a Backend decide whether a workdir belongs to its workspace. That should be modeled explicitly as workspace identity on materialization records, not through a vague managed/unmanaged kind. - -Until that exists, do not infer cross-workspace ownership from display names, local paths, Runtime labels, or Backend registry provenance. - -## Memory backend launch pitfall - -Workspace memory backend HTTP operations run inside async Worker/runtime paths. They must use async HTTP clients. Blocking HTTP clients such as `reqwest::blocking` can panic when created/dropped inside a Tokio async context and can kill Worker controller tasks. - -The current memory backend HTTP path uses async `reqwest::Client`. New Backend/Runtime launch code should preserve that boundary. - -## Current validation surface - -The local-image Compose path has been checked by building/loading the Nix Docker images, running `docker compose config`, starting the Compose stack, loading WebUI `/`, and confirming `/api/runtimes` sees the remote Runtime. - -Code-level changes around Runtime/Backend/WebUI boundaries should generally be checked with the relevant Rust crates, WebUI check, and whitespace validation, for example: - -```text -nix develop -c cargo check -p yoi-workspace-server -p worker-runtime -p client -nix develop -c cargo test -p yoi-workspace-server --lib -cd web/workspace && deno task check -git diff --check -``` diff --git a/docs/development/dogfooding.md b/docs/development/dogfooding.md index 4bdad7c14..54396224e 100644 --- a/docs/development/dogfooding.md +++ b/docs/development/dogfooding.md @@ -1,40 +1,17 @@ -# Dogfooding Yoi +# ドッグフーディング中の安全規則 -This repository is developed with Yoi itself. Dogfooding is valuable because it exposes orchestration, memory, TUI, and workflow problems under real use. +YoiでYoiを開発するときも、稼働中のServer・Runtimeを新しいbinaryの最初の起動試験に使わない。自身を実行しているプロセスは停止せず、マージ後の稼働環境の更新は利用者が行う。 -## Pre-restart gate +## 更新前の検証 -Never use the live dogfood Server or Runtime as the first startup test for a new binary. A dogfood restart is allowed only after this sequence succeeds: +製品のServer・Runtimeエントリーポイントをビルドし、変更した契約と依存箇所のテスト、root check、format、差分検査を行う。別のDB、Runtimeデータ、ポートを使った隔離環境で起動と実際の署名要求を確認する。過去の接続成功を保存して認証済みと見なすfixtureでは、本番の要求単位の認証を検証できない。 -1. Build the production entrypoints: `cargo build -p worker-runtime --bin yoi-runtime -p yoi-workspace-server --bin yoi-server`. -2. Run the focused and dependent tests for the changed contracts, followed by workspace-root `cargo check`, `cargo fmt --all -- --check`, and `git diff --check HEAD`. -3. Exercise the provisioning and operational checks in [Workspace ↔ Runtime authentication](server-runtime-auth.md) against isolated Server DB, Runtime data, and ports. The Workspace owner must configure the Runtime endpoint and pinned Runtime key, the Runtime operator must install the Workspace issuer bundle, and actual signed requests must authenticate. A connection-test result describes only that attempt; no stored verification state is a readiness prerequisite. -4. Have an external supervisor or operator restart Server and Runtime from the same validated source commit/build artifacts. A Worker hosted by the target Runtime must never terminate its own Runtime. -5. Verify post-restart readiness through the Workspace Runtime projection, ping, Worker list/create, protocol subscription, and a Runtime-to-Server source-proof operation before treating the environment as healthy. +稼働環境を更新する際は、検証したsourceとartifactを特定できるようにする。更新後は、疎通だけでなく、Workerの一覧・起動・protocol購読など変更に関係する操作を確認する。正常なHTTP応答だけで、画面、console、APIに残るエラーを無視しない。 -The former isolated startup shell harness depended on removed Server-global trust commands and is intentionally not a fallback smoke path. New automated startup coverage must configure the same Workspace-scoped connection and explicit Runtime-side Workspace trust used by production, then exercise per-request authentication rather than recreating global trust or seeding a past verification result. +検証用のWeb buildや生成処理が、稼働中の開発サーバーの生成物を書き換えないようにする。別のsource treeを検証する場合も、生成物やbuild cacheの混用によって結果を誤認しないように隔離する。 -## What to record +## 問題の記録 -When a tool limitation, workflow obstacle, or model-facing policy problem blocks work, record it under `docs/report/` or a work item artifact. Do not turn every minor annoyance into a maintained design doc. +作業を妨げるツール制限、誤解を招く指示、危険な復旧手順を確認した場合は、`docs/report/`または作業記録に報告する。試みたこと、観測した事実、影響、原因の確度、必要な設計判断を区別する。小さな不便ごとに維持対象の設計文書を増やさない。 -A report is useful when it explains: - -- what the agent/user tried to do -- what made the work unsafe, confusing, or slow -- what design boundary was missing -- what evidence was observed - -For a Remote Runtime rollout, also record the source commit, binary artifact digest, Server schema version, Runtime `binding_id` and pinned key fingerprint, the current Workspace `key_id` and public fingerprint, the Runtime-side trust registration/revocation, and typed HTTP/WebSocket outcomes. Do not copy a Runtime trust ID into the Server connection configuration or treat a successful connection test as persisted authentication. A successful document response does not outweigh visible UI, console, or API errors. - -## Runtime command caveat - -After rebuilding and restarting during dogfooding, `current_exe()` can point at a deleted binary path. Use typed runtime-command configuration and the development-only `YOI_POD_RUNTIME_COMMAND` executable override rather than reviving shell-command overrides. - -## Multi-Worker work - -Use child Workers for scoped tasks and reviews, but keep orchestration decisions in visible project records. Do not merge, close, or clean up merely because a child notification arrived. - -## Secrets and logs - -Do not put secrets, private prompts, or ignored secret-like file contents into diagnostics, work items, docs, session logs, or model context. During broad audits, existence/path checks are enough unless the user explicitly asks to inspect content. +記録には調査に必要なsource・artifact・操作結果の識別情報だけを残す。秘密情報、認証情報、非公開プロンプト、秘密を含むファイルの内容を文書・診断・セッションへ転載しない。広範な監査では、明示的な内容確認が必要でない限り、存在とパスの確認にとどめる。 diff --git a/docs/development/environment.md b/docs/development/environment.md deleted file mode 100644 index 334c3ee05..000000000 --- a/docs/development/environment.md +++ /dev/null @@ -1,24 +0,0 @@ -# Environment boundary - -Environment variables are a minimized runtime boundary. Prefer explicit profile/manifest configuration and secret references over ambient process state. - -## Why minimize environment variables - -Ambient environment is hard to audit: it can differ between shells, services, spawned Workers, tests, and restored processes. If important runtime behavior depends on it, reproducing a session becomes harder. - -Yoi keeps environment variables for narrow bootstrap and development cases, while normal provider credentials and runtime configuration should be explicit records. - -## Principles - -- Distinguish data/runtime paths from resource/config paths. -- Prefer embedded builtin resources over installed runtime resource directories. -- Use explicit secret refs for provider and WebSearch credentials. -- Keep dev-only executable overrides clearly named and documented. -- Avoid shell-command parser overrides for runtime Worker launch. -- Tests should prefer typed fixtures/injection and mutate process environment only around thin env-reader behavior. - -## Current surface - -Use `YOI_*` for current environment variables. Old project prefixes should not be reintroduced. - -`YOI_POD_RUNTIME_COMMAND` is a legacy-named development-only executable-path override for typed `yoi worker` launch. It is not a general shell-command override. diff --git a/docs/development/external-workdir-sharing.md b/docs/development/external-workdir-sharing.md deleted file mode 100644 index 1030bc6ef..000000000 --- a/docs/development/external-workdir-sharing.md +++ /dev/null @@ -1,57 +0,0 @@ -# External Workdir sharing permissions - -`yoi workdir share` exposes a client-hosted directory through an authenticated, -outbound Backend connection. The operator selects exactly one permission level: - -- **READ** (capability bits `25`): file reads and read-only discovery operations - such as Stat, List, ViewImage, Glob, and Grep. -- **WRITE** (capability bits `31`): READ plus Write and Edit. Including READ - preserves the read-before-write content-hash contract. WRITE never enables - COMMAND. -- **COMMAND** (capability bits `63`): WRITE plus READ plus command start, status, - bounded output, and cancellation. It executes on the sharing host with the OS - authority of the user running the CLI; External Workdir command execution is - **not a sandbox** and does not confine the process to the shared directory. - -In short, `COMMAND ⇒ WRITE ⇒ READ`. READ is the default. Interactive mode asks -for one level and shows its included permissions before creating the grant. For -automation, pass `--permission` once: - -```sh -yoi --backend https://example.invalid workdir share ./project \ - --workspace-id workspace-a \ - --permission command \ - --non-interactive -``` - -`--read-only` remains an alias for the READ level. `--read-write` remains an -alias for the WRITE level and never enables COMMAND. These legacy flags conflict -with `--permission`, and multiple `--permission` arguments are rejected so -ambiguous automation fails closed. - -The grant API retains the boolean projection for compatibility, but accepts only -these combinations when creating new grants: - -| Level | `read` | `write` | `command` | -|---|---:|---:|---:| -| READ | `true` | `false` | `false` | -| WRITE | `true` | `true` | `false` | -| COMMAND | `true` | `true` | `true` | - -Older non-hierarchical grants such as command-only or READ+COMMAND are not -silently widened. The Backend refuses their projection, provider registration, -and subsequent restore/reconnect with guidance to revoke and re-share the -directory at one of the supported levels. General Workdir attachment/session -capability intersections may still produce partial capability sets; that -restriction applies only to External grant authority. - -Omitting `--ttl` creates no automatic expiry. An explicit TTL must be between -60 seconds and 24 hours. Ctrl-C, revoke, expiry, provider disconnect, attachment -detach, and provider-generation replacement terminate and reap active commands. -A reconnect creates a fresh command namespace; old handles and results are not -accepted by the new generation. - -Command output uses the existing bounded inline contract. Worker-local spill -paths are not sent to the sharing provider, and provider-local paths are never -returned to the Backend or model. Truncated External command output is therefore -reported without a full-output host path. diff --git a/docs/development/file-operations.md b/docs/development/file-operations.md deleted file mode 100644 index 29adb523e..000000000 --- a/docs/development/file-operations.md +++ /dev/null @@ -1,49 +0,0 @@ -# Shared text file operations - -`fs-operation::text` owns the provider-neutral read/edit/write argument contracts, -JSON decoding, argument validation, pure text outcomes and byte-bound calculations. -The argument schemas also supply WIP parameter declarations. Ordinary Tools flatten -these arguments alongside their own Workdir routing fields; checkout WIP uses the -same Tool execution functions, not a third editing implementation. - -Edit requires a nonempty `old_string`, a different `new_string`, and a unique -non-overlapping match unless `replace_all` is true. Limits count UTF-8 bytes, are -supplied by each provider, and are checked (including arithmetic/capacity overflow) -before allocating a replacement result. Outcomes carry replacement count and byte -size independently of persistence metadata. - -## Provider-owned differences - -- Workdir read exposes optional line offset/limit and numbered, bounded output. - Workspace settings read remains a whole-text read with no arguments, preserving - content type and digest. Line range defaults and numbered rendering live in the - core but are only selected by the Workdir adapter. -- Workdir routing, attachment generation, Scope and permission identity remain in - Worker/Tools/Workdir. Read history and content hashes stay in the existing - attachment-scoped tracker. Descriptor-protected reads and checked edits/writes - retain their existing validator, identity and save boundary in the provider. -- Workspace settings grants, access mode and connection lifetime remain in Backend. - The settings adapter binds the read to captured entry metadata and generates a - canonical commit using the same tree digest. Decodal validation, - formatting and atomic multi-file persistence remain in the canonical Backend - commit. Content type, create/delete/rename/apply_changes and attach are settings - operations, not ordinary OS file operations. - -The core does not perform I/O, authorize anything, or own content/existence -mirrors. Providers invoke pure processing inside their existing consistency -boundary; it must not be used to turn a checked edit into an unchecked -Read–Write sequence. - -## External contracts - -Operation names, argument shapes and successful responses remain unchanged. -Workspace settings Edit now rejects identical old/new strings before reading or -committing (the intended behavior change). Pure text failures are explicitly -mapped to Tool argument errors or WIP `InvalidArguments`/`ResourceLimitExceeded` -by the adapters. A mutation's post-dispatch transport or response failure is still -`OutcomeUnknown`; the core does not retry it or reinterpret it as a validation -rejection. - -Pure policy matrices belong in `fs-operation::text` tests. Provider tests cover -entrypoint wiring and the existing authorization, observation, checked-save, -configuration CAS and unknown-outcome boundaries rather than copying that matrix. diff --git a/docs/development/plugin-development.md b/docs/development/plugin-development.md deleted file mode 100644 index a94993f9a..000000000 --- a/docs/development/plugin-development.md +++ /dev/null @@ -1,45 +0,0 @@ -# Plugin package authoring - -Dynamic Worker Plugins are not currently installable or executable. This guide covers only the retained offline `.yoi-plugin` authoring format. - -## Commands - -Every retained command uses an explicit local input or destination: - -```sh -yoi plugin new rust-component-tool ./example-plugin -yoi plugin new rust-component-service ./example-service -yoi plugin check ./example-plugin -yoi plugin pack ./example-plugin --output ./example-plugin.yoi-plugin -yoi plugin check ./example-plugin.yoi-plugin -``` - -- `new` writes an embedded starter template to the named destination and refuses unsafe or non-empty destinations. -- `check` parses and validates the named directory or package without executing Plugin code. -- `pack` validates the named directory and writes a deterministic constrained archive. - -`list`, `show`, `--workspace`, and `--profile` are intentionally unavailable. They previously implied ambient Workspace/user catalog discovery. - -## Safety and authority - -Offline package commands do not: - -- inspect repository or ancestor `.yoi/plugins` directories; -- inspect a user-data Plugin store; -- enable or install a package; -- mutate Profile or Manifest configuration; -- register Worker Tools, Services, or Ingress handlers; -- instantiate or execute a WASM component; or -- grant filesystem, network, secret, Ticket, or Workspace authority. - -`plugins` and `feature.plugins` are rejected by current Worker Manifest/Profile resolution. Statically compiled built-in Features are the only current Worker capability source. - -## Package format - -A package directory contains `plugin.toml` plus the files named by that manifest. A packed `.yoi-plugin` uses the constrained deterministic archive format documented in [`../design/plugin-packages.md`](../design/plugin-packages.md). Validation rejects malformed metadata, unsafe paths, links, unsupported entries, bounds violations, digest inconsistencies, and legacy raw core-Wasm runtime declarations. - -Generated templates include a placeholder `plugin.component.wasm`. Replace it with a real built component before `check` can report the package as verified. A verified package is still only an offline artifact; verification does not install or authorize it. - -## Future Server Plugin platform - -Do not copy packages into repository or user-data catalogs. Future installation must go through Server-owned package/artifact authority, immutable identity and digest selection, Backend-authored Worker execution plans, Runtime verification, and sandboxed execution. That platform is separate work and must not reintroduce filesystem catalog fallback. diff --git a/docs/development/runtime-request-proof-diagnostics.md b/docs/development/runtime-request-proof-diagnostics.md deleted file mode 100644 index c6ffb8de1..000000000 --- a/docs/development/runtime-request-proof-diagnostics.md +++ /dev/null @@ -1,83 +0,0 @@ -# Runtime request proof rejection diagnostics - -`runtime_request_proof_rejected` (`yoi::auth`, WARN) is an internal operational -record. HTTP responses are unchanged: 401 with no proof details (repository -routes retain their existing generic API error envelope). It does **not** establish -why the reported same-machine intermittent 401 occurred. No reproduction or -causal measurement of that incident has been obtained. - -## Verification and correlation - -- `request_id`: Server-generated UUID identifying this rejection event. It is not - a caller-supplied ID or a response header, and is not a cross-service trace ID. -- `claimed_runtime_id_hash`, `claimed_worker_id_hash`, `claimed_token_id_hash`: - lowercase 64-character hex SHA-256 of - `b"yoi-runtime-proof-diagnostic-id-v1\0" + id.as_bytes()`. - `worker_source::diagnostic_id_hash` exposes this correlation function. Hash a - known Runtime/Worker ID to find its failures; token ID hashes correlate replay - attempts without logging the proof or JTI itself. These are fingerprints, not - encryption: low-entropy IDs can be guessed. Do not place secrets in IDs. -- All claims-derived fields are prefixed `claimed_`, even after signature checks. - `proof_verification` describes check progress, **not an authorized principal**: - - `unverified`: decoding may have succeeded, but signature verification has not. - - `signature_verified`: signature passed; request binding or time failed. - - `request_verified`: signature, request bindings and time passed, but replay, - membership or authority failed. -- When decoding fails, there are no claims fingerprints. Do not infer a Runtime - from a malformed proof. No raw claims, proof, key, header, body or raw error is - logged. The body digest and claim mismatch values are also deliberately absent. -- Method, query-free path, and requested workspace are request metadata, not - authenticated claims. They are bounded to 32, 512 and 128 UTF-8 bytes, with - control characters removed. Truncated paths are not unique identifiers. Query - names and values are never included. Paths are operational metadata, not a - place to put secrets. - -## Reasons and their limits - -Decoding distinguishes `invalid_format`, `invalid_encoding`, and -`malformed_claims`; no parser error text is included. Signature checks distinguish -`invalid_runtime_public_key` from `invalid_signature` (which cannot tell whether -an attacker, wrong key, corruption, or configuration mismatch caused the failure). - -Request binding reports the first mismatch, in the existing check order: -`issuer_mismatch`, `audience_mismatch`, `workspace_mismatch`, `worker_mismatch`, -`permission_mismatch`, `method_mismatch`, `path_mismatch`, `body_digest_mismatch`. -The server selects the trusted key by the unverified issuer and expects that -same issuer/worker; issuer and worker mismatches can be diagnosed by the lower -verifier but are not independently detectable by the current server caller. -These diagnostics do not add new identity checks or change authorization. - -`backend_audience_unavailable` denotes missing Backend public URL configuration. -`runtime_trust_missing_or_revoked` deliberately covers both absence and revocation, -as the existing store lookup/filter does not preserve that distinction. -`replay` means JTI consumption returned false. `worker_catalog_membership` means -neither registry membership nor an active create reservation was found. - -Store failures identify their check stage without exposing raw errors: -`runtime_trust_authority_unavailable`, `replay_authority_unavailable`, -`worker_catalog_authority_unavailable`, `worker_reservation_authority_unavailable`, -`worker_singleton_authority_denied`. The last can represent either an owner -conflict or an authority failure: the store currently returns a shared error, -so the diagnostic does not guess which. `verification_time_out_of_range` means -that conversion to the JTI consumption timestamp failed; it is not proof expiry. -`invalid` is a fallback for lower verifier errors not classified for this path. - -## Time contract and T-728 coordination - -This change preserves strict integer-second checks, inclusive bounds, and -verification order: signature, request binding, then `iat > now`, then `exp < now`. -No skew tolerance is added. If both time predicates fail, future issuance wins. - -- `issued_in_future`: `time_delta_seconds = iat - now` (positive). -- `expired`: `time_delta_seconds = now - exp` (positive). -- Only these reasons include `issued_at_unix`, `expires_at_unix`, - `verified_at_unix` and `time_tolerance_seconds` (currently zero). -- Deltas use `i64::abs_diff`, preserving the full positive `u64` range without - overflow. All times come from the actual signature-verified rejection and - the exact clock sample used for that decision, not a second log-time sample. - -T-728 owns any change to tolerance/validity policy. If it changes these checks, -the error time data, logged effective tolerance, and boundary tests must change -together. Do not label future issuance as expiry or independently recompute the -rejection using a different clock sample. This is instrumentation for the next -incident, not evidence that the original incident has been resolved. diff --git a/docs/development/rust-testing-strategy.md b/docs/development/rust-testing-strategy.md index 62df02ad5..03d616642 100644 --- a/docs/development/rust-testing-strategy.md +++ b/docs/development/rust-testing-strategy.md @@ -1,252 +1,81 @@ -# Rust testing strategy for Yoi +# Rustテストの設計規則 -Yoi tests protect behavior that Rust's type system cannot prove and deliberate compile-time API constraints that must not regress. A test must name the contract it protects and fail for one understandable reason. +テストは、Rustの型だけでは保証できない製品の振舞いと、維持すべきコンパイル時のAPI制約を検証する。名前から守る規則が分かり、一つの失敗から原因となる契約を絞れるようにする。件数、配置場所、smoke testの成功を網羅性の根拠にしない。 -This document defines how to choose test scope for Yoi's architecture: `yoi` CLI, Workspace Server, Worker Runtime, Worker tools/WIP adapters, SQLite authorities, Web/API DTOs, and local/runtime process boundaries. Cargo target classification and architectural test scope are separate concepts. Test the product logic and the connections that carry it; neither a directory name nor a full-stack fixture is evidence of coverage. +## 検証範囲の選択 -References used while writing this policy: +利用者が観測する振舞いと、起きた場合に問題となる失敗を先に定める。その失敗を検出できる最小の範囲を、影響・発生可能性・保守コストから選ぶ。「最小」は、mockに置き換えやすい範囲ではなく、必要な接続まで含む範囲である。 -- Rust Book, "Test Organization": unit tests are focused and may test module-private behavior; integration tests use public APIs the way external code does. -- Cargo Book, `cargo test`: package, target, and feature selection determine which tests Cargo builds and runs. - -## Test scope is not Cargo target classification - -Cargo integration-test targets, normally under `tests/`, compile as separate crates and exercise the library's public API. In-crate tests can access private implementation details. This distinction determines compilation and visibility, not how many production components a test connects. - -Use the following scopes when explaining what a test proves: - -| Scope | What it proves | +| 範囲 | 検証する内容 | | --- | --- | -| Unit / module-local | One local rule without composing the surrounding subsystem | -| Component | One named subsystem's behavior through its boundary, with external dependencies substituted where appropriate | -| Integration / composition | Agreement between connected production components, including propagation of data, authority, and effects | -| E2E | An explicitly selected product journey through the deployed interfaces and execution boundaries relevant to that journey | +| 単体 | 周辺の仕組みを組み立てずに確認できる局所的な規則 | +| コンポーネント | 対象のサブシステムを、その境界から操作した振舞い | +| 結合 | 本番の構成要素を接続した際のデータ・権限・副作用の一致 | +| E2E | 選定した利用場面を、必要な製品インターフェースと実行境界を通して確認した結果 | -These scopes are relative to the named subject. Describe the production path and substitutions rather than relying on the label. A public-API test against a fake dependency does not prove agreement with that dependency's production implementation. Yoi's placement and execution rules below make these scopes operational rather than leaving each author to choose a layout. +Cargoのintegration-test targetは別crateとしてコンパイルされ、公開APIを使う。この区分は可視性とコンパイル単位を決めるものであり、上の検証範囲とは別である。実装内のテストでも本番の構成要素を接続すれば結合テストとなる。 -Process boundaries are an execution concern, not a separate test level. An isolated child process can be part of an ordinary integration test when the connection under test requires it. It does not require an E2E exception merely because it is a real product binary. Assembling the product to exercise a user journey belongs to the E2E suite, not a crate-local process harness. +プロセス境界も検証レベルとは別である。接続の検証に必要な隔離子プロセスは通常の結合テストに含める。利用場面を試すために製品全体を組み立てる場合は、crate内に別の起動基盤を作らず`tests/e2e`を使う。 -## Choose scope from risk and observable behavior +## 本番の接続と代替実装 -Start with the observable behavior and the failure that matters to the user. Select the smallest scope that can expose that failure, considering its likelihood, impact, and the cost of maintaining the test. Smallest means sufficient, not merely easiest to mock. +結合テストには、入口、本番の構成要素、代替した境界、観測する結果を明示する。主張する接続は本番実装のままにし、その外側だけを代替する。双方を別々のfakeでテストしても、実際の組合せが動くことは証明できない。 -- Keep local policy matrices in unit/component tests. Use integration tests when the risk lies in agreement or handoff between components. -- Test domain logic directly. UI interaction and rendering tests protect UI behavior; they do not replace proof of state transitions, authority decisions, or durable effects. -- Select retry, concurrency, cancellation, and recovery cases from the product's guarantees and failure impact. Do not expand every change into all failure modes or all component combinations. -- Prefer deterministic boundary failures for logical recovery tests. Use process termination only when process death itself is material to the guarantee and narrower tests cannot prove it. Logical recovery tests do not establish arbitrary OS-crash durability. -- Reserve E2E for behavior that requires the wider product path. Explain the additional risk it covers beyond component and integration tests. +永続化・トランザクションの保証には一時DBなどの実ストレージを使い、正本からの再読取りで結果を確認する。呼出し記録だけでは永続効果を証明できない。認証・権限を主張するテストでは、その境界をfixtureで迂回しない。障害注入でも、検証対象の復旧処理を代替しない。 -## Production connections and test doubles +再試行、競合、取消、復旧は製品の保証と失敗時の影響から選ぶ。全変更へ全障害の組合せを追加しない。通常は境界で決定的に失敗させ、プロセス死そのものが保証に関わる場合だけ終了を使う。論理的な復旧テストを、任意のOSクラッシュへの耐性の証拠にしない。 -For integration tests, make the entry point, real components, substituted boundaries, and observable result clear in the test or a short fixture comment. +## 型の保証と外部契約 -1. Keep the connection under test real. Independently testing both components against fakes does not prove they work together. -2. Substitute dependencies outside the behavior being claimed. Do not reproduce the product rule in a fake and then assert that it succeeds. -3. Exercise the production operation and observe its result through the boundary that owns the guarantee. Use real temporary storage for persistence and transaction claims; captured calls alone do not prove durable effects. -4. Preserve the identity and authority context relevant to the claim. A fixture that bypasses an authorization boundary cannot prove that boundary. -5. Inject failures without replacing the recovery logic being tested. Synchronize through observable events or conditions, not timing guesses. +構築済みの型について、enumの値域、必須フィールドの存在、所有権、`Option`の有無、網羅的な`match`、通常の引数・戻り値・trait適合を実行時に再検証するテストは追加しない。 -## Coverage responsibility and maintenance +JSON、TOML、DB行、HTTP、WebSocket、CLI引数、環境、パス、LLMのツール入力から型へ変換する境界は別である。Serde表現、schema、経路、範囲制限、拒否条件は外部契約として検証する。コンパイルの受理・拒否自体が公開APIの契約なら、macro展開、typestate、trait boundなどの最小のcompile-pass/compile-failテストを置く。 -Before adding tests, inspect existing coverage for the changed contract. Identify which tests own the local rules and which prove the relevant connections. Keep this explanation proportional to the change; it does not require a new coverage registry or a centralized integration suite. +## 配置と所有者 -Give each invariant a primary test at the boundary that owns it. Integration tests should prove connections rather than repeat every local input matrix. Strengthen existing tests where possible; if they already prove the contract, report that evidence instead of adding redundant tests. Test counts, directory placement, and a passing smoke test are not proof of coverage. - -When a product contract changes, update affected fixtures and assertions, including opt-in suites. Preserve obsolete behavior only for an explicitly supported compatibility boundary. A skipped or unexecuted test is not evidence that the current behavior works. - -## What Rust types already guarantee - -Do not write tests that only re-prove these facts for values already constructed as Rust types: - -- enum values are one of the declared variants; -- required struct fields exist; -- references obey Rust ownership and lifetime rules; -- `Option` is present or absent explicitly; -- exhaustive `match` expressions handle every variant; -- ordinary function argument and return types agree; -- a `Box` or trait object satisfies the trait at compile time; -- a typed DTO has fields of the declared Rust type after successful construction. - -A typed value still needs a **boundary** test when it is converted from untrusted JSON, TOML, files, SQLite rows, WebSocket frames, HTTP requests, CLI argv, environment variables, model tool input, external process output, or platform paths. Serde representation, path containment, HTTP route shape, SQLite schema, and prompt/tool JSON schemas are external contracts, not compiler guarantees. - -Compile-time behavior may itself be a public API contract. Compile-pass or compile-fail tests are appropriate when downstream code must continue to compile or be rejected, including macro expansion, typestate transitions, public trait bounds, and unavailable methods in a restricted state. Such a test protects the shape of the public API; it does not re-run a compiler guarantee as a runtime assertion. - -Yoi currently uses `trybuild` for compile-fail API-shape tests in `agen`. Add new `trybuild` tests only when the compile-time accept/reject behavior is the product contract being protected. Keep fixtures minimal and name them after the API rule, not after the implementation helper that happens to trigger the compiler error. - -## Test names - -A test name must describe an observable rule. Prefer names of the form `_` or `_`. Include the input class or state when omitting it would make the rule ambiguous. - -Good names: - -- `backend_target_rejects_local_worker_operations` -- `cli_backend_url_overrides_config_file_value` -- `repository_path_rejects_symlink_escape` -- `second_enter_does_not_submit_rewind_twice` -- `workspace_server_returns_not_found_for_unknown_worker` - -Bad names: - -- `test_target`, `test_parse`, or `test_create`: names the subject but not the rule. -- `works`, `basic`, `happy_path`, or `success`: does not say what outcome is protected. -- `regression_1234`: records provenance but loses the lasting product behavior. -- `ticket_test` or `server_test_2`: adds no information beyond the module or file. -- a private helper name alone, such as `normalize_path`: couples the name to implementation without stating the externally relevant result. - -No prefix is required. `contract_` is normally redundant because every test in this policy protects a contract. The following narrower labels are optional: - -- `boundary_` when distinguishing external input handling from typed internal behavior is useful in that test target; -- `regression_` when the original failure symptom is itself the clearest durable rule; -- `smoke_` when the deliberately limited contract is successful wiring to a minimum state. - -Examples such as `boundary_repository_path_rejects_symlink_escape`, `regression_second_enter_does_not_submit_rewind_twice`, and `smoke_workspace_server_reaches_health_route` remain meaningful after removing the prefix. A label does not rescue a vague name such as `boundary_path`, `regression_1234`, or `smoke_works`. - -## Yoi contract map - -This map identifies responsibility areas, not an exhaustive scenario checklist or a snapshot of implementation details. Select tests from the current product contract and the risks of the change; do not turn an old fixture or state name into a new product requirement. - -| Area | Guaranteed by types | Must be tested | +| 範囲 | 配置 | 実行区分 | | --- | --- | --- | -| CLI parser and Target selection | parsed `Mode`, `LaunchMode`, `TargetKind`, and command enum variants exist | top-level target option precedence, mutually exclusive options, default config merge, local/backend-only rejection, help/usage text for public command surface | -| Client config | typed config structs after TOML/environment parse | data-dir and cwd config locations, environment snapshot to startup config, property-wise overlay, missing/empty backend URL diagnostics, workspace-to-backend resolution, explicit CLI override over config | -| Workspace Server HTTP API | handler function signatures and response DTO field types | route shape, scoped workspace authority, status-code mapping, request validation, bounded list/detail bodies, mutation round trips, compatibility routes only where explicitly adopted as product contracts | -| Runtime HTTP/WebSocket protocol | protocol enum variants and typed frames after decode | JSON wire compatibility, frame ordering, snapshot/backlog/live semantics, bounded output, cancellation/restore lifecycle, unknown id/status errors | -| Client crate Target operations | trait method signatures and result DTO types | unsupported operation errors, Local/Backend behavior parity where promised, stopped-worker semantics, URL/workspace/routing normalization | -| Worker tool definitions | `Tool` trait implementation and JSON value construction compile | tool name/schema stability, input validation, Backend authority requirement, bounded output, no direct local authority fallback, side effects/audit records | -| Tickets | typed Ticket inputs, state values, and assignment references after parse | authority mutation/readback, lifecycle and resolution semantics, assignment identity and authority, relations and audit history, separation of lifecycle, work progress, and delivery | -| Objectives | typed Objective summaries/details after authority read | SQLite authority mutation round trip, canonical id allocation, state/title/body validation, ticket link/unlink, audit/event rows, scoped API/tool behavior | -| Memory | typed memory operation inputs after parse | single-document edit semantics, old_string uniqueness, staging close lifecycle, bounded reads, Backend Workspace authority use, no filesystem fallback in Worker tools | -| Worker lifecycle and workdirs | Rust structs for worker/workdir records | restore/dry-restore decisions, stopped/corrupted state transitions, occupancy projection, repository path containment, cleanup idempotence | -| Workspace authority and SQLite stores | Rust records and trait methods compile | schema migrations, foreign-key/unique constraints, transaction rollback behavior, authority boundary selection, import-only legacy paths | -| Nix/Docker/dev scripts | Nix attr names and shell strings are strings | binary names, package/bin target selectors, image entrypoints, fixed-output hashes, smoke help checks | -| Web frontend TypeScript/Rust DTO boundary | generated/declared TS types after decode | API representation and route agreement, projection logic, validation and error handling; source-text assertions do not prove runtime or rendered behavior | -| Prompts and tool schema resources | resource files exist as strings | prompt inclusion policy, feature-driven instruction contribution, schema names/descriptions visible to the model, no stale command/tool references | +| 単体 | 所有module内の`#[cfg(test)] mod tests`、または`src/`配下の子テストmodule | 通常のcrateテスト | +| コンポーネント | 所有crateの`src//tests.rs`など。公開APIを外部利用することが目的なら`tests/` | 通常のcrateテスト | +| 結合 | 接続を所有するcrateの`tests/`。内部アクセスが必要なら結合範囲を明記したcrate内module | 対象の製品featureを含む通常のcrateテスト | +| E2E | Workspaceの`tests/e2e` packageと共有harness | 専用の明示選択target/job | -## Test design rules +Cargo test-target間の補助処理は`tests/common/mod.rs`、crate内ではテスト支援moduleにまとめる。関連する結合ケースは同じtargetのmoduleに分け、ケースごとに実行ファイルを増やさない。公開範囲を広げてまで内部テストを`tests/`へ動かさない。 -1. Arrange only the state relevant to the named contract. -2. Exercise public or module-boundary behavior where possible. Test a helper directly only when that helper is the policy boundary. -3. Assert outcomes, rejected inputs, API responses, durable records, or state transitions rather than private call order and incidental ids. -4. Use a table in one test when several inputs describe one policy matrix. Use separate tests when failures would represent different product rules. -5. Boundary tests must include the boundary input in the fixture: argv slice, JSON body, TOML text, explicit environment snapshot, SQLite row/migration, path, HTTP request, protocol frame, or tool input JSON. -6. Filesystem tests use an RAII temporary directory and leave no files behind, including after panic. Do not write tests against the user's real Yoi data dir. -7. SQLite tests use temporary databases or in-memory stores and explicitly seed the authority records they need. Do not depend on the developer's live `server.db`. -8. Tool tests assert the model-visible contract: tool name, schema, validation, bounded content, authority errors, and durable side effects. Do not only assert that a Rust helper was called. -9. API tests should assert status codes and response bodies for both success and rejection. If a helper hides response bodies, include the body in assertion failure output. -10. Serde/DTO tests should exist only for external representation contracts: renamed fields, wire enums, bounded truncation, or explicitly adopted compatibility with existing persisted data. Do not add aliases, legacy variants, routes, or tests for them without an explicit compatibility decision. -11. Snapshot-like assertions are allowed only when the exact text/wire format is the contract. Prefer semantic assertions for large JSON/API responses. -12. Built-in resources and prompts should be tested by deriving expected references from manifests or feature config, not by duplicating mutable inventory counts in Rust assertions. -13. Do not test private helpers just to improve coverage numbers. If a private helper carries a policy boundary, name the contract in the test and keep the fixture minimal. -14. Every bug fix either strengthens an existing test or adds one focused test whose name states the lasting rule or the original failure mode. A `regression_` prefix is not required. -15. When a test needs a production resource, state which compatibility contract that resource represents. Generic fixtures should use minimal generated definitions. -16. Do not introduce environment variables whose purpose is test control, test fixtures, or test observation. -17. Code that consumes environment variables must snapshot them into typed startup configuration once. Test the environment-to-configuration mapping as a boundary over an explicit snapshot; all downstream logic tests pass typed configuration and do not read, set, remove, or depend on process-global environment variables. -18. Tests must not depend on execution order. Ordinary tests must be safe under Cargo's default parallel execution and must not coordinate through process-global state. -19. Synchronize on observable events or conditions. Do not use a fixed sleep to guess that asynchronous work has completed; use a timeout only as an upper bound that turns a hang into a diagnosable failure. -20. Tests must not depend on live external services or unrestricted network access. Use in-process components or loopback transport; substitute only dependencies outside the connection being proved. -21. Keep boundary integration in the ordinary test suite, including isolated process boundaries where needed. Add or extend E2E only for an explicitly selected product journey that narrower tests cannot prove, using the shared `tests/e2e` harness rather than assembling the product independently in a crate-local suite. +複数crateの結合も、接続の契約を所有するcrateに置く。専用packageは依存制約または実際の共有需要がある場合に限る。通常の結合テストを配置の都合でE2Eへ移さない。parser・ツール方針は所有module、保存制約はstore、製品の権限判断はauthority、routerとauthorityの接続はAPIの結合suiteが担当する。 -## Placement, fixtures, and execution +## 共有fixtureと隔離 -The following are Yoi's default operating rules, not interchangeable layout suggestions. The crate that owns a boundary also owns its test suite. Moving a test into another file does not change its scope. +各結合・E2E suiteは、検証範囲、本番の構成要素、代替境界、実行target、前提条件を短いmoduleドキュメントに記す。資源の作成・解放を各ケースで重複させず、suiteの共有支援処理が次を所有する。 -| Scope | Placement | Allowed dependencies | Execution class | -| --- | --- | --- | --- | -| Unit | Owning module's inline `#[cfg(test)] mod tests` or extracted child test module under `src/` | Local logic and minimal fixtures | Ordinary crate tests | -| Component | Owning crate's `src//tests.rs` or equivalent named test module | Component implementation, temporary resources, doubles outside the component boundary | Ordinary crate tests | -| Integration | A named test target under the owning crate's `tests/`; a named in-crate integration module when crate-internal access is required | Connected production implementations, temporary databases, loopback/IPC, isolated child processes when relevant to the boundary | Ordinary crate tests, with applicable product features | -| E2E | Workspace `tests/e2e` package, with journey tests using its shared harness | Product assembly and actual execution interfaces needed by the selected journey, with isolated data and resources | Dedicated opt-in E2E target/job | +- RAIIで管理する一時ディレクトリ・DB、隔離した識別子・通信endpoint。開発者の稼働環境や実データを借りない。 +- 明示的な実行ファイル、子のcwd・環境。親テストプロセスの環境を変更しない。 +- 起動から終了・回収までの子プロセス管理。assert失敗やtimeoutでも後始末する。 +- 観測可能なreadiness・イベントによる同期、待機の上限、障害診断の採取。 -### Placement details and exceptions +固定sleepで処理完了を推測しない。timeoutはハングを失敗へ変える上限として使う。通常テストは順序非依存・並列安全とし、live外部サービスや制限のないネットワークへ依存させない。通信には必要に応じてloopbackやIPCを使う。 -- A component test whose purpose is external public-API use belongs under the owning crate's `tests/`. State its component scope in the target/module documentation; being a Cargo integration-test target does not expand its guarantee. -- In-crate integration modules must be identified as such and document the production connection they cover. Do not expose private implementation solely to move tests under `tests/`. -- Share Cargo test-target helpers under `tests/common/mod.rs`; use a crate-local test-support module for in-crate suites. Group related integration cases into modules of one target instead of creating a separate test executable for every case. -- Cross-crate integration belongs to the crate that owns the composed boundary. Introduce a separate test-support or integration package only when dependency constraints or genuinely shared use justify it; do not place ordinary composition tests in `tests/e2e` for convenience. -- Parser/help policy tests and Worker tool policy tests stay with their owning modules. Store tests prove persistence constraints; authority tests prove product policy. Route tests that connect router and authority implementations belong to the owning API's integration suite. -- Keep product-assembly fixtures in `tests/e2e`. Do not create parallel E2E launch stacks under individual crates. +環境変数は起動境界で一度だけsnapshotから型付き設定へ解決する。その変換を明示的snapshotでテストし、後続のロジックは型付き設定を受け取る。テスト制御・fixture・観測用の環境変数を追加せず、テストでプロセス全体の環境を読み書きしない。 -### Shared fixture responsibilities +## 名前とassertion -Each integration or E2E suite must use shared support for the resources it owns, rather than repeating setup and cleanup in individual cases. A short suite/module description must identify its scope, production components, substitutions, execution target, and prerequisites. +名前は`_`または`_`を基本とし、`repository_path_rejects_symlink_escape`のように観測可能な規則を示す。`works`、`test_parse`、Ticket番号、private helper名だけでは規則を特定できない。`boundary_`、`regression_`、`smoke_`は意味がある場合だけ使い、必須接頭辞にはしない。 -- Allocate isolated directories, databases, identities, and communication endpoints. Never borrow a running developer instance or live user data. -- Resolve binaries explicitly and construct each child's working directory and environment without modifying the parent test process environment. Keep product environment inputs separate from test-control hooks. -- Own child processes from launch through termination and reaping, including assertion failure and timeout. Synchronize startup and work completion on observable readiness/events, not fixed sleeps. -- Bound waits and collect useful failure diagnostics. Reuse the suite's cleanup and artifact policy rather than making each case invent one. +- fixtureには契約に必要な状態だけを用意する。境界テストには実際のargv、JSON、DB行、パスなどを含める。 +- 結果、拒否、応答、永続記録、状態遷移をassertする。privateな呼出し順や偶然のIDに依存させない。 +- 一つの入力規則は表形式にまとめ、別の製品規則は別のテストにする。 +- APIは成功と拒否のstatus・body、ツールはLLMに見える名前・schema・検証・出力上限・権限・副作用を確認する。 +- migrationは番号だけでなく、保存値、制約、変換、rollbackなど変更した保証を確認する。 +- 完全文字列やsnapshot比較は、その表現自体が契約の場合に限る。大きなJSONでは意味をassertする。 +- promptや組込み資源は所有者と包含条件を検証する。可変な一覧・件数・長い文言をassertionへ複写しない。 +- 互換処理は明示的な製品判断がある場合だけ維持し、古いfixtureを通すためにaliasや旧経路を増やさない。 -Ordinary integration tests must remain deterministic and parallel-safe. A need for a database, a socket, or a child process alone is not a reason to make a test opt-in. Tests needing special tools, privileges, or platform support require an explicit prerequisite and execution selection; that exception does not automatically make them E2E. +## 保守と実行責任 -Execution commands and CI selection belong in [validation.md](validation.md). A suite's default/feature/opt-in classification must be reflected in its Cargo target and runner/CI configuration, not just its prose label. +追加前に既存テストを確認し、各規則の主たるテストと接続を証明するテストを特定する。結合テストで局所的な全入力表を繰り返さない。バグ修正では既存テストを強化するか、継続的に守る規則を示す回帰テストを追加する。 -## Boundary-specific guidance +製品契約が変わったら、opt-in suiteを含め、影響するfixtureとassertionを更新する。製品feature限定の契約は同じfeatureで選択する。通常のテストを無関係なテスト専用featureや環境変数で隠さない。 -### CLI +DB・socket・子プロセスが必要なだけではopt-inにしない。特殊なtool・権限・platformが必要なら、その前提と実行選択を定義する。この例外も自動的にE2Eを意味しない。target、CIの選択、timeout、cleanup、診断を共有harnessとrunnerの設定に反映する。 -Parser tests must focus on user-visible command contracts: accepted argv shapes, target precedence, config defaults, and clear rejection. Do not assert that a parser took a particular private branch. - -Help tests should prevent stale public surface: command names, binary names, config locations, and removed shims. They should not duplicate the entire help text unless the exact text is the compatibility contract. - -### Environment-backed startup configuration - -Environment lookup belongs at startup, before operational logic begins. Materialize an explicit environment snapshot, resolve it into typed configuration once, and pass that configuration inward. - -Test parsing, precedence, missing values, and invalid values against the explicit snapshot. Do not repeatedly exercise the same environment lookup through unrelated logic tests, and do not mutate the test process environment. - -### Workspace Server and Runtime APIs - -Route tests should use HTTP-like requests against the router and assert status plus JSON body. A direct handler test is appropriate only when route wiring is irrelevant and the handler is the boundary. - -Protocol tests should assert observable event/order/state semantics, not internal channel choreography. - -### SQLite authority - -Migration tests must assert the current schema version and the persisted behavior introduced by the migration. A schema version bump without a behavior test is insufficient when the migration adds a table, index, constraint, or compatibility transform. - -Authority mutation tests should prove durable state after re-read and audit/event side effects where the authority promises them. - -### Worker tools - -A Worker tool exposes an LLM-facing contract. Tests should cover: - -- exact tool name when the name is model-visible; -- required/optional schema fields and bounds; -- rejected malformed input; -- authority errors when Backend Workspace authority is required; -- bounded and useful output content; -- durable side effects for mutation tools. - -Do not expose local filesystem fallback in a tool test unless that fallback is the intended product contract. - -### Prompts and resources - -Prompt tests should assert inclusion/exclusion and feature ownership, not incidental whitespace. If exact wording is a product policy, keep the asserted snippet short and name the policy. - -## Execution assumptions for test authors - -An ordinary test is part of the normal Cargo test suite. It must be deterministic, order-independent, parallel-safe, hermetic, and fast enough for repeated local execution. - -If a contract exists only under a Cargo feature, gate the test with the same feature and place it in a target that Cargo actually builds for that feature. Do not hide ordinary product tests behind an unrelated test-only feature or environment variable. - -Opt-in, ignored, privileged, platform-specific, and E2E execution must be explicit design decisions. Their targets, selection conditions, prerequisites, timeouts, and diagnostics must be documented and configured. Ordinary process-spawning integration tests use the same shared-fixture isolation and cleanup rules; process spawning alone is not an opt-in condition. - -Commands and validation scope used while developing are repository workflow rules and belong in `AGENTS.md` or `development/validation.md`, not in this test-authoring policy. - -## Review checklist - -When reviewing or adding a test, ask: - -1. What product contract does this test protect? -2. Could Rust types alone already prove this? -3. Is this the right boundary: parser/API/store/tool/protocol/public library? -4. Would one failure point to one understandable rule? -5. Does the test avoid live user data, real data dirs, and incidental ids? -6. If this is a bug fix, did we encode the lasting rule or the original regression symptom? -7. Does the test avoid test-only environment variables and process-global environment access? -8. Is the test order-independent, parallel-safe, and synchronized without guessed sleeps? -9. Is any compatibility behavior backed by an explicit product decision? -10. Does the test use the prescribed owning suite and shared fixtures? If it assembles the product, is it in `tests/e2e` with a selected journey that narrower tests cannot prove? -11. Are the connection under test and its production implementations real, with substitutions outside the claim? -12. Does the test scope match the failure's likelihood and impact, including any recovery or fault-injection cases? -13. Does existing coverage already prove this rule, and have affected fixtures and assertions been updated for the current contract? -14. Are prerequisites, process/resource cleanup, execution selection, and CI responsibility explicit rather than inferred from a test label? +実行方法と結果の扱いは[検証規則](validation.md)に従う。未実行、skip、未配線のCIを成功の証拠にしない。 diff --git a/docs/development/server-runtime-auth.md b/docs/development/server-runtime-auth.md deleted file mode 100644 index 6058836eb..000000000 --- a/docs/development/server-runtime-auth.md +++ /dev/null @@ -1,91 +0,0 @@ -# Workspace ↔ Runtime 認証 - -Yoi の Remote Runtime 認証は Workspace ごとの署名 identity を authority とする。 -認証の成否は実際の HTTP / WebSocket request ごとに決まり、保存済みの「verified」状態や過去の接続テスト結果を前提にしない。 -Server-global な署名鍵や Runtime 側の trusted-Server catalog は使わない。 - -## Authority - -- Server DB は Workspace ごとの signing identity と Runtime 接続設定（binding）を保持する。 -- binding は endpoint、固定した Runtime public key/fingerprint、`binding_id`、authentication mode、管理者による失効を保持する。Workspace key や Runtime 側の trust ID のコピー、認証成功状態は保持しない。 -- Runtime は管理者が `trust-workspace` で受理した `WorkspacePublicIdentityBundle` を保持する。 -- bundle は `workspace_id`、Workspace `key_id`、Workspace public key/fingerprint、Backend URL を固定する。 -- Server → Runtime の各 HTTP / WebSocket request は、その時点の Workspace signing identity で短命な capability token を発行する。署名時には claims の Workspace・key ID・fingerprint が選択した identity と一致することを検証する。秘密鍵materialがない場合は fallback せず拒否する。 -- Runtime は現在の明示的な Workspace trust、署名、request method、`path_and_query`、body digest、permission、Workspace、Runtime、必要に応じ Worker、key ID/fingerprint、expiry、JTI を検証する。過去の handshake や verification record は不要である。 -- Runtime → Server の source proof は Runtime identity で署名し、対象 Workspace と bundle の Backend URL を audience に固定する。この proof の契約は変わらない。 -- Server は現在の Workspace Runtime binding、Runtime public key、Backend public URL、request target、body digest、permission、expiry、replay state を検証する。 - -旧 Server identity/trust 管理 command と旧 Runtime-side Server trust command、旧 Runtime auth key flags は廃止済みである。これらに相当する Server-global trust を fallback として使ってはならない。 - -## Provisioning - -Runtime 側の Workspace trust 登録と Server 側の Runtime 接続設定は独立した管理操作である。Runtime が発行する `trust_id` を Server にコピーする必要はない。 - -1. Runtime identity を初期化する。 - - ```sh - yoi-runtime identity init --runtime-id - yoi-runtime identity show - ``` - -2. Workspace owner が Workspace identity settings から現在の Workspace issuer public bundle を export する。 -3. Runtime 管理者が bundle を確認し、Runtime に明示的に登録する。 - - ```sh - yoi-runtime trust-workspace add --bundle - yoi-runtime trust-workspace show --workspace-id - ``` - -4. Workspace owner が Settings > Runtimes に Runtime public bundle と endpoint を登録する。接続設定には Runtime key を固定するが、Workspace key や Runtime の trust ID は登録しない。 -5. 接続テストまたは通常操作を実行する。実際の Workspace-signed request が Runtime の現在の trust と照合される。接続テスト結果はその試行の診断であり、永続的な認証状態を作らず、以後の request の成功を保証しない。 - -同じ Runtime identity は異なる Workspace から独立して信頼できる。trust record、replay protection、binding、失効はすべて Workspace scope で評価する。 - -## Runtime auth file - -`runtime-auth.toml` は Runtime identity と Workspace issuer records のみを authority とする。 -旧 Server trust entry は読み飛ばされ、以後の identity / `trust-workspace` 更新時に書き戻されない。旧 entry を残しても認証には使用されない。 - -`trust-workspace` の file store は次を fail closed で検証する。 - -- 最大 8 MiB -- 最大 4,096 records -- exact Workspace / Runtime identity -- `key_id` と public key fingerprint、管理 record の opaque `trust_id` -- normalized Backend URL -- 同じ `key_id` に異なる鍵materialを割り当てないこと。実際の置換・失効は新しい `trust_id` を発行するが、この ID は request token や Server binding に含めない -- list は `offset` / `limit` 必須で、1 page 最大 100 records - -## Local token - -`--local-token` は明示的な local Runtime 呼び出し専用であり、Remote Workspace binding の代替ではない。Workspace issuer auth が有効な Remote Runtime request は Workspace capability token を使う。 - -## Key 変更と失効 - -Workspace signing identity は各 request の発行時に現在の authority から選択する。Workspace public key が変更された場合、Runtime 管理者が現在の public bundle を明示的に `trust-workspace replace` で受理する必要がある。Workspace key のコピーを更新するためだけに Server の Runtime 接続設定を置き換える操作は不要である。これは Workspace key rotation の新しい production API を定義するものではない。 - -Runtime 管理者は `trust-workspace revoke` で Workspace trust を失効できる。Runtime は現在の trust と一致しない key、失効済み trust、expired token、replayed JTI を拒否する。同じ key の再登録も管理者の明示的な操作で行い、request の自己申告による trust 登録は行わない。 - -Runtime key や endpoint の変更は Server の接続設定を明示的に置き換える。Server の Runtime cache と authorizer は現在の接続設定（endpoint、Runtime public key/fingerprint、`binding_id`、authentication mode、管理者による失効）と照合し、replaced/revoked binding を使う stale client を拒否する。接続設定の一致は送信先の選択であって、認証成功の証拠ではない。 - -接続先が到達不能・認証失敗でも、owner は観測した `binding_id` を指定して接続設定を修復できる。取得できた active Worker は置換を遮断するが、Worker 一覧の取得失敗を「idle の証明」とは扱わず、認証成功を修復の前提にもしない。接続設定の置換は Worker の停止・削除ではなく、Runtime 削除の資源確認とは別の操作である。 - -## 保存形式の移行 - -Server schema90 は既存 binding の endpoint、Runtime key、binding ID、明示的失効を保持し、古い認証状態と検証記録を履歴専用 archive に移す。履歴は要求の認可には参照しない。既存 Workspace identity や接続設定の作り直しは不要である。 - -Runtime の旧形式の Workspace trust は、検証済みの公開鍵と管理者の Active/Revoked 設定を保持して読み込む。形式の移行だけでは失効させず、すでに保存されている Revoked 設定を自動で解除することもしない。旧 verification file は認可に使用しない。消費済み token の replay 記録は引き継ぎ、同じ鍵の再登録でリセットしない。 - -API と capability claims の旧 handshake 契約は維持しないため、Backend・Runtime・Web は対応する版へ更新する。接続テストは更新後の実認証を確認するものであり、登録を有効化する手順ではない。 - -## 運用確認 - -Remote Runtime を設定した後は次を確認する。 - -1. `yoi-runtime trust-workspace show --workspace-id ` が期待する active bundle を表示する。 -2. Workspace Settings の Runtime 接続設定が期待する endpoint と固定 Runtime key を示し、失効していない。 -3. 接続テストの今回の結果を確認し、Runtime ping、Worker list/create、`worker.protocol` subscription が実際の Workspace-signed request で成功することを確認する。保存済み「verified」表示や verification evidence は確認項目ではない。 -4. wrong Workspace、wrong Runtime、wrong target/body、expired token、revoked Runtime trust、replaced/revoked binding、replayed JTI が拒否される。 -5. Runtime → Server source proof が configured Backend public URL audience と一致し、spoofed headers だけでは認証されない。 - -Server / Runtime の再起動は live reload ではない authority 変更を反映するときだけ、通常の運用権限と migration gate に従って行う。実行中プロセスを開発 Worker が無断で停止してはならない。 diff --git a/docs/development/subjektiv-product-cutover.md b/docs/development/subjektiv-product-cutover.md deleted file mode 100644 index 78c297090..000000000 --- a/docs/development/subjektiv-product-cutover.md +++ /dev/null @@ -1,98 +0,0 @@ -# subjektiv product cutover runbook - -This runbook defines the product and operator boundary for moving a Workspace from legacy single-document Memory to subject-scoped subjektiv Memory. The cutover changes which product path is active; it does not reinterpret legacy tool or API inputs, migrate legacy records into subjektiv, or make historical Worker attribution into current authority. - -T-672 documents and validates the contract only. It does **not** authorize or perform a reset in any real environment. - -## Authorities and persistence - -Two independent Memory systems exist during the transition: - -| Product path | Persistence owner | Durable content | Status | -| --- | --- | --- | --- | -| Legacy Workspace Memory | Workspace Server control-plane database | one Workspace document, staging candidates, and staging resolutions | legacy/deprecated; retained only for compatibility and explicit reset | -| Subject Memory | Server-managed `FeatureStorage`, in the Workspace-scoped `subjektiv` Feature database | subjects, immutable candidate and resolution records, confirmed Memory changes, Session attribution, surface generation records, and immutable surface snapshots | current product path | - -The shared `memory` crate and Worker Features do not own either database. `FeatureStorage` owns the subjektiv SQLite lifecycle (path, connection, migration, backup/restore, shutdown, and Workspace deletion), while the subjektiv repository owns its schema and domain transactions. The legacy control-plane tables are not imported or dual-written into subjektiv. - -Every subject operation is bounded first by the authenticated Workspace and then by an explicitly authorized subject. A subject ID is a selector, not a credential. The Host derives subject scope from authenticated Runtime/Worker source proof and current ownership of the opaque `subjektiv:` Worker key; model-visible requests do not supply or expand that authority. - -The current Worker remains Yoi's keyed-singleton responsibility. The subjektiv database records immutable historical Worker and Session attribution, but it contains no second `current_worker_id` mapping and historical attribution is never used to choose or authorize the current Worker. Stopping or replacing a Worker does not retire its subject, and retiring a subject does not stop a Worker. - -Workspace Memory settings remain Workspace-owned configuration. `feature.subjektiv` in a Profile is reusable policy, not subject authority: it remains inert for ordinary Ticket/Orchestrator Workers and Internal SubWorkers. Only a Backend-attested subject Worker (`subjektiv:`) or dedicated subject consolidator receives the trusted settings snapshot that activates subjektiv tools, resident load, and extraction. The snapshot is not subject identity and cannot be authored by a Profile, Browser, or model; Reviewer children never inherit the parent's attachment or run independent extraction. - -## Session, Memory, and correction semantics - -- **Session attribution and recall:** only a committed primary-Worker Session with Host-recorded subject attribution enters subject Session discovery. Attribution is immutable and idempotent. Recall reads the retained or committed-archive Session projection under current Workspace, subject, observation, and retention authorization; it does not copy Session bodies into subjektiv, infer attribution for older Sessions, restore a Worker, or turn recalled tool output into `HumanInput`. -- **`SubjektivMemoryRemember`:** stages a new candidate; it does not directly create confirmed Memory. Explicit committed evidence is resolved against the Session capture. A non-preference request without an entry is staged only after its tool call commits, using the committed call as model-origin evidence. An uncommitted operation creates no candidate. -- **`SubjektivMemoryProposeChange`:** stages a candidate with typed target metadata: exact Memory ID, `expected_change_id`, intent, and change reason. It does not mutate the target or silently retarget to a newer change. -- **Candidate consolidation:** a restricted, subject-bound consolidator compares an immutable pending candidate with confirmed Memory and makes one idempotent atomic decision. An applied decision writes the new confirmed change, exact provenance and affected change references, and the immutable candidate resolution in one subjektiv transaction. Non-applied dispositions are explicit (`discarded`, `invalid`, `duplicate`, or `already_covered`). A failed or empty model run is not a disposition, and surface generation failure cannot undo an already committed decision. -- **Corrections:** refinement or correction of the same experience appends the new immutable change under the same Memory ID using an `expected_change_id` check. Stale proposals remain unresolved/conflicted and are never rebased automatically. Retraction is terminal; a later corrected experience receives a new Memory ID and may derive from the old fixed change. Old changes, candidates, resolutions, provenance, and surface snapshots remain immutable history. - -Confirmed Memory is useful subject context, not authorization and not exact implementation authority. Tickets, Objectives, repository state, Git history, and append-only Session records remain the exact sources for implementation state. - -## Resident surface and history - -A subject surface is a bounded rendering of current active confirmed records, not another Memory authority. The injection contract is: - -- **ready:** inject the current non-empty snapshot only when it was built from the subject's exact current `memory_fingerprint`; -- **ready-empty:** explicitly record that a successful current generation produced no Markdown and inject no Memory prose; do not conflate this with missing or failed generation; -- **ungenerated:** inject nothing and report no current surface; -- **stale:** inject nothing until regeneration publishes a snapshot for the current Memory-input fingerprint; -- **failed:** inject nothing; retain the failure for diagnosis and retry rather than presenting it as an empty Memory. - -A new Worker receives the current ready or ready-empty result in its initial durable prompt construction. A restored Worker does not rewrite its persisted prompt or prior Session history. It appends one durable restore-boundary system item that supersedes prior resident summaries with the latest ready surface, explicit ready-empty state, or a no-current-surface tombstone. Later turns in the same restored process do not append the boundary again. This preserves append-only history: no stale summary, failed generation, correction, or cutover operation edits previously committed Session entries or immutable subjektiv records. - -## Legacy/deprecated compatibility boundary - -The existing legacy tools (`MemoryQuery`, `MemoryReadDocument`, `MemoryUpdateDocument`, and the legacy `MemoryStaging*` operations) and `/api/w/{workspace_id}/memory...` APIs retain their established Workspace-document meanings while they exist. They are legacy/deprecated surfaces, not aliases for subject Memory. Do not silently route an old request to subjektiv, infer a subject from old input, or change an old input field's meaning. - -New product work uses the explicit subject/session/confirmed-Memory/change-history/source and surface operations. Old clients must be stopped or upgraded before reset; a post-reset legacy read may observe the legacy empty/default state, but it must never expose subjektiv records through the old schema. - -## Explicit legacy reset contract - -The legacy reset is a destructive, operator-triggered API operation scoped to one authorized Workspace. Startup, Profile resolution, enabling `feature.subjektiv`, Worker restore, consolidation, and ordinary reads must never trigger it implicitly. - -In one control-plane database transaction, the reset deletes all of the following legacy rows for the Workspace: - -1. the legacy Memory document; -2. all legacy Memory staging candidates; -3. all legacy Memory staging resolutions. - -The operation is all-or-nothing. A failure before commit leaves all three legacy sets unchanged. A retry after success is idempotent: absence of those rows is success and must not delete anything outside the legacy set. - -Legacy `AppendAudit` was acknowledgement-only and created no dedicated durable Memory-audit rows, so there is no fourth legacy audit table to erase. General Session logs and diagnostics remain retained under their own policies. - -The reset must retain: - -- Workspace Memory settings content; -- Sessions, Session logs, archives, and diagnostics; -- Tickets, Objectives, and their audit/history; -- Knowledge and Skills; -- the entire Workspace-scoped subjektiv Feature database, including subjects, candidate/resolution history, confirmed changes, attribution, generation runs, failures, and surface snapshots; -- all other new-product and unrelated Workspace data. - -The reset is not a subject deletion API, a Workspace deletion API, a settings reset, a retention override, or a data migration. - -## Operator procedure - -Perform the cutover in this order: - -1. **Stop legacy activity.** Disable legacy `feature.memory` Profiles and automatic extraction/consolidation, stop or drain old Workers, and block Browser/API clients from writing `/memory` routes. Wait for in-flight legacy writes to finish. Do not run legacy and new automatic extraction for the same Worker. -2. **Back up while writers are stopped.** Capture the control-plane database containing legacy Memory and a `FeatureStorage` backup containing subjektiv. Record the Workspace and backup identities and verify the backup artifacts. Feature backup is internally consistent, but there is no cross-database snapshot instant; stopping both writer sets is what makes the pair a usable cutover checkpoint. -3. **Trigger reset explicitly.** An authenticated operator invokes the legacy reset API for the intended Workspace. Do not automate this as a startup migration or infer consent from `feature.subjektiv` enablement. -4. **Verify both sides.** Confirm the legacy document, staging, and resolutions are absent (or return their defined empty/default projection); confirm Workspace Memory settings and unrelated control-plane data remain; reopen/read the subjektiv store and verify the expected subject, confirmed changes, attribution, and current surface state are unchanged. Verify a subject Worker resolves through the `subjektiv:` singleton and has only the intended new tools. -5. **Resume only the new path.** Start subject Workers with `feature.subjektiv`, keep legacy write surfaces disabled, and monitor typed surface/Session diagnostics. Do not treat an ungenerated, stale, or failed surface as ready-empty. - -### Failure and recovery - -- If reset returns a definite failure, the transaction rollback means legacy document, staging, and resolutions remain together. Correct the cause and rerun the same explicit operation. -- If communication is lost and the result is ambiguous, keep legacy writers stopped, inspect all three legacy sets, and rerun. Idempotency makes a committed first attempt safe to repeat and atomicity prevents a valid partial state. -- If verification of retained data or the new path fails, keep both write paths stopped. Restore the coordinated pre-reset backups according to the normal control-plane and `FeatureStorage` restore procedures, verify them, and only then resume. Never "recover" by deleting or recreating the subjektiv database. -- Preserve Session logs and diagnostics throughout; they are recovery evidence, not reset targets. - -## Validation boundary - -The validation supporting this cutover proves substrate-level store shutdown/reopen behavior and restoration of a Worker object with its persisted history. It does **not** claim a restart of the Server or Runtime process, behavior during communication loss, real-model consolidation or surface quality, or a full-process end-to-end cutover. Those require separate operational validation before a production reset. - -No real environment reset is performed or authorized by T-672. diff --git a/docs/development/ui-ux/application-architecture.md b/docs/development/ui-ux/application-architecture.md deleted file mode 100644 index c5312c1dd..000000000 --- a/docs/development/ui-ux/application-architecture.md +++ /dev/null @@ -1,104 +0,0 @@ -# Workspace Web Application Architecture - -この文書は、Workspace Webのapplication shell、Header、Sidebar、nested -layoutのcomposition authorityを定義する。Navigation -IAは[`product-ux.md`](product-ux.md)、視覚とinteractionの共通規則は[`design-language.md`](design-language.md)、具体的なsource -pathとCSS -ownershipは[`../../../web/workspace/README.md`](../../../web/workspace/README.md)を参照する。 - -## Application shell - -root layoutが唯一のapplication shellを所有する。 - -- Sidebar frame -- global fallback navigation -- Header frame -- main content -- SidebarとHeaderのoverride context - -page routeはapplication shell、global Header、Sidebar、fold -controlを再実装しない。Pageのmarkupは標準shellのmainへ配置するcontentから始める。 - -## Header composition - -Headerはroot shellが一度だけrenderする。Nested -layoutは`HeaderOverride`を通じて現在scopeのlocation contentを登録する。 - -- page componentはHeader frameを作らない。 -- 最も深いactive overrideを表示する。 -- layoutが破棄されたら直前のoverrideへ戻る。 -- overrideがなければroot fallbackを表示する。 -- route titleをmain contentへ複製しない。 - -## Sidebar composition - -### Root - -root shellは`SidebarFrame`と`GlobalSidebar`を一度だけrenderし、root -`SidebarController`を提供する。Overrideがない場合、`GlobalNavSections`がfallbackになる。 - -### Workspace - -Workspace layoutは、親のroot -slotへ`WorkspaceSidebar`を登録する。同時に、さらに深いlayoutが使う新しい`SidebarController`をcontextへ設定する。 - -`WorkspaceSidebar`はWorkspace shortcut headerを保持し、その下へchild -slotをrenderする。Child overrideがない場合だけ、Workspace navigation -fallbackを表示する。 - -### Settings - -Settings layoutは、Workspace layoutが提供したchild -slotへ`SettingsSidebar`を登録する。Workspace shortcut headerは残り、Workspace -navigation fallbackだけがSettings navigationへ置き換わる。 - -さらに深いscopeも同じcontractを使う。 - -## Override stack - -`HeaderOverride`と`SidebarOverride`は、layout lifecycleにbindしたLIFO -stackとして扱う。 - -1. Layout mount時にparent controllerへregisterする。 -2. Controllerは最も新しいactive registrationを表示する。 -3. Nested layoutはchild controllerを新しく提供する。 -4. Layout destroy時に自分のregistrationをremoveする。 -5. 一つ前のregistrationがあれば復元する。 -6. Stackが空ならfallbackへ戻る。 - -同じregistrationを複数回removeしても結果を変えない。古いcleanupが新しいactive -registrationを削除してはならない。 - -## Ownership - -- `SidebarFrame`だけが`aside`、外枠、scroll領域、fold controlを所有する。 -- `GlobalSidebar`、`WorkspaceSidebar`、`SettingsSidebar`は各scopeのcontentを所有する。 -- `SidebarOverride`を登録できるのはscopeを定義するlayoutだけとする。 -- page component、一時的なwidget、dialogはSidebarを差し替えない。 -- child layoutは親navigationをcopyせず、自分のsidebar layerだけをregisterする。 -- Global、Workspace、Settingsを一つのsnippetへ平坦化しない。 - -## Data authority - -Sidebar compositionはnavigation -structureを決めるが、Workspace、permission、Worker -stateのauthorityにはならない。 - -- Workspace identityとpermissionはBackend projectionを使う。 -- Worker listとstateはWorkspace protocol projectionを使う。 -- route pathやdisplay labelからauthorityを推測しない。 -- unavailable dataを架空の正常stateで置き換えない。 -- permissionにより利用できないnavigationは、権威あるpermission - projectionに基づいて除外する。 - -## Design-lab - -認証不要のstatic design-labはroot shellを再利用し、実際と同じnested -controllerとoverride stackを構成する。 - -Live Workspace protocolへbindするresource listだけは、同じscope layer内のstatic -fixtureへ置き換えてよい。Static fixtureへ別scopeのnavigationを混ぜず、production -componentと同じ順序、state表現を使う。 - -Design-lab固有のswitch controlでSidebarを差し替えず、route -hierarchyによってlayout lifecycleを発生させる。 diff --git a/docs/development/ui-ux/product-ux.md b/docs/development/ui-ux/product-ux.md deleted file mode 100644 index ab7b44408..000000000 --- a/docs/development/ui-ux/product-ux.md +++ /dev/null @@ -1,156 +0,0 @@ -# Workspace Web Product UX - -この文書は、Workspace Webで扱うresource、navigationのinformation -architecture、resourceごとの主要taskを定義する。見せ方の共通文法は[`design-language.md`](design-language.md)、application -shellとSidebar -slotは[`application-architecture.md`](application-architecture.md)を参照する。 - -## Product position - -Workspace -Webは、Workspace内の権威ある状態を確認し、注意が必要な箇所を判断し、一度に一つの操作を行うための作業面である。Marketing -dashboard、analytics画面、Backend DTOのraw viewerにはしない。 - -基本的な操作の流れは次とする。 - -```text -scopeを選ぶ -→ resourceを選ぶ -→ current stateを確認する -→ 次の有効なactionを行う -→ 結果を確認する -``` - -## Resource - -主要resourceは次のとおり。 - -- Workspace -- Ticket -- Objective -- Merge Request -- Memory -- Worker -- Workdir -- Runtime -- Repository - -通常の表示とURLにはcanonical resource keyを使う。内部UUIDをlabelやfallback -linkとして露出しない。 - -## Navigation IA - -### Global - -Workspaceを選ぶ前のscopeを扱う。 - -- Workspace catalog -- Account -- Device Login -- Workspace作成 - -### Workspace - -一つのWorkspace内で日常的に扱うresourceを置く。 - -1. Tickets -2. Objectives -3. Merge Requests -4. Memory - - Subjects - - Surface - - Committed Memory - - Changes - - Sources -5. Workers - -### Settings - -Workspaceの管理と構成を扱う。 - -- Runtimes -- Configuration Sources -- Repositories -- Repository Access -- Profile Sources -- Workspace Identity - -`Runtimes`、`Repositories`、`Repository Access`をWorkspace primary -navigationへ混ぜない。permissionにより利用できないSettings itemは表示しない。 - -## Page model - -各routeは、主要な目的を一つだけ持つ。その目的は「Ticketを選ぶ」「このWorkerを確認する」「Repository -accessを設定する」のように、動詞と対象を一つずつ使って表現できなければならない。 - -Workspace pageは、次のいずれかを基本形とする。 - -1. **一覧またはboard** — resourceを探し、注意が必要な項目を見つける。 -2. **詳細** — 一つのresourceを理解し、現在実行できる次の操作を行う。 -3. **formまたはeditor** — 範囲の明確な設定変更を行う。 -4. **Console** — 一つのWorkerを操作しながら、履歴とcurrent controlを確認する。 - -同じ目的を支える場合に限り、一つのrouteで複数の基本形を組み合わせてよい。 - -## Product pattern - -### Tickets - -- boardを一つの横方向の作業面として扱う。 -- laneはworkflow stateを表し、隣接laneとの関係を維持する。 -- Ticketはlane内でDesign Languageの`Resource row` patternを使う。 -- Ticket key、title、優先判断を変えるmetadataだけを表示する。 -- blockerとattentionはtextでも示す。 -- narrow viewportでは、bounded horizontal scrollでlane比較を維持する。 - -### Resource index - -- 同じfieldをresource間で比較する場合、flat listまたはtableを使う。 -- 人が読むnameとcanonical keyを最初のcolumnに置く。 -- columnは判断に必要な値だけに絞る。 -- provider detail、変更ID、digestはdetailまたはtechnical disclosureへ移す。 - -### Resource detail - -- identity、current state、attentionを最初に置く。 -- narrativeまたはhistoryをprimary領域に置く。 -- relation、target、review、workflowが別のauthorityを持つ場合、それぞれを独立したsectionにする。 -- raw metadataはtechnical disclosureへ置く。 - -### Settingsとform - -- 一つのSettings routeは一つの設定関心事を所有する。 -- edit actionの前または同時に、現在有効な値を示す。 -- 一つの判断に必要なfieldをgroup化する。 -- narrow viewportでは一列にする。 -- form内でfilled actionにするのはsave actionだけとする。 - -### Memory - -- Memoryの入口はWorkspace全体のlegacy `Document` / `Staging`ではなく、明示的に選択したsubjectとする。 -- subject detailではcurrent Workerをkeyed-singleton projectionとして表示し、historical Worker／Session attributionをcurrent authorityとして扱わない。 -- `Surface`はresident contextのcurrent stateを表示する。`ready`、`ready-empty`、`ungenerated`、`stale`、`failed`を区別し、空表示やrequest failureへまとめない。 -- `Committed Memory`はcurrent confirmed Memoryの一覧とdetailを持つ。変更履歴はappend-only viewとして分離し、過去の変更内容をcurrentへ暗黙fallbackしない。 -- `Sources`は候補、resolution、exact `{memory_id, change_id}` refs、Host-recorded Session attribution、閲覧可能なSession sourceをたどるviewとする。source IDをpermissionとして扱わず、現在のWorkspace／subject authorizationを毎回適用する。 -- legacy `/memory` の`Document` / `Staging` viewはdeprecated compatibility surfaceとして明示し、新しいsubject viewへ同じinputを別の意味で転送しない。 -- Markdown本文とdetail presentationはT-661の共有rendererを再利用する。Memory route独自のrenderer、application shell、Header、Sidebarを再実装せず、既存shellのmain contentとnested navigationを保つ。 - -### Worker Console - -- transcriptをprimary surfaceとする。 -- direct SubWorkerを表示している場合も、Composerとrun - controlは親Workerへbindしたままにする。 -- OverviewとNormalはpresentationだけを変え、authorityを変えない。 -- command outputはterminalに近いcompactな表示とし、prose messageから分離する。 -- live stateは権威あるWorker snapshotから導出する。 -- connection stateやtransient eventでcurrent stateを置き換えない。 -- hidden reasoningとraw system promptはdebug modeでも表示しない。 - -## Stateとpermission - -- 存在しない、閲覧できない、読み込みに失敗したWorkspaceを別の状態として扱う。 -- 別のWorkspaceへ暗黙に切り替えない。 -- owner-only mutation controlをnon-ownerへ表示しない。 -- read-onlyの場合は、有用なread viewを残す。 -- Workerのdomain stateとWebSocket connection stateを混同しない。 -- Mergeなど不可逆または重要なoperationは、対象と結果を明示してから実行する。 diff --git a/docs/development/ui-ux/subject-memory-information.md b/docs/development/ui-ux/subject-memory-information.md deleted file mode 100644 index 9040405df..000000000 --- a/docs/development/ui-ux/subject-memory-information.md +++ /dev/null @@ -1,135 +0,0 @@ -# Subject / Memory information audit (T-692) - -This document records T-692's UI decisions and validation at the commits cited below. Its counter values describe that historical implementation, not the current API. Obsolete UI labels are paraphrased by purpose below, not quoted as current copy. Current Memory uses immutable `change_id` references, `memory_fingerprint` and `built_from_memory_fingerprint`; see [the current store contract](../../design/subjektiv-store.md). Historical validation results are not new executions. - -## Decision and scope - -The previous Subjects index exposed a committed-Memory update counter alongside identity, state and -update time; Subject detail led with technical metadata and raw surface states. -These numbers require implementation knowledge before they help a user choose a -Subject or understand what is available. The revised hierarchy starts with -identity, connection, committed records and generated-context availability. - -This is a presentation change, not a change to storage, freshness comparisons, -permissions, generation, or Worker history. T-688's ordinary Worker launch and -T-689's independent Subject creation remain the actual operation paths. T-693 -owns editable user-managed behavior: that content must remain distinct from this -page's context generated from committed Memories. A successful read or a current -surface does not attest what is already in any running Worker's history. - -## Field classification and before/after priorities - -| Information | Classification and reason | Presentation decision | -| --- | --- | --- | -| Subject role and ID | Normal identification. Similar or identical roles need a stable identifier for selection and comparison. | Keep role as heading and the complete wrapping ID visible in index and detail. | -| Subject active/retired state | Normal operation decision: whether a new Worker connection can use the Subject. | Replace raw `active` with `Available`; keep `Retired`. Detail explains connection eligibility, not an assertion that a Worker is running. | -| Current Worker | Normal connection decision, derived from the existing Backend singleton projection. | Promote `Connected` / `Not connected` and display name. Do not infer ownership from historical store counter or claim all Workers received current Memory. Worker start remains the ordinary launch form. | -| Committed-Memory presence | Normal reading decision, independent from surface generation or candidates. | Show `Available` / `None` only on the successful first bounded page. On later pages say `N on this page`; errors say `Unavailable`. Never infer counts from an update counter or scan all pages. | -| List counts | Normal navigation context, not a store-wide statistic. | Label `N shown` and `more available`, preserving existing bounded cursor pagination. | -| Resident availability | Normal context decision. A missing surface is not missing committed Memory. | `Ready to use`, `Ready, no context`, `Not generated`, `Needs refresh`, `Generation failed`, and request `Unavailable` are distinct. These describe the current generated store artifact, not content truth or Worker-history propagation. | -| Resident body and generated time | Normal content/read freshness context. | Keep the shared Markdown renderer, generated date and exact snapshot reference count. Ready-empty explains that a valid snapshot has no body; stale/failed do not display a partial or old snapshot. | -| Historical Subject store counter | Diagnostic only: internal committed-Memory change number, not record count, schema version or quality score. | Remove it from the index's normal columns. Retain the exact value under detail's `Technical details`, with its meaning explained. Backend comparisons are untouched. | -| Subject update time | Secondary metadata; not a context-injection or generation completion timestamp. | Label `Last changed`; on detail put it with Subject diagnostics rather than competing with the availability summary. | -| Snapshot ID and historical built-from counter | Diagnostic correlation, distinct from individual Memory history entries. | Retain under `Surface sources and diagnostics`, labelled the historical generated-from counter label and explained. | -| Surface Memory references | Detailed audit/reachability. | Preserve reference IDs, immutable history references and Memory links in the disclosure. | -| Memory claim, kind, lifecycle state, excerpt and body | Normal content decision. Active, resolved and retracted are not interchangeable. | Retain list scanning and body reading. Keep the Markdown renderer and current/historical distinction. | -| Individual Memory history entries | Detailed content comparison, also useful for identifying a selected version. | The historical UI labelled each Memory history entry separately from Subject counters. Preserve immutable history, selected history-entry links and history pagination. | -| Candidate provenance, evidence, origin, source refs and derivations | Detailed audit, required to understand why a record exists. | Retain IDs, ranges, summaries, excerpts, origins and derived-history-entry links. Counts say `shown` except where the API provides exact per-candidate totals. Preserve body and evidence continuation links. | -| Raw field names / duplicated general update-counter terminology | Unnecessary in the normal hierarchy; multiple unrelated counters are misleading. | Replace bare counter labels and raw surface enums with purpose-specific labels. Do not delete API fields, stored information, diagnostics or their detailed access. | - -No diagnostic datum was removed from Subject detail, surface detail or Memory -provenance/history. The index no longer shows a historical store counter, but the exact -value remains one click away in Subject detail. No invented regeneration, -editing or recovery action was added; `Needs refresh` does not mean an update job -is running. Creation still creates only a Subject, and selection still uses the -existing ordinary launch contract. - -## Candidate-only verification - -Candidates are not committed records and are not an input to this page. The -Subject loader performs just three bounded reads: Subject, resident surface and -`/memories?limit=100[&cursor=...]`. It does not read or count staging records. - -The existing real-store regression -`subjektiv::tests::candidate_decisions_reject_conflicts_and_roll_back_partial_writes` -stages a candidate and attempts an invalid apply. It asserts the candidate has no -resolution and the committed Memory list remains empty. The component regression -`candidate-only Subjects show no committed Memories, not an inferred candidate count` -uses that read projection (historical store counter 0, ungenerated surface, empty successful -committed list) and checks `None`, `No committed Memories yet.`, `Not connected` -and `Not generated`, with no resident body. Thus a staged candidate does not turn -into a committed-Memory count or a claim that context has been generated. - -The store test proves the persistence boundary; the component test proves the -rendering of its public projection. These are separate boundary tests, **not** a -live browser-to-Backend candidate-creation E2E. The existing `ungenerated` capture -represents the same visible projection, but does not itself stage a candidate. - -## Visual review and validation evidence - -The restored implementation is in `fbb09e65` and `4cf6c130`; this report/test -supplement does not change rendered markup, styling, copy or data contracts. - -- Scenarios: `tools/web-ux/scenarios/memory-subjects-{light,dark}.json`. -- Before: `target/web-ux/t-692-before-{light,dark}` at `cc979874`. -- After: `target/web-ux/t-692-final2-{light,dark}` at clean `4cf6c130`. -- Comparisons: `target/web-ux/t-692-final2-{light,dark}-comparison`. -- Browser interactions: `target/web-ux/t-692-browser-interactions`. -- Persona: synthetic Workspace fixture; permitted creation and intentional - permission-rejection responses. No real account or private store data used. -- Viewports: 1440, 768, 390 and 320px; light and dark themes. -- States: connected/unconnected, active/retired, successful empty lists, ready - long Markdown, ready-empty, ungenerated, stale, failed, unavailable HTTP503, - create form and create permission error; expanded technical disclosures. -- The takeover implementer personally inspected individual before/after images - for index at 768px, representative detail at desktop/mobile and Memory detail, - then ready-empty mobile, ungenerated narrow, stale desktop, failed mobile, - endpoint error at 320px, mobile surface sources, compact Subject diagnostics - and mobile permission error. Oversized contact sheets were rejected by the - image provider; individual viewport PNGs were inspected instead. -- Findings/result: primary state groups replace counters without losing IDs or - document access; long IDs wrap, status is textual rather than color-only, - mobile groups stack, diagnostics remain readable, and error state regions stay - separate. No new visual defect identified. **Pass for the audited scope.** -- Final bundles contain 64 captures each. `completed-with-errors` is expected: - intentional creation HTTP403, endpoint HTTP503/associated aborted requests, - and the deliberately failed surface's alert are retained as evidence. Healthy - capture states have empty error arrays; this is not a blanket claim that all - captures were error-free. Review-context diagnostics are empty. -- The focused production-shell browser test proves main-owned vertical scrolling - with wheel/keyboard/touch, 320px/mobile shell behavior, no page-wide overflow, - local table/code scroll, Markdown safety, creation/launch separation, bounded - pagination and Memory/source/history-entry reachability. - -Independent review at source `4cf6c130` ran root `cargo check`, -`cargo fmt --all -- --check`, diff hygiene, Web check (0 errors/warnings), 12 -Memory API tests, 452 full Web unit tests, 134 component tests (8 Memory tests), -production build, the focused production-shell browser test (1), and Web-UX unit -tests (16), all passing. - -The takeover implementer then validated this report/test supplement with: - -- `cargo test -p yoi-workspace-server --lib candidate_decisions_reject_conflicts_and_roll_back_partial_writes`: 1 passed. -- Root `cargo check`, `cargo fmt --all -- --check`, and `git diff --check HEAD`: passed. -- Targeted Memory component tests: 9 passed, including the added candidate-only projection test. -- Web `deno task check`: 0 errors/warnings; `deno task test`: 452 passed; - `deno task test:component`: 135 passed. -- Scoped `deno fmt --check` on the changed TypeScript test: passed. -- JSON audit of both final2 bundles: 64 captures each, exact clean source - `4cf6c130`, and no errors outside the deliberate permission/error/failed routes. - -The supplement changes only a document and a test, so it does not require new -rendered-UI captures. Rereview records and the final approved Ticket handoff link -this report to the new exact source. - -Known non-blocking validation diagnostics: component runs emitted incidental -localhost:3000 connection-refused messages while passing; the production build -warned about a large chunk. A broader Web-UX typecheck fails in unchanged -`tools/web-ux/src/processes.ts:44` on a `TextDecoderStream` generic incompatibility; -this report does not claim that broad check passed. - -No live-Backend browser interactions, real Worker model-request propagation, -regeneration, or server/runtime process E2E were performed. No Rust production -crate changed in T-692; the supplemental store test exercises an existing -contract without changing that contract. Backend counters, comparison logic, -storage schema and feature behavior remain intact. diff --git a/docs/development/ui-ux/visual-review.md b/docs/development/ui-ux/visual-review.md index 8d4fe726f..ac911f67d 100644 --- a/docs/development/ui-ux/visual-review.md +++ b/docs/development/ui-ux/visual-review.md @@ -1,272 +1,32 @@ -# Web visual review +# Web UI変更の目視確認 -この文書は、Web UIを変更する実装者が完了前に行うvisual -reviewの必須手順を定義する。Visual reviewは任意のpolish工程ではなく、type -check、linter、testと並ぶvalidationである。 +描画結果が変わる変更は、実装者自身が変更前後を目視する。type check、linter、testと別の検証であり、後段Reviewerや利用者に最初の発見を委ねない。画像生成やHTTP成功だけでは確認済みにならない。 -[`design-language.md`](design-language.md)はWorkspace -Webが満たす視覚とinteractionの共通規則を定義する。Product固有の構成は[`product-ux.md`](product-ux.md)を参照する。[`tools/web-ux/README.md`](../../../tools/web-ux/README.md)はcapture -toolの操作方法を定義する。この文書は、誰が、いつ、何を目視し、どの証拠を残さなければ実装完了とみなせないかを定義する。 +## 対象と条件 -## 実装者の責任 +page、layout、navigation、componentのmarkup・style、色・書体・余白、文言・状態表示、可変幅、keyboard操作、表示に影響するAPI projectionを変えた場合に行う。描画に影響しない文書・型・Backend変更だけなら省略できるが、影響しない理由を検証記録に残す。 -Visual -reviewは、後段のReviewerやユーザーへ最初の発見を委ねない。UIを変更した実装者自身が、同じ変更の中で次を行う。 +変更箇所に関係する条件を選び、都合のよい一画面だけで済ませない。 -1. 変更前をcaptureする。 -2. 変更後を同じ条件でcaptureする。 -3. screenshotとbrowser evidenceを自分で確認する。 -4. 発見した問題を修正する。 -5. 修正後を再captureして確認する。 -6. 変更と無関係な既存問題は、根拠を付けて分離する。 +- 権限やデータが表示を変える場合は、owner、non-owner、anonymous、read-onlyなど該当する利用主体を分ける。 +- 幅はdesktop `1440px`、狭い幅`768px`、mobile `390px`を基準とする。`320px`対応やbreakpoint付近が変更対象なら追加し、Sidebarなどを除いた実内容幅も確認する。 +- 色・背景・線・focus・状態・code・terminalを変える場合はlightとdarkを確認する。 +- 通常、空、読込み中、失敗、権限不足、長文・長い識別子、少数・多数の一覧など、該当するデータ状態を用意する。 -Agentが実装者の場合、そのAgent自身が`ViewImage`または同等のVLM入力で実画面を確認する。captureを生成しただけ、別のAgentへ画像を渡しただけ、HTTP -200を確認しただけでは完了しない。 +## 実施と確認項目 -## 適用範囲 +1. route、利用主体、幅、theme、データ状態、撮影点を固定する。既存scenarioを再利用し、不足分は同じ変更で更新する。 +2. 変更前を撮影して画像を開き、既存問題と変更対象を把握する。 +3. [Design Language](../../principle/design-language.md)と[UX設計](../../principle/ux-design.md)に沿って実装する。撮影専用のCSS・shellや、本番と異なる権限判定で見栄えを整えない。 +4. 同条件で変更後を撮影し、実装者自身が画像と`review-context.json`を確認する。Agentも`ViewImage`などで実画像を見る。 +5. 発見した新規問題を修正し、再撮影して確認する。既存の無関係な問題は根拠を付けて分離する。 -次の変更ではvisual reviewを必須とする。 +画像では、情報のまとまり、区切り線の役割、余白の大小、重複見出し、文字の強弱、折返し、欠落、overflow、主要操作の位置を確認する。DOM・表示・keyboardの順、focusの維持、色以外の状態表現、必須情報への到達も確認する。画素差の大小だけで合否を決めない。 -- page、layout、navigation、Sidebar、Header、Console、dialog -- componentのmarkupまたはstyle -- color、typography、spacing、border、radius、shadow、theme token -- copy、label、status、empty/loading/error/permission表示 -- responsive breakpoint、container、overflow、scroll -- 表示dataの追加、削除、並び替え、progressive disclosure -- icon、focus、keyboard interaction、motion -- UIへ影響するgenerated typeまたはAPI projection +`review-context.json`では、document status、可視エラー、console・page・requestエラー、accessibility snapshot、撮影点と画像hashを確認する。HTTP 200でも画面が権限エラーなら成功ではない。意図的な障害fixtureは予期したエラーと予期しないエラーを分けて判定する。 -型、Backend、CLI、文書だけの変更でrendered Web -UIが変わらない場合は省略できる。省略する場合は、UIへ影響しない理由をvalidation -evidenceへ一文で残す。 +## 証拠と合否 -## 必須の確認条件 +引継ぎにはscenario、変更前後のartifact、source、確認した条件、修正した問題、根拠付きの既存問題、最終の`pass | fail`を残す。必要な条件を自分で確認し、新規問題を修正し、意図しない描画回帰がない場合だけ合格とする。自動テストと目視確認のどちらも、他方の代替にはしない。 -変更したsurfaceに関係する条件だけを選ぶが、都合のよい一画面だけに限定しない。 - -### Persona - -権限やdataが表示を変える場合は、該当するpersonaを分ける。 - -- owner -- non-owner -- anonymous -- read-onlyまたは操作権限なし - -owner表示だけでpermission-aware UIを承認しない。 - -### Viewport - -最低限、次を確認する。 - -- desktop: `1440px`幅を基準 -- narrow: Sidebarとmainが競合する幅。現在は`768px`を基準 -- mobile: `390px`幅を基準 - -`320px`対応、table、long form、Console、Sidebar -foldなどが変更対象なら、その境界も追加する。media -queryのviewport幅だけでなく、Sidebarやpanelを差し引いた実際のcontent幅を確認する。 - -### Theme - -color、border、background、focus、status、code、terminalを変更した場合はlightとdarkの両方を確認する。片方のthemeで読めることを、もう片方のcontrastの証拠にしない。 - -### Data state - -該当するstateを明示的に用意する。 - -- representative data -- empty -- loading -- error -- permission deniedまたはread-only -- long title、long key、long error、複数行content -- collectionの最小件数と、実際に起こりうる多い件数 - -happy pathだけでcomponentを承認しない。 - -## 目視確認の観点 - -### 情報のまとまり - -- 同じ判断に使う情報がproximity、alignment、typographyによって一つのgroupに見えるか。 -- 別の意味を持つ情報が、十分な距離または一つの明確なboundaryで分離されているか。 -- 親子、peer、metadata、actionの関係が、DOMを読まなくても判別できるか。 -- card、background、border、shadow、余白を重ねて同じgroupingを重複表現していないか。 -- page title、Header、breadcrumb、section - heading、説明文が同じ意味を繰り返していないか。 - -### Separatorとborder - -画面上のすべての線について、何と何を区切る線か説明できなければならない。 - -- Headerの終端、peer row間、section開始、table - cellなど、線の役割が一つに定まっているか。 -- 近接した二本の線が同じboundaryを重複表現していないか。 -- 線の上下の余白から、その線が前後どちらのgroupに属するか分かるか。 -- すべてのsectionへ機械的に同じ線を置き、hierarchyを平坦化していないか。 -- borderがなくてもspacingだけで十分な箇所へ線を追加していないか。 - -### Spacingとdensity - -- 余白がtoken scaleに沿い、近い関係ほど狭く、別groupほど広くなっているか。 -- labelとvalue、headingとcontent、rowとrow、sectionとsectionの間に一貫した大小関係があるか。 -- 線の前後へ同程度の大きな余白を置き、boundaryの所属を曖昧にしていないか。 -- 初期viewportにprimary contentまたは主要操作が存在するか。 -- 空きすぎた領域が情報不足やhierarchyの弱さを隠していないか。 -- 密度を上げるためにfont、touch target、line heightを過度に縮めていないか。 - -### Wrapping、overflow、可変長content - -- 実際に起こりうる最長のlabel、title、key、ref、error、翻訳で確認したか。 -- 一行を前提とするcontrolが意図せず二行にならないか。 -- button labelが折り返されたとき、row heightとalignmentが壊れないか。 -- title、metadata、code、URLが隣接actionを押し出さないか。 -- truncateする場合、完全な値へaccessできるか。 -- horizontal - scrollはtable、board、terminalなど、その意味を所有する領域だけに閉じているか。 -- page全体へhorizontal overflowが発生していないか。 -- `overflow-wrap`だけで問題を隠さず、可読性と比較可能性を維持しているか。 - -### Responsive layout - -- DOM順、視覚順、keyboard順が一致しているか。 -- Sidebar、Header、primary contentの順序がmobileでも目的に合うか。 -- Sidebarだけで初期viewportを占有していないか。 -- breakpoint直前と直後の両方で、content幅とcontrol配置が成立するか。 -- viewport - queryだけでなく、containerの実幅に応じて一列化またはscrollを選べているか。 -- desktopの比較関係を、理由なくmobile cardへ変換していないか。 - -### Typographyとcopy - -- route titleが画面上で一度だけ表示されているか。 -- page purpose、componentの存在理由、設計意図を説明するmeta - copyが残っていないか。 -- proseが判断、warning、constraint、permission、recoveryのいずれかに実際に必要か。 -- heading level、font size、weight、colorが同じhierarchyで揃っているか。 -- metadataがbodyより強く見えていないか。 -- uppercase、letter spacing、monospaceを装飾目的で多用していないか。 - -### State、permission、action - -- loading、empty、error、permission denied、unavailableが見分けられるか。 -- statusがcolorだけに依存していないか。 -- 利用できないowner-only actionをdisabledで並べるのではなく、page - compositionから除外できているか。 -- filled primary actionが同じscopeに複数ないか。 -- destructive actionが通常のprimary pathと誤認されないか。 -- errorとrecovery actionが、失敗したsurfaceの近くにあるか。 - -### Interactionとaccessibility - -- keyboard focusが全controlで見えるか。 -- focus順が画面の作業順と一致するか。 -- fold、disclosure、dialog、menuを操作した後にfocusが失われないか。 -- hoverでしか得られない必須情報がないか。 -- scroll region、icon-only control、status markerにaccessible - nameまたはtextがあるか。 -- reduced motionで意味が失われないか。 - -## 実行手順 - -### 1. Scenarioを決める - -変更対象route、persona、viewport、theme、data state、capture -pointを列挙する。既存scenarioが目的を満たす場合は再利用し、満たさない場合は同じ変更内で追加または更新する。 - -### 2. 変更前をcaptureする - -`tools/web-ux`を使い、変更前のsourceとscenarioを固定する。出力はrepositoryのproduction -sourceへ混ぜず、`target/web-ux`などの開発artifact領域へ置く。 - -### 3. 変更前を目視する - -既存問題も含め、変更対象付近のgrouping、separator、spacing、wrapping、responsive、stateを確認する。変更前を見ずに、変更後の印象だけで改善を主張しない。 - -### 4. 実装する - -発見した問題と、適用するUX規則を対応付けて修正する。screenshotだけを整えるfixture固有hackや、productionと異なるshell、CSS、data -authorityを作らない。 - -### 5. 同条件で再captureする - -persona、route、viewport、theme、data -stateを揃える。変更後だけ別の好条件へ変えない。 - -### 6. Screenshotとreview contextを確認する - -実装者自身が画像を開き、`review-context.json`も読む。次を確認する。 - -- document status -- visible UI error -- console error -- page error -- failed request -- accessibility snapshot -- screenshot hashとcapture point - -HTTP 200でも画面に`401 Unauthorized`などが表示されていれば失敗とする。 - -### 7. Compareする - -before/afterを比較し、差分が意図した領域に限定されているか確認する。pixel -differenceの大小だけで合否を決めない。意図しない移動、折返し、欠落、色変化を目視する。 - -### 8. Testとevidenceを揃える - -source test、component test、E2E、type checkを実行し、visual -evidenceと一緒に記録する。test成功はvisual reviewの代替ではなく、visual -review成功もtestの代替ではない。 - -## 合格条件 - -次をすべて満たした場合だけvisual reviewを合格とする。 - -- 実装者自身がbefore/afterを目視した。 -- 必要なpersona、viewport、theme、data stateを確認した。 -- grouping、separator、spacing、wrapping、responsiveを確認した。 -- visible、console、page、request errorが解決または明確にdispositionされた。 -- 発見した新規問題を修正し、修正後を再captureした。 -- 意図しないvisual regressionがない。 -- artifact path、scenario、確認結果をhandoffへ記録した。 - -次は不合格である。 - -- capture commandが成功しただけ。 -- screenshotを生成したが開いていない。 -- desktopだけ、ownerだけ、happy pathだけを見た。 -- 後段Reviewerまたはユーザーが目視する前提でhandoffした。 -- 「好みの問題」として、説明できないseparator、spacing、overflowを未確認のまま残した。 -- testまたはlinterが成功したため、visual reviewを省略した。 - -## Evidenceの記録 - -handoffには最低限、次を記録する。 - -```text -Visual review -- Scenario: -- Before: -- After: -- Personas: -- Viewports: -- Themes: -- Data states: -- Findings fixed: -- Known issues: -- Result: pass | fail -``` - -画像をTicketやGitへ大量にcommitしない。artifact pathとsource -commitを結び付け、必要なreviewerが同じ条件を再現できるようにする。認証profile、cookie、token、secret、private -response bodyをartifactやhandoffへ含めない。 - -## Toolの位置付け - -`tools/web-ux`はbrowser起動、scenario再現、capture、review -context、compareを提供する。Toolは視覚的な良否を自動決定しない。 - -Linterが構文規則を検査し、testが特定のbehaviorを検査するのと同様に、visual -reviewはrendered -resultの関係性を検査する。三者は補完関係であり、どれか一つで他を代替しない。 +成果物は`target/web-ux`などの開発artifact領域へ置き、画像を大量にGitやTicketへ保存しない。認証profile、cookie、token、秘密情報、非公開response bodyを含めない。確認支援ツールは[tools/web-ux](../../../tools/web-ux/README.md)が所有し、合否判断は実装者が行う。 diff --git a/docs/development/validation.md b/docs/development/validation.md index 4f5b34f7b..768630eb2 100644 --- a/docs/development/validation.md +++ b/docs/development/validation.md @@ -1,32 +1,21 @@ -# Validation +# 変更の検証 -Validation should match the change. Do not run expensive broad checks just to look thorough, but do not skip checks that prove the changed boundary still works. +変更した契約を確認できる検証を選び、実行した範囲と未確認の範囲を区別する。広いcheckの成功で、変更した境界のテストを代替しない。 -## Docs-only changes +## 文書とrustdoc -Minimum checks: +文書だけの変更では`git diff --check HEAD`に加え、移動・削除したファイルへのリンク、ソースのinclude、テストや生成処理の参照を確認する。crate READMEやrustdocを変えた場合は、対象crateのドキュメント生成とdoctestも確認する。通常無効のfeatureでだけ存在するitemへのリンクは、その構成でも検証する。 -```sh -git diff --check HEAD -``` +## Rustコード -Useful stale-surface sweeps: - -```sh -STALE_DOC_PATTERN='TODO[.]md|obsolete doc dirs|old ticket surface' -rg -n "$STALE_DOC_PATTERN" README.md docs crates/*/README.md --glob '!docs/report/**' -``` - -## Code changes - -Follow `AGENTS.md` and the [test strategy](rust-testing-strategy.md). Start with the smallest target/filter that proves the changed contract: +[テスト設計規則](rust-testing-strategy.md)に従い、開発中は変更を証明する最小のtarget・filterを使う。 ```sh cargo test -p --lib cargo test -p --test ``` -Before completion, run the root compile check and the changed crate's full tests, plus formatting and diff checks: +完了前は[AGENTS.md](../../AGENTS.md)に従ってrootのcompile closureと変更crate全体を確認する。 ```sh cargo check @@ -35,49 +24,26 @@ cargo fmt --all -- --check git diff --check HEAD ``` -Unit, component, and integration suites are ordinary tests. Do not exclude a named integration target merely because it uses temporary SQLite, loopback/IPC, or an isolated child process. Include affected product features and cross-crate integration suites beyond the changed crate when they own the connection being changed. `cargo check` does not execute tests, and `--lib` alone does not run external integration-test targets. +単体・コンポーネント・結合は通常のテストに含める。SQLite、loopback、隔離子プロセスを使うtargetも省略しない。影響する製品feature、test-only target、別crateが所有する結合suiteを追加する。`cargo check`はテストを実行せず、`--lib`だけでは外部integration-test targetを実行しない。 -Workspace-wide tests and heavyweight builds are selected when the changed boundary requires them, not as a substitute for identifying the relevant suites. Avoid repository-wide formatting churn when a validation failure is caused by pre-existing unrelated formatting. +Workspace全体、`--all-targets`、Nix/Dockerなどの重い検証は、変更した境界を狭い検証では確認できない場合か、明示的に求められた場合に選ぶ。既存の無関係なformat差分を混ぜない。公開APIや生成元を変えた場合は、所有する生成処理と差分検査で生成物との一致を確認する。 -## Web UI changes +## Webとレビュー -Rendered Web UI changes require implementer-owned visual review in addition to type check, linter, and test. Follow [`ui-ux/visual-review.md`](ui-ux/visual-review.md). +描画結果が変わるWeb UI変更には、type check・testに加えて実装者自身の[目視確認](ui-ux/visual-review.md)を必要とする。画像の生成、HTTP成功、後段Reviewerへの委託だけでは代替できない。 -Minimum evidence includes: +Ticketに対するレビューでは、差分のもっともらしさだけでなく、現行の要求と受入条件を満たすかを確認する。レビュー判断と実行可能な検証は互いの代替にしない。 -- before/after capture under the relevant persona, viewport, theme, and data state; -- the implementer’s own inspection of grouping, separator meaning, spacing, wrapping, overflow, responsive order, and visible errors; -- `review-context.json` inspection; -- disposition of every observed problem; -- artifact paths and a final `pass | fail` result. +## E2EとCI -A successful capture command, HTTP 200, screenshot generation, or later Reviewer inspection does not replace this validation. - -## Work item checks - -Use the authorized Ticket tools and boundaries described in `AGENTS.md`. Work item review verifies that the implementation satisfies the Ticket, not only that the diff looks plausible; it does not replace executable tests. - -## E2E validation - -Product-assembly journeys belong to the workspace `tests/e2e` package and use its shared harness. Add or extend these tests for an explicitly selected journey that ordinary component/integration tests cannot prove. A crate-local test using an isolated child process is not E2E merely because it crosses a process boundary. - -E2E has a dedicated opt-in entry point: +製品を組み立てる利用場面の検証は、`tests/e2e`の共有harnessを使う。必要なbinary・製品feature、隔離・回収、timeout、診断をsuiteに明記する。稼働中の開発環境を借りたり、crateごとに別の製品起動基盤を作ったりしない。 ```sh cargo test -p yoi-e2e --features e2e --test ``` -Select the target for the affected journey. Its suite documentation must describe required binaries/build features, runtime prerequisites, isolation and cleanup, timeouts, and diagnostic artifacts. Do not substitute a live developer deployment or create a second product-launch harness in an individual crate. +このpackageは`default-members`に含まれず、対象には`e2e` featureが必要である。通常のroot実行やfeatureなしのWorkspace実行をE2E成功として報告しない。選んだ利用場面と実結果、または未実行を記録する。 -The package is outside workspace `default-members`, and its test targets require the `e2e` feature. An ordinary root test run does not execute these targets; even a workspace-wide test run without the feature does not prove E2E behavior. Record the selected target and actual result, or state that E2E was not run. +通常suiteのCI jobは、主張する結合targetと製品featureを実際に選択する。E2Eは専用jobとし、必須になる変更、手動・定期実行の条件を定める。特殊なplatform・tool・権限が必要なsuiteも適切なjobを持ち、可搬なテストまで例外扱いしない。 -## CI execution policy - -Suite ownership includes execution selection, not only source placement: - -- Ordinary unit/component/integration suites belong in normal validation jobs. Jobs must include named integration targets and the product-feature configurations they claim to validate; a `--lib` job is not whole-crate coverage. -- E2E belongs in a separate job that explicitly selects the package, feature, and journey targets. State its trigger policy: which changes require it and whether it also runs manually or on a schedule. Opt-in locally must not mean an unowned suite that is never exercised. -- Suites with special platform, tool, or privilege requirements need a suitable job and an explicit selection rule. Keep the portable portion in normal validation rather than hiding an entire suite behind the exceptional case. -- Timeouts, concurrency limits, resource cleanup, and failure artifacts belong to the shared harness/runner configuration. Do not rely on undocumented local setup. - -When adding or changing a suite, identify the job that executes it and update its selection if needed. If execution is not yet wired, record that gap; a policy statement or a successful local command is not evidence that CI runs the suite. +suiteの追加・変更時は実行するjobを確認し、選択条件を更新する。未配線なら不足として記録する。CI方針の記載やローカルの成功だけから、CIで実行済みと判断しない。 diff --git a/docs/development/web-state-preconditions.md b/docs/development/web-state-preconditions.md deleted file mode 100644 index 2a678ea2e..000000000 --- a/docs/development/web-state-preconditions.md +++ /dev/null @@ -1,20 +0,0 @@ -# Web state checks and retained identities - -The Web client uses purpose-specific content and operation identities, not an independent update counter. The checks below describe current boundaries, not a claim that the full cleanup or final generated-contract validation is complete. - -| Boundary | What is compared, how, and when | Why order or time alone is insufficient | -| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| Config source commit | The editor sends its saved `base_digest` and staged changes; update/rename/delete carry each observed entry's `expected_digest`. Local WASM checks entry conditions before replacing local content. The Server owns persistence admission. Success installs the returned snapshot as the new base; rejection retains the draft until explicit reload/reapply. | A later request can be authored from old content. Digests identify the actual source tree/entry, including a return to identical content; wall-clock age is not the condition. | -| Config diagnostics | Before publication the editor requires a non-aborted request and filters diagnostics by the analyzed tree digest. | A late result is not evidence about currently displayed source. Content identity and asynchronous invalidation are different checks. | -| Drive update/relocate/delete | The controller pins `expectedMutationId` from the user's observed node. The adapter sends it unchanged. `workspace-drive::mutate` checks current `last_mutation_id` via `changeable` in the same SQLite transaction before mutation. Publication stores the actual successful request ID and a durable receipt. | This identifies a real mutation, not a separately allocated state number. Content-only equality misses move/rename or ABA return to identical bytes. Last-arriving writes can overwrite another operation's result; timestamps do not prove operation identity. | -| Drive read/preview | Each chunk checks the current node's `last_mutation_id` against the observed operation inside the store transaction. Conflict does not silently switch to latest bytes. | Chunks from different writes must not be combined, regardless of arrival order. | -| Drive response-loss recovery | Before sending, the controller records one pending request ID. Pending/unknown receipts block another write for that selection. Recovery queries the same request. The store checks actor and typed payload against the existing receipt fingerprint before returning its result. | Unknown outcome does not mean the write failed. A fresh ID can duplicate a committed side effect; ordering cannot identify which payload committed. | -| UI asynchronous publication | Selection and write/page invalidation fences are checked before publishing returned data. These are local request-lifetime fences, not persisted resource-state conditions. | The projection follows the latest selected request, not the last response to arrive. No server-visible opaque state number is needed here. | -| Subject behavior save | The client sends exact `expected_behavior_md`, including empty text, with the replacement. A successful response must describe the requested text; mismatch is an unknown outcome and is not automatically retried. | Observed text already supplies the content condition; no extra counter is needed. Application to Worker history is a separate operation. | -| Memory history display | `change_id` and `current_change_id` identify immutable entries. The parser preserves Server order; current/historical is exact identity equality. | Stable links need immutable identities, but IDs do not establish chronology and must not be parsed or sorted as numbers. | -| Runtime trust display | The parser requires verification `binding_id` to equal the displayed binding. Mutations use the observed enrollment identity; revoke confirmation also compares the exact displayed key fingerprint before sending. | Re-enrolling the same public key creates a different binding lifetime. Matching key bytes or a recent response cannot make old verification current. | -| Merge Request review freshness | Review/request `subject_ref` must match current Git source ref. Revoked verdicts are excluded; effective events follow supplied thread order. Merged results use validated immutable integration evidence, not a new live observation. | Source equality identifies what was reviewed. Arrival order or target-only movement cannot confer approval on another source commit. | - -`ConfigProjectionValidator` is a typed static-template-catalog validation policy (`kind`, namespace, key aliases), not an opaque freshness token. Source tree, projection and toolchain digests describe different inputs/results. Key fingerprints identify key bytes; binding IDs identify enrollment lifetimes; mutation IDs identify actual committed requests. These are not interchangeable and are not increasing numbers. - -WIP remains `0.2.0`; naming cleanup alone does not justify a new package/protocol version or generic validator number. Final API/OpenAPI/protocol freshness must be checked after authoritative Rust changes are integrated. Earlier Web build or test success does not prove that final generation check. diff --git a/docs/development/wip-integration-validation.md b/docs/development/wip-integration-validation.md deleted file mode 100644 index 671646df7..000000000 --- a/docs/development/wip-integration-validation.md +++ /dev/null @@ -1,212 +0,0 @@ -# Scoped WIP integration validation - -## Released dependency and resolution - -The adopted Client, HTTP, Protocol and official Text View are **crates.io 0.2.0**, -with exact `=0.2.0` workspace declarations and registry checksums in Cargo.lock. -`cargo info @0.2.0` confirmed all four published packages (Rust 1.88). -`node scripts/verify-wip-dependencies.mjs` checks actual locked/offline Cargo -resolution: all WIP packages use registry 0.2.0, each required package occurs -once, and Protocol has one source. No path/Git override or attached checkout is -used. The earlier unreleased 0.1.0 source snapshot and vendor verifier were -removed when T-713 thread sequence 26 specified 0.2.0. - -Canonical contract reference: wip-reference -`6086f3c5ef10aa1464ce9c824750d7f667217bfd`. This reference documents the contract; -Cargo resolves the release, not that checkout or an unreleased Rust source commit. - -## Executed checks on registry 0.2.0 - -- `cargo test -p worker --locked --offline`: **894 unit + 110 integration tests - passed**, no ignored/failed tests. This includes ordinary Tools mode, Feature - permission/revocation, restore, session/history and controller integration. -- Narrow development checks: `wip::binding` (25), `wip::provider_tests` (12), existing - `wip::tests` and workdir projection tests (43), checkout transports/providers - (20), workspace-config production/authority tests (19). These are included - in the complete worker run. -- Published SDK tests via `cargo test -p wip-client -p wip-http -p wip-protocol - -p wip-text-view --locked --offline`: Text View **16 public signature/token/golden - tests + 4 doctests**; Client **63 adapter/runtime tests**; HTTP **38 tests + 1 - doctest**; Protocol **22 tests + 1 doctest**. Tests run from the registry packages, - not substituted source or removed vendor directories. -- Root `cargo check --locked --offline` passed, including normal TUI, CLI, - Runtime and Workspace Server compile closure. -- `cargo fmt --all -- --check` and `git diff --check HEAD` passed. -- `nix build --no-link .#yoi.cargoDeps` passed with the updated registry dependency - hash. This validates credential-free dependency acquisition, not a full Nix - binary/image build or deployment update. - -## Current target compatibility (preview, not integration) - -Source runtime at `23cf6c4f882ce78985ba4bd673751cebe64e2d87` and current develop -`b5ea162f838fc36e364edec9ab6689292b59cffd` produce the conflict-free `git merge-tree` -preview tree `7ef3545b635cbdfc5e24658f6d1532e0f8b0947c`. An archive of this tree, -without a branch/ref change, passed root locked/offline Cargo check, all **91** -Server API TypeScript tests (including OpenAPI/checked-in generator freshness), -registry WIP resolution checks, and Nix dependency acquisition with the same hash. -T-722's API macros, Server API/Drive/Server source, OpenAPI and generated TS blobs -are unchanged from develop in the combined tree; no artifact overwrite or -regeneration is necessary for this source change. The later added same-name scope -regression changes tests only, not the validated runtime or shared contracts. - -The initial archive check reused a Cargo target directory; returning to the older -main checkout exposed stale preview Server API artifacts. All local workspace -artifacts were invalidated, and a fresh main root check and complete **894 + 110** -Worker suite passed. Use an isolated target directory for future cross-tree checks. -No target push, integration branch, deployment, or live dogfood update was made. -Orchestrator readiness/integration must still resolve the then-current target. - -## Contract evidence - -`crates/worker/src/wip/binding.rs` tests exercise fresh/loading/stale/failed and -capacity-blocked acquisition, explicit refresh, superseded responses, retrieval -ownership cleanup, output limits, direct unseen-path Inspect, official unmodified -signature/path envelopes, structured Interface input and complete edge-only trees. -They invoke the registered three-tool surface for a search returning typed Entry -paths, inspect only a selected nonindexable path, dispatch the selected operation, -and verify that a later Inspect failure neither fails nor replays an earlier -successful operation. Interface acquisition failures are not empty operation lists. - -`provider_tests.rs` exercises the actual Yoi adapter for root/self scope, optional -refs, copied scope_ref, different scopes with identical names/operations, shape vs -membership errors, validator/ref absence/mismatch and protocol failure precedence. -It replaces a dynamic scope Object identity, observes both stale sides after -InterfaceMismatch, reacquires without replay, and confirms direct NotFound ends -safe Interface reuse. An in-flight operation survives descriptor refresh using -its exact frozen Host and Client descriptors rather than the newer return type. - -`ancestor_tests.rs` covers the independent review finding P1_CURRENT_ANCESTOR_SCOPE: -valid live ancestor refs that differ from registration placeholders, root `/` and -intermediate provider-owned ancestors, ref-less and ref-bearing scope deletion -without a prior scope Inspect, rejected cached dispatch and orphan observation, -replacement/republication without restoring old Interface registrations, -same-name scope replacement or loss of public ref rejecting copied old metadata -without replay (and later explicit Inspect/Invoke copying only the new optional ref), -shape/target/Object-validator/Interface-mismatch precedence, and frozen in-flight -ancestor Descriptor/handler/result validation after replacement. Valid Invoke -makes one provider publication call for the target, not a separate scope lookup. - -Existing checkout tests retain commit-time captured Workdir/connection/validator -checks, alias reuse/restore fencing, inode replacement rejection, authorization, -shared prior-read tracking, exact post-operation validators, cooperative -cancellation and unknown/lost/malformed responses without retries. Existing native -Ticket/Objective/Merge Request tests retain scoped Workspace authority and Feature -permission separation; compatibility Invoke retains original schema validation, -permission identity and ToolOutput/attachments. - -## Downstream provider seams - -- Public projection/contribution Interface fields use - `wip_protocol::InterfaceReference { scope, name }` from registry 0.2.0. -- `WipSubtreeProvider::publication` returns a `WipPublication` that captures target, - current scope Object, Descriptor/validator and handler at one provider boundary. - Mutable ancestors must not be resolved independently after selecting a target. - `scope: None` ends usable publication, even without a public ref; it is not a - ref-less published Object. Host checks and observation never substitute the - root registration placeholder, and Interface fetch never falls back to retained - provider registrations. Old registrations must end when their scope lifetime - ends. Published ancestor scopes outside the subtree remain Host-owned. -- Contextual/self-scope publishers use `WipPublication::self_scoped` on their exact - provider snapshot; references are explicit paths, not an encoded `/@/` suffix. - Checkout and Workspace Config retain their captured provider/Backend execution - and commit fences. Ref-less publication remains legitimate. -- `WipRuntime::{tree, inspect, invoke}` is the object-centered integration seam. - Inspect uses official summary-only signatures and typed rendering failures. -- `WipMountRegistry::mount_interface` adds a separate static Interface namespace - without replacing Object ownership or flattening same-named operations. It does - not overlay stale registration metadata onto a live subtree provider. -- FS indexable boundaries and directory.list are provider policy. T-714 now - implements entrance-only trees and bounded typed discovery (details below). - Transport/commit tests are `checkout_wip_tests.rs`, `checkout_http_tests.rs`, - and `checkout_race_tests.rs`. -- Worldspace state is runtime-only. Historical opaque references in append-only - session/tool history are not migrated into cache or parsed for a guessed scope; - a restored Client starts with fresh observations and direct path Inspect. - -## T-714 checkout discovery validation - -Based on integrated T-713 at `f9d4c97f`, still using the exact released crates.io -0.2.0 SDK and official renderer. No alternate wire, renderer or upstream version -was introduced. The checkout projection design documents provider live pagination, -its ordering/cursor/byte ceilings, special-entry policy and the distinction between -indexable emptiness and empty filesystem directories. - -Focused regressions: - -```sh -cargo test -p worker --lib checkout_wip -cargo test -p worker --lib checkout::race_tests -cargo test -p worker --lib checkout_http_tests -cargo test -p worker --lib ancestor_tests -cargo test -p tools --lib checkout_list -cargo test -p fs-operation --test list_pagination -cargo test -p workdir --test list_pagination -``` - -Host depth-zero omission / positive-depth `children: []`, empty/small/1100-entry -folders, direct deep roots and zero provider content-List calls during observation -are tested independently of Client output. Registered model tools exercise entrance -self-scope and deep target ancestor-scope, paginated List/Glob/Grep typed entries, -lazy selected Inspect with official signatures/path context, and Read without -COMMAND. Acquired paths never become tree edges; a stalled result observation is -never awaited and disappearance only fails a later explicit Inspect. Remote HTTP -checks verify List continuation on the wire with no ordinary List fallback or -response credentials/host-path leakage. Provider tests cover scoped External and -nested cwd/output_root rebasing, nonrecursive enumeration/read authority, ignore -policy, live mutations, byte bounds, cancellation and malicious cursor/result DTOs. - -`WipSubtreeProvider::interface_target` is a routing hint for target-qualified local -names under ancestor scopes. The default retains self/shared-scope behavior. Host -checks provider ownership and resolves **one** coherent current target/scope -publication; it never treats the hint as authorization or fetches a scope in an -independent mutable lookup. Focused Host tests reject forged/outside/prefix/missing -hints, ref-less deletion and absent membership while preserving existing ancestor -ref/validator and in-flight snapshots. Actual checkout tests reject sibling/prefix -scopes, supplied refs when current publication has none and stale alias generations; -ordinary content updates do not terminate the entrance Interface lifetime. - -Completion validation: - -- Root `cargo check` passed. -- `cargo test -p fs-operation -p workdir -p tools -p worker` passed: FS **58 unit + - 6 pagination**; Workdir **125 unit + 5 pagination**; Tools **84 unit + 39 - integration + 1 doctest**; Worker **901 unit + 110 integration**. -- `cargo test -p workdir --features http-client` passed (**125 + 5**). -- `cargo test -p yoi-workspace-server` passed (**691 unit + 11 CLI**). -- `cargo test -p server-api` passed (**83**, including checked-in OpenAPI/TypeScript - freshness). No generated API artifact changes were needed. -- `node scripts/verify-wip-dependencies.mjs` confirmed one crates.io 0.2.0 Protocol - source and all four exact SDK packages. `cargo fmt --all -- --check` and - `git diff --check HEAD` passed. - -The first dependent Server test build exposed predecessor fixtures still using -removed `discover`/opaque strings/private `call`. Those tests now select structured -references from actual path Inspect and use public Tree/Inspect/Invoke; the complete -Server suite above includes configuration integration and the typed Workdir proxy. -Its dev-only Protocol dependency reuses the same locked SDK source; the resolved -package/version set is unchanged. - -No full workspace test, full Nix/image build, deployment, target integration or -live dogfood update is claimed. Target movement remains Orchestrator authority. - -### Post-success connection-loss review regression - -The first independent source review identified `POST_COMMIT_CONNECTION_FENCE`: -lazy result publication had propagated a connection mismatch as a pre-effect -rejection even after successful Write/Edit/Create. The production result -publication boundary now maps post-success connection loss for mutations to -`OutcomeUnknown`, while Read/List/Glob/Grep keep their validator mismatch. Lazy -entry mapping is unchanged; there is still no result-wide acquisition. - -`committed_checkout_mutation_losing_connection_is_unknown_not_rejected_or_replayed` -executes real checked Tools/provider mutations, deterministically detaches after -the routed execution guard finishes but before the same production publication -boundary, then passes the error through WIP Invoke/Client. It checks committed -file contents, unknown audit (not rejection), and exactly one dispatch for each -Write/Edit/Create. `readonly_checkout_result_losing_connection_keeps_validator_mismatch` -protects the non-mutating error policy. No sleep or production test hook is used. - -After this fix, all **9** checkout race tests and the complete Worker suite -(**903 unit + 110 integration**) passed, as did root Cargo check, formatting, -diff checks and released SDK resolution verification. The earlier **901** Worker -count above describes the pre-review combined-crate run. diff --git a/docs/development/work-items.md b/docs/development/work-items.md index 6e4c4d5df..820fa6d66 100644 --- a/docs/development/work-items.md +++ b/docs/development/work-items.md @@ -1,95 +1,25 @@ -# Tickets and work outcomes +# 作業記録と判断の責任 -A Ticket is a durable record of natural-language user intent and a concrete work outcome, not a mandatory code/review/merge pipeline. Research, analysis, planning, documentation, operations, and code can all be Ticket work. Requirements, acceptance criteria, and binding decisions govern what must be done; the requested return path governs whether to continue, complete, close, or return results without concluding the Ticket. +Ticketには、利用者の意図と、それに対して判定できる成果を記録する。調査、設計、文書、運用も成果となる。コード、レビュー、マージをすべてのTicketに必須とせず、要求と受入条件に従って結果の返却・完了・終了を判断する。 -Workspace Server control-plane records, append-only Ticket events, and any linked Merge Request evidence are authoritative. Ticket and Objective records live in the Workspace Server database. Repository-local `.yoi/tickets`, `.yoi/objectives`, and config files are ignored legacy input, not a debugging or compatibility backend. Notifications, chat summaries, and Memory are hints to inspect current authority, not proof of completion. +## 記録の粒度 -## Concepts +一つのTicketは、単独で価値と達成を判断できる成果にする。中期的な目標はObjective、一時的な進捗はセッションのTaskで扱う。進捗を束ねるだけのTicketや、その代替となる階層関係を増やさない。 -- `Ticket`: durable work record containing intent, requirements, decisions, useful results/evidence, and resolution history. -- `Objective`: a first-class medium-term goal with background, strategy, success criteria, and canonical Ticket context links. It neither schedules nor authorizes Ticket work. -- `Task`: session-local progress tracking, not the project record. -- `Assignment`: an authorized delegation of Ticket work to a Worker, independent from its Profile or Flow. -- `IntentPacket`: a bounded natural-language handoff of intent, requirements, binding decisions, latitude, escalation conditions, and validation. -- `Ticket relation`: non-hierarchical project metadata (`depends_on`, `blocks`, `related`, `supersedes`, `duplicate_of`). Inverse views are derived. Relations do not grant resources or start Workers. +作成・大幅な変更の前に、関連記録と重複を確認する。背景、要求、受入条件、決定事項、実装の裁量、未解決の問いを区別して書く。確認済みの事実、利用者の説明、仮説を混ぜない。担当者の役割制限、そのターンの計画、文書作成後に停止する判断を製品の禁止事項へ転記しない。 -One Ticket should describe an independently useful outcome that can be judged on its own terms. Do not convert temporary execution steps or a broad effort's progress container into mandatory product deliverables. +実装の承認待ちは、利用者が求めた場合にだけ作業の条件として記録する。未確認事項は必要な調査で絞り、実装手段を早期に必須要件へ変えない。依存関係は成果間の関係を示すものであり、資源所有や実行順の記録の代わりにはしない。 -## Entry points and authority +## 正本の確認と成果の報告 -Use the highest-level interface matching the request: Workspace Dashboard/Ticket UI, authenticated Workspace APIs, product `yoi ticket ...` or `yoi objective ...` commands, and typed Worker tools. Inside Workers, use the supplied typed Ticket tools; do not substitute a CLI or direct storage access for missing tools. Maintainers inspect authority through the Workspace API, Server administration surfaces, and database diagnostics, not repository-local Ticket files. +実装・routing・レビュー・終了の判断前には、現行Ticketと必要な関連記録を読む。通知、要約、過去の実装報告を現行の要求や承認の証拠にしない。Worker内の操作は、提供されたtyped toolsとsystem instructionsに従う。CLIや保存先の直接編集で欠けた権限を代替しない。 -The Server resolves Workspace access, active configuration, and immutable Profile launch material; Runtime does not infer identity or authority from a checkout's `.yoi` files, cwd, or launch prose. The first committed user request supplies bounded action context, not actor identity or new grants. Profile/Flow selection is configuration, not review, merge, or resource authority. Availability of a tool is not authorization to use it for unrelated work. +成果には、実際の変更、検証結果、未確認事項、判断に必要な証拠を残す。すべての作業にコメントやartifactを義務付けず、利用者が必要とする返却方法を選ぶ。長い調査結果、生ログ、秘密情報はTicket本文へ転載しない。 -## Starting and returning Ticket work +## コードの公開とレビュー -`SpawnTicketWorker` starts and atomically assigns a generic Worker with an `initial_request`. Its registered `profile` is optional (omitted/null selects `builtin:ticket-worker`); `flow` and alias-keyed `workdir_attachments` are optional. No repository or Flow is required for non-code work. Choose an explicit Profile when its tools and instructions fit the task; assignment does not depend on choosing the Coder role. Backend validates claims, derives repository attachment access from declared Ticket targets, and preserves explicit ExternalGrant ceilings. Callers cannot request capabilities through attachment selection. +公開が作業範囲なら、まとまりのあるcommitを作り、正確なsourceを通常の非force更新で公開する。レビューでは要求、変更全体、検証証拠を独立に確認する。口頭の承認、Profile、Flow遷移を権限の代用にしない。 -The guarded start records acceptance only after spawn, initial input, assignment, and attachment finalization. Do not pre-write `inprogress` or mistake a state update for a launch. On a failed or unknown outcome, reread durable operation evidence before retrying or spawning a replacement. After acceptance, reread Ticket state and current Worker responsibility. +source変更後は変更後のsourceを再レビューする。targetだけの変化では、未変更sourceへの承認と統合可否の再確認を区別する。通常の修正・再レビュー証拠はMerge Requestに置き、Ticketには調整が必要な阻害要因、決定、最終引継ぎを残す。 -Results can be returned in conversation, optionally recorded in a `TicketComment`, optionally published as a Drive document/artifact when that surface is available, or written/published to an authorized repository. Neither comments nor Drive artifacts are universally required. Preserve concise evidence useful to the user without fabricating work, tests, or conclusions. `implementation_report` is optional historical/audit context, not integration readiness or completion authority. - -The return choice is independent of result format. Examples: - -- Research with review not required: answer the question and optionally record a useful summary; no MR or approval is required for an authorized completion decision. -- Code requiring independent review before merge: use the trusted review path and guarded MR integration. A Profile, Flow transition, tool invocation, or prose approval is not the review judgment. -- Return without conclusion: report the result/limitations and leave the Ticket unconcluded. A finished turn, an artifact, or an approved MR does not itself decide completion. - -These are instruction/tool contracts, not a claim that a mechanism proves real agent judgment. Backend authenticates actors and enforces typed boundaries; the responsible Worker/human must actually judge satisfaction against the natural-language request. - -## Ticket tools and decisions - -`QueryTicket` provides bounded discovery/filtering; `ShowTicket` provides authoritative item content and `content_digest`, paged thread, relations, Objectives, and any current MR context. Read the relevant Ticket before implementation, routing, review, state, or conclusion decisions. Check potential duplicates before creation or material rescope. Use `QueryObjective` and `ShowObjective` for broader context without replacing the Ticket read. - -Useful mutation surfaces include `TicketCreate`, `TicketComment`, relation/plan tools, and authorized `TicketWorkflowState`, `CompleteTicket`, and `TicketClose` decisions. Exposure depends on the effective Profile/Feature configuration. Decision tools use current `content_digest` as `expected_content_digest`, expected state, a reason/resolution, stable operation key, and optional supporting references. Backend checks actor authority, CAS, replay, and resource boundaries. It does not enforce a universal state sequence, MR/approval gate, or natural-language satisfaction. - -`CompleteTicket` records completion; `TicketClose` records closure. Neither integrates an MR or expands resource grants. Ticket progress/conclusion can occur without commits, MRs, or approvals when consistent with the request. Conversely, code's default publication/review practices must not be erased by confusing an independent Ticket decision with merge authority. Do not claim an unmerged result was integrated. Reread after conflicts and replay a recorded operation only with the same fingerprint. - -Relations remain metadata, not a scheduling mechanism: record actual project dependencies, but do not encode capacity, Worker/worktree ownership, or execution ordering as relations. Use orchestration/session records for those concerns and inspect current Backend claim rules rather than inferring launch eligibility from a relation alone. - -## Optional code Git/MR recipe - -For code work using the default publication/review recipe, select `builtin:coder` and optionally `builtin:coder-review`. The Flow guides implement, review, fix, and handoff; it is not a Backend Ticket completion requirement and grants no authority. Read current repository state, use coherent commits and normal non-force publication only when authorized, validate the changed contracts, and report changed files, validation, and limitations. No commits/push should be inferred from a request explicitly prohibiting them. - -Use one open repository-scoped MR per Ticket/repository. `OpenMergeRequest` creates selectors; once an MR exists, explicitly address its ID with `ShowMergeRequest` and preserve its selectors. Advance only the existing source selector with a normal non-force push. Do not invent an operation to append another source candidate, replacement MR, or fresh integration branch for every fix. Publish the exact committed source and verify provider resolution before requesting review. - -The assigned Coder launches the Reviewer as an actual direct-child `builtin:reviewer` SubWorker, with explicit command grant and appropriate Workdir scope for inspection/validation, plus the structured Ticket/MR handoff. Server authority verifies assignment, Runtime-owned child identity, effective Profile, one-shot attempt, and captured subject. Review capability is injected by the trusted layer, not supplied by the model. The Reviewer makes an independent judgment against intent, acceptance criteria, the complete captured result snapshot, and validation, then records `ReviewMergeRequest`; prose and observation are not approval authority. - -Source movement requires fresh review for the exact new source. Target-only movement preserves unchanged-source approval but requires refreshed integration evidence. Keep routine review/fix/rereview evidence on the MR rather than repeating each iteration in Ticket comments. - -MR integration remains separate Orchestrator authority. Immediately before integration, reread Ticket/MR authority and `CheckMergeRequestReadiness`. Verify exact approved source, current target, strategy, and approval event. Apply and validate integration through ordinary source control in the bound Workdir, push normally, and verify provider resolution. `CompleteMergeRequest` records that already-applied repository result; it does not move a branch, complete the Ticket, or release assignment. If push succeeded but recording failed, reread/replay the same operation rather than pushing again. A Ticket request saying review is not required does not bypass MR integration guards. - -## Responsibility, unfinished work, and cleanup (T-715) - -Terminal current responsibility is retained for display after unfinished work ends. It is not the same as an unfinished assignment and does not require an invented unassignment operation before cleanup. A stopped or idle Worker may still own unfinished work; a completed/closed Ticket does not automatically stop/remove the Worker, release its attachments, or delete a Workdir. A result report or Flow terminal state is neither conclusion nor cleanup authority. - -Worker stop, retention/removal, attachment release, and Workdir deletion are independent guarded decisions. Do not predeclare `delete_on_completion` or `retain_on_completion` at launch. Retain Workers while a requested review/fix or other handoff can still return; do not remove them just because one turn ended. - -When authorized cleanup is in scope: - -1. Reread Ticket responsibility, active unfinished work, Worker state, handoffs/notifications, and current retention constraints. Use the exact Worker subject from `WorkerList`. -2. Stop the Worker if needed and confirm terminal state. Stop alone does not end unfinished work. `UnfinishedWork` (`unfinished_work`) requires explicit authorized work end or reassignment and a fresh authority read. -3. Call `WorkerRemove` only with no unfinished work, running/restoring state, pin/legal hold, pending notification, or handoff. Backend revalidates guards; removal releases attachments but preserves Workdir materialization. -4. Reread actual Workdir attachment release, occupancy, cleanliness, ownership, provider availability, and other use. Retain still-needed/existing resources. Only then use `WorkdirDelete` for a proven clean, unoccupied, no-longer-needed Ticket-dedicated Workdir owned/selected for this work. - -Never delete before attachment release, force removal, discard changes, or advance on a partial/unknown failure. Reread authority before a bounded retry. Cleanup failures do not roll back recorded Ticket or MR outcomes; report concrete blockers rather than adding routine success comments. - -## Authoring and granularity - -Intake clarifies ambiguous intent, checks duplicates and relevant context, and records agreed work. Create/update only after user agreement or explicit instruction to record the draft. A minimum investigation gate is an authoring responsibility, not a universal implementation approval gate. Intake is not scheduling, implementation, review, merge, or cleanup authority. - -A useful Ticket records background/motivation, requirements, acceptance criteria, binding decisions/invariants, implementation latitude, and relevant open questions/risks. Include useful result evidence and final resolution when appropriate. Separate confirmed facts, user claims, hypotheses, and undecided questions. Do not copy Worker role boundaries, a temporary plan, or a decision to stop after authoring into the Ticket as a product prohibition. Record an approval gate only when the user asks for it; do not prematurely prescribe implementation tactics. - -Risk flags prompt context checks and reviewer focus, not automatic stop gates. Return to planning only for a named missing decision/information after bounded checks; escalate beyond-scope decisions without inventing new constraints. - -Do not create new umbrella/progress-container Tickets. Split broad work into concrete useful outcomes, record the split in Ticket/Objective context, and use Objectives for enduring medium-term goals. Do not replace umbrellas with parent/child, sub-ticket, part-of, contains, or other hierarchy relations. A concrete planning/design/investigation Ticket is valid when requested. Existing umbrellas can be retired as superseded/decomposed without rewriting history; distinguish container retirement from completion of future work. - -Keep long research dumps outside the item body and summarize necessary bounded artifacts. Never store secrets, credentials, private prompt contents, or secret-bearing logs in Ticket/MR/Drive records or model-visible evidence. - -## Configuration and validation - -Workspace Ticket data and authority live in Server SQLite and active Workspace configuration. `yoi init --display-name --repository-key ` registers a repository through Backend and writes global client routing only. Repository-local `.yoi/workspace.toml` and `.yoi/ticket.config.toml` do not grant Workspace identity, Backend connection, role selection, or Ticket authority. Legacy `ticket init` and `ticket import-local` are unsupported; no automatic migration reads local Ticket trees. - -Dashboard actions are explicit user actions, not scheduler/auto-maintainer authority. Launch success without a visible result calls for attaching to or observing the Worker and rereading authority, not assuming conclusion. Select an accessible Workspace and current registered Profile when a launch reports missing configuration. - -Validate the actual result with checks appropriate to its acceptance criteria. Code changes follow `AGENTS.md` and [Rust testing strategy](rust-testing-strategy.md): begin with the smallest contract-proving target/filter, then the required compile/test/format checks. Record only validation actually run. Narrow effective instruction/tool fixtures protect inclusion, schema, and policy wording; they do not prove real Worker judgment or end-to-end behavior. +統合、Ticketの結論、Worker停止、接続解除、資源削除は別の判断である。終了通知や最終回答だけから後続操作を開始せず、依頼範囲と現在の正本を確認する。未反映の変更を統合済みと報告しない。 diff --git a/docs/development/workdir-refusal-diagnostics.md b/docs/development/workdir-refusal-diagnostics.md deleted file mode 100644 index b3faf706f..000000000 --- a/docs/development/workdir-refusal-diagnostics.md +++ /dev/null @@ -1,159 +0,0 @@ -# Workdir refusal diagnostics - -This change improves the internal `api_error` record for -`POST /api/w/{workspace_id}/workers/self/workdir-session/operations`. -It does **not** identify or resolve the reported 2026-10-09 403: the running -binary source commit and actual request are still unverified. Fixture success is -not evidence about that incident. No deployment, live asset cleanup, privilege -change, retry policy, or WIP dependency/version change is included (WIP remains -0.2.0). - -## Diagnostic path and public contract - -Original refusing branches attach a closed `WorkdirDenialReason`, rather than -recovering meaning from an error string. `WorkdirError::Denied` carries local -message plus optional typed reason. `DenialContext` retains an existing error -classification while enriching its diagnostics (notably checkout's historical -permission-to-out-of-scope mapping). Typed provider markers survive the -`FsAccessPolicy`/`io::Error` → `FsError` → `WorkdirError` boundaries. - -`WorkdirTransportError.denial_reason` and -`ExternalWorkdirOperationError.denial_reason` carry only an optional snake-case -enum. Remote HTTP and External errors preserve that reason when converted back -to `WorkdirError` and then forwarded again. Provider text is not the reason. -The External codec continues dropping provider-authored message text. A missing -reason stays unknown for generic `denied`; existing code-level path/symlink/ -read-only classifications can still identify their original category. Unknown -or oversized reason strings fail deserialization rather than becoming log data. - -Runtime registry session-open failures now retain a safe transport error **before** -the former raw error-to-string conversion. Their existing Workspace HTTP meaning -is still 502 (`workdir_session_open_failed`); operation-level permission failures -remain 403. Session-open messages are normalized to the same safe provider -message vocabulary instead of forwarding raw local errors. No authorization, -scope, capability, session ownership, command ownership, or retry conditions -are relaxed. - -The self handler attaches internal context to its error. Both the handwritten -response adapter and the generated route preserve it through the typed error -response extension to `ApiErrorLog` and the logging middleware. Server API -`workdir_log` and `denial_reason` are `serde(skip)` and omitted from schemas: -the external error envelope remains `{code, message}` for provider failures or -the existing Repository-style envelope for API/domain failures. Workspace -responses do not expose the new internal diagnostics. Authorized Runtime/provider -transport carries the enum to its consumer; it contains no free text or paths. - -## Fields in `yoi::api` / `api_error` - -- `operation`: fixed label from the typed operation (read, read_bytes, write, - edit, command_start/status/output/cancel, authorize_scope, scope_rules_overlap, - checkout_search/observe/execute, stat/list/glob/grep). No operation Debug dump. -- `workdir_stage`: workspace_scope, worker_identity, attachment_validation, - session_open, command_lookup, provider_dispatch, or command_registration. API - failures identify the encountered stage without copying raw alias/handle/error - text into the log. Registration failures can follow successful command start; - they are not evidence of a pre-dispatch refusal or absence of effects. -- `denial_reason`: optional closed reason reported by the provider/error chain. - It describes the encountered refusing branch, not a permission grant and not - an independent attestation of an external provider's explanation. -- `workspace_id_hash`, `runtime_id_hash`, `worker_id_hash`, `workdir_id_hash`, - `attachment_alias_hash`: 64-character lowercase SHA-256 fingerprints via - `worker_source::diagnostic_id_hash`, using the existing - `yoi-runtime-proof-diagnostic-id-v1\0` domain prefix. Hash a known catalog ID - or attachment alias to locate its logs. Workspace and alias represent route/ - request selection; Runtime/Worker are added only after current identity checks, - and Workdir only after a validated attachment is found. Missing information is - omitted, never inferred from a failed lookup or command handle. -- `request_token_id_hash`: fingerprint of the **existing verified Runtime proof - JTI**, reusing the T-729 hash algorithm. It is not the proof, raw token ID, - caller-supplied trace header, or a newly invented cross-service trace ID. - In-process/test callers without that proof correlation omit it. Authentication - still consumes the JTI using the existing policy; logging does not retry it. -- Existing `method`, query-free `path`, `status`, `kind`, `message`, `diagnostics` - remain. Workdir-context method/path are bounded to 32/512 UTF-8 bytes with - controls removed. Provider operation messages remain normalized safe text. - Contextual API failures use a fixed internal message and status-based kind, - with no raw domain diagnostics copied into this log. Empty `diagnostics` does - **not** mean the optional typed reason/context is absent. - -Fingerprints are bounded correlation, not encryption: low-entropy identifiers -can be guessed. Do not put credentials in IDs. Neither hashes nor stage labels -prove the reported incident's cause. The request JTI identifies the incoming -request; External operation IDs/generation/audit records remain their existing -provider-side mechanisms, not a cross-service trace automatically joined here. - -## Refusal vocabulary and limits - -Reasons distinguish scope resolution/comparison unavailable, attenuated -capability refusal, logical/parent/attachment/delegation scope exceeded, -empty scope, invalid scoped path, writable-scope requirements for commands, -child write-lease conflicts, unreadable cwd, read-only session, external root -symlink/change/move, external scope symlink, checkout output-root escape, -provider root escape, checkout search/enumeration refusal, provider symlink/ -checked-target refusal, path out of scope, symlink target out of scope, and -read-only path. Original local messages remain local; arbitrary message text -is never parsed into a reason. - -- `os_permission_denied`: PermissionDenied with an actual OS error code, without - a recognized policy marker. It does not determine whether modes, ACLs, - sandboxing, mount policy, or another OS mechanism caused the refusal. -- `unclassified_permission_denied`: custom PermissionDenied with neither typed - policy origin nor raw OS code. Do not label an arbitrary - `Error::new(PermissionDenied, "...")` as an OS refusal by guessing from text. -- External/remote providers that do not send a reason cannot supply distinctions - erased in their own binary. A registered provider's typed explanation is - reported data, not proof of what happened inside that provider. -- Authentication/JSON ingress failures before a typed Workdir operation and - target can be obtained continue using their existing rejection diagnostics; - no operation/body is guessed or decoded from rejected proof data. - -## Regression proof boundaries - -- Workdir provider fixtures exercise actual local authorization, broker scope/ - capability/read-only/delegation/lease refusals and confinement. Permission - fixtures distinguish typed policy origins, OS-coded PermissionDenied, and - origin-unknown custom errors, including checkout's existing public mapping. -- Loopback HTTP fake exercises remote open/operation decoding and typed reason - forwarding, including missing-reason providers and hostile provider text. - External JSON/frame codecs exercise the bounded error envelope and reject - arbitrary reason strings; the Server's External adapter drives the real self - generated route through command-session dispatch and response/log conversion. -- Server router tests use signed fixture requests and capture actual tracing - events. They cover read/write/command_start/authorize_scope, session-open - failures, External command status failures, no-identity rejection, oversized - UTF-8 alias, oversized command/file/tool-call inputs, query/header/proof/key/ - credential markers, and raw provider/host paths. Identity hashes and the - existing verified JTI fingerprint are checked against fixture identities. -- The Server API boundary proves internal fields are absent from body/schema. - Worker HTTP→WIP coverage verifies diagnostic wrappers do not change refusal - classification or induce retries; tools retain established checkout codes. - Normal-tool JSON→transport→WorkdirError→ToolsError→ToolError coverage checks all - 18 codes with no reason, a typed reason, and nested diagnostic wrappers. In - particular, OutcomeUnknown/Conflict remain execution failures with unchanged - display and inspection-before-retry guidance, not argument errors. - Operational consumers share `WorkdirError::classification_source` to ignore - only diagnostic wrappers while retaining the original error for diagnostics. - Public Write tests cross JSON stat errors into the tool and the checked local - provider: missing files are created identically without/with/nested reasons; - denied/read-only/conflict/unknown-outcome stat failures never write or retry. - This protects actual create control flow in addition to error conversion. -- Real Remote HTTP and channel-backed External session dispatch tests cover - Read/Create/Write/Edit crossed with missing/typed reasons and five error codes. - After a dispatched mutation, Unavailable/Transport still become OutcomeUnknown - with the original reason preserved: forwarded 500 and inspect-before-retry - guidance are unchanged. Read and definite refusal/conflict controls retain - their existing codes. The External path also checks actual HTTP error body/ - status conversion and absence of a new dispatch after the failure. -- Fake-command JSON fixtures cover attachment detach and scoped-command close - without launching processes. Missing/typed/nested UnknownCommand all retain - disappearance/lease-release semantics. Unavailable remains a busy detach or - failed scope cleanup; scope close revokes new work immediately but retains the - write lease while cleanup is unresolved. Diagnostic data never grants cleanup. -- A signed real self-handler test observes command dispatch, then removes only - its fixture session-registration entry before returning the start handle. - The 502 remains a post-start `command_registration` failure with a neutral - safe log message; no process is launched and no retry is performed. - -These are hermetic Rust/API/protocol fixtures, not a full product-process E2E, -not a deployed binary test, and not reproduction of the user's asset/request. -Validation commands and concrete counts are recorded on the implementation MR. diff --git a/docs/development/workdir-removal.md b/docs/development/workdir-removal.md deleted file mode 100644 index d0f6f6a9f..000000000 --- a/docs/development/workdir-removal.md +++ /dev/null @@ -1,91 +0,0 @@ -# Workdir removal and ordinary retry - -Use the Runtime **Workdirs** page or the normal typed Workdir removal tool to -remove an unused repository Workdir. Deletion is authorized by the Backend's -existing durable removal operation; it is not a registry-only hide operation. - -A failed attempt does not require a new recovery state or a manual edit of -`materialization.json`. Retry the same removal request after resolving the -reported cause. Each attempt rechecks current attachments, attachment -reservations, retention holds, materialization identity and current file changes. -A previous clean observation is not permission to discard new changes. A dirty -Workdir must be preserved or its changes resolved by its owner before deletion. -When the same repository checkout is still usable, an authorized Worker may -reattach it through normal `WorkdirAttach` to preserve or resolve changes, then -detach and retry removal. An executing removal fences new attachments; a failed -attempt does not permanently prevent reconnection. Missing or replaced checkout -identity is not made usable by this permission. - -## Interpreting a retained Workdir - -- **Mount remains:** Git status being clean does not mean mounted resources are - released. Confirm the mount's owner, users and data destination. Have that - owner safely release the resource, then retry normal deletion. Yoi does not - recursively delete mounted data or implicitly unmount external/unknown - resources. Do not use force-clean, recursive chmod or raw recursive deletion - to override this boundary. -- **Permission denied / resource busy / storage unavailable:** have the owning - operator resolve access, release the resource or restore storage. Retry the - same normal action; registry retention is intentional until cleanup is proven. -- **Changes present / changes cannot be verified:** preserve current files and - restore repository inspection where possible. Do not treat a previous cleanup - failure or missing checkout metadata as proof that all remaining files are - expendable. -- **Identity or ownership unavailable:** verify the Workdir target with its - owner. Do not rewrite registry or materialization identity to bypass checks. -- **Runtime unavailable / outcome unknown:** reconnect the Runtime and retry. - Absence of a provider response is not authoritative not-found evidence. -- **Attachment, reservation or hold:** release the current owning authority - through its normal API before retrying. A stale failure does not freeze these - conditions; newly attached Workers remain protected. - -The Web page refreshes authoritative inventory after each attempt. Successful -removal commits registry deletion only after Runtime confirms physical deletion -or authoritative not-found. An unknown result leaves the registry in place so -retry can reconcile the provider's current state. - -A retained partial-removal Workdir is listed as `cleanup_pending` with cleanliness -`unknown`. List/detail observation checks its saved identity but does not read or -hash surviving checkout content. The saved removal witness is not a current -cleanliness assertion; only an actual removal retry runs the full content checks. - -## Runtime safety boundary - -The Runtime keeps removal content/identity evidence outside the tree being -removed. This is a factual witness for partial deletion, not a new recovery -state or cleanup stage. Surviving files are rechecked against it; an intact, -currently clean checkout can renew obsolete evidence without discarding newly -ignored/untracked content. Do not edit this witness manually. - -Mount-safe removal currently requires Linux `openat2` and `/proc` inspection. -Unavailable inspection fails closed with `mount_check_unavailable`; there is no -fallback to recursive deletion across an unchecked filesystem boundary. -Runtime HTTP list, detail and removal requests run synchronous provider work on -blocking workers rather than the HTTP executor. Removal reserves only the target -Workdir under the Runtime state mutex, then releases that mutex before content -inspection and unlink. Unrelated Worker list/detail, session observation and -subscription snapshots remain available while removal runs. Conflicting use, -attachment, creation or removal of the same Workdir is rejected promptly rather -than waiting under the global mutex. - -Provider leases protect in-flight creation/access and live session resources, -including their command tasks. Cloned sessions retain protection until safe close -or final resource release; passive bindings alone do not prevent removal. Removed -bindings cannot activate against a recreated Workdir with the same ID. Creation -owns a fresh root exclusively so failed creation cannot roll back another -creator's checkout or adopt an existing partial-removal directory. Reservations -are transient and released on provider success, error or unwind; they do not -replace the durable cleanup witness or attachment authority. - -## Diagnostics - -Public responses expose a bounded cause category and safe guidance, not arbitrary -provider errors or host paths. For an OS failure, correlate Backend removal logs -(`operation_id`, `workdir_id`, Runtime) with Runtime cleanup logs for that Workdir -and cleanup action. Runtime logs retain the concrete OS error/errno for the -owning operator. These private diagnostics must not be copied unredacted into -public responses or Tickets. - -This procedure is not authorization to manipulate a live mount or another -Worker's storage. If ownership or required operator authority is unclear, stop -and request that specific owner/operator's decision. diff --git a/docs/development/worker-drive.md b/docs/development/worker-drive.md deleted file mode 100644 index b0dd936c4..000000000 --- a/docs/development/worker-drive.md +++ /dev/null @@ -1,171 +0,0 @@ -# Worker Drive tools and WIP - -Drive is Workspace-owned DB hierarchy and local blob storage exposed through the -authorized Server API. It is **not** a Workdir, mount, Git checkout, or a Session -attachment store. Worker code never accesses the Server's physical blob root. - -## Enablement and authority - -Set `feature.drive = { enabled = true; };` in a DCDL Profile (or -`[feature.drive] enabled = true` in TOML). Only activation is accepted; Profile -fields cannot specify Workspace, grant, actor, access, or host paths. Default, -Coder, Companion, Intake, Orchestrator and Ticket Worker builtin Profiles select -the feature. Standalone, Reviewer and Job defaults do not. Host must inject an -available Workspace-bound `WorkspaceClient`; standalone activation is inert. - -An authorized Workspace owner must independently grant the **registered Runtime -Worker** whole-Drive `read_only` or `read_write` access through the T-722 grant API. -Every read, metadata resolution, mutation, chunk and receipt query passes current -Backend authorization. An enabled feature, Subject, Ticket assignment, path, ID, -or observed validator is not permission. A read-only grant is never upgraded. -The current API has no read-only capability descriptor endpoint: an authorized -WIP inspect describes the operations for the entry kind, while Backend rejects -writes without a write grant. Tool permission configuration also applies to -native Invoke, including fail-closed `ask` where approval cannot be obtained. - -Internal SubWorkers (including Reviewer children) have no separate Backend Drive -identity/grant contract today. Spawn suppresses Drive activation **after every -selector resolution**, including `inherit` and explicit builtin Profiles, and -wraps the Workspace client to deny Drive JSON/binary operations. They cannot use -the parent's shared client to inherit Drive read or write. Perform such work in -the authorized parent, or a separately registered Runtime Worker with its own -current Backend grant. Any future internal delegation must introduce an explicit -parent-bounded scope and Backend reauthorization, not remove this fence based on -Profile activation. The wrapper preserves non-Drive Workspace operations and -Reviewer attestation. - -## Normal Tools - -`DriveRoot {}` discovers the authorized root and returns `{path, metadata}`. -Public entry inputs are **Workspace-bound** references: - -```json -{"workspace_id":"workspace-id","node_id":"9007199254740993"} -``` - -Node IDs remain canonical positive decimal **strings**, including -values above JavaScript's safe-integer range. Renaming/moving preserves IDs and -authenticated latest URLs. Deleting/recreating the same name allocates a new ID. - -| Tool | Input/use | -| --- | --- | -| `DriveMetadata` | `entry`: one live entry and its last committed request ID | -| `DriveList` | optional `parent` (default root), `limit` 1–200, opaque `after` | -| `DriveSearch` | `query`, optional `include_text`, `limit` 1–128, opaque `after` | -| `DriveRead` | `entry`, optional `max_bytes` 1–65,536; reports `truncated` | -| `DriveCreateFolder` | `parent`, logical `name` | -| `DriveCreateText` | `parent`, `name`, `content` <=64 KiB, optional `content_type` (Markdown by default) | -| `DriveWrite` | observed `entry`, complete replacement `content` <=64 KiB | -| `DriveEdit` | observed `entry`, `old_string`, `new_string`, optional `replace_all` | -| `DriveRelocate` | observed `entry`, destination `parent`, new/current `name` | -| `DriveDelete` | observed `entry`; folder deletion follows Backend rules | -| `DriveViewImage` | `entry`; bounded committed-request-bound image bytes, not a URL | -| `DriveSaveWorkdir` | explicit `target_workdir` alias, relative `path`, destination `parent`, `name`, optional `content_type` (octet-stream by default) | -| `DriveRequestStatus` | `request_id` from an uncertain mutation | - -List/search are paged discovery only, not implicit mutation observations. Omitting -`limit` uses the Backend maximum (200 for list, 128 for search) in both Tools and -native WIP; use the returned `next_after` to continue. Read or -metadata-inspect an entry before replacing, editing, moving or deleting it. The -Tools adapter retains at most 256 observations, scoped to its injected Workspace. -Eviction requires a new explicit observation; it never substitutes another entry. -A successful mutation removes the prior Tools observation. This is a bounded -precondition cache, not a grant cache, second hierarchy DB, or durable history of -all Drive versions. Restored workers observe again before mutating. - -`DriveEdit` uses the same pure `fs_operation::text` rules as Workdir and Workspace -config: nonempty `old_string`, different replacement, exactly one nonoverlapping -match unless `replace_all=true`, and bounded resulting text. It fetches a full -<=64 KiB preimage and verifies the **previously observed** committed request ID. It cannot edit -truncated content. The Backend's atomic CAS still decides the first winner. - -## Native WIP 0.2.0 - -Start with `Inspect {"path":"/drive"}`. `/drive` offers root/metadata, list/search, -creation, Workdir save and receipt query. Invoke selects the structured Interface -reference shown by Inspect: - -```json -{ - "path":"/drive", - "interface":{"scope":"/drive","name":"yoi.drive/entry/v1"}, - "operation":"list", - "arguments":{"limit":20} -} -``` - -Direct entry paths returned by discovery are: - -```text -/drive// -``` - -Use the returned path rather than hand-constructing it. A file exposes metadata, -read, write/edit, relocate/delete and view_image; a folder exposes metadata, -list, create_folder/create_text, save_workdir and relocate/delete where valid. -A native target binds its entry or creation parent: no ID or mutation-precondition argument is -needed. Relocate's `parent` is a WIP typed entry path. Each entry is self-scoped; -Host captures target, scope object, descriptor, validator and handler together. -Client retains and supplies metadata automatically. Do not copy validators or -`scope_ref` through model arguments. - -Drive has **no indexable entry children**, regardless of the size of the Drive or -the Client's already-inspected entries. Tree shows the Drive entrance, not a file -inventory. List/search return bounded metadata and paths; only inspect the -selected path. No Worker index/embedding or eager body injection is added. - -## Conflicts, unknown outcomes and publication - -Mutations use a Host-generated request identity, observed node ID and last -committed request ID. Conflict never causes an adapter to replace that observation with -latest and retry. Observe explicitly, reconsider the edit and submit a deliberate -new operation. Native Client stale handling likewise does not replay dispatch. - -Timeout, lost response, oversized/invalid mutation response or invalid completion -may mean **OutcomeUnknown**, including a completion with a mismatched request ID -or invalid entry metadata. The adapter queries the *same* request receipt once; -only a committed receipt with matching IDs and valid metadata returns confirmed -DB success. Otherwise it preserves unknown and the `request_id`. Query -`DriveRequestStatus`; an `uncommitted` snapshot is not -proof that an in-flight request cannot later commit. Do not retry blindly with a -fresh request ID. Backend grants also protect receipt lookup/replay. Safe typed -Denied/NotFound/Conflict/Invalid/Limit/StorageUnavailable/OutcomeUnknown errors are -classified without reflecting provider paths, SQL, transport URLs or secrets. - -Saving a Workdir artifact reads through the selected alias's scoped session -`read_bytes`, checks path/offset/size/hash on every bounded chunk and verifies the -assembled SHA256. Source read and destination write are independent. Unknown -aliases, outside-scope paths, traversal/absolute host paths, changed sources, -source read denial, oversized bodies and destination denial cannot publish a -success. Upload transfers raw bytes internally with size/SHA256 metadata; the -model never has to use Bash, a raw host path, or generate base64. Success is the -Backend's **DB-publication commit**, not merely blob transfer completion. - -## Images and historical observations - -ViewImage reads <=10 MiB PNG/JPEG/GIF/WebP bytes in <=64 KiB Drive chunks at the -metadata’s last committed request ID. Every chunk is authorized and committed-request-bound. An intervening -update expires the read; no mixed-generation image is attached. Media is detected -from bytes using the existing image helper, not the filename or a download URL. - -Tools and native Invoke return `ImageAttachment` through `ToolOutput`. Native -attachments accompany—not replace or bypass validation of—the WIP return value. -Engine history, durable Session capture/log replay and normal ToolResult pruning -own those bytes. Later Drive updates/deletion/revocation do not rewrite the text -or image already recorded in a Session; restore reproduces that observation until -normal pruning/correction removes details append-only. This is not full Drive -version history and does not use unrecorded per-request message insertion. - -## Validation seams - -- `cargo test -p worker --lib feature::builtin::drive` -- `cargo test -p worker --lib native_invoke` -- `cargo test -p worker --lib native_attachments_do_not_bypass` -- `cargo test -p worker-runtime --lib binary` -- `cargo test -p yoi-workspace-server --lib worker_drive_tests` - -Server adapter tests use isolated DB + LocalFileSystem + production typed HTTP -router and fresh Runtime proofs/current Worker grants. Worker tests cover -bounded observations, exact CAS inputs, no blind resend, Tools/WIP schemas, -conditional registration, child denial and native image capture/replay. See -`AGENTS.md` for root check, changed-crate tests, format and diff requirements. diff --git a/docs/development/workspace-request-admission.md b/docs/development/workspace-request-admission.md deleted file mode 100644 index de48b6f92..000000000 --- a/docs/development/workspace-request-admission.md +++ /dev/null @@ -1,34 +0,0 @@ -# Workspace request admission and deletion - -Workspace HTTP dispatch uses `WorkspaceAdmission` for **lifecycle admission**, not a Workspace-wide operation mutex. Each admitted handler owns a drop-safe lease. Independent requests (including read-oriented POSTs) may execute concurrently; proof validation occurs before any business/resource wait in the inner router. - -## Ordering and authority - -1. Acquire an immediate lifecycle lease. While deletion drains, new ordinary requests fail closed instead of queueing behind the deletion operation. -2. Validate the bounded request body, Runtime signature, audience/Workspace/permission/method/target/body binding, expiry, transactional JTI consumption, current Runtime trust, Worker membership/reservation and singleton ownership. Browser actor/origin checks are unchanged. No TTL extension or 401 retry policy is introduced. -3. Check the persisted Workspace state. Scoped requests, including GETs, reject a deleting Workspace, including after Server reconstruction. An absent Workspace remains 404. Server-owned deletion status/polling is outside scoped dispatch. -4. Execute the inner router. Resource owners retain their existing exclusion: per-Worker session/control/removal locks; transactional Workdir attachment exclusivity and removal fences; SQLite domain transactions and content/identity precondition checks; the resource broker's one-shot handle consumption. Legacy memory document read/replace/write uses a short, resource-specific mutex because it spans multiple store calls. -5. Drop the lifecycle lease on response, error, or cancellation. An admitted long operation is not rechecked for proof expiry merely because its business execution lasts past the proof TTL. This does not grant subsequent requests or disable provider/attachment authority checks. - -The proof verifier's clock is injected into the Server instance for deterministic route tests, not obtained from test environment variables. Production uses the same wall-clock seconds and existing proof rules. Clock-skew policy remains a separate scope (T-728). - -## Deletion and callbacks - -Deletion start serializes only with other deletion starts. Before awaiting active requests it closes ordinary admission, so ongoing arrivals cannot starve the drain. The fence itself is RAII: a cancelled wait or a rejected preflight reopens admission. Successful reservation persists the deleting state before cleanup is scheduled; the persisted state remains the fence after the temporary drain lease is released. - -While at least one ordinary handler remains admitted, these exact callback paths may obtain their own tracked leases: - -- POST `/api/w/{workspace_id}/subjektiv/sessions` (create/restore session attribution) -- POST `/api/runtime/v1/workspaces/{workspace_id}/resources/fetch` (bound Runtime resource handles) -- GET `/api/w/{workspace_id}/runtime-config` -- GET `/api/w/{workspace_id}/worker-discovery/workers` - -They do not bypass authentication or authorization. Callback admission closes atomically when the last ordinary lease leaves; deletion then waits for already-admitted callbacks too. A callback stream cannot keep adding work once all parents have finished. There is no deletion write-lock queue in front of callbacks required by a parent. - -## Evidence boundary - -`server/tests/workspace_admission_tests.rs` exercises the real outer dispatcher and generated inner routes with isolated SQLite stores, authenticated Runtime proofs and controlled external-provider fixtures. The long-operation fixture stops A at its provider response and proves B's ShowTicket/QueryTicket POSTs and separate Workdir read finish before releasing A. It also covers same-session exclusion, conflicting Workdir attachments, memory edit and JTI atomicity, drain/fence/cancellation, real create/restore with session callbacks overlapping deletion start, resource/config/discovery callbacks, injected-clock proof rejections and provider failures/disconnect/timeout recovery. - -Timeout uses Tokio's test clock; proof tests advance an independent injected clock by 120 seconds. Bounds detect hangs, not guessed sleeps. Dispatch task cancellation represents cancellation of the HTTP handler future; provider disconnect is a dropped transport response. These are in-process boundary tests, not a real-socket disconnect or live browser/Runtime E2E test. Background command and websocket lifetimes retain their existing domain-specific cleanup; request admission covers the handler until its response, not the entire upgraded connection. - -Past production `401 expired` incidents have **not** been correlated with live binary-source/request traces. The structural failure path and deterministic regression evidence do not establish that all prior 401s had this cause. T-731 and its retained session/MR/source/Workdir are unrelated and untouched. WIP remains 0.2.0; no generated DTO/schema/route artifacts change. diff --git a/docs/development/workspace-schema-migrations.md b/docs/development/workspace-schema-migrations.md index 294919b37..450757548 100644 --- a/docs/development/workspace-schema-migrations.md +++ b/docs/development/workspace-schema-migrations.md @@ -1,59 +1,9 @@ -# Workspace schema migrations +# 保存schema変更の開発規則 -Workspace schema authority belongs to `crates/workspace-server/src/store.rs`. The canonical -schema, ordered migrations, migration-history validation, startup upgrade path, and explicit -`yoi-server migrate` command must remain one contract. +公開またはドッグフーディングで使用したmigrationは、対応する保存データを更新できる限り保持する。新しいmigrationを追加したことだけを理由に、直前のmigrationを削除しない。 -## Retained migration chain +schema変更では、既存の順序付きmigrationへ新しい処理を追加し、新規DBの作成結果も同じschemaへ揃える。適用済みmigrationの名前や内容を、語彙を統一するだけの目的で書き換えない。schema一覧はコードを正本とし、文書へ複製しない。 -Released or dogfooded schema migrations are retained and composed in version order. Adding a new -schema version does not authorize deleting the preceding migration. A migration may be removed -only as an explicit baseline-retirement operation after the supported installations that depend on -it have been migrated or intentionally discarded. +検証には、最古の対応schemaからの更新、保存値と制約の維持、新規作成結果との一致、失敗時のrollbackを含める。途中まで完了した更新の再開や、元DBを変更しないdry-runを提供する場合は、その保証も実装経路で確認する。schema番号だけのassertionでは不足する。 -The current retained Workspace chain is: - -1. schema 50: `workspace schema baseline` -2. schema 51: `workspace runtime bindings` -3. schema 52: Runtime-binding counter and audit (historical migration purpose; the persisted migration name remains frozen). -4. schema 53: `durable Workspace deletion operations` - -A database may begin at any retained baseline. Its following history rows must be the exact prefix -of the ordered migration chain from that baseline. This allows both a freshly created current -database and a database upgraded across several releases while rejecting edited, reordered, or -unknown histories. - -## Runtime behavior - -`SqliteWorkspaceStore::open` computes all pending retained migrations and applies them in order. -Each migration is transactional and restartable: if a later step fails, completed steps remain a -valid canonical prefix and the next run resumes from that version. - -`yoi-server migrate` invokes the same store migration path without starting the Server: - -```sh -# Copy the DB into memory and validate the complete pending path without changing the source. -yoi-server migrate --dry-run - -# Preflight the complete path, then apply it to the source DB. -yoi-server migrate -``` - -Use `--database ` for a non-default Server DB. Stop `yoi-server` before applying migrations -and make an external backup before an operational upgrade. - -## Development workflow - -When changing the Workspace schema: - -1. increment `LATEST_SCHEMA_VERSION`; -2. append one `Migration` entry with the new version, stable name, and apply function; -3. preserve all migrations at or above `OLDEST_SCHEMA_VERSION`; -4. update the canonical latest-schema creator for fresh databases; -5. add a fixture at the oldest retained version and prove migration through every retained step; -6. prove that `--dry-run` leaves the source DB unchanged; -7. keep DDL validation and cross-schema foreign-key checks in the shared store preparation path. - -A deliberate baseline retirement must be a separately reviewed change. It must identify the oldest -remaining version, provide an operational migration/discard plan for older databases, update tests -and this document, and must not be inferred merely because a new migration was added. +古いbaselineの対応終了は、対象となる保存データを移行または明示的に破棄する判断を含め、独立にレビューする。残す最古のschema、移行・破棄の責任、影響するテストを明記する。所有storeのmigration経路と検証を共用し、起動・管理操作ごとに別のDDL判定を増やさない。 diff --git a/docs/manifest.toml b/docs/manifest.toml deleted file mode 100644 index 7c43b97da..000000000 --- a/docs/manifest.toml +++ /dev/null @@ -1,328 +0,0 @@ -# ============================================================================ -# Worker Manifest リファレンス -# ============================================================================ -# Worker の宣言的設定 (`WorkerManifest` / `WorkerManifestConfig`)。 -# -# このファイル形式は低レベル runtime manifest。通常起動は profile discovery/default -# (`profiles.toml` と bundled builtin profile) から manifest を生成する。 -# `yoi worker --manifest ` の one-file compatibility/debug mode では、 -# 指定した TOML 1 枚に builtin defaults を merge し、required validation を行う。 -# user/project `manifest.toml` を暗黙に merge する通常起動 cascade は使わない。 -# -# `WorkerManifestConfig` の merge 規則: 上の層が同名フィールドを上書き、scope rule と -# skills.directories は累積マージ、tool_output.per_tool は key 単位でマージ。 -# -# パス解決: `--manifest ` では相対パスはその manifest ファイルの親ディレクトリ -# を base に絶対パスへ解決される。profile artifact でも同じ validation 境界を通る。 -# -# 凡例: -# - 必須 … 値が無いと resolve エラー -# - 任意 … 省略可 -# - デフォルト … 省略時に採用される値 (None なら "なし") -# - 値 … enum 等で取り得る値 -# ---------------------------------------------------------------------------- - - -# ===== [worker] ================================================================ -# Worker メタデータ。 -[worker] - -# 必須。Worker の表示名 (ResolveError::MissingField("worker.name") の対象)。 -name = "example-agent" - - -# ===== [model] ============================================================== -# LLM モデル設定。次の 3 形態を受ける: -# (a) `ref` 単独 — カタログから全部解決 -# (b) `ref` + 一部 override — auth など個別差し替え -# (c) `scheme` + `model_id` 直書き — カタログを使わない inline 指定 -# (b) / (c) では `ref` 未指定なら `scheme` / `model_id` / `auth` が必須。 -# (実際の必須判定は manifest::model_catalog の resolve 側で行う) -[model] - -# 任意。形式: "/"。 -# 最初の `/` だけで split されるので、`openrouter/anthropic/claude-sonnet-4` -# のように内部 model_id に `/` が含まれる場合もそのまま書ける。 -ref = "anthropic/claude-sonnet-4-6" - -# 任意 (ただし ref 未指定時は実質必須)。デフォルト: なし。 -# 値: "anthropic" | "openai_chat" | "openai_responses" | "gemini" -# scheme = "anthropic" - -# 任意 (ref 未指定時は実質必須)。デフォルト: なし。 -# プロバイダが受け付けるモデル ID 文字列。 -# model_id = "claude-sonnet-4-20250514" - -# 任意。デフォルト: scheme ごとの組み込み既定 URL。 -# base_url = "https://api.anthropic.com" - -# 任意 (ref 未指定時は実質必須)。デフォルト: なし。 -# kind の値: "none" | "secret_ref" | "api_key" | "codex_oauth" -# - "none" … 認証不要 (ローカル Ollama 等) -# - "secret_ref" … `yoi keys` の local secret store から key を読む。 -# `ref` はユーザー設定が明示的に選ぶ論理 secret id。 -# store は id -> value のみを持ち、provider 種別を解釈しない。 -# - "api_key" … 明示ファイルから key を読む低レベル形式。通常は -# `secret_ref` を使う。file 指定時、相対パスは manifest base 起点で解決。 -# - "codex_oauth" … ChatGPT OAuth (`~/.codex/auth.json`)。追加フィールドなし。 -# auth = { kind = "none" } -# auth = { kind = "secret_ref", ref = "providers/anthropic/default" } -# auth = { kind = "api_key", file = "./sk-ant.local" } -# auth = { kind = "codex_oauth" } - -# 任意。デフォルト: モデルカタログ → provider.default_capability → scheme 既定 -# の順で解決される。明示 override したいときだけ書く。 -# [model.capability] -# # 値: "none" | "sequential" | "parallel" -# tool_calling = "parallel" -# # 値: "none" | "json_object" | "json_schema" -# structured_output = "json_schema" -# # 任意。値: 省略 (= None) | "effort" | "budget_tokens" | "both" -# reasoning = "both" -# # 任意。デフォルト: false -# vision = false -# # 値: { kind = "explicit", max_breakpoints = } | { kind = "auto" } -# prompt_caching = { kind = "explicit", max_breakpoints = 4 } - - -# ===== [engine] ============================================================= -# ワーカーの生成パラメータ等。セクション自体省略可 (全フィールド任意)。 -[engine] - -# 任意。デフォルト: "default" (`defaults::DEFAULT_INSTRUCTION`)。 -# effective Prompt catalog の exact dotted name を選択する。 -# instruction = "default" - -# 任意。デフォルト: なし (プロバイダ任せ)。 -# 1 レスポンスあたりの出力 token 上限。 -# max_tokens = 4096 - -# 任意。デフォルト: なし (無制限)。 -# 1 セッションの最大ターン数。NonZeroU32 — 0 は parse エラー。 -# max_turns = 50 - -# 任意。デフォルト: なし (プロバイダ既定)。 -# temperature = 0.3 - -# 任意。デフォルト: なし (プロバイダ既定)。 -# top_p = 0.9 - -# 任意。デフォルト: なし (プロバイダ既定)。 -# top_k = 40 - -# 任意。デフォルト: 空配列。 -# stop_sequences = ["\n\n", ""] - -# 任意。デフォルト: なし。 -# 値: -# - 文字列 effort: "minimal" | "low" | "medium" | "high" | "xhigh" -# あるいは provider-native な任意ラベル文字列 -# - 整数: Anthropic 系 thinking.budget_tokens (-1 で dynamic) -# reasoning = "medium" -# reasoning = -1 - -# 任意。デフォルト: "on_tool_call_complete"。 -# provider が個別 ToolCall の完成を保証する場合、そのブロック完成後に実行開始を -# 前倒しする。安全な完成境界がない provider、または応答全体を見て実行前 Pause を -# 判断する turn-end hook がある場合は、応答完了まで警告付きで明示的に遅延する。 -# これは直列実行モードではなく、先行 call の完了を待たず後続 call も開始するため、 -# 完了順・DB 反映順・採番順は保証しない。早期開始後に stream が切断された場合は -# 自動 continuation せず、安全に terminalize して停止する。 -# 従来どおり応答全体の完了後に実行を開始するには "after_response" を明示する。 -# tool_call_dispatch = "after_response" - -# 任意。tool 実行 content の byte 長キャップ。 -# セクション省略時は default_max_bytes = 64 * 1024、per_tool 空。 -# [worker.tool_output] -# # 任意。デフォルト: 65536 (`defaults::TOOL_OUTPUT_MAX_BYTES` = 64 KiB)。 -# default_max_bytes = 65536 -# -# # 任意。デフォルト: 空マップ。tool 名キーで個別キャップ上書き。 -# # キーは tool の登録名 ("Read", "Grep", "Glob", ...)。 -# [worker.tool_output.per_tool] -# Read = 131072 -# Grep = 8192 - -# 任意。submit 時の FileRef (`@`) upload / attachment byte 長キャップ。 -# Tool Output の truncation とは独立している。 -# [worker.file_upload] -# # 任意。デフォルト: 262144 (`defaults::FILE_UPLOAD_MAX_BYTES` = 256 KiB)。 -# max_bytes = 262144 - - -# ===== [scope] ============================================================== -# Worker がアクセスできるディレクトリ／ファイル範囲。 -# - allow: 最低 1 件必要 (空だと ResolveError / ScopeError::EmptyAllow)。 -# 複数 allow がマッチした場合は最大の permission が採用される。 -# - deny : 任意。マッチした deny の最小 permission *未満* に effective を -# 押し下げる (deny.read で完全遮断、deny.write で Read 止まり)。 -# `target` は最終的に絶対パスでなければならない。manifest 内では相対 OK -# (manifest base 起点で resolve)。 - -# 必須: 最低 1 件の allow ルール。 -[[scope.allow]] -# 必須。manifest 内では相対 OK (base 起点で絶対化)。 -target = "./" -# 必須。値: "read" | "write" -permission = "write" -# 任意。デフォルト: true (再帰的にマッチ)。 -# false の場合、ルール自身および直下の child のみマッチ。 -# recursive = true - -# allow は何件でも書ける。 -# [[scope.allow]] -# target = "/abs/docs" -# permission = "read" -# recursive = false - -# 任意。アクセスを *ルール内 permission 未満* に押し下げる。 -# [[scope.deny]] -# target = "./secrets" -# permission = "write" # write を禁止 → 該当パスは Read までに降格 -# recursive = true -# -# [[scope.deny]] -# target = "./secrets/key" -# permission = "read" # read 自体を禁止 → アクセス完全遮断 -# recursive = true - - -# ===== [compaction] ========================================================= -# コンテキスト圧縮 (Prune / Compact)。セクション省略で両方無効。 -# `prune_enabled`でPruneを制御する。Compactは閾値がNoneなら無効。 -# [compaction] -# -# # 任意。デフォルト: false。Compactの設定を保ったままPruneだけ切替できる。 -# prune_enabled = true -# -# # 任意。デフォルト: 8000 (`defaults::PRUNE_PROTECTED_TOKENS`)。 -# # pruning から保護する末尾 token budget。turn 数ではなく usage estimate で境界を引く。 -# prune_protected_tokens = 8000 -# -# # 任意。デフォルト: 4096 (`defaults::PRUNE_MIN_SAVINGS`)。 -# # prune が発火するための最低節約 token 推定値。 -# prune_min_savings = 4096 -# -# # 任意。デフォルト: なし (proactive compact 無効)。 -# # ターン間チェック (Controller post-run)。占有 token > これ で次ターン前に compact。 -# compact_threshold = 80000 -# -# # 任意。デフォルト: なし (safety-net compact 無効)。 -# # ターン中チェック (WorkerInterceptor::pre_llm_request)。期待される関係: -# # compact_threshold < compact_request_threshold (proactive を先に発火)。 -# # 逆順設定は許容するが warn ログを出す。 -# compact_request_threshold = 90000 -# -# # 任意。デフォルト: 8000 (`defaults::COMPACT_RETAINED_TOKENS`)。 -# # compaction 後の history 末尾に verbatim で残す token budget。 -# compact_retained_tokens = 8000 -# -# # 任意。デフォルト: 8000 (`defaults::COMPACT_AUTO_READ_BUDGET`)。 -# # compact worker が `mark_read_required` で取り込める累計 token。 -# compact_auto_read_budget = 8000 -# -# # 任意。デフォルト: 50000 (`defaults::COMPACT_WORKER_MAX_INPUT_TOKENS`)。 -# # compact worker 自身の現在占有 token cap。超過で abort (circuit breaker)。 -# compact_worker_max_input_tokens = 50000 -# -# # 任意。デフォルト: 20 (`defaults::COMPACT_WORKER_MAX_TURNS`)。 -# # compact worker 自身の tool loop 上限。Rust config で None の場合のみ無制限。 -# compact_worker_max_turns = 20 -# -# # 任意。デフォルト: メインモデルを `clone_boxed()` で複製。 -# # compact 専用モデルを使う場合のみ書く ([model] と同じ形式)。 -# # [compaction.model] -# # ref = "anthropic/claude-haiku-4-5" - - -# ===== [feature.memory] (legacy/deprecated) ================================ -# Legacy single-document Workspace Memory compatibility設定。新規product pathは -# `[feature.subjektiv]` を使う。旧`MemoryQuery` / `MemoryReadDocument` / -# `MemoryUpdateDocument` / `MemoryStaging*` toolsと`/memory` APIのinput意味は -# 変更せず、subject Memoryへ暗黙転送しない。 -# -# resolved Worker ManifestではProfile由来の設定を`profile`に、Backend由来の -# Workspace設定snapshotを`workspace_settings`に分離して保存する。source TOMLの -# `[feature.memory]` fieldsはresolve後に`profile`へ入り、`workspace_settings`は -# Backendだけがbindする信頼済み入力で、Profile・Browser・ -# model入力から指定できない。source `enabled = false`（resolvedでは -# `profile.enabled = false`）の場合、legacy Memory tools、 -# resident injection、extract、consolidation requestをすべて無効にし、snapshotも保持しない。 -# `[feature.subjektiv]`の有効化やWorker restoreはlegacy dataをreset/migrateしない。 -# legacy resetは認証済みoperatorが明示APIで別途実行する。 -# -# [feature.memory] -# enabled = true -# staging_tools = false -# -# [feature.memory.resident] -# inject_summary = true -# -# [feature.memory.extraction] -# enabled = true -# reasoning = "medium" -# threshold = 30000 -# worker_max_turns = 8 -# -# # modelは任意。省略時はmain modelをcloneする。 -# # [feature.memory.extraction.model] -# # ref = "codex-oauth/gpt-5.6-luna" -# -# [feature.memory.consolidation] -# request_enabled = true -# -# # Backendがresolved Manifestへbindする。手書き/Profile入力では指定しない。 -# # [feature.memory.workspace_settings] -# # workspace_id = "workspace-id" -# # language = "日本語" -# -# Query結果/抜粋の上限とconsolidation eligibility/thresholdはoperation/Backend -# policyが所有し、通常Worker Manifestには含めない。legacy `[memory]` は拒否する。 - - -# ===== [feature.subjektiv] ================================================ -# Current subject-scoped Memory product。ProfileはFeature activationと自動extract -# policyだけを所有する。subject idはManifest fieldではない。Hostが認証済み -# Runtime/Worker sourceとcurrent `subjektiv:` keyed-singleton ownershipから -# operationごとにbindし、明示Workspace/subject authorizationを再評価する。 -# -# `enabled = true`はsubject tools、Session attribution/recall、confirmed Memory、 -# candidate consolidation、resident surface lifecycleを有効にする。legacy -# `[feature.memory]` dataのimport/reset/dual-writeは行わない。同じWorkerでlegacyと -# subjektivの自動extract destinationを同時に有効化する構成は拒否される。 -# -# [feature.subjektiv] -# enabled = true -# -# [feature.subjektiv.extraction] -# enabled = true -# reasoning = "medium" -# threshold = 50000 -# worker_max_turns = 8 -# -# # modelは任意。省略時はmain modelをcloneする。 -# # [feature.subjektiv.extraction.model] -# # ref = "codex-oauth/gpt-5.6-luna" -# -# # Backendがresolved Manifestへbindする。Profile・Browser・model入力では指定しない。 -# # Workspace identity/language設定のtrusted snapshotであり、subject identityではない。 -# # [feature.subjektiv.workspace_settings] -# # workspace_id = "workspace-id" -# # language = "日本語" -# -# Surface selection/generation/injection、candidate consolidation eligibility、recall -# limitsはHost/Backend policyが所有し、通常Worker Manifestには含めない。current Worker -# authorityはYoi keyed singletonに残り、historical Session attributionを代用しない。 - - -# ===== [skills] ============================================================= -# 外部 Agent Skills (`SKILL.md`) の候補ディレクトリ群。 -# セクション省略 = 何も使用しない (implicit な `$config_dir/skills/` 検索や -# builtin probe は存在しない)。 -# [skills] -# -# # 任意。デフォルト: 空配列。 -# # 各エントリは skills *root* (root の child 各々が `/SKILL.md` を持つ -# # skill バンドル)。root 自身は skill ではない。 -# # 相対パスは manifest base 起点で解決。マージ時は層を跨いで concat される。 -# directories = [".claude/skills", ".cursor/skills"] diff --git a/docs/principle/authority-and-history.md b/docs/principle/authority-and-history.md new file mode 100644 index 000000000..b096eb1a5 --- /dev/null +++ b/docs/principle/authority-and-history.md @@ -0,0 +1,31 @@ +# 正本・権限・実行の分離 + +## 正本と観測 + +永続的な判断の根拠は、その判断を所有する明示的な記録に置く。UIの表示、キャッシュ、通知、プロセスの生存情報は、正本を再確認するための観測として扱う。これらだけから操作の成功、作業の完了、権限の継続を推定しない。 + +永続化する情報と、現在の状態から再構成できる情報を分ける。別の所有者が管理する状態を複写して第二の状態機械を作ると、更新漏れと復旧時の不一致が生じる。親の処理は子の識別子を保持し、必要な時点で子の正本を読む。 + +## 権限と入力 + +操作権限は認証された主体、明示的な付与、対象資源の現在の制約から決定する。プロンプト、Profile名、表示名、cwd、ファイルの存在、ツールが見えることを権限の根拠にしない。別の操作経路を設けても、同じ所有者の検証を通す。 + +委譲では親が実際に委譲できる範囲を上限にする。読取り、書込み、コマンド実行、観測、レビュー、統合を混同しない。探索で対象を発見できても操作権限を得たことにはならない。 + +## 履歴とLLM入力 + +LLMの生成に影響する入力は、リクエストへ加える前に履歴へ確定し、永続化形式からリクエストを再構成できるようにする。揮発的な入力の差込みでは、後のターンから発言やツール呼出しの根拠を追えなくなる。 + +履歴は追記を基本とする。圧縮や保持範囲の変更は、明示的な境界と再構成規則を持つ処理に限定する。過去の記録を黙って書き換えず、キャッシュとトークン消費への影響をセッションの実測に基づいて判断する。 + +## 初期設定と実行処理 + +環境変数などの暗黙的なプロセス状態は、起動境界で明示的な入力へ変換する。通常の処理は解決済みの型付き設定を受け取り、実行中に環境を読み直して判断を変えない。秘密情報は明示的な参照で扱い、設定の説明や診断へ値を複製しない。 + +## 状態と操作の独立性 + +作業の結論、レビュー判断、コード統合、Worker停止、記録の保持、接続解除、資源削除は別の判断である。一つの成功を別の操作の許可として流用しない。自動的に連動させる場合は、その責任を持つ操作として条件と結果を定義する。 + +競合確認では、何を、どの観測と比較し、どの操作の前後で検査するかを名前と処理で示す。到着順や時刻で足りる処理に不透明な更新番号を加えない。内容の一致、同じ操作の再送、鍵の一致、接続の生存期間を確認する場合は、それぞれを識別する根拠を使う。 + +個別の永続操作と再試行の判断は[永続操作の原則](durable-operations.md)に従う。具体的な認証、保存、復旧の仕組みは所有moduleのドキュメントに置く。 diff --git a/docs/development/ui-ux/design-language.md b/docs/principle/design-language.md similarity index 96% rename from docs/development/ui-ux/design-language.md rename to docs/principle/design-language.md index bb70169e6..620180d6f 100644 --- a/docs/development/ui-ux/design-language.md +++ b/docs/principle/design-language.md @@ -1,6 +1,6 @@ # Workspace Web Design Language -この文書は、Workspace Webのrendered resultが満たす設計規則を定義する。Product固有のresource構成は[`product-ux.md`](product-ux.md)、application shellとSidebar slotの実装構造は[`application-architecture.md`](application-architecture.md)、CSSとsource配置は[`../../../web/workspace/README.md`](../../../web/workspace/README.md)、visual review手順は[`visual-review.md`](visual-review.md)をauthorityとする。 +この文書は、Workspace Webのrendered resultが満たす設計規則を定義する。 一つの規則は最も具体的な一箇所だけをauthorityとする。具体的な二案の一方を棄却できない文は規則として残さない。 diff --git a/docs/principle/durable-operations.md b/docs/principle/durable-operations.md new file mode 100644 index 000000000..40a8c2fb7 --- /dev/null +++ b/docs/principle/durable-operations.md @@ -0,0 +1,43 @@ +# 永続操作と再試行 + +永続操作には、同じ依頼を識別する情報と、安全な再試行に必要な事実を保存する。Rustの関数呼出しや待機位置を、そのまま永続状態として記録しない。再読取りで分かる情報や安全に繰り返せる処理は、再構成または再実行する。 + +## 記録の分類 + +| 種類 | 保存する理由 | +| --- | --- | +| 原子的な冪等性記録 | 正本の更新と結果が一つのDBトランザクションで確定する。同一依頼の再送には保存済み結果を返し、異なる依頼による操作IDの再利用は拒否する。途中状態は不要。 | +| 予約 | 後続の結付けに先立ち、資源IDや排他的な権利を確保する。予約先、依頼、権限、確定した結付けを記録する。 | +| 外部副作用を伴う操作 | DB外で起きた効果を再試行で重複させないため、安定した依頼ID、必要なチェックポイント、結果を残す。 | +| 親ワークフロー | 複数の操作を調整する。親の依頼、権限、安定した子操作ID、最終結果を持ち、子の状態は子の正本から読む。 | + +同じDB内で原子的に確定できる処理を、永続操作を設けるためだけに分割しない。補償にも独立した副作用や再試行がある場合は、暗黙的な後始末ではなく、識別可能な子操作として扱う。 + +## 操作の識別 + +操作IDは所有するWorkspaceなどの権限範囲で識別する。依頼のfingerprintには、変わると別の依頼になる入力を含める。Serverが採番した結果ID、時刻、試行回数、変動する観測値、秘密の値は含めない。 + +厳密な再送で固定する権限・設定と、実行時に再検証する制約を分ける。固定した入力があっても、現在の割当、所有、接続、削除制約の検証を省略しない。子の作成も安定したIDまたは原子的な結付けで冪等にし、再送で別の子を増やさない。 + +## チェックポイントの条件 + +チェックポイントは、再試行の方法が変わる事実を記録する。次の条件をすべて満たす場合だけ追加する。 + +1. 現在のトランザクション外で副作用が発生し得る。 +2. その事実を十分確実に再観測できない、または安全に繰り返せない。 +3. 記録の前後で再試行の処理が変わる。 +4. 記録の前後それぞれの挙動をテストできる。 + +「検証中」「終了処理中」などコード位置を表す段階名ではなく、削除確認、子操作との結付け、確定結果の識別子などの事実を保存する。要求を送信した事実と、副作用の完了を確認した事実は区別する。 + +## 状態・失敗・処置 + +操作が未完か完了か、直近の失敗、現在の阻害条件、保持・削除などの処置結果は別の情報である。阻害条件は実行前に現在の正本から再判定する。過去の失敗から再試行可能性を推測せず、所有するdomainの規則に従う。 + +外部providerには、利用できる冪等性キー、期待する内容、安定した資源IDを渡し、権威ある応答または観測で結果を確認する。応答喪失を失敗と決めつけて、新しい操作IDで副作用を繰り返さない。確認できなければ最後に確定した事実を保持し、結果不明または要確認として扱う。 + +## 保証の限界 + +対象は重複要求、返却エラー、timeout、既知の部分成功、最後に確定した記録からの再起動復旧である。任意の命令位置でのpanic、強制終了、マシン喪失、電源断に対して、一度だけ実行されることを一般的には保証しない。すべての`await`の前後に状態を書いて、その保証を装わない。 + +破壊的・機密性の高い操作に強い耐障害性が必要なら、providerの契約、記録順、再送規則、照合根拠を個別に定義する。診断には調査と再試行に必要な識別子・分類だけを残し、秘密、生のプロンプト、セッション全文、内部パスを複製しない。 diff --git a/docs/principle/ux-design.md b/docs/principle/ux-design.md new file mode 100644 index 000000000..9ff3514d7 --- /dev/null +++ b/docs/principle/ux-design.md @@ -0,0 +1,61 @@ +# Workspace WebのUX設計 + +Workspace Webの画面構成と、リソースごとの表示・操作を決める規則を定める。視覚表現と操作の共通規則は[Design Language](design-language.md)に従い、この文書では製品固有の判断を扱う。 + +## 作業面の目的 + +Workspace Webは、Workspaceの確定した状態を確認し、注意が必要な箇所を判断し、次の操作を行うための作業面とする。マーケティング用のdashboard、分析画面、Backend DTOをそのまま並べる画面にはしない。 + +操作の流れは、対象のWorkspaceとリソースを選び、現在の状態を確認し、操作を行い、その結果を確認する順とする。通常の表示とURLには各リソースのcanonical keyを使い、内部UUIDをlabelや代替リンクとして露出しない。 + +## ナビゲーションの区分 + +- Globalでは、Workspaceの選択・作成、Account、Device Loginなど、Workspaceを選ぶ前の操作を扱う。 +- Workspaceでは、Tickets、Objectives、Merge Requests、Drive、Memory、Workersなど、選択中のWorkspaceで日常的に扱う対象へ移動する。 +- Settingsでは、Runtime、Repositoryとそのアクセス設定、Configuration Sources、Profile Sources、Workspace Identityなど、Workspaceの管理と構成を扱う。これらを日常作業のナビゲーションへ混ぜない。 + +存在しないWorkspace、閲覧できないWorkspace、読込みに失敗したWorkspaceは区別し、別のWorkspaceへ暗黙に切り替えない。読取り専用の場合も、利用できる閲覧機能を残す。 + +## ページの目的 + +各routeの主要目的は、「Ticketを選ぶ」「Workerを確認する」「Repositoryのアクセスを設定する」のように、動詞と対象を一つずつ使って表現できるものにする。 + +- 一覧・boardは、対象を探し、注意が必要な項目を見つけるために使う。 +- 詳細は、一つの対象を理解し、現在実行できる操作を行うために使う。 +- form・editorは、範囲の明確な変更を行うために使う。 +- Consoleは、一つのWorkerを操作しながら履歴を確認するために使う。 + +一つのrouteで複数の形式を組み合わせるのは、同じ目的を支える場合に限る。 + +## 一覧と詳細 + +同じ項目をリソース間で比較する一覧には、平坦なlistまたはtableを使う。人が読む名前とcanonical keyを最初の列に置く。providerの詳細、変更ID、digestは比較用の列へ常設せず、詳細表示や診断用の開閉領域へ置く。 + +詳細では、対象の識別情報、現在の状態、注意が必要な事項を最初に示し、本文や履歴を主領域に置く。関連、target、review、workflowの状態をそれぞれ別の主体が確定する場合は、独立したsectionに分ける。 + +## Ticket board + +boardは横方向にlaneを比較する作業面とする。laneはworkflow stateを表し、隣接laneとの関係を維持する。狭い画面でもboard内の横スクロールで比較できるようにする。 + +各Ticketにはkey、title、優先判断を変える情報を表示する。blockerとattentionは文字でも示す。 + +## Settingsと編集 + +一つのSettings routeは一つの設定関心事を扱う。編集操作の前または同時に現在有効な値を示し、一つの判断に必要な入力項目をまとめる。form内で塗りつぶしによって強調する操作は保存だけとする。 + +## Memory + +Memoryは、明示的に選択したsubjectを起点に閲覧する。subjectの詳細では現在接続しているWorkerを示し、過去のWorkerやSessionの帰属情報を現在の接続として表示しない。 + +- `Surface`は生成されたresident contextの現在の状態を示す。生成済み、生成済みだが空、未生成、古くなった状態、生成失敗を区別する。取得要求の失敗を生成状態として扱わない。 +- `Committed Memory`は現在の確定Memoryを一覧と詳細で示す。変更履歴は現在の内容とは別に閲覧できるようにし、過去の変更内容を現在値の代わりに表示しない。 +- `Sources`では、候補とその処理結果、根拠となったMemoryと変更、Hostが記録したSessionへの帰属をたどれるようにする。Sessionの内容を表示するのは閲覧権限がある場合に限る。 + +## Worker Console + +Consoleは会話と実行の履歴を主領域とする。コマンド出力はterminalに近い簡潔な表示にし、通常のメッセージ本文と分ける。 + +- 直接のSubWorkerの履歴を表示している間も、Composerと実行制御の対象は親Workerのままとする。 +- OverviewとNormalの切替では見せ方だけを変え、操作対象や権限を変えない。 +- 現在のWorker状態はWorkerの確定したsnapshotから表示し、WebSocketの接続状態や一時的なeventで置き換えない。 +- 非公開の推論と生のsystem promptは、debug表示でも公開しない。 diff --git a/docs/standalone.md b/docs/standalone.md deleted file mode 100644 index 524aa08f6..000000000 --- a/docs/standalone.md +++ /dev/null @@ -1,52 +0,0 @@ -# Standalone local Subjects - -Standalone runs one client-owned Worker in the current process, without a Backend or Workspace. Local Subjects are explicitly created and selected; enabling a subjektiv Profile alone neither creates nor attaches a Subject. - -## Create and list - -```sh -yoi --local subject create assistant --behavior-md 'Keep durable decisions and preferences.' -yoi --local subject list -``` - -`create ` prints the created Subject record as JSON. `--behavior-md ` is optional inline text, not a file path. `list` prints a JSON array of records in the selected standalone state scope. Both commands require explicit `--local` and reject Backend/Workspace selectors. - -The state directory is obtained from the same local connection target as normal standalone launch. Its base follows the existing `YOI_DATA_DIR`, `YOI_HOME`, `XDG_DATA_HOME`, or `HOME` resolution, with `client/standalone/workers` appended. Subjects live under `/subjektiv/`; cwd, repository files, Worker names, and Profiles are not Subject identity or sharing boundaries. Use the returned common-domain Subject ID (`subject-`), not the role, for selection. - -The user-managed state root must be operator-owned and private (mode **0700**). A missing root is created with mode 0700; an existing root is validated, not automatically chmodded. An older nonprivate root is rejected. The operator can explicitly run `chmod 700 ` on their own state root before retrying; do not substitute a repository directory or assume the CLI repairs permissions. Symlink and nonprivate Subject storage nodes are also rejected. - -## Select for a new Worker - -```sh -yoi --local --subject --profile builtin:standalone-subjektiv -``` - -`--subject=` is also accepted. The selector must be nonempty and may only be supplied once. It applies only to fresh local spawn: Backend selection, Runtime attach, socket/session attach, and resume combinations are rejected. A selected Subject requires a compatible subjektiv policy; launch validation rejects incompatible policy before Worker startup side effects. - -The interactive local spawn form has an optional `subject` field, initially `(none)`. Press **F2** to switch between editing the Worker name and Subject ID; **Tab/Up/Down** continue to cycle Profiles. Leave the Subject field empty for the existing Subject-free launch path. The form does not create Subjects or open Subject storage just to display the field. The optional built-in `builtin:standalone-subjektiv` Profile enables local Subject behavior; its consolidation Profile is `builtin:standalone-subjektiv-consolidation` and does not require Backend resources. - -## Resume - -```sh -yoi --local resume -``` - -Resume restores the original persisted Subject/storage-scope binding and reacquires its execution lease. It does not accept a new `--subject` selector or silently change Subjects. The Host holds an exclusive per-Subject kernel lease for body execution and delegated Jobs until confirmed shutdown; one Subject cannot be executed concurrently by two processes. Process death releases the kernel lease, while uncertain in-process cleanup retains ownership. Separate Subjects may run concurrently in the same state scope. - -## Subject Jobs and recovery - -A selected Subject uses `/subjektiv/jobs//jobs.sqlite3`, shared across Worker IDs for that Subject and protected by its exclusive execution lease. Subject-free Hosts retain the existing Worker-local Job location. There is no separate Subject runner or background daemon. - -On explicit selected-Subject Host startup, reserved Jobs resume automatically. An interrupted Job whose execution outcome is unknown is **never automatically replayed**. A successful result that has not yet been acknowledged is delivered from durable storage without retrying the model. Recovery is driven by an explicit Host startup, not an automatic process restart or independent scheduler. - -## Memory and historical Sessions - -The normal body Profile has recall, exact Memory change reads, explicit candidate tools, and bounded historical Session exploration. It cannot directly apply Memory. Explicit Remember/Revise without `entry_refs` returns a pending-commit receipt; the Host stages its exact tool-call evidence only after the logical Run is append-only committed. Pre-request retry may defer an open Run, but commit/rewrite failures are not silently treated as success. A candidate remains unconfirmed until the restricted consolidation Job applies or rejects it using the shared CAS/receipt rules. - -Normal extraction uses the existing committed-capture lifecycle, restricted extraction tools, and append-only `subjektiv.extract.v1` pointer. The selected Profile owns extraction thresholds/model policy. An explicit extraction model override chooses that provider, not an injected dialog transport. Local Memory language defaults to English and is stored in operator-managed `subjektiv/language.json`, never loaded from repository policy files. - -The Host stamps human input on its owned in-process client channel as persisted `local_human_input`, projected as the existing `HumanInput` provenance. It neither invents a Backend account nor trusts serialized source claims. Ordinary SubWorker/Job input retains its own nonhuman origin. Local evidence has no Workspace/account/Runtime identity; local storage-scope identity and Worker binding authorize discovery separately. Historical Sessions are not copied into SQLite. Reads/search exclude system prompts, hidden reasoning, incomplete logical Runs, partial JSONL tails, and foreign bindings; cursor continuations revalidate scope/Subject/Worker attribution and committed generation. - -## Validation boundary - -Host injection tests pass a clone of the injected model transport into delegated Jobs, so scripted body and Job calls use the same test-owned transport. These are local contract tests, not live-provider validation or evidence of model/consolidation quality. Generic Job finalizer and Host integration proofs are validated separately from the CLI/parser and spawn-form tests. diff --git a/resources/plugin/templates/rust-component-instance/README.md b/resources/plugin/templates/rust-component-instance/README.md index a68d591ca..7d0c2fd01 100644 --- a/resources/plugin/templates/rust-component-instance/README.md +++ b/resources/plugin/templates/rust-component-instance/README.md @@ -1,7 +1,5 @@ -# Rust Component Model instance Plugin template +# Rust Component Instance Plugin雛形 -This offline authoring template declares the proposed `yoi:plugin/instance@1.0.0` world. Yoi currently provides no Plugin installation or Worker execution path. +提案段階の`yoi:plugin/instance@1.0.0`を対象にしたオフラインpackageの雛形である。YoiはPluginのインストールやWorker内での実行を提供していない。 -The example contains request/response Tool and Service ingress shapes that a future sandboxed Server Plugin platform may support. Their host API and grant metadata is inert in the current product and grants no authority. - -Build with `cargo component build --release` (or the project-specific build command used by your package), then run `yoi plugin check .` and `yoi plugin pack .` against the explicit directory. Passing offline validation does not install or authorize the package. +Tool・Serviceの要求と応答は将来の実行境界を検討するための例である。host APIやgrantのmetadataは現在の製品では権限を付与せず、オフライン検査の成功も実行の許可を意味しない。 diff --git a/resources/plugin/templates/rust-component-tool/README.md b/resources/plugin/templates/rust-component-tool/README.md index 2dcf25d5d..506ff64ee 100644 --- a/resources/plugin/templates/rust-component-tool/README.md +++ b/resources/plugin/templates/rust-component-tool/README.md @@ -1,37 +1,5 @@ -# Rust Component Tool Template +# Rust Component Plugin雛形 -This is the embedded starter template for a Yoi Component Model Tool Plugin written with the first-party Rust PDK. +提案段階の`yoi:plugin/tool@1.0.0`を対象にしたオフラインpackageの雛形である。YoiはPluginのインストールやWorker内での実行を提供していない。 -## What this template demonstrates - -- Offline package metadata targeting the proposed `yoi:plugin/tool@1.0.0` world. -- Guest-side runtime binding setup through the PDK. -- Typed JSON input parsing through `run_json_tool` via `export_component_tool!`. -- Typed JSON output serialization with `ToolOutput::json`. -- Structured, bounded `ToolError` output for user-visible Tool failures. - -The PDK is guest-side only. It does not grant filesystem, network, or environment authority. Yoi currently provides no Plugin installation or Worker execution path; this template is retained for offline package authoring. - -## Checkout/development dependency - -Inside the Yoi checkout this template uses a local path dependency and declares an empty `[workspace]` so it can be checked in place without becoming a member of Yoi's root workspace: - -```toml -yoi-plugin-pdk = { path = "../../../../crates/plugin-pdk" } -``` - -If this template is copied to an independent Plugin repository, pin a Yoi source commit with `rev` instead of tracking a branch. Use the repository root `.git` URL, not the browser `/src/branch/...` URL: - -```toml -yoi-plugin-pdk = { git = "https://gitea.hareworks.net/Hare/yoi.git", package = "yoi-plugin-pdk", rev = "" } -``` - -`plugin.component.wasm` in the template is a text placeholder so `yoi plugin check` and `yoi plugin pack` can exercise deterministic local package validation immediately. Replace it with a real built component before treating the offline package as verified. - -## Next steps - -1. Replace package/plugin ids, names, descriptions, and Tool schema. -2. Replace `EchoInput` / `EchoOutput` and `handle_echo` with your Tool logic. -3. Build the Rust component artifact for `wasm32-unknown-unknown`, replacing the placeholder `plugin.component.wasm`. -4. Run `yoi plugin check .` and `yoi plugin pack . --output ./my-plugin.yoi-plugin`. -5. Retain the package as an offline artifact. Yoi does not currently install or execute it. +guest側の型付き入出力とエラー表現はRust PDKが担当し、ファイル・ネットワーク・環境への権限は付与しない。`plugin.component.wasm`はpackage検査用のテキストplaceholderであり、実行可能なcomponentの証拠にはならない。 diff --git a/tests/e2e/README.md b/tests/e2e/README.md new file mode 100644 index 000000000..678300006 --- /dev/null +++ b/tests/e2e/README.md @@ -0,0 +1,9 @@ +# yoi E2E + +実際の製品プロセスを隔離したfixtureで検証する。PTY操作とBackend／Runtimeプロセスの検証を担当し、通常のunit testとは分離する。 + +## 実装の入口 + +- [`PanelHarness`]: PTY経由の入力、観測イベントによる同期、失敗時のartifact保存。 +- [`FixtureWorkspace`]: TUI用WorkspaceとHOME／XDG領域の隔離。 +- [`WorkerE2eEnvironment`]: Backend／Runtime用プロセスとfixtureの管理。 diff --git a/tests/e2e/src/lib.rs b/tests/e2e/src/lib.rs index 6719adbaa..176c7c2ff 100644 --- a/tests/e2e/src/lib.rs +++ b/tests/e2e/src/lib.rs @@ -1,8 +1,4 @@ -//! Opt-in E2E helpers for driving the real `yoi panel` process through a PTY. -//! -//! The harness intentionally sends keyboard and mouse input only through the PTY. -//! Structured JSONL events emitted by the TUI are used for synchronization, -//! assertions, and failure artifacts; they are not an input or authority channel. +#![doc = include_str!("../README.md")] use std::fs::{self, File, OpenOptions}; use std::io::{self, Read, Write}; @@ -567,6 +563,10 @@ pub struct PanelArtifacts { pub run_json: PathBuf, } +/// 実TUIへの入力をPTYだけで送るharness。 +/// +/// JSONL観測イベントは同期、assertion、失敗時のartifactに使う。製品状態を +/// 変更する入力には使わない。最初の描画と具体的なfixture内容の表示準備は別に検査する。 pub struct PanelHarness { child: Child, master: File, diff --git a/tests/e2e/src/worker_environment.rs b/tests/e2e/src/worker_environment.rs index 5ab78e053..e0c4eafb2 100644 --- a/tests/e2e/src/worker_environment.rs +++ b/tests/e2e/src/worker_environment.rs @@ -207,6 +207,10 @@ pub struct WorkerEnvironmentCleanupReport { pub report_path: PathBuf, } +/// 隔離したHOME／XDG領域と実Backend／Runtime subprocessを所有するfixture。 +/// +/// HTTP clientはproxyとredirectを使わず、テスト対象の接続先を明示する。 +/// 終了時に所有processと一時領域を片付け、artifactとcleanup結果を残す。 #[derive(Debug)] pub struct WorkerE2eEnvironment { temp_root: Option, diff --git a/tools/web-ux/README.md b/tools/web-ux/README.md index 8fe963bac..ed34cb958 100644 --- a/tools/web-ux/README.md +++ b/tools/web-ux/README.md @@ -1,275 +1,13 @@ -# Web UX inspection workbench +# Web UI検証支援 -`tools/web-ux` is a development-only Playwright workbench for repeatable visual inspection of the -real Web Workspace. It does not add a Yoi product Skill, Flow, Runtime capability, or browser -automation route. +`tools/web-ux`は、実際のWeb画面とブラウザの診断を同じ条件で採取する開発用ツールである。製品のSkill・FlowやRuntime権限は追加しない。合否は[目視確認の規則](../../docs/development/ui-ux/visual-review.md)に従って実装者が判断する。 -The required implementer-owned review workflow and pass/fail policy are defined in -[`../../docs/development/ui-ux/visual-review.md`](../../docs/development/ui-ux/visual-review.md). This -README documents the tool; running the tool is not itself visual review completion. +入口は[`cli.ts`](cli.ts)、採取条件は[`scenarios/`](scenarios/)、保存証拠の構造は[`src/types.ts`](src/types.ts)に置く。撮影・認証fixture・プロセス管理の理由は対応する`src/`のコメントで管理する。 -The workbench produces a **review context bundle** rather than treating a screenshot as evidence by -itself. Every capture records the persona, route, viewport, theme, intended user goal, expected data -state, sanitized document URL/status, console/page/request failures, screenshot hashes, an -accessibility snapshot, source Git commit SHA, and browser version. +## 維持規則 -## Environment - -Enter the repository dev shell. The shell supplies the Nix-pinned Chromium build and sets -`PLAYWRIGHT_BROWSERS_PATH`; Playwright does not download a browser at runtime. - -```sh -nix develop -cd tools/web-ux -deno task check -deno task test -deno task test:browser -``` - -`test:browser` starts a deterministic fixture server owned by the test, captures distinct owner and -non-owner contexts, verifies the review bundle, and proves server/browser cleanup. It must run -inside `nix develop` so it uses the pinned browser. - -The npm Playwright version in `deno.json` must match `pkgs.playwright-driver.version` in the pinned -Nixpkgs input. Update both as one toolchain change. - -## Scenario contract - -Scenarios are reviewed JSON files under `scenarios/`. A scenario fixes: - -- personas and whether each uses an isolated anonymous context or a local Playwright storage-state - file; -- explicit routes and user goals; -- expected data state, viewports, theme, locale, timezone, and reduced-motion mode; -- an explicit readiness condition for every route and optional interaction/capture-point conditions; -- selectors and exact environment-derived text that must be redacted; -- optional processes owned by the capture command, including an HTTP readiness URL. - -`${UPPER_CASE_ENV}` values are expanded at runtime. URLs with embedded credentials are rejected. -Route readiness is bounded and retried twice; it never relies on a fixed sleep. `network-idle` is -available but should be used only for screens whose contract actually reaches idle. Prefer a stable -screen-owned selector. - -`workspace-control-plane.json` expects: - -```sh -export WEB_UX_BASE_URL='http://127.0.0.1:5173' -export WORKSPACE_ID='' -export XDG_STATE_HOME="${XDG_STATE_HOME:-$HOME/.local/state}" -``` - -## Authentication fixtures - -Authentication state is local sensitive material stored under `$XDG_STATE_HOME/yoi/web-ux/auth/`, -outside the Repository and Workdir. Files are written with mode `0600`, state contents are never -copied into a review bundle, and the CLI never prints cookies or credentials. Each profile has a -sidecar binding it to the exact persona and base URL origin with a 12-hour default expiry. Capture -fails explicitly when metadata is missing, the origin differs, or the profile has expired; it never -silently reuses or refreshes that state. - -For an interactive Passkey/browser login: - -```sh -deno task web-ux auth \ - --scenario scenarios/workspace-control-plane.json \ - --persona owner -``` - -The command opens Chromium at the configured login route, waits up to five minutes for the -scenario's success URL, saves `storageState`, and closes the browser in `finally`. Repeat for -`non-owner` using a real account with that permission projection. - -A test fixture may already provide Playwright-compatible `{ cookies, origins }` state. Import it -without putting its value on the command line: - -```sh -deno task web-ux auth \ - --scenario scenarios/workspace-control-plane.json \ - --persona owner \ - --import-state /private/path/owner-state.json \ - --expires-in-hours 8 -``` - -Delete both the profile and its metadata when it is no longer needed: - -```sh -deno task web-ux auth \ - --scenario scenarios/workspace-control-plane.json \ - --persona owner \ - --delete -``` - -Do not place passwords, bearer tokens, private keys, WebAuthn material, or inline cookies in a -scenario, process arguments, a Repository URL, or `redact.text`. `redact.text` is only a final -defense for a secret already supplied through an environment-owned fixture; it is not a credential -transport. - -## Capture and inspect - -Capture a stable multi-persona bundle: - -```sh -deno task web-ux capture \ - --scenario scenarios/workspace-control-plane.json \ - --output ../../target/web-ux \ - --run-id before-change -``` - -Use filters for a bounded feedback loop: - -```sh -deno task web-ux capture \ - --scenario scenarios/workspace-control-plane.json \ - --output ../../target/web-ux \ - --run-id ticket-list-after \ - --personas owner,non-owner \ - --routes tickets \ - --viewports desktop -``` - -The command exits `2` when it produced evidence but observed UI/tool errors, and exits `1` when -capture itself failed. It continues other route/persona captures after a bounded route failure. -Inspect: - -- `review-context.json` for the exact context, hashes, HTTP status, retained/truncated diagnostic - counts, route and capture-point readiness, and the redacted interaction sequence; -- `contact-sheet.png` through its manifest `workdirPath` with an image-capable reviewer for - composition, hierarchy, density, clipping, empty/error states, and permission-specific - affordances; -- each `accessibility.md` through its manifest `workdirPath` for landmark/name/state evidence that a - screenshot cannot prove; -- `process-logs/` when the scenario owns a server process. Each stdout/stderr stream is redacted, - capped at 1 MiB, and paired with truncation metadata. - -The implementing agent must inspect the actual contact sheet (for example with `ViewImage`), record -concrete findings, fix them, recapture under the same persona/route/viewport filters, and inspect -the new evidence. Playwright success alone is not visual acceptance. - -## Compare before and after - -```sh -deno task web-ux compare \ - --before ../../target/web-ux/before-change/review-context.json \ - --after ../../target/web-ux/after-change/review-context.json \ - --output ../../target/web-ux/before-vs-after -``` - -`comparison.html` and `comparison.png` show before, after, and pixel diff side by side. -`comparison.json` records changed-pixel counts, dimension mismatches, unmatched capture keys, and -diff hashes. Pixel differences are orientation evidence, not a correctness verdict; explain expected -animation/font/data changes and inspect the actual UI. - -Capture keys are stable across runs: `persona / route / viewport / capture-point`. Keep those -identities unchanged when comparing the same user task. - -## Process and artifact cleanup - -The capture command owns only processes declared in its scenario. It starts them without a shell, -records bounded/redacted output, and terminates the process and descendants on success, capture -failure, or interruption observed by the command. It never stops an existing Yoi Server or Runtime -that it did not start. - -Old complete review bundles can be removed without touching auth state or arbitrary directories: - -```sh -deno task web-ux cleanup --output ../../target/web-ux --keep 5 --older-than-days 14 --dry-run -deno task web-ux cleanup --output ../../target/web-ux --keep 5 --older-than-days 14 -``` - -Cleanup recognizes only directories containing `review-context.json`. The repository `target/` tree -is ignored by Git, while authentication state remains outside the repository. `capture` defaults to -`target/web-ux` when `--output` is omitted. Keep a bundle outside Git or publish it through the -approved immutable artifact channel when durable review evidence is required. - -## Workspace Home fixture - -`scenarios/workspace-home.json` captures the production work dashboard with owner, non-owner, -long-name, empty, and intentionally failing-review Workspace projections at 1440px, 768px, and 390px. -The partial-error route deliberately returns HTTP 503 only for Merge Requests; captures should retain -that request/console/visible error while showing successful active-work and recent-update feeds. -It uses no real account or private Workspace data. Build `web/workspace` first, then run from `tools/web-ux`: - -```sh -deno task web-ux capture --scenario scenarios/workspace-home.json --output ../../target/web-ux --run-id home-review -``` - -The scenario owns a fixture server on localhost port 15281 and stops only that server. The production -Home browser regression (`browser-tests/workspace_home_shell_test.ts`, included in `test:browser`) -also checks 320px, both themes, keyboard record navigation, bounded loading, independent slow/error -feeds, explicit refresh, and absence of Host requests. - -## Workspace Identity settings fixture - -`scenarios/workspace-settings.json` uses the same owned fixture server with synthetic metadata, -public identity, and deletion-preflight responses. It covers owner, member, long name, pending -provisioning, and intentional signing-identity HTTP503 states at 1440px, 768px, and 390px. -Build `web/workspace` before capture. No real keys, accounts, or deletion requests are used. -`browser-tests/workspace_settings_shell_test.ts` also checks 320px and both themes, Edit/Save/Cancel, -keyboard disclosure/focus return, full public bundle access, scoped errors, and exact-name deletion -confirmation. Interaction screenshots are saved under `target/web-ux/settings-interactions`. - -## Console pending input fixture - -`scenarios/console-pending.json` uses the owned Console history fixture with `?pending=both`, -`queue`, `notifications`, `legacy`, or `empty`. Each state includes a task for typography comparison. -It captures 1440px, 768px, and 390px; `browser-tests/console_pending_shell_test.ts` additionally checks -320px and both themes, independent group visibility, full-width single groups, uniform 16px rows, -monospace inheritance, truncation, and retained-session cancellation gating. Live ordered-FIFO-content-fenced -cancellation and state transitions are covered by Console component tests. No real Worker is used. - -## Console live Feature invocation fixture - -`scenarios/console-live-light.json` and `console-live-dark.json` capture the **production Console** -with a synthetic live Worker at 1440px, 768px, and 390px. The fixture provides metadata for `review` -(enum, integer, boolean, and Worker-file arguments) and `attach` (the generic attachment client -adapter), a real workspace protocol WebSocket, request-ID-preserving completion responses, and -synthetic upload staging. It never executes a Worker or Feature invocation. No fixture CSS, shell, -or editor overrides are used. - -Build production sources first, optionally from an isolated copy under `target/web-ux`, then select -the build explicitly. From `tools/web-ux`: - -```sh -export WEB_UX_BUILD_ROOT='' -deno task web-ux capture --scenario scenarios/console-live-light.json --output ../../target/web-ux --run-id console-live-light -deno task web-ux capture --scenario scenarios/console-live-dark.json --output ../../target/web-ux --run-id console-live-dark -deno test --allow-env --allow-net --allow-read --allow-write --allow-run --allow-sys browser-tests/console_live_shell_test.ts browser-tests/console_live_admission_shell_test.ts -``` - -`WEB_UX_LIFECYCLE_OUTPUT` optionally places browser interaction screenshots and JSON evidence at an -explicit artifact path. Tests consume an existing build; they do not rebuild or mutate Web sources. -The lifecycle test covers argument completion, editing/removing/undoing typed chips, native picker -cancellation, generic-adapter staging, literal unselected slash prose, typed Submit values, and -abandoned-file DELETE. Admission tests verify rejection preserves drafts and staged files, -transport send alone cannot clear the draft, authoritative acceptance transfers ownership, and -failed/in-flight-cancelled uploads cannot Submit. A retry/late-success matrix verifies stable upload -IDs, deleted reservation non-resurrection, and abandoned resource cleanup at accepted draft boundaries. -Intentional upload HTTP503 browser errors are recorded separately from unexpected failures. -The Chromium picker helper awaits interception acknowledgement before real gestures (not a timeout -extension or a replacement input); diagnostics retain user activation, input connectivity, and DOM -lifecycle evidence. - -The localhost-only fixture control endpoint sets deterministic `submit: accept | reject | hold` -and `upload: normal | fail | hold` modes; `accept_pending: true` releases held admission responses and -`release_uploads: true` delivers real successful responses for held uploads. -`/fixture-state` exposes actual received protocol methods, HTTP requests, emitted admission events, -upload attempts (including retry IDs), completed uploads, and DELETE IDs for assertions. Only synthetic data is recorded. - -`console-live-baseline-light.json` and `console-live-baseline-dark.json` use the same production -shell/data and capture initial plus unselected literal slash text. They support a pre-Feature HEAD -build, where no implicit Feature completion or typed chip exists. Initial and `slash-plain` capture -keys are shared with the current scenarios for like-for-like comparison; Feature-specific states -are explicitly new behavior, not fabricated baseline controls. - -## Adding a scenario - -1. Name the concrete user task and expected data state; do not write “looks correct”. -2. Use the smallest persona/route/viewport matrix that proves the intended contract, including - owner/non-owner/anonymous boundaries when permissions affect composition. -3. Choose a screen-owned readiness selector or response. Avoid arbitrary sleeps. -4. Add capture points only for meaningful visual states (initial, expanded detail, error, empty, and - so on). -5. Mark sensitive DOM regions with `[data-web-ux-redact]` or scenario selectors; never use review - artifacts to transport secrets. -6. Run `deno task check`, `deno task test`, one real capture, and inspect `contact-sheet.png` plus - `review-context.json`. +- scenarioは利用者の具体的な判断とデータ状態を定める。権限差・幅・themeは変更の契約を確認する範囲で選ぶ。 +- readinessは画面または応答で確認し、固定sleepで推測しない。本番と異なるCSSやshellを撮影用に作らない。 +- 認証状態と成果物を分離し、scenario・引数・成果物に秘密の値を含めない。 +- PlaywrightとNixで固定するbrowserの対応を、toolchain変更として同時に検証する。 +- 変更時は`deno.json`のcheck・testと必要なbrowser testを選ぶ。scenario変更では実際の画像と`review-context.json`を確認する。browser testの成功を目視の代わりにしない。 diff --git a/tools/web-ux/src/auth_state.ts b/tools/web-ux/src/auth_state.ts index 25a9288e1..facc779c6 100644 --- a/tools/web-ux/src/auth_state.ts +++ b/tools/web-ux/src/auth_state.ts @@ -1,3 +1,8 @@ +/** + * 認証fixtureは撮影成果物から分離する。別の利用主体やoriginへの流用と、 + * 期限切れの状態による誤判定を防ぐため、撮影前にsidecarの結付けを確認する。 + * metadataが欠けても暗黙に再認証せず、明示的な準備を要求する。 + */ import { writePrivateJson } from "./artifacts.ts"; export type AuthStateMetadata = { diff --git a/tools/web-ux/src/capture.ts b/tools/web-ux/src/capture.ts index 8c353eae8..b63f47a4d 100644 --- a/tools/web-ux/src/capture.ts +++ b/tools/web-ux/src/capture.ts @@ -1,3 +1,8 @@ +/** + * 画像だけでは再現条件や失敗を判断できないため、撮影条件とsource、画像hash、 + * 上限付きのブラウザ診断を同じreview contextに記録する。採取成功と画面の + * 正しさは別であり、合否は実装者が画像と診断を確認して判断する。 + */ import { basename, dirname, join, relative, resolve } from "@std/path"; import { type Browser, chromium, type Page, type Response } from "playwright"; import { validateAuthState } from "./auth_state.ts"; diff --git a/tools/web-ux/src/processes.ts b/tools/web-ux/src/processes.ts index 43c0f35d0..26c3721c2 100644 --- a/tools/web-ux/src/processes.ts +++ b/tools/web-ux/src/processes.ts @@ -1,3 +1,8 @@ +/** + * scenarioから起動したプロセスだけを所有し、終了・失敗時に回収する。 + * 外部の稼働ServerやRuntimeを停止対象にしない。診断出力は上限と秘匿処理を + * 適用し、切詰めた事実も記録して、途中のログを完全な証拠と誤認させない。 + */ import { dirname, isAbsolute, resolve } from "@std/path"; import { bounded, redactText, writePrivateJson } from "./artifacts.ts"; import type { CaptureError, OwnedProcess } from "./types.ts"; diff --git a/web/workspace/README.md b/web/workspace/README.md index f8f3b8f2e..8da3e83ed 100644 --- a/web/workspace/README.md +++ b/web/workspace/README.md @@ -1,106 +1,20 @@ -# Workspace web UI +# Workspace Web -SvelteKit static SPA for the Yoi workspace control plane. +Workspaceの状態確認と操作を担当するstatic SPA。Backend projectionを表示し、認可と永続化はBackendへ委ねる。独立したWorkspace／Worker registryは持たない。 -The frontend is intentionally static. Workspace authority, validation, and API behavior live in the Rust `yoi-server` backend. +## 実装の入口 -## Toolchain +- `src/routes/+layout.svelte`: application shellとHeader／Sidebarの枠。 +- `src/lib/workspace/sidebar/`: scope別navigationとoverride stack。 +- `src/lib/workspace/console/`: Worker transcriptと入力編集。 +- `src/lib/workspace/drive/`: Drive通信とdraft／receipt管理。 +- `src/lib/workspace/markdown/`: 共有Markdown rendererとリンク検査。 +- [`src/lib/generated/`](src/lib/generated/README.md): Rust契約からの生成物。 -Dependencies are pinned in `deno.json` and `package.json`, with resolutions in `deno.lock`. Run `deno install --frozen` to reproduce the installed versions. The current toolchain is SvelteKit 3, Svelte 5, Vite 8, and Vitest 5. TypeScript is kept on 6.0.3 because SvelteKit 3.0.0 and svelte-check 4.7.6 do not yet declare support for TypeScript 7. +## 維持規則 -SvelteKit and adapter settings live in `vite.config.ts`. Internal library imports use the `#lib/*` package subpath mapping with explicit file extensions, shared by Vite, TypeScript, and Deno. `tsconfig.json` extends `$app/tsconfig`; generated WASM JavaScript is consumed through its generated `.d.ts`, not checked as handwritten JavaScript. Deno-only tests under `test/` are type-checked by `deno task test`. +`src/app.css` はfont、cascade layer、semantic token、resetと基礎typographyを所有する。画面layoutと機能別styleは所有CSSまたはSvelte componentに置き、共通tokenを使う。 -## Development +生成契約は手で編集しない。通常のWeb build／devもSvelteKit生成物を書き換えるため、稼働中のcheckoutを検証専用buildに使わない。 -Use two terminals from the repository checkout. - -Backend terminal: - -```bash -cd web/workspace -deno task dev:backend -``` - -Frontend terminal: - -```bash -cd web/workspace -deno task dev -``` - -The Vite dev server proxies `/api/*` to `http://127.0.0.1:8787`, so frontend hot reload works while the Rust backend serves the workspace API. Open the Vite URL printed by `deno task dev`. - -If you want to run the backend from the repository root instead: - -```bash -cargo run -p yoi-workspace-server --bin yoi-server -- serve --listen 127.0.0.1:8787 -``` - -The backend reads Workspace records from the Yoi server DB at `/server/server.db`. Run `cargo run -p yoi-workspace-server --bin yoi-server -- init --workspace .` first when the server DB has not been initialized. - -## Static build - -Build the SPA: - -```bash -deno task build -``` - -Static asset packaging is a deployment concern. Local development normally uses the Vite dev server proxy plus the Rust backend command above. - -## Checks - -```bash -deno task check -deno task test -deno task test:component -deno task build -``` - -## Design and architecture authority - -- [`../../docs/development/ui-ux/design-language.md`](../../docs/development/ui-ux/design-language.md): visual and interaction language. -- [`../../docs/development/ui-ux/product-ux.md`](../../docs/development/ui-ux/product-ux.md): resource and navigation IA. -- [`../../docs/development/ui-ux/application-architecture.md`](../../docs/development/ui-ux/application-architecture.md): shell and nested override architecture. -- [`../../docs/development/ui-ux/visual-review.md`](../../docs/development/ui-ux/visual-review.md): implementer-owned visual validation. - -This README owns frontend implementation guidance and concrete source paths. It does not redefine product or visual rules from those documents. - -## CSS ownership - -`src/app.css` is the global foundation and owns only: - -- font imports; -- cascade layer order; -- semantic tokens and light/dark theme; -- reset and base element typography; -- global focus and text-selection behavior. - -It must not own page layout, feature class selectors, table or definition-list layout, button variants, status presentation, helper spacing, or anchor color. Navigation, inline links, and action links own their color in the relevant component stylesheet. - -Feature styles are owned by their implementation area: - -- `src/lib/workspace/styles/workspace-pages.css`: Workspace main-content primitives. -- `src/lib/workspace/styles/tickets.css`: Ticket surfaces. -- `src/lib/workspace/styles/workers.css`: Worker surfaces. -- `src/lib/workspace/styles/settings.css`: Settings, Account, and form surfaces. -- `src/lib/workspace/styles/workspace-catalog.css`: Workspace catalog. -- `src/lib/workspace/sidebar/sidebar.css`: Sidebar components. -- Svelte-local styles: behavior-specific presentation that is not reused outside that component. - -Use semantic tokens from `app.css`. Do not introduce component-local color, spacing, font, or z-index systems. - -## Source map - -- `src/routes/+layout.svelte`: root application shell and override contexts. -- `src/routes/w/[workspaceId]/+layout.svelte`: Workspace Header and Sidebar registration. -- `src/routes/w/[workspaceId]/settings/+layout.svelte`: Settings Sidebar registration. -- `src/lib/workspace/home/`: bounded, read-only work dashboard feeds and their presentation. -- `src/lib/workspace/settings/WorkspaceIdentitySettings.svelte`: Workspace-scoped name editing, public identity disclosure, and deletion confirmation. -- `src/lib/workspace/header/`: Header frame, context, and overrides. -- `src/lib/workspace/sidebar/`: Sidebar frame, scoped content, contexts, and override stack. -- `src/lib/workspace/markdown/`: shared safe Markdown parsing plus presentation-specific document renderers. -- `src/lib/workspace/ui/`: generic presentation components such as `Tooltip`, `Bevel`, and `BevelLine`. -- `src/routes/design-lab/workspace-web-ux/`: static Design Language showroom. - -When one of these paths changes, update this source map in the same change. Stable architecture and UX authority remain in `docs/`; this map follows the current implementation. +視覚・操作の共通規則は[Design Language](../../docs/principle/design-language.md)、製品固有の画面構成と操作は[UX設計](../../docs/principle/ux-design.md)、画像と実装条件の検証は[目視確認の規則](../../docs/development/ui-ux/visual-review.md)に従う。 diff --git a/web/workspace/src/lib/generated/README.md b/web/workspace/src/lib/generated/README.md index 65613dbe7..e51a4692b 100644 --- a/web/workspace/src/lib/generated/README.md +++ b/web/workspace/src/lib/generated/README.md @@ -1,35 +1,9 @@ -# Generated frontend contracts +# 生成されたWeb契約 -Checked-in files in this directory are generated API contracts. Do not edit them by hand. +このディレクトリのAPI型とvalidatorはRust契約からの生成物であり、手で編集しない。生成元とgeneratorは各ファイルのheaderに記録する。 -## Repository list/detail/create API +- `repository-api.ts`: canonical OpenAPIから生成するRepository契約。 +- `protocol.ts` / `protocol-validator.ts`: `protocol::subscription::SubscriptionFrame`から一緒に生成する型と受信検査。 +- `legacy-server-api.ts`: 未分離の契約。新しいdomainを追加するときは、対象を限定した生成契約へ分ける。 -`repository-api.ts` is the only TypeScript declaration authority for the Repository list, detail, and create request/response/error wire types. It is generated from the schema closure of the five Repository operations in the canonical `openapi/server-api.json` artifact. The generated header pins the in-repository generator name/version/options, canonical input path and digest, and output path. - -Regenerate or verify it from the repository root: - -```sh -cargo run -q -p server-api --example generate_repository_openapi_types -cargo run -q -p server-api --example generate_repository_openapi_types -- --check -``` - -The generator fails closed on unsupported or lossy OpenAPI constructs, including ambiguous nullable unions, external references, semantic `$ref` siblings, unimplemented ordinary schema keywords, unsupported enum literals/formats, and integer ranges that cannot be represented safely by JavaScript numbers. Its one explicit format mapping is canonical `uint64` to a nonnegative JavaScript safe `number`; the runtime parser enforces the same range. The legacy TypeScript generator imports the two shared Repository source projection types it still needs and no longer declares the migrated Repository list/detail types. - -## Workspace subscription / Worker protocol - -`protocol.ts` and `protocol-validator.ts` are generated together from the Serde/`schemars` closure of `protocol::subscription::SubscriptionFrame`. Rust DTOs and validation constants in `crates/protocol` remain the sole wire authority. The generator normalizes Serde defaults to the actual serialized required/nullable shape, tightens every supported Rust integer format to its JavaScript-safe wire range, and fails on an unknown integer format. The validator closes every generated object shape, rejects unknown tagged variants and unsafe integers, applies Rust-owned aggregate bounds to object keys as well as values, and mirrors the Browser-relevant `SubscriptionFrame::validate()` selector/payload checks. Rust-serialized compatibility fixtures are embedded for Browser regression tests. The Workspace multiplexer additionally checks live event payloads against its locally retained selector. - -Inbound Browser frames only are validated: Browser-to-Server methods remain compile-time typed and are decoded plus semantically validated by the Rust server. Invalid inbound frames close the shared socket with a fixed, non-reflective diagnostic; the existing reconnect path clears routing state and requests fresh snapshots. A valid `subscription_closed` frame continues to isolate and resubscribe only that subscription. - -Regenerate or verify both artifacts from the repository root: - -```sh -cargo run -q -p protocol --example generate_typescript --features typescript -cargo run -q -p protocol --example generate_typescript --features typescript -- --check -``` - -The `protocol` crate tests also compare both checked-in files byte-for-byte with fresh generator output. - -## Other generated contracts - -The remaining files retain their generator command in their header. `legacy-server-api.ts` is transitional: new bounded domain slices should use a focused contract artifact instead of adding another API surface to it. +契約変更時はRustの生成元と出力を同じ変更で更新し、generatorの検査と互換fixtureで一致を確認する。Browser固有の受信処理は `../workspace/multiplexer.ts` が所有する。 diff --git a/web/workspace/src/lib/workspace/api/runtime-management.ts b/web/workspace/src/lib/workspace/api/runtime-management.ts index a34a83871..039d2a385 100644 --- a/web/workspace/src/lib/workspace/api/runtime-management.ts +++ b/web/workspace/src/lib/workspace/api/runtime-management.ts @@ -1,3 +1,9 @@ +/** Runtime管理ではbinding_idを観測した登録条件として送る。 + * fingerprintはkey bytesの識別であり、bindingの再登録を識別する条件には使わない。 + * revoke adapterは確認値とcurrentFingerprintの一致を検査するが、Serverへの条件は + * expected_binding_idである。画面は現在のfingerprintを両引数へ渡す。 + * 接続テストの表示条件は呼出側の画面が現在bindingと照合する。 + */ import type { CreateRemoteRuntimeRequest, Diagnostic, diff --git a/web/workspace/src/lib/workspace/config-source/ConfigSourceEditor.svelte b/web/workspace/src/lib/workspace/config-source/ConfigSourceEditor.svelte index a5c35b4ab..de04b71d7 100644 --- a/web/workspace/src/lib/workspace/config-source/ConfigSourceEditor.svelte +++ b/web/workspace/src/lib/workspace/config-source/ConfigSourceEditor.svelte @@ -1,4 +1,9 @@ [Image: {text || 'unnamed'} — open its Drive URL to view] diff --git a/web/workspace/src/lib/workspace/drive-ui/DrivePage.svelte b/web/workspace/src/lib/workspace/drive-ui/DrivePage.svelte index ec978cc05..f435f1d67 100644 --- a/web/workspace/src/lib/workspace/drive-ui/DrivePage.svelte +++ b/web/workspace/src/lib/workspace/drive-ui/DrivePage.svelte @@ -1,4 +1,7 @@