fix: merge worker restore into develop host lifecycle
Integrate integration/worker-restore without reviving removed Worker socket/discovery paths. Keep restore candidate allocation and canonical effects deferred until activation, with registry reclamation and observer publication fences on the in-process controller. Preserve develop BackendJob/provenance, immutable Journal body capture and erasure, and Workdir diagnostics. Adapt obsolete socket fault tests to in-process publication and registry drift boundaries. Validation: root cargo check; ordinary worker, worker-runtime and standalone package tests; Server restore (43) and Journal (49) filters; mcp/server-api/runtime-api/session-store tests; optional Runtime feature check; OpenAPI and six generated TypeScript checks; Web restore unit tests (15); runner tests (21); fmt, naming and diff checks. Full workspace/Server suites, E2E, live/model and fixture generation not run.
This commit is contained in:
@@ -19,15 +19,19 @@ required-features = ["ws-server", "fs-store"]
|
||||
[features]
|
||||
default = ["ws-server", "fs-store"]
|
||||
fs-store = ["dep:fs4"]
|
||||
# Offline fixture tooling only; never enabled by the product defaults.
|
||||
test-support = ["fs-store", "ws-server"]
|
||||
# Explicit process-test synchronization; independent of offline fixture tooling.
|
||||
restore-checkpoints = ["fs-store"]
|
||||
http-server = ["dep:axum", "dep:tower", "dep:reqwest"]
|
||||
ws-server = ["http-server", "axum/ws", "dep:futures", "tokio/sync"]
|
||||
ws-server = ["http-server", "axum/ws", "tokio/sync"]
|
||||
|
||||
[dependencies]
|
||||
agen.workspace = true
|
||||
async-trait.workspace = true
|
||||
base64.workspace = true
|
||||
axum = { workspace = true, optional = true }
|
||||
futures = { workspace = true, optional = true }
|
||||
futures.workspace = true
|
||||
decodal.workspace = true
|
||||
flow = { path = "../flow" }
|
||||
fs4 = { workspace = true, optional = true }
|
||||
|
||||
@@ -8,3 +8,61 @@ Manages Worker creation, stopping, and Restore, execution connections within the
|
||||
- [`execution`] — execution backends and operation-specific results.
|
||||
- [`working_directory`] — repository materialization, session resources, and safe deletion.
|
||||
- [`workspace_issuer`] — Workspace signature verification and replay prevention.
|
||||
|
||||
## Offline stopped-Worker fixtures
|
||||
|
||||
The nondefault `test-support` feature exposes `test_support::StoppedWorkerTemplate`.
|
||||
It is intentionally limited to current-schema, stopped, attachment-free artificial
|
||||
conversations with one Segment, text history, and no pending execution/activation.
|
||||
It does not add a product endpoint or change create/restore authorization.
|
||||
|
||||
1. With the source Runtime offline, call `StoppedWorkerTemplate::extract(worker_dir)`.
|
||||
The source must have its existing Runtime owner lock and identity. Extraction is
|
||||
read-only; the returned serde template excludes source auth, saved Manifest,
|
||||
request/config references, run directories, and Runtime operation history.
|
||||
2. Initialize **new** Runtime identity and Workspace trust in a fresh destination
|
||||
using normal setup. Preserve the template's Runtime and Workspace IDs.
|
||||
3. Call `template.install(&options, request, &resolved_manifest)` before starting
|
||||
the Runtime. Supply a fresh typed `CreateWorkerRequest` (including config/profile
|
||||
references and trusted memory settings) and `manifest::WorkerManifest` resolved
|
||||
for that environment. Its Worker name must equal `template.worker_name()`.
|
||||
4. Start normally and use the ordinary explicit restore operation. Installation
|
||||
itself creates no execution and leaves the Worker stopped with a fresh guard.
|
||||
|
||||
The template preserves the original Session manifest/JSONL bytes, including stable
|
||||
entry IDs, timestamps, and settled submission receipts. It rejects resumable queues
|
||||
and Feature invocation history. A saved system prompt is required for restore;
|
||||
checkpoint receipts must match the Worker's typed receipt schema. Preserved records
|
||||
must use canonical writer field shapes: unknown/duplicate fields (including nested
|
||||
fields), discarded optional fields, and opaque Feature history payloads are rejected,
|
||||
not stripped or rewritten. Portability/secret-marker checks do not replace
|
||||
review of the supplied artificial conversation or its independently verified hashes.
|
||||
The installer refuses any destination content other than initialized `auth.toml`;
|
||||
interrupted publication is fail-closed, not an overwrite/recovery facility.
|
||||
|
||||
## Owned-process Restore checkpoints
|
||||
|
||||
The independent, nondefault `restore-checkpoints` feature exists only for explicit
|
||||
process tests in the shared `tests/e2e` harness. Offline `test-support` does not enable
|
||||
it. Normal binaries reject `--test-restore-checkpoint`; no environment variable or
|
||||
HTTP endpoint activates this facility. Do not deploy a checkpoint-enabled binary
|
||||
as a replacement for the default product.
|
||||
|
||||
A test-enabled Runtime accepts `--test-restore-checkpoint <config.json>` with a
|
||||
bounded regular JSON file selecting a loopback socket, Worker ID, one stage and a
|
||||
diagnostic nonce. The nonce is only a handshake label, never execution identity,
|
||||
request authority or routing state. The one-shot stages are real prepared candidate
|
||||
with `NotCommitted`, and durable `Committed` before activation/terminal receipt.
|
||||
Before acknowledging, the existing bounded canonical reader validates the exact
|
||||
Worker/operation/decision, syncs its file/directories and requires no saved receipt.
|
||||
No pending JSON is fabricated and no lifecycle work is skipped.
|
||||
|
||||
The Runtime sends bounded newline-delimited JSON evidence, then waits at most eight
|
||||
seconds for byte `R`. EOF, invalid reply, unavailable peer or deadline releases the
|
||||
wait into the original operation. The Runtime-state mutex is not held across that
|
||||
wait; existing per-Worker ownership still covers the actual candidate and cleanup.
|
||||
The harness verifies the ack, kills/reaps only its owned child and inspects the
|
||||
canonical aggregate offline before restarting the same roots/identity without the
|
||||
checkpoint flag or another fixture install. These tests are a separate cfg/binary
|
||||
lane from featureless product regression; they do not certify arbitrary OS failure,
|
||||
external power loss or a real model call.
|
||||
|
||||
@@ -215,6 +215,38 @@ impl WorkerExecutionResult {
|
||||
}
|
||||
}
|
||||
|
||||
// Private one-shot synchronization at real publication boundaries. Disconnect
|
||||
// releases the waiter during unwinding; timeout bounds a forgotten test owner.
|
||||
#[cfg(test)]
|
||||
#[derive(Debug)]
|
||||
pub(crate) struct RestoreTestGate<T> {
|
||||
pub(crate) entered: std::sync::mpsc::Sender<T>,
|
||||
pub(crate) release: std::sync::mpsc::Receiver<()>,
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
impl<T> RestoreTestGate<T> {
|
||||
pub(crate) fn wait(self, evidence: T) {
|
||||
let _ = self.entered.send(evidence);
|
||||
assert!(
|
||||
!matches!(
|
||||
self.release
|
||||
.recv_timeout(std::time::Duration::from_secs(30)),
|
||||
Err(std::sync::mpsc::RecvTimeoutError::Timeout)
|
||||
),
|
||||
"publication test gate timed out"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(all(test, feature = "ws-server"))]
|
||||
#[derive(Debug)]
|
||||
pub(crate) struct RestoreFlushFault {
|
||||
pub(crate) remaining_successes: usize,
|
||||
pub(crate) gate: RestoreTestGate<protocol::Event>,
|
||||
pub(crate) restored: std::sync::mpsc::Sender<Vec<protocol::Event>>,
|
||||
}
|
||||
|
||||
/// Runtime hooks available to an execution backend for one Worker.
|
||||
#[cfg(feature = "ws-server")]
|
||||
type WorkerObservationPublisher = Arc<
|
||||
@@ -242,10 +274,15 @@ enum WorkerExecutionPublicationState {
|
||||
#[derive(Clone)]
|
||||
pub struct WorkerExecutionContext {
|
||||
worker_ref: WorkerRef,
|
||||
restore_progress: Option<Arc<dyn Fn(WorkerLifecycleOperationId) + Send + Sync>>,
|
||||
restore_repair_authorizer:
|
||||
Option<Arc<dyn Fn(WorkerLifecycleOperationId) -> Result<(), RuntimeError> + Send + Sync>>,
|
||||
#[cfg(feature = "ws-server")]
|
||||
observation_publisher: WorkerObservationPublisher,
|
||||
#[cfg(feature = "ws-server")]
|
||||
publication_state: Arc<Mutex<WorkerExecutionPublicationState>>,
|
||||
#[cfg(all(test, feature = "ws-server"))]
|
||||
flush_fault: Arc<Mutex<Option<RestoreFlushFault>>>,
|
||||
}
|
||||
|
||||
impl WorkerExecutionContext {
|
||||
@@ -256,7 +293,11 @@ impl WorkerExecutionContext {
|
||||
) -> Self {
|
||||
Self {
|
||||
worker_ref,
|
||||
restore_progress: None,
|
||||
restore_repair_authorizer: None,
|
||||
observation_publisher,
|
||||
#[cfg(test)]
|
||||
flush_fault: Arc::default(),
|
||||
publication_state: Arc::new(Mutex::new(WorkerExecutionPublicationState::Active)),
|
||||
}
|
||||
}
|
||||
@@ -268,7 +309,11 @@ impl WorkerExecutionContext {
|
||||
) -> Self {
|
||||
Self {
|
||||
worker_ref,
|
||||
restore_progress: None,
|
||||
restore_repair_authorizer: None,
|
||||
observation_publisher,
|
||||
#[cfg(test)]
|
||||
flush_fault: Arc::default(),
|
||||
publication_state: Arc::new(Mutex::new(WorkerExecutionPublicationState::Pending(
|
||||
VecDeque::new(),
|
||||
))),
|
||||
@@ -277,12 +322,56 @@ impl WorkerExecutionContext {
|
||||
|
||||
#[cfg(not(feature = "ws-server"))]
|
||||
pub(crate) fn new(worker_ref: WorkerRef) -> Self {
|
||||
Self { worker_ref }
|
||||
Self {
|
||||
worker_ref,
|
||||
restore_progress: None,
|
||||
restore_repair_authorizer: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "ws-server"))]
|
||||
pub(crate) fn candidate(worker_ref: WorkerRef) -> Self {
|
||||
Self { worker_ref }
|
||||
Self {
|
||||
worker_ref,
|
||||
restore_progress: None,
|
||||
restore_repair_authorizer: None,
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn with_restore_repair_authorizer(
|
||||
mut self,
|
||||
authorizer: Arc<
|
||||
dyn Fn(WorkerLifecycleOperationId) -> Result<(), RuntimeError> + Send + Sync,
|
||||
>,
|
||||
) -> Self {
|
||||
self.restore_repair_authorizer = Some(authorizer);
|
||||
self
|
||||
}
|
||||
|
||||
pub(crate) fn authorize_restore_source_repair(
|
||||
&self,
|
||||
operation: WorkerLifecycleOperationId,
|
||||
) -> Result<(), RuntimeError> {
|
||||
match &self.restore_repair_authorizer {
|
||||
Some(authorizer) => authorizer(operation),
|
||||
// Direct factory/component users own their source repair separately;
|
||||
// only Runtime contexts carry a Runtime aggregate authority callback.
|
||||
None => Ok(()),
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn with_restore_progress(
|
||||
mut self,
|
||||
progress: Arc<dyn Fn(WorkerLifecycleOperationId) + Send + Sync>,
|
||||
) -> Self {
|
||||
self.restore_progress = Some(progress);
|
||||
self
|
||||
}
|
||||
|
||||
pub(crate) fn restore_progressed(&self, operation_id: WorkerLifecycleOperationId) {
|
||||
if let Some(progress) = &self.restore_progress {
|
||||
progress(operation_id);
|
||||
}
|
||||
}
|
||||
|
||||
pub fn worker_ref(&self) -> &WorkerRef {
|
||||
@@ -335,20 +424,65 @@ impl WorkerExecutionContext {
|
||||
let Some(event) = event else {
|
||||
return Ok(());
|
||||
};
|
||||
if let Err(error) = (self.observation_publisher)(self.worker_ref.clone(), event.clone())
|
||||
{
|
||||
#[cfg(test)]
|
||||
let fault = {
|
||||
let mut fault = self.flush_fault.lock().unwrap();
|
||||
if fault.as_ref().is_some_and(|f| f.remaining_successes == 0) {
|
||||
fault.take()
|
||||
} else {
|
||||
if let Some(fault) = fault.as_mut() {
|
||||
fault.remaining_successes -= 1;
|
||||
}
|
||||
None
|
||||
}
|
||||
};
|
||||
#[cfg(test)]
|
||||
let mut restored = None;
|
||||
#[cfg(test)]
|
||||
let publication = if let Some(fault) = fault {
|
||||
restored = Some(fault.restored);
|
||||
fault.gate.wait(event.clone());
|
||||
Err(RuntimeError::InvalidRequest(
|
||||
"injected buffered publication failure".into(),
|
||||
))
|
||||
} else {
|
||||
(self.observation_publisher)(self.worker_ref.clone(), event.clone())
|
||||
};
|
||||
#[cfg(not(test))]
|
||||
let publication = (self.observation_publisher)(self.worker_ref.clone(), event.clone());
|
||||
if let Err(error) = publication {
|
||||
let mut state = self
|
||||
.publication_state
|
||||
.lock()
|
||||
.map_err(|_| RuntimeError::StatePoisoned)?;
|
||||
if let WorkerExecutionPublicationState::Activating(events) = &mut *state {
|
||||
events.push_front(event);
|
||||
#[cfg(test)]
|
||||
if let Some(restored) = restored {
|
||||
let _ = restored.send(events.iter().cloned().collect());
|
||||
}
|
||||
}
|
||||
return Err(error);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(all(test, feature = "ws-server"))]
|
||||
pub(crate) fn test_fail_flush(&self, fault: RestoreFlushFault) {
|
||||
assert!(self.flush_fault.lock().unwrap().replace(fault).is_none());
|
||||
}
|
||||
|
||||
#[cfg(all(test, feature = "ws-server"))]
|
||||
pub(crate) fn test_buffered_events(&self) -> Vec<protocol::Event> {
|
||||
match &*self.publication_state.lock().unwrap() {
|
||||
WorkerExecutionPublicationState::Pending(events)
|
||||
| WorkerExecutionPublicationState::Activating(events) => {
|
||||
events.iter().cloned().collect()
|
||||
}
|
||||
_ => Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "ws-server"))]
|
||||
pub(crate) fn activate_candidate(&self) -> Result<(), RuntimeError> {
|
||||
Ok(())
|
||||
@@ -424,6 +558,7 @@ pub struct WorkerExecutionSpawnRequest {
|
||||
/// Request passed to a [`WorkerExecutionBackend`] when restoring a persisted Worker.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct WorkerExecutionRestoreRequest {
|
||||
pub source_repair_possible: bool,
|
||||
pub operation_id: WorkerLifecycleOperationId,
|
||||
pub worker_ref: WorkerRef,
|
||||
pub request: crate::catalog::CreateWorkerRequest,
|
||||
@@ -565,6 +700,19 @@ impl fmt::Debug for WorkerProtocolTransport {
|
||||
}
|
||||
}
|
||||
|
||||
/// Derived resource disposition for an existing restore owner. Not durable
|
||||
/// authority: Runtime must reread its decision before choosing any action.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum RestoreRecoveryDisposition {
|
||||
/// No live or pending owner for this exact operation. Not cleanup evidence.
|
||||
Absent,
|
||||
InFlight,
|
||||
Continue,
|
||||
Discard,
|
||||
/// A retained cleanup-only owner; never reconstruct or activate it.
|
||||
Reclaim,
|
||||
}
|
||||
|
||||
/// The connection selects this backend; WorkerRef identifies an execution,
|
||||
/// never a capability or authorization token.
|
||||
pub trait WorkerExecutionBackend: Send + Sync + 'static {
|
||||
@@ -633,6 +781,33 @@ pub trait WorkerExecutionBackend: Send + Sync + 'static {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Completed resource-owner evidence retained while Runtime settlement may
|
||||
/// still need retry. This is not a new request or execution authorization.
|
||||
fn completed_restore_failure(
|
||||
&self,
|
||||
_operation_id: WorkerLifecycleOperationId,
|
||||
_worker_ref: &WorkerRef,
|
||||
) -> Option<WorkerExecutionSpawnResult> {
|
||||
None
|
||||
}
|
||||
|
||||
/// Opt in to Runtime-owned autonomous restore continuation. Pending tasks
|
||||
/// signal completion through the original execution context; callbacks must
|
||||
/// not wait for Runtime locks. Independent of resource disposition support.
|
||||
fn supports_restore_continuation(&self) -> bool {
|
||||
false
|
||||
}
|
||||
|
||||
/// Describe exact operation resource ownership for warm/cold recovery
|
||||
/// dispatch. This does not opt in to callbacks or autonomous continuation.
|
||||
fn restore_recovery_disposition(
|
||||
&self,
|
||||
_operation_id: WorkerLifecycleOperationId,
|
||||
_worker_ref: &WorkerRef,
|
||||
) -> Option<RestoreRecoveryDisposition> {
|
||||
None
|
||||
}
|
||||
|
||||
fn reconcile_restore(
|
||||
&self,
|
||||
request: WorkerExecutionRestoreRequest,
|
||||
@@ -640,6 +815,42 @@ pub trait WorkerExecutionBackend: Send + Sync + 'static {
|
||||
self.restore_worker(request)
|
||||
}
|
||||
|
||||
/// Reclaim an uncommitted operation after restart without preparing,
|
||||
/// repairing, or activating a replacement candidate. Runtime must establish
|
||||
/// the durable uncommitted decision before calling this seam. Unsupported
|
||||
/// cleanup is uncertainty, never a successful rollback.
|
||||
fn reclaim_uncommitted_restore(
|
||||
&self,
|
||||
_request: WorkerExecutionRestoreRequest,
|
||||
) -> WorkerExecutionSpawnResult {
|
||||
WorkerExecutionSpawnResult::ReconciliationRequired {
|
||||
result: WorkerExecutionResult::unsupported(
|
||||
WorkerExecutionOperation::Restore,
|
||||
"backend cannot reclaim an uncommitted restore after restart",
|
||||
),
|
||||
worker_state: None,
|
||||
workdir_attachments: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
/// Discard only the exact unpublished operation after Runtime has proved
|
||||
/// its durable decision uncommitted. This is not Stop and must not affect a
|
||||
/// connected execution. RolledBack requires confirmed cleanup and no repair.
|
||||
fn discard_uncommitted_restore(
|
||||
&self,
|
||||
_operation_id: WorkerLifecycleOperationId,
|
||||
_worker_ref: &WorkerRef,
|
||||
) -> WorkerExecutionSpawnResult {
|
||||
WorkerExecutionSpawnResult::ReconciliationRequired {
|
||||
result: WorkerExecutionResult::unsupported(
|
||||
WorkerExecutionOperation::Restore,
|
||||
"backend cannot confirm unpublished restore cleanup",
|
||||
),
|
||||
worker_state: None,
|
||||
workdir_attachments: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
fn activate_restored_worker(
|
||||
&self,
|
||||
_operation_id: WorkerLifecycleOperationId,
|
||||
@@ -901,6 +1112,28 @@ impl WorkerExecutionBackendRef {
|
||||
self.backend.preflight_restore(request)
|
||||
}
|
||||
|
||||
pub(crate) fn completed_restore_failure(
|
||||
&self,
|
||||
operation_id: WorkerLifecycleOperationId,
|
||||
worker_ref: &WorkerRef,
|
||||
) -> Option<WorkerExecutionSpawnResult> {
|
||||
self.backend
|
||||
.completed_restore_failure(operation_id, worker_ref)
|
||||
}
|
||||
|
||||
pub(crate) fn supports_restore_continuation(&self) -> bool {
|
||||
self.backend.supports_restore_continuation()
|
||||
}
|
||||
|
||||
pub(crate) fn restore_recovery_disposition(
|
||||
&self,
|
||||
operation_id: WorkerLifecycleOperationId,
|
||||
worker_ref: &WorkerRef,
|
||||
) -> Option<RestoreRecoveryDisposition> {
|
||||
self.backend
|
||||
.restore_recovery_disposition(operation_id, worker_ref)
|
||||
}
|
||||
|
||||
pub(crate) fn reconcile_restore(
|
||||
&self,
|
||||
request: WorkerExecutionRestoreRequest,
|
||||
@@ -908,6 +1141,22 @@ impl WorkerExecutionBackendRef {
|
||||
self.backend.reconcile_restore(request)
|
||||
}
|
||||
|
||||
pub(crate) fn reclaim_uncommitted_restore(
|
||||
&self,
|
||||
request: WorkerExecutionRestoreRequest,
|
||||
) -> WorkerExecutionSpawnResult {
|
||||
self.backend.reclaim_uncommitted_restore(request)
|
||||
}
|
||||
|
||||
pub(crate) fn discard_uncommitted_restore(
|
||||
&self,
|
||||
operation_id: WorkerLifecycleOperationId,
|
||||
worker_ref: &WorkerRef,
|
||||
) -> WorkerExecutionSpawnResult {
|
||||
self.backend
|
||||
.discard_uncommitted_restore(operation_id, worker_ref)
|
||||
}
|
||||
|
||||
pub(crate) fn activate_restored_worker(
|
||||
&self,
|
||||
operation_id: WorkerLifecycleOperationId,
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -25,6 +25,7 @@ use crate::interaction::{WorkerInput, WorkerInteractionAck};
|
||||
use crate::management::{RuntimeSummary, WorkerDeleteResult};
|
||||
#[cfg(feature = "ws-server")]
|
||||
use crate::observation::WorkerObservationCursor;
|
||||
#[cfg(feature = "fs-store")]
|
||||
use crate::retention::{
|
||||
WorkerRetentionExecutionRequest, WorkerRetentionExecutionResult, WorkerRetentionInventory,
|
||||
};
|
||||
@@ -1568,6 +1569,7 @@ fn protocol_error_event(message: impl Into<String>) -> protocol::Event {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "fs-store")]
|
||||
#[allow(dead_code)]
|
||||
async fn worker_retention_inventory(
|
||||
State(state): State<RuntimeHttpState>,
|
||||
@@ -1589,6 +1591,7 @@ async fn worker_retention_inventory(
|
||||
.map_err(RuntimeHttpRestError::runtime)
|
||||
}
|
||||
|
||||
#[cfg(feature = "fs-store")]
|
||||
#[allow(dead_code)]
|
||||
async fn execute_worker_retention(
|
||||
State(state): State<RuntimeHttpState>,
|
||||
@@ -2654,6 +2657,87 @@ mod tests {
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
// Generated management routes and real Workspace capability middleware stay
|
||||
// connected even when this process has no filesystem retention provider.
|
||||
#[cfg(not(feature = "fs-store"))]
|
||||
#[tokio::test]
|
||||
async fn memory_retention_routes_preserve_auth_and_return_typed_unavailable() {
|
||||
let identity = RuntimeIdentityMaterial::generate("workspace-key").unwrap();
|
||||
let trust = workspace_trust(&identity);
|
||||
let runtime = Runtime::new_memory();
|
||||
let app = runtime_http_router_with_workspace_auth(
|
||||
runtime,
|
||||
None,
|
||||
workspace_test_auth(vec![trust.clone()]),
|
||||
);
|
||||
let worker_id = WorkerId::now_v7();
|
||||
let body = runtime_api::WorkerRetentionExecutionRequest {
|
||||
operation_id: "memory-retention".into(),
|
||||
input_fingerprint: "input".into(),
|
||||
archive_id: None,
|
||||
workspace_id: "workspace-a".into(),
|
||||
source_runtime_id: "runtime-test".into(),
|
||||
worker_id,
|
||||
source_created_at: "2026-01-01T00:00:00Z".into(),
|
||||
removed_at: "2026-01-02T00:00:00Z".into(),
|
||||
effective_profile: None,
|
||||
retention_class: None,
|
||||
policy_id: "policy".into(),
|
||||
session_disposition: runtime_api::SessionDisposition::Purge,
|
||||
diagnostics_disposition: runtime_api::DiagnosticsDisposition::Purge,
|
||||
};
|
||||
for (method, path, body) in [
|
||||
(
|
||||
"GET",
|
||||
format!("/v1/workers/{worker_id}/retention/inventory"),
|
||||
vec![],
|
||||
),
|
||||
(
|
||||
"POST",
|
||||
format!("/v1/workers/{worker_id}/retention/execute"),
|
||||
serde_json::to_vec(&body).unwrap(),
|
||||
),
|
||||
] {
|
||||
let unsigned = app
|
||||
.clone()
|
||||
.oneshot(
|
||||
Request::builder()
|
||||
.method(method)
|
||||
.uri(&path)
|
||||
.header(header::CONTENT_TYPE, "application/json")
|
||||
.body(Body::from(body.clone()))
|
||||
.unwrap(),
|
||||
)
|
||||
.await
|
||||
.unwrap();
|
||||
assert_eq!(unsigned.status(), StatusCode::UNAUTHORIZED);
|
||||
let claims = signed_claims(&trust, method, &path, &body);
|
||||
let response =
|
||||
signed_request(app.clone(), &identity, &claims, body, "workspace-a").await;
|
||||
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
|
||||
let error: RuntimeHttpErrorResponse = read_json(response).await;
|
||||
assert_eq!(error.error.code, "worker_retention_unavailable");
|
||||
}
|
||||
for (mut value, expected_status, expected_code) in [
|
||||
(body.clone(), StatusCode::NOT_FOUND, "worker_not_found"),
|
||||
(body.clone(), StatusCode::BAD_REQUEST, "worker_id_mismatch"),
|
||||
] {
|
||||
if expected_status == StatusCode::NOT_FOUND {
|
||||
value.workspace_id = "workspace-b".into();
|
||||
} else {
|
||||
value.worker_id = WorkerId::now_v7();
|
||||
}
|
||||
let path = format!("/v1/workers/{worker_id}/retention/execute");
|
||||
let bytes = serde_json::to_vec(&value).unwrap();
|
||||
let claims = signed_claims(&trust, "POST", &path, &bytes);
|
||||
let response =
|
||||
signed_request(app.clone(), &identity, &claims, bytes, "workspace-a").await;
|
||||
assert_eq!(response.status(), expected_status);
|
||||
let error: RuntimeHttpErrorResponse = read_json(response).await;
|
||||
assert_eq!(error.error.code, expected_code);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn ordinary_signed_ping_succeeds_without_handshake_and_rejects_replay() {
|
||||
let identity = RuntimeIdentityMaterial::generate("workspace-key").unwrap();
|
||||
|
||||
@@ -719,6 +719,9 @@ impl runtime_api::RuntimeApi for RuntimeManagementApi {
|
||||
server_api::WorkerRestoreState::RolledBack => {
|
||||
runtime_api::WorkerRestoreState::RolledBack
|
||||
}
|
||||
server_api::WorkerRestoreState::Superseded => {
|
||||
runtime_api::WorkerRestoreState::Superseded
|
||||
}
|
||||
server_api::WorkerRestoreState::ReconciliationRequired => {
|
||||
runtime_api::WorkerRestoreState::ReconciliationRequired
|
||||
}
|
||||
@@ -769,6 +772,9 @@ impl runtime_api::RuntimeApi for RuntimeManagementApi {
|
||||
server_api::WorkerRestoreState::RolledBack => {
|
||||
runtime_api::WorkerRestoreState::RolledBack
|
||||
}
|
||||
server_api::WorkerRestoreState::Superseded => {
|
||||
runtime_api::WorkerRestoreState::Superseded
|
||||
}
|
||||
server_api::WorkerRestoreState::ReconciliationRequired => {
|
||||
runtime_api::WorkerRestoreState::ReconciliationRequired
|
||||
}
|
||||
@@ -881,13 +887,25 @@ impl runtime_api::RuntimeApi for RuntimeManagementApi {
|
||||
)
|
||||
})?;
|
||||
let worker_ref = worker_ref_for(&self.state.runtime, worker_id).map_err(api_error)?;
|
||||
let inventory = self
|
||||
.state
|
||||
.runtime
|
||||
.worker_retention_inventory(&scope.workspace_id, &worker_ref)
|
||||
.map_err(RuntimeHttpRestError::runtime)
|
||||
.map_err(api_error)?;
|
||||
response(inventory)
|
||||
#[cfg(feature = "fs-store")]
|
||||
{
|
||||
let inventory = self
|
||||
.state
|
||||
.runtime
|
||||
.worker_retention_inventory(&scope.workspace_id, &worker_ref)
|
||||
.map_err(RuntimeHttpRestError::runtime)
|
||||
.map_err(api_error)?;
|
||||
response(inventory)
|
||||
}
|
||||
#[cfg(not(feature = "fs-store"))]
|
||||
{
|
||||
let _ = (scope, worker_ref);
|
||||
Err(runtime_api::RuntimeApiError::new(
|
||||
503,
|
||||
"worker_retention_unavailable",
|
||||
"Worker retention requires filesystem storage",
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
async fn execute_retention(
|
||||
@@ -905,27 +923,36 @@ impl runtime_api::RuntimeApi for RuntimeManagementApi {
|
||||
)
|
||||
})?;
|
||||
let worker_ref = worker_ref_for(&self.state.runtime, worker_id).map_err(api_error)?;
|
||||
let request: WorkerRetentionExecutionRequest = request(value)?;
|
||||
if request.workspace_id != scope.workspace_id {
|
||||
if value.workspace_id != scope.workspace_id {
|
||||
return Err(runtime_api::RuntimeApiError::new(
|
||||
StatusCode::NOT_FOUND.as_u16(),
|
||||
"worker_not_found",
|
||||
"worker does not belong to the authenticated Workspace",
|
||||
));
|
||||
}
|
||||
if request.worker_id != worker_ref.worker_id {
|
||||
if value.worker_id != worker_ref.worker_id {
|
||||
return Err(runtime_api::RuntimeApiError::new(
|
||||
StatusCode::BAD_REQUEST.as_u16(),
|
||||
"worker_id_mismatch",
|
||||
"Worker retention request worker_id does not match route worker_id",
|
||||
));
|
||||
}
|
||||
let result = self
|
||||
.state
|
||||
.runtime
|
||||
.execute_worker_retention(&request)
|
||||
.map_err(RuntimeHttpRestError::runtime)
|
||||
.map_err(api_error)?;
|
||||
response(result)
|
||||
#[cfg(feature = "fs-store")]
|
||||
{
|
||||
let request: WorkerRetentionExecutionRequest = request(value)?;
|
||||
let result = self
|
||||
.state
|
||||
.runtime
|
||||
.execute_worker_retention(&request)
|
||||
.map_err(RuntimeHttpRestError::runtime)
|
||||
.map_err(api_error)?;
|
||||
response(result)
|
||||
}
|
||||
#[cfg(not(feature = "fs-store"))]
|
||||
Err(runtime_api::RuntimeApiError::new(
|
||||
503,
|
||||
"worker_retention_unavailable",
|
||||
"Worker retention requires filesystem storage",
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -16,6 +16,11 @@ pub mod management;
|
||||
pub mod observation;
|
||||
pub mod profile_archive;
|
||||
pub mod resource;
|
||||
#[cfg(feature = "restore-checkpoints")]
|
||||
pub mod restore_checkpoints;
|
||||
#[cfg(test)]
|
||||
#[path = "../../../tests/support/restore_environment.rs"]
|
||||
mod restore_test_environment;
|
||||
#[cfg(feature = "fs-store")]
|
||||
pub mod retention;
|
||||
mod runtime;
|
||||
@@ -32,6 +37,8 @@ pub mod workspace_issuer;
|
||||
#[cfg(feature = "http-server")]
|
||||
pub mod workspace_request;
|
||||
|
||||
#[cfg(feature = "test-support")]
|
||||
pub use fs_store::test_support;
|
||||
#[cfg(feature = "fs-store")]
|
||||
pub use fs_store::{FsRuntimeStore, FsRuntimeStoreOptions};
|
||||
pub use management::RuntimeOptions;
|
||||
|
||||
@@ -117,6 +117,12 @@ fn run() -> Result<(), ProcessError> {
|
||||
let listener = tokio::net::TcpListener::bind(config.http.bind_addr).await?;
|
||||
let local_addr = listener.local_addr()?;
|
||||
let worker_runtime = build_runtime(&config)?;
|
||||
#[cfg(feature = "restore-checkpoints")]
|
||||
if let Some(checkpoint) = &config.test_restore_checkpoint {
|
||||
worker_runtime
|
||||
.install_test_restore_checkpoint(checkpoint.clone())
|
||||
.map_err(ProcessError::Runtime)?;
|
||||
}
|
||||
eprintln!(
|
||||
"yoi-runtime listening on {local_addr}; intended client is a trusted backend/proxy, not a browser"
|
||||
);
|
||||
@@ -391,6 +397,21 @@ where
|
||||
let (flag, inline_value) = split_flag_value(arg)?;
|
||||
match flag.as_str() {
|
||||
"--help" | "-h" => return Ok(None),
|
||||
#[cfg(feature = "restore-checkpoints")]
|
||||
"--test-restore-checkpoint" => {
|
||||
let path = take_value(&flag, inline_value, &mut args)?;
|
||||
if config.test_restore_checkpoint.is_some() {
|
||||
return Err(ProcessError::usage(
|
||||
"--test-restore-checkpoint may only be specified once".into(),
|
||||
));
|
||||
}
|
||||
config.test_restore_checkpoint = Some(
|
||||
worker_runtime::restore_checkpoints::RestoreCheckpointConfig::from_path(
|
||||
Path::new(&path),
|
||||
)
|
||||
.map_err(ProcessError::Runtime)?,
|
||||
);
|
||||
}
|
||||
"--bind" => {
|
||||
let value = take_value(&flag, inline_value, &mut args)?;
|
||||
config.http.bind_addr = value.parse::<SocketAddr>().map_err(|error| {
|
||||
@@ -465,6 +486,12 @@ where
|
||||
}
|
||||
|
||||
apply_store_selection(&mut config);
|
||||
#[cfg(feature = "restore-checkpoints")]
|
||||
if config.test_restore_checkpoint.is_some() && config.no_store {
|
||||
return Err(ProcessError::usage(
|
||||
"--test-restore-checkpoint requires an fs Runtime store".into(),
|
||||
));
|
||||
}
|
||||
Ok(Some(config))
|
||||
}
|
||||
|
||||
@@ -521,6 +548,8 @@ struct ProcessConfig {
|
||||
no_store: bool,
|
||||
backend_resource_endpoint: Option<String>,
|
||||
backend_resource_token: Option<String>,
|
||||
#[cfg(feature = "restore-checkpoints")]
|
||||
test_restore_checkpoint: Option<worker_runtime::restore_checkpoints::RestoreCheckpointConfig>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
@@ -542,6 +571,8 @@ impl ProcessConfig {
|
||||
no_store: false,
|
||||
backend_resource_endpoint: None,
|
||||
backend_resource_token: None,
|
||||
#[cfg(feature = "restore-checkpoints")]
|
||||
test_restore_checkpoint: None,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -1227,6 +1258,47 @@ Auth commands:
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn checkpoint_cli_is_absent_by_default() {
|
||||
let config = parse_args([] as [&str; 0]).unwrap().unwrap();
|
||||
#[cfg(feature = "restore-checkpoints")]
|
||||
assert!(config.test_restore_checkpoint.is_none());
|
||||
#[cfg(not(feature = "restore-checkpoints"))]
|
||||
{
|
||||
let _ = config;
|
||||
let error = parse_args(["--test-restore-checkpoint", "never-opened.json"]).unwrap_err();
|
||||
assert!(error.to_string().contains("unknown argument"));
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "restore-checkpoints")]
|
||||
#[test]
|
||||
fn checkpoint_cli_requires_valid_explicit_file_and_fs_store() {
|
||||
use worker_runtime::restore_checkpoints::{
|
||||
RestoreCheckpointConfig, RestoreCheckpointStage,
|
||||
};
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("checkpoint.json");
|
||||
let checkpoint = RestoreCheckpointConfig {
|
||||
endpoint: "127.0.0.1:1234".parse().unwrap(),
|
||||
worker_id: protocol::WorkerId::now_v7(),
|
||||
stage: RestoreCheckpointStage::CommittedPreterminal,
|
||||
nonce: "cli-test".into(),
|
||||
};
|
||||
std::fs::write(&path, serde_json::to_vec(&checkpoint).unwrap()).unwrap();
|
||||
let args = ["--test-restore-checkpoint", path.to_str().unwrap()];
|
||||
let config = parse_args(args).unwrap().unwrap();
|
||||
assert_eq!(config.test_restore_checkpoint, Some(checkpoint));
|
||||
assert!(parse_args([args[0], args[1], "--no-store"]).is_err());
|
||||
assert!(parse_args([args[0], args[1], args[0], args[1]]).is_err());
|
||||
assert!(parse_args([args[0]]).is_err());
|
||||
std::fs::write(&path, b"private-invalid-body").unwrap();
|
||||
let error = parse_args(args).unwrap_err().to_string();
|
||||
assert!(!error.contains("private-invalid-body"));
|
||||
std::fs::remove_file(&path).unwrap();
|
||||
assert!(parse_args(args).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn help_does_not_start_server() {
|
||||
assert!(parse_args(["--help"]).unwrap().is_none());
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
#[cfg(feature = "http-server")]
|
||||
use crate::auth::BACKEND_RESOURCE_FETCH_PERMISSION;
|
||||
use crate::identity::WorkerId;
|
||||
use crate::profile_archive::{ProfileSourceArchive, ProfileSourceArchiveRef, sha256_hex};
|
||||
#[cfg(feature = "http-server")]
|
||||
use crate::workspace_request::RuntimeWorkspaceRequestClient;
|
||||
use async_trait::async_trait;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
//! Explicit, ephemeral synchronization for owned Runtime process tests only.
|
||||
//!
|
||||
//! The nonce authenticates no Runtime action and is never a routing identity.
|
||||
//! Socket failure, EOF and the fixed deadline only release the gate; the lifecycle
|
||||
//! owner continues its original operation and remains responsible for cleanup.
|
||||
|
||||
use std::fs::OpenOptions;
|
||||
use std::io::{Read, Write};
|
||||
use std::net::{SocketAddr, TcpStream};
|
||||
use std::path::Path;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use protocol::WorkerId;
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::error::RuntimeError;
|
||||
use crate::execution::WorkerLifecycleOperationId;
|
||||
|
||||
pub const RESTORE_CHECKPOINT_TIMEOUT: Duration = Duration::from_secs(8);
|
||||
pub const MAX_RESTORE_CHECKPOINT_CONFIG_BYTES: usize = 4096;
|
||||
pub const MAX_RESTORE_CHECKPOINT_ACK_BYTES: usize = 4096;
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum RestoreCheckpointStage {
|
||||
PreparedNotCommitted,
|
||||
CommittedPreterminal,
|
||||
ActivatedPreterminal,
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum RestoreCheckpointCommit {
|
||||
NotCommitted,
|
||||
Committed,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct RestoreCheckpointConfig {
|
||||
pub endpoint: SocketAddr,
|
||||
pub worker_id: WorkerId,
|
||||
pub stage: RestoreCheckpointStage,
|
||||
pub nonce: String,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct RestoreCheckpointAck {
|
||||
pub nonce: String,
|
||||
pub stage: RestoreCheckpointStage,
|
||||
pub runtime_id: String,
|
||||
pub worker_id: WorkerId,
|
||||
pub operation_id: WorkerLifecycleOperationId,
|
||||
pub commit: RestoreCheckpointCommit,
|
||||
pub receipt_absent: bool,
|
||||
pub pid: u32,
|
||||
}
|
||||
|
||||
fn invalid_config() -> RuntimeError {
|
||||
// Do not include serde errors, paths or input contents in process diagnostics.
|
||||
RuntimeError::InvalidRequest("invalid test restore checkpoint configuration".into())
|
||||
}
|
||||
|
||||
impl RestoreCheckpointConfig {
|
||||
pub fn validate(&self) -> Result<(), RuntimeError> {
|
||||
if !self.endpoint.ip().is_loopback()
|
||||
|| self.endpoint.port() == 0
|
||||
|| self.nonce.is_empty()
|
||||
|| self.nonce.len() > 256
|
||||
|| self.nonce.chars().any(char::is_control)
|
||||
{
|
||||
return Err(invalid_config());
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn from_path(path: &Path) -> Result<Self, RuntimeError> {
|
||||
let mut options = OpenOptions::new();
|
||||
options.read(true);
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
// A FIFO must not stall startup before its regular-file check.
|
||||
options.custom_flags(libc::O_NONBLOCK | libc::O_NOFOLLOW);
|
||||
}
|
||||
let file = options.open(path).map_err(|_| invalid_config())?;
|
||||
let metadata = file.metadata().map_err(|_| invalid_config())?;
|
||||
if !metadata.is_file() || metadata.len() > MAX_RESTORE_CHECKPOINT_CONFIG_BYTES as u64 {
|
||||
return Err(invalid_config());
|
||||
}
|
||||
let mut bytes = Vec::new();
|
||||
file.take(MAX_RESTORE_CHECKPOINT_CONFIG_BYTES as u64 + 1)
|
||||
.read_to_end(&mut bytes)
|
||||
.map_err(|_| invalid_config())?;
|
||||
if bytes.len() > MAX_RESTORE_CHECKPOINT_CONFIG_BYTES {
|
||||
return Err(invalid_config());
|
||||
}
|
||||
let config: Self = serde_json::from_slice(&bytes).map_err(|_| invalid_config())?;
|
||||
config.validate()?;
|
||||
Ok(config)
|
||||
}
|
||||
}
|
||||
|
||||
pub(crate) fn wait_for_release(config: &RestoreCheckpointConfig, ack: &RestoreCheckpointAck) {
|
||||
let _ = exchange(config, ack);
|
||||
}
|
||||
|
||||
fn remaining(deadline: Instant) -> std::io::Result<Duration> {
|
||||
deadline
|
||||
.checked_duration_since(Instant::now())
|
||||
.filter(|remaining| !remaining.is_zero())
|
||||
.ok_or_else(|| std::io::Error::from(std::io::ErrorKind::TimedOut))
|
||||
}
|
||||
|
||||
fn exchange(config: &RestoreCheckpointConfig, ack: &RestoreCheckpointAck) -> std::io::Result<()> {
|
||||
let deadline = Instant::now() + RESTORE_CHECKPOINT_TIMEOUT;
|
||||
let mut bytes = serde_json::to_vec(ack)?;
|
||||
bytes.push(b'\n');
|
||||
if bytes.len() > MAX_RESTORE_CHECKPOINT_ACK_BYTES {
|
||||
return Err(std::io::Error::from(std::io::ErrorKind::InvalidData));
|
||||
}
|
||||
let mut stream = TcpStream::connect_timeout(&config.endpoint, remaining(deadline)?)?;
|
||||
// Each syscall shares one budget; partial writes cannot reset the deadline.
|
||||
let mut pending = bytes.as_slice();
|
||||
while !pending.is_empty() {
|
||||
stream.set_write_timeout(Some(remaining(deadline)?))?;
|
||||
let written = stream.write(pending)?;
|
||||
if written == 0 {
|
||||
return Err(std::io::Error::from(std::io::ErrorKind::WriteZero));
|
||||
}
|
||||
pending = &pending[written..];
|
||||
}
|
||||
stream.set_read_timeout(Some(remaining(deadline)?))?;
|
||||
let mut release = [0];
|
||||
stream.read_exact(&mut release)?;
|
||||
if release != [b'R'] {
|
||||
return Err(std::io::Error::from(std::io::ErrorKind::InvalidData));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
@@ -0,0 +1,139 @@
|
||||
use super::*;
|
||||
use std::io::{BufRead, BufReader};
|
||||
use std::net::TcpListener;
|
||||
use std::thread;
|
||||
|
||||
fn config(endpoint: SocketAddr) -> RestoreCheckpointConfig {
|
||||
RestoreCheckpointConfig {
|
||||
endpoint,
|
||||
worker_id: WorkerId::now_v7(),
|
||||
stage: RestoreCheckpointStage::PreparedNotCommitted,
|
||||
nonce: "checkpoint-test-handshake".into(),
|
||||
}
|
||||
}
|
||||
|
||||
fn ack(config: &RestoreCheckpointConfig) -> RestoreCheckpointAck {
|
||||
RestoreCheckpointAck {
|
||||
nonce: config.nonce.clone(),
|
||||
stage: config.stage,
|
||||
runtime_id: "checkpoint-runtime".into(),
|
||||
worker_id: config.worker_id,
|
||||
operation_id: WorkerLifecycleOperationId::new(),
|
||||
commit: RestoreCheckpointCommit::NotCommitted,
|
||||
receipt_absent: true,
|
||||
pid: std::process::id(),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn config_rejects_nonliteral_nonloopback_bad_stage_unknown_fields_and_invalid_worker() {
|
||||
let valid = serde_json::to_value(config("127.0.0.1:1234".parse().unwrap())).unwrap();
|
||||
for (field, value) in [
|
||||
("endpoint", "localhost:1234"),
|
||||
("endpoint", "192.0.2.1:1234"),
|
||||
("endpoint", "0.0.0.0:1234"),
|
||||
("endpoint", "127.0.0.1:0"),
|
||||
("endpoint", "[::ffff:127.0.0.1]:1234"),
|
||||
("worker_id", "not-a-worker-id"),
|
||||
("stage", "after_terminal"),
|
||||
("nonce", ""),
|
||||
("nonce", "secret\nbody"),
|
||||
("timeout", "unbounded"),
|
||||
] {
|
||||
let mut invalid = valid.clone();
|
||||
invalid[field] = value.into();
|
||||
let parsed = serde_json::from_value::<RestoreCheckpointConfig>(invalid);
|
||||
assert!(
|
||||
parsed.is_err() || parsed.unwrap().validate().is_err(),
|
||||
"{field}"
|
||||
);
|
||||
}
|
||||
for endpoint in ["127.0.0.1:1234", "[::1]:1234"] {
|
||||
config(endpoint.parse().unwrap()).validate().unwrap();
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn config_file_bounds_and_errors_do_not_disclose_contents() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let path = dir.path().join("config.json");
|
||||
assert!(RestoreCheckpointConfig::from_path(&path).is_err());
|
||||
std::fs::write(&path, vec![b'x'; MAX_RESTORE_CHECKPOINT_CONFIG_BYTES + 1]).unwrap();
|
||||
assert!(RestoreCheckpointConfig::from_path(&path).is_err());
|
||||
std::fs::write(
|
||||
&path,
|
||||
br#"{"nonce":"private-config-body","stage":"secret"}"#,
|
||||
)
|
||||
.unwrap();
|
||||
let error = RestoreCheckpointConfig::from_path(&path)
|
||||
.unwrap_err()
|
||||
.to_string();
|
||||
assert!(!error.contains("private-config-body"));
|
||||
assert!(!error.contains("secret"));
|
||||
let mut valid = config("127.0.0.1:1234".parse().unwrap());
|
||||
valid.nonce = "x".repeat(257);
|
||||
assert!(valid.validate().is_err());
|
||||
valid.nonce = "valid".into();
|
||||
std::fs::write(&path, serde_json::to_vec(&valid).unwrap()).unwrap();
|
||||
assert_eq!(RestoreCheckpointConfig::from_path(&path).unwrap(), valid);
|
||||
assert!(RestoreCheckpointConfig::from_path(dir.path()).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn explicit_release_eof_and_invalid_release_close_the_owned_stream() {
|
||||
for release in [Some(b'R'), None, Some(b'X')] {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
|
||||
let config = config(listener.local_addr().unwrap());
|
||||
let expected = ack(&config);
|
||||
let sent = expected.clone();
|
||||
let task = thread::spawn(move || exchange(&config, &sent));
|
||||
let (mut stream, _) = listener.accept().unwrap();
|
||||
stream
|
||||
.set_read_timeout(Some(Duration::from_secs(2)))
|
||||
.unwrap();
|
||||
let mut line = String::new();
|
||||
BufReader::new(&mut stream).read_line(&mut line).unwrap();
|
||||
assert!(line.len() <= MAX_RESTORE_CHECKPOINT_ACK_BYTES);
|
||||
assert_eq!(
|
||||
serde_json::from_str::<RestoreCheckpointAck>(&line).unwrap(),
|
||||
expected
|
||||
);
|
||||
if let Some(byte) = release {
|
||||
stream.write_all(&[byte]).unwrap();
|
||||
} else {
|
||||
stream.shutdown(std::net::Shutdown::Write).unwrap();
|
||||
}
|
||||
assert_eq!(task.join().unwrap().is_ok(), release == Some(b'R'));
|
||||
let mut byte = [0];
|
||||
assert_eq!(stream.read(&mut byte).unwrap(), 0);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn absent_listener_resumes_without_waiting_or_retrying() {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
|
||||
let config = config(listener.local_addr().unwrap());
|
||||
drop(listener);
|
||||
let start = Instant::now();
|
||||
wait_for_release(&config, &ack(&config));
|
||||
assert!(start.elapsed() < Duration::from_secs(2));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn silent_peer_hits_fixed_deadline_and_stream_is_dropped() {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
|
||||
let config = config(listener.local_addr().unwrap());
|
||||
let start = Instant::now();
|
||||
let task = thread::spawn(move || exchange(&config, &ack(&config)));
|
||||
let (mut stream, _) = listener.accept().unwrap();
|
||||
stream
|
||||
.set_read_timeout(Some(Duration::from_secs(12)))
|
||||
.unwrap();
|
||||
let mut reader = BufReader::new(&mut stream);
|
||||
let mut line = String::new();
|
||||
reader.read_line(&mut line).unwrap();
|
||||
assert!(task.join().unwrap().is_err());
|
||||
assert!(start.elapsed() >= RESTORE_CHECKPOINT_TIMEOUT - Duration::from_millis(100));
|
||||
assert!(start.elapsed() < RESTORE_CHECKPOINT_TIMEOUT + Duration::from_secs(3));
|
||||
assert_eq!(reader.read(&mut [0]).unwrap(), 0);
|
||||
}
|
||||
@@ -3,7 +3,7 @@
|
||||
//! only the empty segment envelope is built with current APIs.
|
||||
use super::tests::{request, write_json};
|
||||
use super::*;
|
||||
use crate::fs_store::{FsRuntimeStore, FsRuntimeStoreOptions};
|
||||
use crate::fs_store::{FsRuntimeStore, FsRuntimeStoreOptions, SCHEMA_VERSION};
|
||||
use serde_json::{Value, json};
|
||||
use session_store::{LogEntry, SegmentId, SessionId, Store, WorkerSessionStore};
|
||||
use std::collections::BTreeMap;
|
||||
@@ -226,7 +226,7 @@ fn completed_v4_archive(
|
||||
|
||||
#[test]
|
||||
fn runtime_v11_and_current_route_v4_stopped_orphan_archive_through_staging() {
|
||||
for runtime_schema in [11, 12] {
|
||||
for runtime_schema in [11, 12, SCHEMA_VERSION] {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let root = temp.path().join("runtime");
|
||||
let (worker, session, segment) = runtime_fixture(&root, runtime_schema);
|
||||
@@ -329,7 +329,7 @@ fn runtime_v11_and_current_route_v4_stopped_orphan_archive_through_staging() {
|
||||
(storage.checksum_sha256, storage.bytes, storage.file_count),
|
||||
(checksum, bytes, count)
|
||||
);
|
||||
if runtime_schema == 12 {
|
||||
if runtime_schema == SCHEMA_VERSION {
|
||||
for name in [
|
||||
PathBuf::from("runtime.json"),
|
||||
PathBuf::from(format!("workers/{worker}/worker.json")),
|
||||
@@ -346,8 +346,8 @@ fn runtime_v11_and_current_route_v4_stopped_orphan_archive_through_staging() {
|
||||
);
|
||||
drop(FsRuntimeStore::open_or_create(root.clone(), "runtime-a").unwrap());
|
||||
assert_eq!(tree_bytes(&root), once, "retry rewrote migrated evidence");
|
||||
assert!(!temp.path().join(".runtime.schema-v12-staging").exists());
|
||||
assert!(!temp.path().join(".runtime.pre-schema-v12-backup").exists());
|
||||
assert!(!temp.path().join(".runtime.schema-v13-staging").exists());
|
||||
assert!(!temp.path().join(".runtime.pre-schema-v13-backup").exists());
|
||||
// session-storage, not the immutable receipt's old checksum, attests
|
||||
// current bytes. Tampering with those bytes must remain detectable.
|
||||
let path = archive_session
|
||||
@@ -360,7 +360,7 @@ fn runtime_v11_and_current_route_v4_stopped_orphan_archive_through_staging() {
|
||||
|
||||
#[test]
|
||||
fn corrupt_archive_does_not_block_startup_or_healthy_session_migration_and_repair_converges() {
|
||||
for runtime_schema in [11, 12] {
|
||||
for runtime_schema in [11, 12, SCHEMA_VERSION] {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let root = temp.path().join("runtime");
|
||||
let (worker, session, segment) = runtime_fixture(&root, runtime_schema);
|
||||
@@ -440,8 +440,8 @@ fn unmigratable_archive_preserves_evidence_across_startup_and_reopen() {
|
||||
] {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let root = temp.path().join("runtime");
|
||||
let (live_worker, live_session, live_segment) = runtime_fixture(&root, 12);
|
||||
let (worker, _, segment) = stopped_worker(&root, 12);
|
||||
let (live_worker, live_session, live_segment) = runtime_fixture(&root, SCHEMA_VERSION);
|
||||
let (worker, _, segment) = stopped_worker(&root, SCHEMA_VERSION);
|
||||
let provider = FsWorkerRetentionProvider::new(&root);
|
||||
let path = provider
|
||||
.worker_dir(worker)
|
||||
@@ -542,8 +542,8 @@ fn unreadable_archive_metadata_does_not_block_archive_only_runtime_startup() {
|
||||
] {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let root = temp.path().join("runtime");
|
||||
let (worker, _, _) = runtime_fixture(&root, 12);
|
||||
completed_v4_archive(&root, 12);
|
||||
let (worker, _, _) = runtime_fixture(&root, SCHEMA_VERSION);
|
||||
completed_v4_archive(&root, SCHEMA_VERSION);
|
||||
let provider = FsWorkerRetentionProvider::new(&root);
|
||||
fs::remove_dir_all(provider.worker_dir(worker)).unwrap();
|
||||
let archive = provider.archive_dir("archive-a").unwrap();
|
||||
@@ -575,8 +575,8 @@ fn unreadable_archive_metadata_does_not_block_archive_only_runtime_startup() {
|
||||
fn archive_isolation_does_not_bypass_corrupt_erasure_authority() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let root = temp.path().join("runtime");
|
||||
runtime_fixture(&root, 12);
|
||||
let (session, _, _, _) = completed_v4_archive(&root, 12);
|
||||
runtime_fixture(&root, SCHEMA_VERSION);
|
||||
let (session, _, _, _) = completed_v4_archive(&root, SCHEMA_VERSION);
|
||||
let ledger = root.join("retention/session-sources");
|
||||
fs::create_dir_all(&ledger).unwrap();
|
||||
fs::write(ledger.join(format!("{session}.json")), b"{broken\n").unwrap();
|
||||
@@ -591,7 +591,7 @@ fn archive_isolation_does_not_bypass_corrupt_erasure_authority() {
|
||||
fn runtime_v5_probe_and_staging_skip_erased_originals() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let root = temp.path().join("runtime");
|
||||
let (worker, live_session, live_segment) = runtime_fixture(&root, 12);
|
||||
let (worker, live_session, live_segment) = runtime_fixture(&root, SCHEMA_VERSION);
|
||||
let erased = root
|
||||
.join("orphaned-workers")
|
||||
.join(WorkerId::now_v7().to_string());
|
||||
@@ -643,7 +643,7 @@ fn runtime_v5_rejects_broken_or_future_source_without_rewriting_evidence() {
|
||||
for corruption in ["broken-json", "future-schema"] {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let root = temp.path().join("runtime");
|
||||
let (worker, session, segment) = runtime_fixture(&root, 12);
|
||||
let (worker, session, segment) = runtime_fixture(&root, SCHEMA_VERSION);
|
||||
let session_root = root
|
||||
.join("workers")
|
||||
.join(worker.to_string())
|
||||
@@ -674,19 +674,19 @@ fn runtime_v5_rejects_broken_or_future_source_without_rewriting_evidence() {
|
||||
assert_eq!(tree_bytes(&root), before);
|
||||
assert!(FsRuntimeStore::migrate(&options).is_err());
|
||||
assert_eq!(tree_bytes(&root), before);
|
||||
assert!(!temp.path().join(".runtime.schema-v12-staging").exists());
|
||||
assert!(!temp.path().join(".runtime.pre-schema-v12-backup").exists());
|
||||
assert!(!temp.path().join(".runtime.schema-v13-staging").exists());
|
||||
assert!(!temp.path().join(".runtime.pre-schema-v13-backup").exists());
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn current_runtime_session_migration_still_rejects_live_worker_sockets() {
|
||||
// Relative socket names keep this fixture independent of the checkout/TMPDIR
|
||||
// length. The TempDir still owns the same tree; no process cwd is changed.
|
||||
// The same TempDir owns the relative bind/connect reference. No cwd change
|
||||
// or alias is needed, and the real live-socket migration refusal stays tested.
|
||||
let temp = tempfile::tempdir_in(".").unwrap();
|
||||
let root = Path::new(temp.path().file_name().unwrap()).join("runtime");
|
||||
let (worker, _, _) = runtime_fixture(&root, 12);
|
||||
let (worker, _, _) = runtime_fixture(&root, SCHEMA_VERSION);
|
||||
let before = tree_bytes(&root);
|
||||
let runs = root.join("workers").join(worker.to_string()).join("runs");
|
||||
fs::create_dir_all(runs.join("1")).unwrap();
|
||||
@@ -697,7 +697,7 @@ fn current_runtime_session_migration_still_rejects_live_worker_sockets() {
|
||||
drop(listener);
|
||||
fs::remove_dir_all(runs).unwrap();
|
||||
assert_eq!(tree_bytes(&root), before);
|
||||
assert!(!temp.path().join(".runtime.schema-v12-staging").exists());
|
||||
assert!(!temp.path().join(".runtime.schema-v13-staging").exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -705,7 +705,7 @@ fn current_runtime_upgrades_v5_role_metadata_through_staging_without_reserializi
|
||||
{
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let root = temp.path().join("runtime");
|
||||
let (worker, session, segment) = runtime_fixture(&root, 12);
|
||||
let (worker, session, segment) = runtime_fixture(&root, SCHEMA_VERSION);
|
||||
let aggregate = root.join("workers").join(worker.to_string());
|
||||
let path = aggregate
|
||||
.join("session/segments")
|
||||
|
||||
+1847
-488
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,112 @@
|
||||
use super::*;
|
||||
use crate::restore_checkpoints::{
|
||||
RestoreCheckpointAck, RestoreCheckpointCommit, RestoreCheckpointConfig, RestoreCheckpointStage,
|
||||
wait_for_release,
|
||||
};
|
||||
|
||||
impl Runtime {
|
||||
/// Install an explicitly selected, one-shot process-test gate after normal startup.
|
||||
/// It is not persisted or consulted by admission, routing or recovery authority.
|
||||
pub fn install_test_restore_checkpoint(
|
||||
&self,
|
||||
config: RestoreCheckpointConfig,
|
||||
) -> Result<(), RuntimeError> {
|
||||
config.validate()?;
|
||||
let mut state = self.lock()?;
|
||||
if !matches!(state.persistence, RuntimePersistence::Fs(_)) {
|
||||
return Err(RuntimeError::InvalidRequest(
|
||||
"test restore checkpoint requires an fs Runtime store".into(),
|
||||
));
|
||||
}
|
||||
state.restore_checkpoint = Some(config);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// The lifecycle caller holds the per-Worker operation lock. No Runtime-state
|
||||
// mutex is retained while waiting; the real backend candidate remains owned by
|
||||
// the same operation throughout this synchronous call.
|
||||
pub(super) fn test_restore_checkpoint(
|
||||
&self,
|
||||
worker_ref: &WorkerRef,
|
||||
operation_id: WorkerLifecycleOperationId,
|
||||
stage: RestoreCheckpointStage,
|
||||
) {
|
||||
let config = {
|
||||
let Ok(mut state) = self.lock() else {
|
||||
return;
|
||||
};
|
||||
let slot = &mut state.restore_checkpoint;
|
||||
if !slot.as_ref().is_some_and(|config| {
|
||||
config.worker_id == worker_ref.worker_id && config.stage == stage
|
||||
}) {
|
||||
return;
|
||||
}
|
||||
// Consume even if canonical verification or the handshake fails.
|
||||
slot.take().expect("matched checkpoint")
|
||||
};
|
||||
let Ok(ack) = self.restore_checkpoint_ack(&config, worker_ref, operation_id) else {
|
||||
return;
|
||||
};
|
||||
wait_for_release(&config, &ack);
|
||||
}
|
||||
|
||||
pub(super) fn restore_checkpoint_ack(
|
||||
&self,
|
||||
config: &RestoreCheckpointConfig,
|
||||
worker_ref: &WorkerRef,
|
||||
operation_id: WorkerLifecycleOperationId,
|
||||
) -> Result<RestoreCheckpointAck, RuntimeError> {
|
||||
let invalid = || {
|
||||
RuntimeError::InvalidRequest("test restore checkpoint authority did not match".into())
|
||||
};
|
||||
let state = self.lock()?;
|
||||
let RuntimePersistence::Fs(store) = &state.persistence else {
|
||||
return Err(invalid());
|
||||
};
|
||||
// The existing helper validates the exact bounded canonical aggregate and
|
||||
// syncs that same file plus its owning directories. Never manufacture a
|
||||
// journal marker or infer commit from an in-memory projection.
|
||||
let record = store.read_durable_worker_record(worker_ref)?;
|
||||
let PersistedWorkerExecutionState::ReconciliationRequired(
|
||||
PersistedWorkerLifecycleOperation::Restore(pending),
|
||||
) = &record.execution_state
|
||||
else {
|
||||
return Err(invalid());
|
||||
};
|
||||
let expected = match config.stage {
|
||||
RestoreCheckpointStage::PreparedNotCommitted => RestoreCommitFact::NotCommitted,
|
||||
RestoreCheckpointStage::CommittedPreterminal
|
||||
| RestoreCheckpointStage::ActivatedPreterminal => RestoreCommitFact::Committed,
|
||||
};
|
||||
if config.worker_id != worker_ref.worker_id
|
||||
|| pending.operation_id != operation_id
|
||||
|| pending.commit != expected
|
||||
{
|
||||
return Err(invalid());
|
||||
}
|
||||
let guard = &record.restore_guard;
|
||||
let id = guard.active_request_id.as_ref().ok_or_else(invalid)?;
|
||||
let owner = guard.owners.get(id).ok_or_else(invalid)?;
|
||||
if owner.operation_id != operation_id
|
||||
|| owner.request.request_id != *id
|
||||
|| owner.receipt.is_some()
|
||||
{
|
||||
return Err(invalid());
|
||||
}
|
||||
let runtime_id = state.runtime_identity.clone().ok_or_else(invalid)?;
|
||||
Ok(RestoreCheckpointAck {
|
||||
nonce: config.nonce.clone(),
|
||||
stage: config.stage,
|
||||
runtime_id,
|
||||
worker_id: record.worker_id,
|
||||
operation_id,
|
||||
commit: match expected {
|
||||
RestoreCommitFact::NotCommitted => RestoreCheckpointCommit::NotCommitted,
|
||||
RestoreCommitFact::Committed => RestoreCheckpointCommit::Committed,
|
||||
RestoreCommitFact::LegacyUnknown => unreachable!(),
|
||||
},
|
||||
receipt_absent: true,
|
||||
pid: std::process::id(),
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,304 @@
|
||||
use super::*;
|
||||
use crate::restore_checkpoints::{
|
||||
RestoreCheckpointAck, RestoreCheckpointCommit, RestoreCheckpointConfig, RestoreCheckpointStage,
|
||||
};
|
||||
use std::io::{BufRead, BufReader, Write};
|
||||
use std::net::TcpListener;
|
||||
use std::time::Duration;
|
||||
|
||||
#[test]
|
||||
fn checkpoint_stages_ack_canonical_decision_and_eof_resumes_same_operation_once() {
|
||||
// Component scope: durable Runtime with a substituted execution backend.
|
||||
// Real prepared Controllers and process death are owned by the E2E suite.
|
||||
for (stage, release) in [
|
||||
(RestoreCheckpointStage::PreparedNotCommitted, "release"),
|
||||
(RestoreCheckpointStage::CommittedPreterminal, "eof"),
|
||||
(RestoreCheckpointStage::CommittedPreterminal, "timeout"),
|
||||
(RestoreCheckpointStage::ActivatedPreterminal, "release"),
|
||||
] {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let backend = Arc::new(TestExecutionBackend::default());
|
||||
let runtime = Runtime::with_fs_store_and_execution_backend(
|
||||
FsRuntimeStoreOptions::new(dir.path().join("runtime"))
|
||||
.with_runtime_id("checkpoint-runtime"),
|
||||
backend.clone(),
|
||||
)
|
||||
.unwrap();
|
||||
runtime.bind_runtime_identity("checkpoint-runtime").unwrap();
|
||||
runtime.store_config_bundle(test_bundle()).unwrap();
|
||||
let worker = runtime
|
||||
.create_worker(task_request("checkpoint original operation"))
|
||||
.unwrap();
|
||||
runtime.stop_worker(&worker.worker_ref, None).unwrap();
|
||||
let request = runtime.test_restore_request(&worker.worker_ref);
|
||||
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
|
||||
runtime
|
||||
.install_test_restore_checkpoint(RestoreCheckpointConfig {
|
||||
endpoint: listener.local_addr().unwrap(),
|
||||
worker_id: worker.worker_id,
|
||||
stage,
|
||||
nonce: "runtime-stage-test".into(),
|
||||
})
|
||||
.unwrap();
|
||||
let cloned = runtime.clone();
|
||||
let worker_ref = worker.worker_ref.clone();
|
||||
let sent_request = request.clone();
|
||||
let (sent, received) = std::sync::mpsc::channel();
|
||||
let task = std::thread::spawn(move || {
|
||||
sent.send(cloned.restore_worker_operation(&worker_ref, sent_request))
|
||||
.unwrap();
|
||||
});
|
||||
listener.set_nonblocking(true).unwrap();
|
||||
let stream = tokio::runtime::Builder::new_current_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.unwrap()
|
||||
.block_on(async {
|
||||
let listener = tokio::net::TcpListener::from_std(listener).unwrap();
|
||||
let (stream, _) = tokio::time::timeout(Duration::from_secs(5), listener.accept())
|
||||
.await
|
||||
.expect("checkpoint did not connect")
|
||||
.unwrap();
|
||||
stream.into_std().unwrap()
|
||||
});
|
||||
stream.set_nonblocking(false).unwrap();
|
||||
stream
|
||||
.set_read_timeout(Some(Duration::from_secs(3)))
|
||||
.unwrap();
|
||||
let mut reader = BufReader::new(stream);
|
||||
let mut line = String::new();
|
||||
reader.read_line(&mut line).unwrap();
|
||||
let ack: RestoreCheckpointAck = serde_json::from_str(&line).unwrap();
|
||||
assert_eq!(ack.stage, stage);
|
||||
assert_eq!(ack.nonce, "runtime-stage-test");
|
||||
assert_eq!(ack.runtime_id, "checkpoint-runtime");
|
||||
assert_eq!(ack.worker_id, worker.worker_id);
|
||||
assert_eq!(ack.pid, std::process::id());
|
||||
assert!(ack.receipt_absent);
|
||||
assert_eq!(
|
||||
ack.commit,
|
||||
match stage {
|
||||
RestoreCheckpointStage::PreparedNotCommitted =>
|
||||
RestoreCheckpointCommit::NotCommitted,
|
||||
RestoreCheckpointStage::CommittedPreterminal
|
||||
| RestoreCheckpointStage::ActivatedPreterminal => RestoreCheckpointCommit::Committed,
|
||||
}
|
||||
);
|
||||
assert_eq!(
|
||||
backend.restore_operation_ids.lock().unwrap().as_slice(),
|
||||
&[ack.operation_id]
|
||||
);
|
||||
if stage == RestoreCheckpointStage::ActivatedPreterminal {
|
||||
assert_eq!(backend.activation_operation_ids.lock().unwrap().as_slice(), &[ack.operation_id]);
|
||||
} else {
|
||||
assert!(backend.activation_operation_ids.lock().unwrap().is_empty());
|
||||
}
|
||||
assert!(received.try_recv().is_err());
|
||||
let canonical = runtime_store(&runtime)
|
||||
.read_durable_worker_record(&worker.worker_ref)
|
||||
.unwrap();
|
||||
let id = canonical.restore_guard.active_request_id.as_ref().unwrap();
|
||||
assert_eq!(id, &request.request_id);
|
||||
assert!(canonical.restore_guard.owners[id].receipt.is_none());
|
||||
// Release, EOF and timeout all resume the original lifecycle owner.
|
||||
let start = std::time::Instant::now();
|
||||
if release == "release" {
|
||||
reader.get_mut().write_all(b"R").unwrap();
|
||||
} else if release == "eof" {
|
||||
reader
|
||||
.get_mut()
|
||||
.shutdown(std::net::Shutdown::Write)
|
||||
.unwrap();
|
||||
}
|
||||
let result = received
|
||||
.recv_timeout(Duration::from_secs(12))
|
||||
.unwrap()
|
||||
.unwrap();
|
||||
assert!(start.elapsed() < Duration::from_secs(11));
|
||||
task.join().unwrap();
|
||||
assert_eq!(result.state, WorkerRestoreState::Accepted);
|
||||
assert_eq!(
|
||||
backend.activation_operation_ids.lock().unwrap().as_slice(),
|
||||
&[ack.operation_id]
|
||||
);
|
||||
assert_eq!(
|
||||
runtime
|
||||
.restore_worker_operation(&worker.worker_ref, request)
|
||||
.unwrap(),
|
||||
result
|
||||
);
|
||||
assert_eq!(*backend.restore_count.lock().unwrap(), 1);
|
||||
assert!(runtime.lock().unwrap().restore_checkpoint.is_none());
|
||||
runtime.stop_worker(&worker.worker_ref, None).unwrap();
|
||||
let next = runtime.test_restore_request(&worker.worker_ref);
|
||||
assert_eq!(
|
||||
runtime
|
||||
.restore_worker_operation(&worker.worker_ref, next)
|
||||
.unwrap()
|
||||
.state,
|
||||
WorkerRestoreState::Accepted
|
||||
);
|
||||
assert_eq!(*backend.restore_count.lock().unwrap(), 2);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn checkpoint_disabled_mismatched_or_unreachable_does_not_change_restore_result() {
|
||||
for mode in ["disabled", "mismatch", "unreachable"] {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let backend = Arc::new(TestExecutionBackend::default());
|
||||
let runtime = Runtime::with_fs_store_and_execution_backend(
|
||||
FsRuntimeStoreOptions::new(dir.path().join("runtime"))
|
||||
.with_runtime_id("checkpoint-runtime"),
|
||||
backend.clone(),
|
||||
)
|
||||
.unwrap();
|
||||
runtime.bind_runtime_identity("checkpoint-runtime").unwrap();
|
||||
runtime.store_config_bundle(test_bundle()).unwrap();
|
||||
let worker = runtime
|
||||
.create_worker(task_request("checkpoint absent"))
|
||||
.unwrap();
|
||||
runtime.stop_worker(&worker.worker_ref, None).unwrap();
|
||||
let request = runtime.test_restore_request(&worker.worker_ref);
|
||||
if mode != "disabled" {
|
||||
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
|
||||
let endpoint = listener.local_addr().unwrap();
|
||||
drop(listener);
|
||||
runtime
|
||||
.install_test_restore_checkpoint(RestoreCheckpointConfig {
|
||||
endpoint,
|
||||
worker_id: if mode == "mismatch" {
|
||||
WorkerId::now_v7()
|
||||
} else {
|
||||
worker.worker_id
|
||||
},
|
||||
stage: RestoreCheckpointStage::PreparedNotCommitted,
|
||||
nonce: "absent-test".into(),
|
||||
})
|
||||
.unwrap();
|
||||
}
|
||||
let result = runtime
|
||||
.restore_worker_operation(&worker.worker_ref, request.clone())
|
||||
.unwrap();
|
||||
assert_eq!(result.state, WorkerRestoreState::Accepted, "{mode}");
|
||||
assert_eq!(
|
||||
runtime
|
||||
.restore_worker_operation(&worker.worker_ref, request)
|
||||
.unwrap(),
|
||||
result
|
||||
);
|
||||
assert_eq!(*backend.restore_count.lock().unwrap(), 1);
|
||||
assert_eq!(
|
||||
runtime.lock().unwrap().restore_checkpoint.is_some(),
|
||||
mode == "mismatch"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn checkpoint_verification_requires_canonical_worker_operation_commit_and_absent_receipt() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let root = dir.path().join("runtime");
|
||||
let backend = Arc::new(TestExecutionBackend::default());
|
||||
let runtime = Runtime::with_fs_store_and_execution_backend(
|
||||
FsRuntimeStoreOptions::new(&root).with_runtime_id("checkpoint-runtime"),
|
||||
backend.clone(),
|
||||
)
|
||||
.unwrap();
|
||||
runtime.bind_runtime_identity("checkpoint-runtime").unwrap();
|
||||
runtime.store_config_bundle(test_bundle()).unwrap();
|
||||
let worker = runtime
|
||||
.create_worker(task_request("checkpoint validation"))
|
||||
.unwrap();
|
||||
runtime.stop_worker(&worker.worker_ref, None).unwrap();
|
||||
let request = runtime.test_restore_request(&worker.worker_ref);
|
||||
*backend.restore_result.lock().unwrap() = Some(WorkerExecutionSpawnResult::Errored(
|
||||
WorkerExecutionResult::errored(WorkerExecutionOperation::Restore, "not yet connected"),
|
||||
));
|
||||
assert_eq!(
|
||||
runtime
|
||||
.restore_worker_operation(&worker.worker_ref, request.clone())
|
||||
.unwrap()
|
||||
.state,
|
||||
WorkerRestoreState::ReconciliationRequired
|
||||
);
|
||||
let operation = backend.restore_operation_ids.lock().unwrap()[0];
|
||||
let mut config = RestoreCheckpointConfig {
|
||||
endpoint: "127.0.0.1:1234".parse().unwrap(),
|
||||
worker_id: worker.worker_id,
|
||||
stage: RestoreCheckpointStage::PreparedNotCommitted,
|
||||
nonce: "verification-test".into(),
|
||||
};
|
||||
let path = root
|
||||
.join("workers")
|
||||
.join(worker.worker_id.to_string())
|
||||
.join("worker.json");
|
||||
let before = std::fs::read(&path).unwrap();
|
||||
assert!(
|
||||
runtime
|
||||
.restore_checkpoint_ack(&config, &worker.worker_ref, operation)
|
||||
.is_ok()
|
||||
);
|
||||
assert!(
|
||||
runtime
|
||||
.restore_checkpoint_ack(
|
||||
&config,
|
||||
&worker.worker_ref,
|
||||
WorkerLifecycleOperationId::new()
|
||||
)
|
||||
.is_err()
|
||||
);
|
||||
config.worker_id = WorkerId::now_v7();
|
||||
assert!(
|
||||
runtime
|
||||
.restore_checkpoint_ack(&config, &worker.worker_ref, operation)
|
||||
.is_err()
|
||||
);
|
||||
config.worker_id = worker.worker_id;
|
||||
config.stage = RestoreCheckpointStage::CommittedPreterminal;
|
||||
assert!(
|
||||
runtime
|
||||
.restore_checkpoint_ack(&config, &worker.worker_ref, operation)
|
||||
.is_err()
|
||||
);
|
||||
config.stage = RestoreCheckpointStage::ActivatedPreterminal;
|
||||
assert!(runtime.restore_checkpoint_ack(&config, &worker.worker_ref, operation).is_err());
|
||||
config.stage = RestoreCheckpointStage::PreparedNotCommitted;
|
||||
runtime_store(&runtime).fail_next_worker_reconcile_sync();
|
||||
assert!(
|
||||
runtime
|
||||
.restore_checkpoint_ack(&config, &worker.worker_ref, operation)
|
||||
.is_err()
|
||||
);
|
||||
assert_eq!(
|
||||
std::fs::read(&path).unwrap(),
|
||||
before,
|
||||
"verification must not write authority"
|
||||
);
|
||||
// In-memory equality must not hide an unreadable canonical aggregate.
|
||||
std::fs::write(&path, b"unreadable canonical aggregate").unwrap();
|
||||
assert!(
|
||||
runtime
|
||||
.restore_checkpoint_ack(&config, &worker.worker_ref, operation)
|
||||
.is_err()
|
||||
);
|
||||
assert_eq!(
|
||||
std::fs::read(&path).unwrap(),
|
||||
b"unreadable canonical aggregate"
|
||||
);
|
||||
std::fs::write(&path, before).unwrap();
|
||||
backend.restore_result.lock().unwrap().take();
|
||||
assert_eq!(
|
||||
runtime
|
||||
.restore_worker_operation(&worker.worker_ref, request)
|
||||
.unwrap()
|
||||
.state,
|
||||
WorkerRestoreState::Accepted
|
||||
);
|
||||
assert!(
|
||||
runtime
|
||||
.restore_checkpoint_ack(&config, &worker.worker_ref, operation)
|
||||
.is_err(),
|
||||
"terminal operation cannot ack a preterminal checkpoint"
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,522 @@
|
||||
//! Runtime-owned continuation of an admitted restore, independent of its caller.
|
||||
//!
|
||||
//! Completion callbacks only signal a condition variable. They never acquire a
|
||||
//! Runtime/lifecycle lock and are never joined by this driver while holding one.
|
||||
//! Each step revalidates the canonical owner under the existing Worker lock;
|
||||
//! waiting/backoff and driver teardown happen outside it. A repeated obstruction
|
||||
//! parks this owner with its reason, rather than spinning or inventing rollback.
|
||||
|
||||
use super::*;
|
||||
use crate::execution::RestoreRecoveryDisposition;
|
||||
use std::collections::HashMap;
|
||||
use std::sync::{Condvar, Weak};
|
||||
use std::thread::JoinHandle;
|
||||
use std::time::Duration;
|
||||
|
||||
type Key = (WorkerRef, WorkerLifecycleOperationId);
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
pub(super) struct RestoreContinuations {
|
||||
owners: Mutex<HashMap<Key, Arc<Continuation>>>,
|
||||
#[cfg(test)]
|
||||
paused: std::sync::atomic::AtomicBool,
|
||||
#[cfg(test)]
|
||||
pub(super) terminal_receipt_gate:
|
||||
Mutex<Option<crate::execution::RestoreTestGate<WorkerLifecycleOperationId>>>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
struct Schedule {
|
||||
signaled: bool,
|
||||
cancelled: bool,
|
||||
finished: bool,
|
||||
blocked_reason: Option<String>,
|
||||
parked: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Default)]
|
||||
struct Continuation {
|
||||
schedule: Mutex<Schedule>,
|
||||
wake: Condvar,
|
||||
task: Mutex<Option<JoinHandle<()>>>,
|
||||
}
|
||||
|
||||
impl RestoreContinuations {
|
||||
pub(super) fn signal(&self, worker_ref: &WorkerRef, operation: WorkerLifecycleOperationId) {
|
||||
if let Some(owner) = self
|
||||
.owners
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.get(&(worker_ref.clone(), operation))
|
||||
{
|
||||
let mut schedule = owner
|
||||
.schedule
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
schedule.signaled = true;
|
||||
owner.wake.notify_all();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for RestoreContinuations {
|
||||
fn drop(&mut self) {
|
||||
let owners = self
|
||||
.owners
|
||||
.get_mut()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
for owner in owners.values() {
|
||||
let mut schedule = owner
|
||||
.schedule
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
schedule.cancelled = true;
|
||||
owner.wake.notify_all();
|
||||
}
|
||||
for owner in owners.values() {
|
||||
if let Some(task) = owner
|
||||
.task
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.take()
|
||||
{
|
||||
if task.thread().id() != std::thread::current().id() {
|
||||
let _ = task.join();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) struct ContinuationPause(Arc<RestoreContinuations>);
|
||||
|
||||
#[cfg(test)]
|
||||
impl Drop for ContinuationPause {
|
||||
fn drop(&mut self) {
|
||||
self.0
|
||||
.paused
|
||||
.store(false, std::sync::atomic::Ordering::Release);
|
||||
for owner in self.0.owners.lock().unwrap().values() {
|
||||
owner.schedule.lock().unwrap().signaled = true;
|
||||
owner.wake.notify_all();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
enum Step {
|
||||
Finished,
|
||||
Waiting,
|
||||
Contended,
|
||||
Retry(String),
|
||||
}
|
||||
|
||||
impl Runtime {
|
||||
pub(super) fn restore_repair_authorizer(
|
||||
&self,
|
||||
worker_ref: WorkerRef,
|
||||
) -> Arc<dyn Fn(WorkerLifecycleOperationId) -> Result<(), RuntimeError> + Send + Sync> {
|
||||
let inner = Arc::downgrade(&self.inner);
|
||||
Arc::new(move |operation| {
|
||||
let inner = inner.upgrade().ok_or(RuntimeError::RuntimeStopped)?;
|
||||
// The factory caller owns the lifecycle lock. Never reacquire it
|
||||
// here: it may be awaiting this exact factory task.
|
||||
let mut state = inner.lock().map_err(|_| RuntimeError::StatePoisoned)?;
|
||||
state.ensure_running()?;
|
||||
state.refresh_uncertain_worker_write(&worker_ref)?;
|
||||
let worker = state.worker(&worker_ref)?;
|
||||
state.ensure_subject_not_reserved(&worker_ref, false)?;
|
||||
if worker.pending_stop.is_some()
|
||||
|| worker.pending_restore.is_none_or(|pending| {
|
||||
pending.operation_id != operation
|
||||
|| pending.commit == RestoreCommitFact::LegacyUnknown
|
||||
})
|
||||
{
|
||||
return Err(RuntimeError::RestoreObservationConflict {
|
||||
worker_id: worker_ref.worker_id,
|
||||
});
|
||||
}
|
||||
if let Some(owner) = worker
|
||||
.restore_guard
|
||||
.active_request_id
|
||||
.as_ref()
|
||||
.and_then(|id| worker.restore_guard.owners.get(id))
|
||||
{
|
||||
owner.validate_workdir_preparation_authority(worker)?;
|
||||
}
|
||||
if worker
|
||||
.pending_restore
|
||||
.is_some_and(|pending| pending.source_repair_possible)
|
||||
{
|
||||
return Ok(());
|
||||
}
|
||||
let mut candidate = worker.clone();
|
||||
candidate
|
||||
.pending_restore
|
||||
.as_mut()
|
||||
.expect("validated restore owner")
|
||||
.source_repair_possible = true;
|
||||
// Grant before touching another canonical store. Unknown durability
|
||||
// denies repair until authoritative reconciliation; no old-memory write.
|
||||
state.persist_restore_worker_record(&candidate)?;
|
||||
state.workers.insert(worker_ref.worker_id, candidate);
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn test_wait_restore_continuation_parked(
|
||||
&self,
|
||||
worker_ref: &WorkerRef,
|
||||
operation: WorkerLifecycleOperationId,
|
||||
timeout: Duration,
|
||||
) -> String {
|
||||
let owner = self
|
||||
.restore_continuations
|
||||
.owners
|
||||
.lock()
|
||||
.unwrap()
|
||||
.get(&(worker_ref.clone(), operation))
|
||||
.cloned()
|
||||
.unwrap();
|
||||
let (schedule, waited) = owner
|
||||
.wake
|
||||
.wait_timeout_while(owner.schedule.lock().unwrap(), timeout, |s| {
|
||||
!s.parked && !s.finished
|
||||
})
|
||||
.unwrap();
|
||||
assert!(
|
||||
!waited.timed_out() && schedule.parked,
|
||||
"continuation did not park: {:?}",
|
||||
schedule.blocked_reason
|
||||
);
|
||||
schedule.blocked_reason.clone().expect("parked reason")
|
||||
}
|
||||
|
||||
/// Gate autonomous steps when a component test observes a single fault
|
||||
/// boundary. Dedicated continuation tests run without this gate.
|
||||
#[cfg(test)]
|
||||
pub(crate) fn test_pause_restore_continuation(&self) -> ContinuationPause {
|
||||
assert!(
|
||||
!self
|
||||
.restore_continuations
|
||||
.paused
|
||||
.swap(true, std::sync::atomic::Ordering::AcqRel)
|
||||
);
|
||||
ContinuationPause(self.restore_continuations.clone())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn test_wait_restore_continuation(
|
||||
&self,
|
||||
worker_ref: &WorkerRef,
|
||||
operation: WorkerLifecycleOperationId,
|
||||
timeout: Duration,
|
||||
) {
|
||||
let owner = self
|
||||
.restore_continuations
|
||||
.owners
|
||||
.lock()
|
||||
.unwrap()
|
||||
.get(&(worker_ref.clone(), operation))
|
||||
.cloned()
|
||||
.expect("continuation owner");
|
||||
let schedule = owner.schedule.lock().unwrap();
|
||||
let (schedule, waited) = owner
|
||||
.wake
|
||||
.wait_timeout_while(schedule, timeout, |s| !s.finished && !s.cancelled)
|
||||
.unwrap();
|
||||
assert!(
|
||||
!waited.timed_out() && schedule.finished,
|
||||
"restore continuation did not settle: {:?}",
|
||||
schedule.blocked_reason
|
||||
);
|
||||
}
|
||||
|
||||
pub(super) fn ensure_restore_continuation(
|
||||
&self,
|
||||
worker_ref: &WorkerRef,
|
||||
) -> Result<(), RuntimeError> {
|
||||
let operation = {
|
||||
let state = self.lock()?;
|
||||
let worker = state.worker(worker_ref)?;
|
||||
worker
|
||||
.pending_restore
|
||||
.map(|pending| pending.operation_id)
|
||||
.or_else(|| {
|
||||
worker
|
||||
.restore_guard
|
||||
.active_request_id
|
||||
.as_ref()
|
||||
.and_then(|id| worker.restore_guard.owners.get(id))
|
||||
.filter(|owner| owner.receipt.is_none())
|
||||
.map(|owner| owner.operation_id)
|
||||
})
|
||||
};
|
||||
if let Some(operation) = operation {
|
||||
self.start_restore_continuation(worker_ref.clone(), operation)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(super) fn start_restore_continuation(
|
||||
&self,
|
||||
worker_ref: WorkerRef,
|
||||
operation: WorkerLifecycleOperationId,
|
||||
) -> Result<(), RuntimeError> {
|
||||
{
|
||||
let state = self.lock()?;
|
||||
let Some(backend) = &state.execution_backend else {
|
||||
return Ok(());
|
||||
};
|
||||
if !backend.supports_restore_continuation()
|
||||
|| backend
|
||||
.restore_recovery_disposition(operation, &worker_ref)
|
||||
.is_none()
|
||||
{
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
let key = (worker_ref.clone(), operation);
|
||||
let mut owners = self
|
||||
.restore_continuations
|
||||
.owners
|
||||
.lock()
|
||||
.map_err(|_| RuntimeError::StatePoisoned)?;
|
||||
if owners.contains_key(&key) {
|
||||
return Ok(());
|
||||
}
|
||||
let owner = Arc::new(Continuation::default());
|
||||
let retained = owner.clone();
|
||||
let state = Arc::downgrade(&self.inner);
|
||||
let operations = Arc::downgrade(&self.worker_operations);
|
||||
let continuations = Arc::downgrade(&self.restore_continuations);
|
||||
let task = std::thread::Builder::new()
|
||||
.name("restore-continuation".into())
|
||||
.spawn(move || {
|
||||
run(
|
||||
state,
|
||||
operations,
|
||||
continuations,
|
||||
retained,
|
||||
worker_ref,
|
||||
operation,
|
||||
);
|
||||
})
|
||||
.map_err(|error| {
|
||||
RuntimeError::InvalidRequest(format!(
|
||||
"Restore continuation could not start: {error}"
|
||||
))
|
||||
})?;
|
||||
*owner.task.lock().map_err(|_| RuntimeError::StatePoisoned)? = Some(task);
|
||||
owners.insert(key, owner);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn continue_restore_operation(
|
||||
&self,
|
||||
worker_ref: &WorkerRef,
|
||||
operation: WorkerLifecycleOperationId,
|
||||
) -> Result<Step, RuntimeError> {
|
||||
#[cfg(test)]
|
||||
if self
|
||||
.restore_continuations
|
||||
.paused
|
||||
.load(std::sync::atomic::Ordering::Acquire)
|
||||
{
|
||||
return Ok(Step::Contended);
|
||||
}
|
||||
let lease = self.worker_operation_lock(worker_ref.worker_id)?;
|
||||
// Another caller may be waiting for a backend task. Do not queue a join
|
||||
// of that caller/driver while retaining the lock it needs to complete.
|
||||
let _guard = match lease.try_lock() {
|
||||
Ok(guard) => guard,
|
||||
Err(std::sync::TryLockError::WouldBlock) => return Ok(Step::Contended),
|
||||
Err(_) => return Err(RuntimeError::StatePoisoned),
|
||||
};
|
||||
let stop_is_owned = {
|
||||
let mut state = self.lock()?;
|
||||
state.ensure_running()?;
|
||||
state.refresh_uncertain_worker_write(worker_ref)?;
|
||||
let worker = state.worker(worker_ref)?;
|
||||
worker.pending_stop.is_some()
|
||||
&& worker
|
||||
.pending_restore
|
||||
.is_some_and(|pending| pending.operation_id == operation)
|
||||
};
|
||||
if stop_is_owned {
|
||||
// Stop supersedes publication, not ownership of late resources.
|
||||
// Retain this same continuation until its cleanup is confirmed.
|
||||
self.reconcile_pending_stop_under_lock(worker_ref)?;
|
||||
return Ok(Step::Finished);
|
||||
}
|
||||
let (backend, pending) = {
|
||||
let mut state = self.lock()?;
|
||||
state.ensure_running()?;
|
||||
state.refresh_uncertain_worker_write(worker_ref)?;
|
||||
let worker = state.worker(worker_ref)?;
|
||||
if worker.pending_stop.is_some() {
|
||||
return Ok(Step::Finished);
|
||||
}
|
||||
let owner = worker
|
||||
.restore_guard
|
||||
.active_request_id
|
||||
.as_ref()
|
||||
.and_then(|id| worker.restore_guard.owners.get(id));
|
||||
if owner.is_some_and(|owner| owner.operation_id != operation || owner.receipt.is_some())
|
||||
{
|
||||
return Ok(Step::Finished);
|
||||
}
|
||||
let Some(pending) = worker.pending_restore else {
|
||||
return Ok(Step::Finished);
|
||||
};
|
||||
if pending.operation_id != operation {
|
||||
return Ok(Step::Finished);
|
||||
}
|
||||
if pending.commit == RestoreCommitFact::LegacyUnknown {
|
||||
return Ok(Step::Retry(
|
||||
"Saved Restore has no authoritative commit evidence".into(),
|
||||
));
|
||||
}
|
||||
state.ensure_subject_not_reserved(worker_ref, false)?;
|
||||
if let Some(owner) = owner {
|
||||
owner.validate_workdir_preparation_authority(worker)?;
|
||||
}
|
||||
let Some(backend) = state.execution_backend.clone() else {
|
||||
return Ok(Step::Finished);
|
||||
};
|
||||
(backend, pending)
|
||||
};
|
||||
match backend.restore_recovery_disposition(operation, worker_ref) {
|
||||
None => return Ok(Step::Finished),
|
||||
Some(RestoreRecoveryDisposition::InFlight) => return Ok(Step::Waiting),
|
||||
Some(RestoreRecoveryDisposition::Discard)
|
||||
if pending.commit == RestoreCommitFact::NotCommitted =>
|
||||
{
|
||||
match backend.discard_uncommitted_restore(operation, worker_ref) {
|
||||
WorkerExecutionSpawnResult::RolledBack(result) => {
|
||||
let receipt = RuntimeWorkerRestoreResult::failed(
|
||||
WorkerRestoreState::RolledBack,
|
||||
"worker_restore_rolled_back",
|
||||
result.message_or_default(),
|
||||
);
|
||||
#[cfg(feature = "fs-store")]
|
||||
self.lock()?
|
||||
.record_restore_failure(worker_ref, &result, &receipt)?;
|
||||
#[cfg(not(feature = "fs-store"))]
|
||||
self.settle_restore_rollback(worker_ref, &receipt)?;
|
||||
return Ok(Step::Finished);
|
||||
}
|
||||
result => {
|
||||
return Ok(Step::Retry(format!(
|
||||
"Restore cleanup remains unresolved: {result:?}"
|
||||
)));
|
||||
}
|
||||
}
|
||||
}
|
||||
Some(_) => {}
|
||||
}
|
||||
let result = self.restore_admitted_worker_under_lock(worker_ref, pending.mode)?;
|
||||
if backend.restore_recovery_disposition(operation, worker_ref)
|
||||
== Some(RestoreRecoveryDisposition::InFlight)
|
||||
{
|
||||
return Ok(Step::Waiting);
|
||||
}
|
||||
if result.state == WorkerRestoreState::ReconciliationRequired {
|
||||
Ok(Step::Retry(result.message.unwrap_or_else(|| {
|
||||
"Restore reconciliation remains pending".into()
|
||||
})))
|
||||
} else {
|
||||
Ok(Step::Finished)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn run(
|
||||
inner: Weak<Mutex<RuntimeState>>,
|
||||
operations: Weak<Mutex<BTreeMap<WorkerId, Arc<Mutex<()>>>>>,
|
||||
continuations: Weak<RestoreContinuations>,
|
||||
owner: Arc<Continuation>,
|
||||
worker_ref: WorkerRef,
|
||||
operation: WorkerLifecycleOperationId,
|
||||
) {
|
||||
let mut repeated = 0u32;
|
||||
let mut last_reason = None;
|
||||
loop {
|
||||
let (Some(inner), Some(worker_operations), Some(restore_continuations)) = (
|
||||
inner.upgrade(),
|
||||
operations.upgrade(),
|
||||
continuations.upgrade(),
|
||||
) else {
|
||||
break;
|
||||
};
|
||||
owner
|
||||
.schedule
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.parked = false;
|
||||
let runtime = Runtime {
|
||||
inner,
|
||||
worker_operations,
|
||||
restore_continuations,
|
||||
};
|
||||
let step = runtime.continue_restore_operation(&worker_ref, operation);
|
||||
// No lifecycle lock or strong Runtime/registry reference survives waiting.
|
||||
drop(runtime);
|
||||
let mut schedule = owner
|
||||
.schedule
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
if schedule.cancelled {
|
||||
break;
|
||||
}
|
||||
let contended = matches!(&step, Ok(Step::Contended));
|
||||
let reason = match step {
|
||||
Ok(Step::Finished) => {
|
||||
schedule.finished = true;
|
||||
owner.wake.notify_all();
|
||||
break;
|
||||
}
|
||||
Ok(Step::Waiting) | Ok(Step::Contended) => None,
|
||||
Ok(Step::Retry(reason)) => Some(reason),
|
||||
Err(error) => Some(sanitize_worker_create_failure_message(&error.to_string())),
|
||||
};
|
||||
if let Some(reason) = &reason {
|
||||
repeated = repeated.saturating_add(1);
|
||||
last_reason = Some(reason.clone());
|
||||
schedule.blocked_reason = Some(reason.clone());
|
||||
}
|
||||
// Completed failed attempts can have signaled while this step joined
|
||||
// them. Do not let our own unchanged failure reset the retry budget.
|
||||
if reason.is_some() {
|
||||
schedule.signaled = false;
|
||||
}
|
||||
if (!contended && reason.is_none()) || repeated >= 4 {
|
||||
if repeated >= 4 {
|
||||
schedule.parked = true;
|
||||
owner.wake.notify_all();
|
||||
tracing::warn!(worker_id = %worker_ref.worker_id, operation_id = %operation, reason = ?last_reason,
|
||||
"Restore continuation parked after bounded attempts; operation remains unresolved");
|
||||
}
|
||||
schedule = owner
|
||||
.wake
|
||||
.wait_while(schedule, |s| !s.signaled && !s.cancelled)
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
} else {
|
||||
let delay = Duration::from_millis(100 * (1u64 << repeated.min(4)));
|
||||
schedule = owner
|
||||
.wake
|
||||
.wait_timeout_while(schedule, delay, |s| !s.signaled && !s.cancelled)
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.0;
|
||||
}
|
||||
if schedule.cancelled {
|
||||
break;
|
||||
}
|
||||
if schedule.signaled {
|
||||
schedule.signaled = false;
|
||||
// A completion wakes this same owner, not a new request. Its next
|
||||
// observed result, not the signal itself, determines progress.
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,872 @@
|
||||
//! Offline support for the approved, attachment-free stopped Worker fixture.
|
||||
//!
|
||||
//! Enabled only by `test-support`. This is not an import API: extraction accepts a
|
||||
//! current-schema, settled, single-Segment aggregate; installation accepts only a
|
||||
//! fresh Runtime directory containing its newly initialized `auth.toml`. No
|
||||
//! Runtime or execution backend is constructed. Original Session bytes are never
|
||||
//! rewritten. The caller must separately review/attest the artificial conversation;
|
||||
//! structural checks and secret-marker checks cannot prove arbitrary prose safe.
|
||||
|
||||
use super::*;
|
||||
use crate::catalog::ProfileSourceArchiveSource;
|
||||
use crate::workspace_issuer::{
|
||||
WorkspaceIssuerTrustRecord, WorkspaceIssuerTrustState, validate_workspace_issuer_trust_records,
|
||||
};
|
||||
use session_store::{
|
||||
SegmentId, SessionId, WorkerActiveSegmentRef, WorkerMetadata, WorkerSessionStore,
|
||||
};
|
||||
|
||||
#[cfg(test)]
|
||||
#[path = "test_support_tests.rs"]
|
||||
mod tests;
|
||||
|
||||
const TEMPLATE_VERSION: u32 = 1;
|
||||
const MAX_SESSION_BYTES: usize = 8 * 1024 * 1024;
|
||||
|
||||
/// Portable fixture, intentionally without Debug (Session material is private).
|
||||
/// Fields are private and every deserialized template is validated on install.
|
||||
/// Serialize this value to retain the fixture; do not serialize its source aggregate.
|
||||
#[derive(Clone, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct StoppedWorkerTemplate {
|
||||
template_version: u32,
|
||||
runtime_schema_version: u32,
|
||||
runtime_id: String,
|
||||
workspace_id: String,
|
||||
worker_id: WorkerId,
|
||||
created_at_ms: Option<u64>,
|
||||
session_id: SessionId,
|
||||
segment_id: SegmentId,
|
||||
session_manifest: Vec<u8>,
|
||||
segment_jsonl: Vec<u8>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
struct SessionManifest {
|
||||
schema_version: u32,
|
||||
session_id: SessionId,
|
||||
}
|
||||
|
||||
// Read-only projection: never retain source key material in a template.
|
||||
#[derive(Deserialize)]
|
||||
struct AuthIdentity {
|
||||
identity_id: String,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct SourceAuth {
|
||||
identity: AuthIdentity,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
struct InitializedAuth {
|
||||
identity: crate::auth::RuntimeIdentityMaterial,
|
||||
workspace_issuers: Vec<WorkspaceIssuerTrustRecord>,
|
||||
#[serde(default)]
|
||||
archived_legacy_issuers: Vec<String>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct GuardState {
|
||||
active_request_id: Option<String>,
|
||||
owners: BTreeMap<String, GuardOwner>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct GuardOwner {
|
||||
receipt: Option<serde::de::IgnoredAny>,
|
||||
}
|
||||
|
||||
fn reject(message: &'static str) -> RuntimeError {
|
||||
RuntimeError::InvalidRequest(format!("stopped Worker fixture: {message}"))
|
||||
}
|
||||
|
||||
fn io<T>(result: std::io::Result<T>) -> Result<T, RuntimeError> {
|
||||
result.map_err(|_| reject("filesystem operation failed"))
|
||||
}
|
||||
|
||||
fn decode<T: serde::de::DeserializeOwned>(bytes: &[u8]) -> Result<T, RuntimeError> {
|
||||
serde_json::from_slice(bytes).map_err(|_| reject("invalid typed document"))
|
||||
}
|
||||
|
||||
fn no_symlink_components(path: &Path) -> Result<(), RuntimeError> {
|
||||
let mut current = PathBuf::new();
|
||||
for part in path.components() {
|
||||
if matches!(part, std::path::Component::ParentDir) {
|
||||
return Err(reject("parent traversal is unsupported"));
|
||||
}
|
||||
current.push(part);
|
||||
let metadata = io(fs::symlink_metadata(¤t))?;
|
||||
if metadata.file_type().is_symlink() {
|
||||
return Err(reject("symlinks are unsupported"));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn check_tree(path: &Path, budget: &mut usize) -> Result<(), RuntimeError> {
|
||||
if *budget == 0 {
|
||||
return Err(reject("file count exceeds fixture limit"));
|
||||
}
|
||||
*budget -= 1;
|
||||
let metadata = io(fs::symlink_metadata(path))?;
|
||||
if metadata.is_dir() {
|
||||
for entry in io(fs::read_dir(path))? {
|
||||
check_tree(&io(entry)?.path(), budget)?;
|
||||
}
|
||||
} else if !metadata.is_file() {
|
||||
return Err(reject("symlinks and special files are unsupported"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn regular_file(path: &Path) -> Result<(), RuntimeError> {
|
||||
no_symlink_components(path)?;
|
||||
if !io(fs::symlink_metadata(path))?.is_file() {
|
||||
return Err(reject("expected a regular file"));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn bytes(path: &Path) -> Result<Vec<u8>, RuntimeError> {
|
||||
regular_file(path)?;
|
||||
let mut options = OpenOptions::new();
|
||||
options.read(true);
|
||||
#[cfg(target_os = "linux")]
|
||||
{
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
options.custom_flags(libc::O_NOATIME | libc::O_NOFOLLOW);
|
||||
}
|
||||
let file = io(options.open(path))?;
|
||||
let mut result = Vec::new();
|
||||
io(file
|
||||
.take(MAX_SESSION_BYTES as u64 + 1)
|
||||
.read_to_end(&mut result))?;
|
||||
if result.len() > MAX_SESSION_BYTES {
|
||||
return Err(reject("file exceeds fixture limit"));
|
||||
}
|
||||
Ok(result)
|
||||
}
|
||||
|
||||
fn text(path: &Path) -> Result<String, RuntimeError> {
|
||||
String::from_utf8(bytes(path)?).map_err(|_| reject("invalid UTF-8 document"))
|
||||
}
|
||||
|
||||
fn portable_id(id: &str) -> bool {
|
||||
!id.is_empty()
|
||||
&& id.len() <= 256
|
||||
&& id
|
||||
.bytes()
|
||||
.all(|c| c.is_ascii_alphanumeric() || b"-_".contains(&c))
|
||||
}
|
||||
|
||||
impl StoppedWorkerTemplate {
|
||||
pub fn runtime_id(&self) -> &str {
|
||||
&self.runtime_id
|
||||
}
|
||||
pub fn workspace_id(&self) -> &str {
|
||||
&self.workspace_id
|
||||
}
|
||||
pub fn worker_id(&self) -> WorkerId {
|
||||
self.worker_id
|
||||
}
|
||||
pub fn worker_name(&self) -> String {
|
||||
runtime_worker_name(self.worker_id)
|
||||
}
|
||||
pub fn session_id(&self) -> SessionId {
|
||||
self.session_id
|
||||
}
|
||||
pub fn segment_id(&self) -> SegmentId {
|
||||
self.segment_id
|
||||
}
|
||||
|
||||
/// Extract without migration, rewriting metadata, or copying credentials,
|
||||
/// resolved manifests, execution-operation receipts, or run directories.
|
||||
/// The source Runtime must be offline; its existing owner lock is held for
|
||||
/// the entire read. The path must be `<runtime>/workers/<WorkerId>`.
|
||||
pub fn extract(worker_dir: &Path) -> Result<Self, RuntimeError> {
|
||||
no_symlink_components(worker_dir)?;
|
||||
let workers = worker_dir
|
||||
.parent()
|
||||
.ok_or_else(|| reject("invalid aggregate path"))?;
|
||||
if workers.file_name().and_then(|s| s.to_str()) != Some(WORKERS_DIR) {
|
||||
return Err(reject("invalid aggregate path"));
|
||||
}
|
||||
let root = workers
|
||||
.parent()
|
||||
.ok_or_else(|| reject("invalid Runtime path"))?;
|
||||
// Require the existing lock, rather than creating a file in the source.
|
||||
let lock_path = root.with_file_name(format!(
|
||||
".{}.runtime-owner.lock",
|
||||
root.file_name()
|
||||
.and_then(|s| s.to_str())
|
||||
.ok_or_else(|| reject("invalid Runtime path"))?
|
||||
));
|
||||
regular_file(&lock_path)?;
|
||||
let lock = io(OpenOptions::new().read(true).open(lock_path))?;
|
||||
if !io(FileExt::try_lock_shared(&lock))? {
|
||||
return Err(reject("source Runtime is not offline"));
|
||||
}
|
||||
let _lock = RuntimeStoreOwnerLock { file: lock };
|
||||
check_tree(worker_dir, &mut 256)?;
|
||||
let snapshot: RuntimeSnapshot = decode(&bytes(&root.join(RUNTIME_FILE))?)?;
|
||||
snapshot
|
||||
.validate(&root.join(RUNTIME_FILE))
|
||||
.map_err(|_| reject("unsupported Runtime schema"))?;
|
||||
if !snapshot.subject_create_recovery_fences.is_empty()
|
||||
|| !snapshot.subject_releases.is_empty()
|
||||
{
|
||||
return Err(reject("source has lifecycle operations"));
|
||||
}
|
||||
let record: WorkerAggregateRecord = decode(&bytes(&worker_dir.join(WORKER_FILE))?)?;
|
||||
let record = record
|
||||
.validate(&worker_dir.join(WORKER_FILE))
|
||||
.map_err(|_| reject("invalid Worker schema or identity"))?;
|
||||
let guard: GuardState = decode(
|
||||
&serde_json::to_vec(&record.restore_guard)
|
||||
.map_err(|_| reject("invalid restore guard"))?,
|
||||
)?;
|
||||
if record.status != WorkerStatus::Stopped
|
||||
|| guard.active_request_id.is_some()
|
||||
|| guard.owners.values().any(|owner| owner.receipt.is_none())
|
||||
{
|
||||
return Err(reject("Worker is not settled and stopped"));
|
||||
}
|
||||
let PersistedWorkerExecutionState::Available(execution) = &record.execution_state else {
|
||||
return Err(reject("pending or unavailable execution metadata"));
|
||||
};
|
||||
// A stopped aggregate may retain the empty historical binding marker.
|
||||
// It is deliberately not copied to the installed Worker.
|
||||
if execution.restore_intent != WorkerRestoreIntent::Explicit {
|
||||
return Err(reject("Worker requires explicit restore"));
|
||||
}
|
||||
if !record.workdir_attachments.is_empty() || !record.logical_workdir_attachments.is_empty()
|
||||
{
|
||||
return Err(reject("attachments are unsupported"));
|
||||
}
|
||||
check_request(&execution.request, false)?;
|
||||
let workspace_id = record
|
||||
.workspace_id
|
||||
.ok_or_else(|| reject("missing Workspace identity"))?;
|
||||
let metadata: WorkerMetadata = decode(&bytes(&worker_dir.join(WORKER_METADATA_FILE))?)?;
|
||||
crate::worker_backend::validate_fixture_restore_metadata(&metadata, &execution.request)
|
||||
.map_err(|_| reject("invalid source restore Manifest or settings"))?;
|
||||
if metadata.worker_name != runtime_worker_name(record.worker_id)
|
||||
|| metadata.workspace_id.as_deref() != Some(&workspace_id)
|
||||
|| !metadata.spawned_children.is_empty()
|
||||
|| !metadata.reclaimed_children.is_empty()
|
||||
|| !metadata.peers.is_empty()
|
||||
|| metadata.subjektiv_session_attribution.is_some()
|
||||
{
|
||||
return Err(reject("unsupported or inconsistent Worker metadata"));
|
||||
}
|
||||
let active = metadata
|
||||
.active
|
||||
.ok_or_else(|| reject("missing active Session"))?;
|
||||
let segment_id = active
|
||||
.segment_id
|
||||
.ok_or_else(|| reject("unmaterialized Session"))?;
|
||||
let source_auth: SourceAuth = toml::from_str(&text(&root.join("auth.toml"))?)
|
||||
.map_err(|_| reject("missing Runtime identity"))?;
|
||||
let session = worker_dir.join("session");
|
||||
let fixture = Self {
|
||||
template_version: TEMPLATE_VERSION,
|
||||
runtime_schema_version: SCHEMA_VERSION,
|
||||
runtime_id: source_auth.identity.identity_id,
|
||||
workspace_id,
|
||||
worker_id: record.worker_id,
|
||||
created_at_ms: record.created_at_ms,
|
||||
session_id: active.session_id,
|
||||
segment_id,
|
||||
session_manifest: bytes(&session.join("session.json"))?,
|
||||
segment_jsonl: bytes(&session.join("segments").join(format!("{segment_id}.jsonl")))?,
|
||||
};
|
||||
fixture.validate_shape()?;
|
||||
fixture.validate_session(&session)?;
|
||||
Ok(fixture)
|
||||
}
|
||||
|
||||
/// Install after normal identity/trust initialization, before Runtime start.
|
||||
/// `request` and `manifest` supply NEW backend/config/profile references and
|
||||
/// freshly resolved policy, not a launch. The saved source Manifest is never
|
||||
/// reused. `manifest.worker.name` must equal `worker_name()`.
|
||||
/// Only `auth.toml` may already exist. The owner lock fences live Runtimes;
|
||||
/// staged files are synced and published without overwriting any Worker.
|
||||
/// An interrupted publication fails closed as an incomplete Runtime layout.
|
||||
pub fn install(
|
||||
&self,
|
||||
options: &FsRuntimeStoreOptions,
|
||||
request: CreateWorkerRequest,
|
||||
manifest: &manifest::WorkerManifest,
|
||||
) -> Result<(), RuntimeError> {
|
||||
self.validate_shape()?;
|
||||
check_request(&request, true)?;
|
||||
if manifest.worker.name != self.worker_name() {
|
||||
return Err(reject("resolved Manifest Worker name mismatch"));
|
||||
}
|
||||
let mut metadata = WorkerMetadata::new(
|
||||
self.worker_name(),
|
||||
Some(WorkerActiveSegmentRef {
|
||||
session_id: self.session_id,
|
||||
segment_id: Some(self.segment_id),
|
||||
}),
|
||||
)
|
||||
.with_workspace_id(&self.workspace_id);
|
||||
metadata.resolved_manifest_snapshot = Some(
|
||||
manifest::write_persisted_worker_manifest_snapshot(manifest)
|
||||
.map_err(|_| reject("invalid resolved Manifest"))?,
|
||||
);
|
||||
crate::worker_backend::validate_fixture_restore_metadata(&metadata, &request)
|
||||
.map_err(|_| reject("resolved Manifest does not match fresh request settings"))?;
|
||||
if options.runtime_id != self.runtime_id
|
||||
|| request.worker_id != self.worker_id
|
||||
|| request
|
||||
.workspace_api
|
||||
.as_ref()
|
||||
.map(|api| api.workspace_id.as_str())
|
||||
!= Some(&self.workspace_id)
|
||||
{
|
||||
return Err(reject("installation identity mismatch"));
|
||||
}
|
||||
no_symlink_components(&options.root)?;
|
||||
check_tree(&options.root, &mut 256)?;
|
||||
// Validate the lock location before the ordinary owner-lock opener.
|
||||
let lock_path = options.root.with_file_name(format!(
|
||||
".{}.runtime-owner.lock",
|
||||
options
|
||||
.root
|
||||
.file_name()
|
||||
.and_then(|s| s.to_str())
|
||||
.ok_or_else(|| reject("invalid Runtime path"))?
|
||||
));
|
||||
if fs::symlink_metadata(&lock_path).is_ok() {
|
||||
regular_file(&lock_path)?;
|
||||
}
|
||||
let lock = acquire_runtime_store_owner_lock(&options.root)?;
|
||||
validate_fresh_runtime_store_layout(&options.root)
|
||||
.map_err(|_| reject("destination is not fresh; refusing overwrite"))?;
|
||||
let auth: InitializedAuth = toml::from_str(&text(&options.root.join("auth.toml"))?)
|
||||
.map_err(|_| reject("initialize Runtime identity and trust before install"))?;
|
||||
if auth.identity.identity_id != self.runtime_id
|
||||
|| !auth.archived_legacy_issuers.is_empty()
|
||||
|| auth.identity.signing_key().is_err()
|
||||
{
|
||||
return Err(reject("invalid destination Runtime identity"));
|
||||
}
|
||||
validate_workspace_issuer_trust_records(&auth.workspace_issuers)
|
||||
.map_err(|_| reject("invalid destination Workspace trust"))?;
|
||||
let api = request
|
||||
.workspace_api
|
||||
.as_ref()
|
||||
.expect("validated Workspace binding");
|
||||
if !auth.workspace_issuers.iter().any(|trust| {
|
||||
trust.workspace_id == self.workspace_id
|
||||
&& trust.backend_url == api.base_url
|
||||
&& trust.state == WorkspaceIssuerTrustState::Active
|
||||
}) {
|
||||
return Err(reject("request does not match initialized Workspace trust"));
|
||||
}
|
||||
let stage = options
|
||||
.root
|
||||
.join(format!(".fixture-{}", uuid::Uuid::now_v7()));
|
||||
io(fs::create_dir(&stage))?;
|
||||
let store = FsRuntimeStore {
|
||||
root: stage.clone(),
|
||||
_owner_lock: Some(lock),
|
||||
worker_write_fault: Arc::new(AtomicU8::new(AtomicWriteFault::None as u8)),
|
||||
#[cfg(test)]
|
||||
worker_reconcile_sync_fault: Default::default(),
|
||||
#[cfg(test)]
|
||||
worker_reconcile_read_fault: Default::default(),
|
||||
#[cfg(test)]
|
||||
worker_reconcile_counts: Default::default(),
|
||||
};
|
||||
let result = (|| {
|
||||
let worker = PersistedWorkerRecord {
|
||||
restore_guard: crate::runtime::RestoreGuard::default(),
|
||||
worker_ref: WorkerRef::new(self.worker_id),
|
||||
worker_id: self.worker_id,
|
||||
profile: request.profile.clone(),
|
||||
display_name: request.display_name.clone(),
|
||||
profile_source: request.profile_source.reference(),
|
||||
config_bundle: request.config_bundle.clone(),
|
||||
created_at_ms: self.created_at_ms,
|
||||
status: WorkerStatus::Stopped,
|
||||
last_finished_submission_request_id: None,
|
||||
workspace_id: Some(self.workspace_id.clone()),
|
||||
workdir_attachments: Vec::new(),
|
||||
logical_workdir_attachments: Vec::new(),
|
||||
execution_state: PersistedWorkerExecutionState::Available(
|
||||
PersistedWorkerExecution {
|
||||
request,
|
||||
binding: None,
|
||||
restore_intent: WorkerRestoreIntent::Explicit,
|
||||
},
|
||||
),
|
||||
};
|
||||
WorkerAggregateRecord::from_persisted(&worker)
|
||||
.validate(&store.worker_dir(&self.worker_id).join(WORKER_FILE))
|
||||
.map_err(|_| reject("invalid installation request"))?;
|
||||
store.write_worker_record(&worker)?;
|
||||
let worker_dir = store.worker_dir(&self.worker_id);
|
||||
atomic_write_json(
|
||||
&worker_dir.join(WORKER_METADATA_FILE),
|
||||
&metadata,
|
||||
"write fixture metadata",
|
||||
)?;
|
||||
let session = worker_dir.join("session");
|
||||
io(fs::create_dir_all(session.join("segments")))?;
|
||||
write_new(&session.join("session.json"), &self.session_manifest)?;
|
||||
write_new(
|
||||
&session
|
||||
.join("segments")
|
||||
.join(format!("{}.jsonl", self.segment_id)),
|
||||
&self.segment_jsonl,
|
||||
)?;
|
||||
self.validate_session(&session)?;
|
||||
store.write_runtime_snapshot(&PersistedRuntimeState {
|
||||
subject_create_recovery_fences: BTreeMap::new(),
|
||||
subject_releases: Vec::new(),
|
||||
// Match normal store initialization: this is catalog availability,
|
||||
// not a process or Worker execution start.
|
||||
display_name: options.display_name.clone(),
|
||||
status: RuntimeStatus::Running,
|
||||
next_diagnostic_id: 1,
|
||||
workers: BTreeMap::new(),
|
||||
workspace_owners: BTreeMap::new(),
|
||||
diagnostics: Vec::new(),
|
||||
})?;
|
||||
let loaded = store.load_runtime_state()?;
|
||||
if loaded.workers.len() != 1 || !loaded.diagnostics.is_empty() {
|
||||
return Err(reject("staged aggregate failed ordinary store roundtrip"));
|
||||
}
|
||||
sync_runtime_tree(&stage)?;
|
||||
// No destination existed when checked under the owner lock. Hard-link
|
||||
// the snapshot last with create-new semantics; never replace a file.
|
||||
// Reserve the destination catalog with create-new semantics. Even
|
||||
// an empty preexisting workers directory must never be replaced.
|
||||
let destination_workers = options.root.join(WORKERS_DIR);
|
||||
io(fs::create_dir(&destination_workers))?;
|
||||
io(fs::rename(
|
||||
store.worker_dir(&self.worker_id),
|
||||
destination_workers.join(self.worker_id.to_string()),
|
||||
))?;
|
||||
sync_directory(&destination_workers, "publish stopped Worker aggregate")?;
|
||||
io(fs::hard_link(
|
||||
stage.join(RUNTIME_FILE),
|
||||
options.root.join(RUNTIME_FILE),
|
||||
))?;
|
||||
sync_directory(&options.root, "publish stopped fixture")?;
|
||||
Ok(())
|
||||
})();
|
||||
// Only remove our private staging directory, never existing destination data.
|
||||
let cleanup = fs::remove_dir_all(&stage);
|
||||
result?;
|
||||
io(cleanup)?;
|
||||
sync_directory(&options.root, "finish stopped fixture")
|
||||
}
|
||||
|
||||
fn validate_shape(&self) -> Result<(), RuntimeError> {
|
||||
// Version 1 templates carry only settled Session material, never a
|
||||
// lifecycle operation. Schema 12 -> 13 changes pending decision evidence,
|
||||
// not that payload. Accept the fixed schema-12 artifact read-only and
|
||||
// install through the current aggregate owner; never rewrite its bytes.
|
||||
if self.template_version != TEMPLATE_VERSION
|
||||
|| !matches!(
|
||||
self.runtime_schema_version,
|
||||
RESTORE_ADMISSION_SCHEMA_VERSION | SCHEMA_VERSION
|
||||
)
|
||||
{
|
||||
return Err(reject("unsupported template schema"));
|
||||
}
|
||||
if !portable_id(&self.runtime_id)
|
||||
|| !portable_id(&self.workspace_id)
|
||||
|| self.session_id.get_version_num() != 7
|
||||
|| self.segment_id.get_version_num() != 7
|
||||
|| self.session_manifest.len() > MAX_SESSION_BYTES
|
||||
|| self.segment_jsonl.len() > MAX_SESSION_BYTES
|
||||
{
|
||||
return Err(reject("invalid portable identity or size"));
|
||||
}
|
||||
let manifest: SessionManifest = decode(&self.session_manifest)?;
|
||||
if manifest.session_id != self.session_id
|
||||
|| manifest.schema_version == 0
|
||||
|| !self.segment_jsonl.ends_with(b"\n")
|
||||
{
|
||||
return Err(reject("Session identity or committed boundary mismatch"));
|
||||
}
|
||||
// Read-only JSON traversal, never mutation/reconstruction of Session bytes.
|
||||
// Reject known secret/host markers even in otherwise valid typed records.
|
||||
for line in self
|
||||
.segment_jsonl
|
||||
.split(|b| *b == b'\n')
|
||||
.filter(|line| !line.is_empty())
|
||||
{
|
||||
let _: PortableJson = decode(line)?;
|
||||
validate_preserved_record(line)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_history_entry(
|
||||
&self,
|
||||
entry: &session_store::LoggedHistoryEntry,
|
||||
ids: &mut std::collections::BTreeSet<String>,
|
||||
) -> Result<(), RuntimeError> {
|
||||
if !portable_id(&entry.metadata.entry_id.0)
|
||||
|| !ids.insert(entry.metadata.entry_id.0.clone())
|
||||
{
|
||||
return Err(reject("duplicate or invalid entry identity"));
|
||||
}
|
||||
if entry
|
||||
.metadata
|
||||
.input_metadata
|
||||
.as_ref()
|
||||
.is_some_and(|metadata| {
|
||||
metadata.feature_payload().is_some() || metadata.public_item().is_some()
|
||||
})
|
||||
{
|
||||
// These are opaque owner-specific payloads, not part of the narrow
|
||||
// artificial conversation contract. Never preserve unchecked JSON.
|
||||
return Err(reject("Feature history payloads are unsupported"));
|
||||
}
|
||||
if !matches!(
|
||||
entry.item,
|
||||
session_store::LoggedItem::Message { .. }
|
||||
| session_store::LoggedItem::AssistantResponseBoundary { .. }
|
||||
) {
|
||||
return Err(reject("only text conversation records are supported"));
|
||||
}
|
||||
if let session_store::LoggedSessionHistoryOrigin::ModelOutput { worker } =
|
||||
&entry.metadata.origin
|
||||
{
|
||||
// Session producer IDs identify the original execution, not the
|
||||
// Workspace catalog Worker. Preserve them; do not rebind/rewrite them.
|
||||
if !portable_id(&worker.worker_id)
|
||||
|| worker
|
||||
.workspace_id
|
||||
.as_ref()
|
||||
.is_some_and(|id| id != &self.workspace_id)
|
||||
|| worker
|
||||
.runtime_id
|
||||
.as_ref()
|
||||
.is_some_and(|id| id != &self.runtime_id)
|
||||
{
|
||||
return Err(reject("Session origin identity mismatch"));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn validate_session(&self, root: &Path) -> Result<(), RuntimeError> {
|
||||
for entry in io(fs::read_dir(root))? {
|
||||
let entry = io(entry)?;
|
||||
match entry.file_name().to_str() {
|
||||
Some("session.json" | "segments") => {}
|
||||
Some("artifacts") => check_empty_artifacts(&entry.path())?,
|
||||
_ => return Err(reject("unsupported Session files")),
|
||||
}
|
||||
}
|
||||
let entries = io(fs::read_dir(root.join("segments")))?.collect::<Result<Vec<_>, _>>();
|
||||
let entries = io(entries)?;
|
||||
if entries.len() != 1
|
||||
|| entries[0].file_name() != format!("{}.jsonl", self.segment_id).as_str()
|
||||
{
|
||||
return Err(reject("only the fixed single-Segment fixture is supported"));
|
||||
}
|
||||
let store = WorkerSessionStore::open_read_only(root)
|
||||
.map_err(|_| reject("unsupported Session schema"))?;
|
||||
let logs = store
|
||||
.read_all_read_only_bounded(self.session_id, self.segment_id, MAX_SESSION_BYTES as u64)
|
||||
.map_err(|_| reject("invalid canonical Session records"))?
|
||||
.0;
|
||||
if !matches!(logs.first(), Some(session_store::LogEntry::AnnotatedSegmentStart {
|
||||
session_id, system_prompt: Some(_), forked_from: None, compacted_from: None, ..
|
||||
}) if *session_id == self.session_id)
|
||||
{
|
||||
return Err(reject(
|
||||
"missing system prompt, invalid Segment seed, or unsupported lineage",
|
||||
));
|
||||
}
|
||||
let mut ids = std::collections::BTreeSet::new();
|
||||
for (index, log) in logs.iter().enumerate() {
|
||||
use session_store::LogEntry;
|
||||
match log {
|
||||
LogEntry::AnnotatedSegmentStart { history, .. }
|
||||
if index == 0 && history.is_empty() => {}
|
||||
LogEntry::AnnotatedUserInput {
|
||||
history,
|
||||
extensions,
|
||||
..
|
||||
} => {
|
||||
for extension in extensions {
|
||||
validate_settled_activation(extension)?;
|
||||
}
|
||||
for entry in history {
|
||||
self.validate_history_entry(entry, &mut ids)?;
|
||||
}
|
||||
}
|
||||
LogEntry::AnnotatedAssistantItem { entry, .. } => {
|
||||
self.validate_history_entry(entry, &mut ids)?
|
||||
}
|
||||
LogEntry::RunCompleted {
|
||||
entry_id: Some(id), ..
|
||||
} => {
|
||||
if !portable_id(&id.0) || !ids.insert(id.0.clone()) {
|
||||
return Err(reject("duplicate or invalid entry identity"));
|
||||
}
|
||||
}
|
||||
LogEntry::Invoke { .. } | LogEntry::LlmUsage { .. } | LogEntry::TurnEnd { .. } => {}
|
||||
_ => {
|
||||
return Err(reject(
|
||||
"unsupported conversation record or pending operation",
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
let state = session_store::collect_state(&logs);
|
||||
if state.last_run_interrupted || state.active_run_turn_count.is_some() {
|
||||
return Err(reject("Session has pending execution"));
|
||||
}
|
||||
for segments in &state.user_segments {
|
||||
if segments
|
||||
.iter()
|
||||
.any(|segment| !matches!(segment, protocol::Segment::Text { .. }))
|
||||
{
|
||||
return Err(reject("Session attachments are unsupported"));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
// Compare *fields*, not re-encoded bytes or scalar values (e.g. f32 JSON
|
||||
// normalization). Serde's permissive/untagged/flattened compatibility readers
|
||||
// can drop unknown data without surfacing it to an ignored-field callback. A
|
||||
// retained field must survive the owner's typed projection at every depth.
|
||||
// This intentionally accepts only canonical writer shapes: even an explicit
|
||||
// empty optional field omitted by that writer is rejected, not repaired.
|
||||
// The projection is validation-only; installation always writes original bytes.
|
||||
fn validate_preserved_record(bytes: &[u8]) -> Result<(), RuntimeError> {
|
||||
let original: serde_json::Value = decode(bytes)?;
|
||||
let record: session_store::LogEntry = decode(bytes)?;
|
||||
let projection =
|
||||
serde_json::to_value(&record).map_err(|_| reject("invalid typed Session record"))?;
|
||||
require_preserved_fields(&original, &projection)
|
||||
}
|
||||
|
||||
fn require_preserved_fields(
|
||||
original: &serde_json::Value,
|
||||
projection: &serde_json::Value,
|
||||
) -> Result<(), RuntimeError> {
|
||||
use serde_json::Value;
|
||||
match (original, projection) {
|
||||
(Value::Object(original), Value::Object(projection)) => {
|
||||
for (key, value) in original {
|
||||
let projected = projection
|
||||
.get(key)
|
||||
.ok_or_else(|| reject("unknown or noncanonical Session field"))?;
|
||||
require_preserved_fields(value, projected)?;
|
||||
}
|
||||
}
|
||||
(Value::Array(original), Value::Array(projection))
|
||||
if original.len() == projection.len() =>
|
||||
{
|
||||
for (value, projected) in original.iter().zip(projection) {
|
||||
require_preserved_fields(value, projected)?;
|
||||
}
|
||||
}
|
||||
(Value::Null, Value::Null)
|
||||
| (Value::Bool(_), Value::Bool(_))
|
||||
| (Value::Number(_), Value::Number(_))
|
||||
| (Value::String(_), Value::String(_)) => {}
|
||||
_ => return Err(reject("noncanonical Session field shape")),
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// Exact receipt wire fields owned by worker::PendingActivationState. Keep these
|
||||
// fixture-only mirrors strict; do not change the production compatibility reader.
|
||||
#[derive(Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
struct SubmissionReceipt {
|
||||
submission_request_id: String,
|
||||
source_namespace: String,
|
||||
submission_id: String,
|
||||
payload_digest: String,
|
||||
disposition: protocol::SubmissionDisposition,
|
||||
}
|
||||
|
||||
#[derive(Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
struct NotificationReceipt {
|
||||
notification_request_id: String,
|
||||
source_namespace: String,
|
||||
payload_digest: String,
|
||||
}
|
||||
|
||||
// The approved conversation has committed submission checkpoints in its original
|
||||
// Session records. Keep their receipt bytes, but reject any resumable queue or
|
||||
// Feature invocation history. This projection confers no execution authority.
|
||||
fn validate_settled_activation(
|
||||
extension: &session_store::SessionExtension,
|
||||
) -> Result<(), RuntimeError> {
|
||||
#[derive(Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
struct Checkpoint {
|
||||
#[serde(rename = "next_activation_sequence")]
|
||||
_next_activation_sequence: u64,
|
||||
activating: Option<serde::de::IgnoredAny>,
|
||||
activating_notification: Vec<serde::de::IgnoredAny>,
|
||||
pending: Vec<serde::de::IgnoredAny>,
|
||||
pending_notifications: Vec<serde::de::IgnoredAny>,
|
||||
#[serde(rename = "receipts")]
|
||||
_receipts: Vec<SubmissionReceipt>,
|
||||
#[serde(rename = "notification_receipts")]
|
||||
_notification_receipts: Vec<NotificationReceipt>,
|
||||
invocation_receipts: Vec<serde::de::IgnoredAny>,
|
||||
invocation_recovery_blocked: bool,
|
||||
}
|
||||
if extension.domain != "worker.pending_activations.v1" {
|
||||
return Err(reject("unsupported Session extension"));
|
||||
}
|
||||
let checkpoint: Checkpoint = serde_json::from_value(extension.payload.clone())
|
||||
.map_err(|_| reject("invalid Session activation checkpoint"))?;
|
||||
if checkpoint.activating.is_some()
|
||||
|| !checkpoint.activating_notification.is_empty()
|
||||
|| !checkpoint.pending.is_empty()
|
||||
|| !checkpoint.pending_notifications.is_empty()
|
||||
|| !checkpoint.invocation_receipts.is_empty()
|
||||
|| checkpoint.invocation_recovery_blocked
|
||||
{
|
||||
return Err(reject(
|
||||
"Session has pending activation or invocation history",
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn check_request(request: &CreateWorkerRequest, install: bool) -> Result<(), RuntimeError> {
|
||||
if !request.workdir_attachment_requests.is_empty()
|
||||
|| !request.workdir_attachments.is_empty()
|
||||
|| request.backend_job.is_some()
|
||||
|| request.subjektiv_attached
|
||||
|| request.worker_observation_enabled
|
||||
|| !request.worker_observation_grants.is_empty()
|
||||
|| (install && request.initial_input.is_some())
|
||||
{
|
||||
return Err(reject(
|
||||
"unsupported capabilities, attachments, or initial input",
|
||||
));
|
||||
}
|
||||
if install
|
||||
&& (request.create_fingerprint.trim().is_empty()
|
||||
|| request.config_bundle.is_none()
|
||||
|| !matches!(
|
||||
request.profile_source,
|
||||
ProfileSourceArchiveSource::WorkspaceConfig { .. }
|
||||
))
|
||||
{
|
||||
return Err(reject(
|
||||
"fresh config/profile references and create fingerprint are required",
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn check_empty_artifacts(root: &Path) -> Result<(), RuntimeError> {
|
||||
for entry in io(fs::read_dir(root))? {
|
||||
let entry = io(entry)?;
|
||||
let metadata = io(entry.metadata())?;
|
||||
if metadata.is_dir() {
|
||||
check_empty_artifacts(&entry.path())?;
|
||||
} else if entry.file_name() != ".aggregate.lock" || metadata.len() != 0 {
|
||||
return Err(reject("Session artifacts are unsupported"));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn write_new(path: &Path, bytes: &[u8]) -> Result<(), RuntimeError> {
|
||||
let mut file = io(OpenOptions::new().write(true).create_new(true).open(path))?;
|
||||
io(file.write_all(bytes))?;
|
||||
io(file.sync_all())
|
||||
}
|
||||
|
||||
// A streaming validation visitor, not a generic document available to callers.
|
||||
struct PortableJson;
|
||||
impl<'de> Deserialize<'de> for PortableJson {
|
||||
fn deserialize<D: serde::Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> {
|
||||
struct Visitor;
|
||||
impl<'de> serde::de::Visitor<'de> for Visitor {
|
||||
type Value = PortableJson;
|
||||
fn expecting(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.write_str("portable fixture JSON")
|
||||
}
|
||||
fn visit_bool<E: serde::de::Error>(self, _: bool) -> Result<PortableJson, E> {
|
||||
Ok(PortableJson)
|
||||
}
|
||||
fn visit_i64<E: serde::de::Error>(self, _: i64) -> Result<PortableJson, E> {
|
||||
Ok(PortableJson)
|
||||
}
|
||||
fn visit_u64<E: serde::de::Error>(self, _: u64) -> Result<PortableJson, E> {
|
||||
Ok(PortableJson)
|
||||
}
|
||||
fn visit_f64<E: serde::de::Error>(self, _: f64) -> Result<PortableJson, E> {
|
||||
Ok(PortableJson)
|
||||
}
|
||||
fn visit_unit<E: serde::de::Error>(self) -> Result<PortableJson, E> {
|
||||
Ok(PortableJson)
|
||||
}
|
||||
fn visit_str<E: serde::de::Error>(self, value: &str) -> Result<PortableJson, E> {
|
||||
let lower = value.to_ascii_lowercase();
|
||||
if [
|
||||
"://",
|
||||
"private key",
|
||||
"bearer ",
|
||||
"sk-",
|
||||
"api_key",
|
||||
"access_token",
|
||||
"refresh_token",
|
||||
"private_key",
|
||||
"authorization",
|
||||
]
|
||||
.iter()
|
||||
.any(|marker| lower.contains(marker))
|
||||
|| value
|
||||
.split_whitespace()
|
||||
.any(|word| word.starts_with('/') || word.contains(":\\"))
|
||||
{
|
||||
return Err(E::custom("nonportable or sensitive fixture material"));
|
||||
}
|
||||
Ok(PortableJson)
|
||||
}
|
||||
fn visit_seq<A: serde::de::SeqAccess<'de>>(
|
||||
self,
|
||||
mut seq: A,
|
||||
) -> Result<PortableJson, A::Error> {
|
||||
while seq.next_element::<PortableJson>()?.is_some() {}
|
||||
Ok(PortableJson)
|
||||
}
|
||||
fn visit_map<A: serde::de::MapAccess<'de>>(
|
||||
self,
|
||||
mut map: A,
|
||||
) -> Result<PortableJson, A::Error> {
|
||||
let mut keys = std::collections::BTreeSet::new();
|
||||
while let Some(key) = map.next_key::<String>()? {
|
||||
// A duplicate key can hide discarded data even inside an
|
||||
// otherwise opaque extension payload. Never retain it.
|
||||
if !keys.insert(key.clone()) {
|
||||
return Err(serde::de::Error::custom("duplicate fixture JSON field"));
|
||||
}
|
||||
serde::de::Visitor::visit_str::<A::Error>(Visitor, &key)?;
|
||||
map.next_value::<PortableJson>()?;
|
||||
}
|
||||
Ok(PortableJson)
|
||||
}
|
||||
}
|
||||
deserializer.deserialize_any(Visitor)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,919 @@
|
||||
use super::*;
|
||||
use crate::auth::RuntimeIdentityMaterial;
|
||||
use crate::catalog::{ProfileSelector, WorkspaceApiRef};
|
||||
use crate::profile_archive::ProfileSourceGraphSummary;
|
||||
use session_store::{LogEntry, Store};
|
||||
use sha2::{Digest, Sha256};
|
||||
|
||||
const RUNTIME: &str = "fixture-runtime";
|
||||
const WORKSPACE: &str = "fixture-workspace";
|
||||
const NEW_URL: &str = "http://127.0.0.1:34567";
|
||||
|
||||
#[derive(Serialize)]
|
||||
#[serde(untagged)]
|
||||
enum ReceiptFixture<T> {
|
||||
Valid(T),
|
||||
Malformed(u8),
|
||||
}
|
||||
|
||||
#[derive(Default, Serialize)]
|
||||
struct CheckpointFixture {
|
||||
next_activation_sequence: u64,
|
||||
activating: Option<u8>,
|
||||
activating_notification: Vec<u8>,
|
||||
pending: Vec<u8>,
|
||||
pending_notifications: Vec<u8>,
|
||||
receipts: Vec<ReceiptFixture<SubmissionReceipt>>,
|
||||
notification_receipts: Vec<ReceiptFixture<NotificationReceipt>>,
|
||||
invocation_receipts: Vec<u8>,
|
||||
invocation_recovery_blocked: bool,
|
||||
}
|
||||
|
||||
fn settled_checkpoint() -> CheckpointFixture {
|
||||
CheckpointFixture {
|
||||
next_activation_sequence: 2,
|
||||
receipts: vec![ReceiptFixture::Valid(SubmissionReceipt {
|
||||
submission_request_id: "completed-submission-request".into(),
|
||||
source_namespace: "fixture-source".into(),
|
||||
submission_id: "completed-submission".into(),
|
||||
payload_digest: "fixture-payload-digest".into(),
|
||||
disposition: protocol::SubmissionDisposition::Started,
|
||||
})],
|
||||
notification_receipts: vec![ReceiptFixture::Valid(NotificationReceipt {
|
||||
notification_request_id: "completed-notification-request".into(),
|
||||
source_namespace: "fixture-source".into(),
|
||||
payload_digest: "fixture-notification-digest".into(),
|
||||
})],
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
fn checkpoint_extension(checkpoint: &CheckpointFixture) -> session_store::SessionExtension {
|
||||
session_store::SessionExtension::new(
|
||||
"worker.pending_activations.v1",
|
||||
serde_json::to_value(checkpoint).unwrap(),
|
||||
)
|
||||
}
|
||||
|
||||
fn worker_id() -> WorkerId {
|
||||
"01a1657d-b762-7791-87ae-a49f28cadbce".parse().unwrap()
|
||||
}
|
||||
|
||||
fn resolved() -> manifest::WorkerManifest {
|
||||
let mut config = manifest::WorkerManifestConfig::resolution_defaults();
|
||||
config.worker.name = Some(runtime_worker_name(worker_id()));
|
||||
manifest::WorkerManifest::try_from(config).unwrap()
|
||||
}
|
||||
|
||||
fn request() -> CreateWorkerRequest {
|
||||
CreateWorkerRequest {
|
||||
worker_id: worker_id(),
|
||||
create_fingerprint: "new-create-fingerprint".into(),
|
||||
profile: ProfileSelector::Builtin("default".into()),
|
||||
display_name: Some("fixture".into()),
|
||||
profile_source: ProfileSourceArchiveSource::WorkspaceConfig {
|
||||
archive: ProfileSourceArchiveRef {
|
||||
id: "new-profile".into(),
|
||||
digest: "new-profile-digest".into(),
|
||||
size_bytes: 1,
|
||||
source_graph: ProfileSourceGraphSummary {
|
||||
source_count: 1,
|
||||
total_source_bytes: 1,
|
||||
entrypoints: BTreeMap::from([("builtin:default".into(), "default.dcl".into())]),
|
||||
import_count: 0,
|
||||
},
|
||||
},
|
||||
},
|
||||
config_bundle: Some(ConfigBundleRef {
|
||||
id: "new-config".into(),
|
||||
digest: "new-config-digest".into(),
|
||||
}),
|
||||
workspace_api: Some(WorkspaceApiRef {
|
||||
workspace_id: WORKSPACE.into(),
|
||||
base_url: NEW_URL.into(),
|
||||
}),
|
||||
initial_input: None,
|
||||
workdir_attachment_requests: Vec::new(),
|
||||
workdir_attachments: Vec::new(),
|
||||
worker_observation_enabled: false,
|
||||
worker_observation_grants: Vec::new(),
|
||||
memory_settings: Some(manifest::WorkspaceMemorySettingsSnapshot {
|
||||
workspace_id: WORKSPACE.into(),
|
||||
language: "en".into(),
|
||||
}),
|
||||
subjektiv_attached: false,
|
||||
backend_job: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn initialize(root: &Path, runtime_id: &str, workspace_id: &str, url: &str) {
|
||||
#[derive(Serialize)]
|
||||
struct Auth {
|
||||
identity: RuntimeIdentityMaterial,
|
||||
workspace_issuers: Vec<WorkspaceIssuerTrustRecord>,
|
||||
}
|
||||
let issuer = RuntimeIdentityMaterial::generate("fixture-issuer").unwrap();
|
||||
let digest = Sha256::digest(crate::auth::decode_public_key(&issuer.public_key).unwrap());
|
||||
let fingerprint = format!(
|
||||
"sha256:{}",
|
||||
digest
|
||||
.iter()
|
||||
.map(|byte| format!("{byte:02x}"))
|
||||
.collect::<String>()
|
||||
);
|
||||
let trust = WorkspaceIssuerTrustRecord::from_bundle(
|
||||
server_api::WorkspacePublicIdentityBundle {
|
||||
workspace_id: workspace_id.into(),
|
||||
backend_url: url.into(),
|
||||
key_id: "fixture-issuer".into(),
|
||||
algorithm: "ed25519".into(),
|
||||
public_key: issuer.public_key,
|
||||
public_key_fingerprint: fingerprint,
|
||||
},
|
||||
"fixture-trust".into(),
|
||||
100,
|
||||
)
|
||||
.unwrap();
|
||||
fs::create_dir_all(root).unwrap();
|
||||
fs::write(
|
||||
root.join("auth.toml"),
|
||||
toml::to_string(&Auth {
|
||||
identity: RuntimeIdentityMaterial::generate(runtime_id).unwrap(),
|
||||
workspace_issuers: vec![trust],
|
||||
})
|
||||
.unwrap(),
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
fn record(request: CreateWorkerRequest) -> PersistedWorkerRecord {
|
||||
PersistedWorkerRecord {
|
||||
restore_guard: crate::runtime::RestoreGuard::default(),
|
||||
worker_ref: WorkerRef::new(request.worker_id),
|
||||
worker_id: request.worker_id,
|
||||
profile: request.profile.clone(),
|
||||
display_name: request.display_name.clone(),
|
||||
profile_source: request.profile_source.reference(),
|
||||
config_bundle: request.config_bundle.clone(),
|
||||
created_at_ms: Some(123456),
|
||||
status: WorkerStatus::Stopped,
|
||||
last_finished_submission_request_id: Some("old-submission-operation".into()),
|
||||
workspace_id: Some(WORKSPACE.into()),
|
||||
workdir_attachments: vec![],
|
||||
logical_workdir_attachments: vec![],
|
||||
execution_state: PersistedWorkerExecutionState::Available(PersistedWorkerExecution {
|
||||
request,
|
||||
binding: Some(PersistedWorkerExecutionBinding {}),
|
||||
restore_intent: WorkerRestoreIntent::Explicit,
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
fn source() -> (tempfile::TempDir, PathBuf, StoppedWorkerTemplate) {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let root = temp.path().join("runtime");
|
||||
initialize(&root, RUNTIME, WORKSPACE, "http://127.0.0.1:23456");
|
||||
drop(
|
||||
crate::Runtime::with_fs_store(FsRuntimeStoreOptions::new(&root).with_runtime_id(RUNTIME))
|
||||
.unwrap(),
|
||||
);
|
||||
let opened = FsRuntimeStore::open_or_create(root.clone(), RUNTIME).unwrap();
|
||||
let mut old = request();
|
||||
old.workspace_api.as_mut().unwrap().base_url = "http://127.0.0.1:23456".into();
|
||||
old.config_bundle.as_mut().unwrap().id = "old-config".into();
|
||||
opened.store.write_worker_record(&record(old)).unwrap();
|
||||
let worker = opened.store.worker_dir(&worker_id());
|
||||
let session_id = "01a1657d-b800-7777-8888-111111111111".parse().unwrap();
|
||||
let segment_id = "01a1657d-b818-77c1-bd73-176172a4bd82".parse().unwrap();
|
||||
let mut metadata = WorkerMetadata::new(
|
||||
runtime_worker_name(worker_id()),
|
||||
Some(WorkerActiveSegmentRef {
|
||||
session_id,
|
||||
segment_id: Some(segment_id),
|
||||
}),
|
||||
)
|
||||
.with_workspace_id(WORKSPACE)
|
||||
.with_workspace_root(PathBuf::from("/old/host/path"));
|
||||
metadata.resolved_manifest_snapshot =
|
||||
Some(manifest::write_persisted_worker_manifest_snapshot(&resolved()).unwrap());
|
||||
atomic_write_json(
|
||||
&worker.join(WORKER_METADATA_FILE),
|
||||
&metadata,
|
||||
"test metadata",
|
||||
)
|
||||
.unwrap();
|
||||
let sessions = WorkerSessionStore::new(worker.join("session")).unwrap();
|
||||
let history = session_store::LoggedHistoryEntry {
|
||||
item: session_store::LoggedItem::Message {
|
||||
role: session_store::LoggedRole::Assistant,
|
||||
content: vec![session_store::LoggedContentPart::Text {
|
||||
text: "artificial fixture reply".into(),
|
||||
}],
|
||||
},
|
||||
metadata: session_store::LoggedSessionHistoryMetadata {
|
||||
entry_id: session_store::LoggedSessionHistoryEntryId("fixed-entry-id".into()),
|
||||
origin: session_store::LoggedSessionHistoryOrigin::ModelOutput {
|
||||
worker: session_store::LoggedWorkerSubject {
|
||||
workspace_id: Some(WORKSPACE.into()),
|
||||
runtime_id: Some(RUNTIME.into()),
|
||||
worker_id: worker_id().to_string(),
|
||||
},
|
||||
},
|
||||
derivation: None,
|
||||
input_metadata: None,
|
||||
},
|
||||
};
|
||||
sessions
|
||||
.create_segment(
|
||||
session_id,
|
||||
segment_id,
|
||||
&[
|
||||
LogEntry::AnnotatedSegmentStart {
|
||||
ts: 123,
|
||||
session_id,
|
||||
system_prompt: Some("Harmless artificial fixture instructions.".into()),
|
||||
config: Default::default(),
|
||||
history: vec![],
|
||||
forked_from: None,
|
||||
compacted_from: None,
|
||||
},
|
||||
LogEntry::Invoke {
|
||||
ts: 124,
|
||||
trigger: protocol::InvokeKind::UserSend,
|
||||
},
|
||||
LogEntry::AnnotatedUserInput {
|
||||
ts: 124,
|
||||
segments: vec![protocol::Segment::text(
|
||||
"Harmless artificial fixture input.",
|
||||
)],
|
||||
extensions: vec![checkpoint_extension(&settled_checkpoint())],
|
||||
history: vec![session_store::LoggedHistoryEntry {
|
||||
item: session_store::LoggedItem::Message {
|
||||
role: session_store::LoggedRole::User,
|
||||
content: vec![session_store::LoggedContentPart::Text {
|
||||
text: "Harmless artificial fixture input.".into(),
|
||||
}],
|
||||
},
|
||||
metadata: session_store::LoggedSessionHistoryMetadata {
|
||||
entry_id: session_store::LoggedSessionHistoryEntryId(
|
||||
"fixed-user-id".into(),
|
||||
),
|
||||
origin: session_store::LoggedSessionHistoryOrigin::BackendInstruction {
|
||||
operation_id: None,
|
||||
},
|
||||
derivation: None,
|
||||
input_metadata: None,
|
||||
},
|
||||
}],
|
||||
},
|
||||
LogEntry::AnnotatedAssistantItem {
|
||||
ts: 125,
|
||||
entry: history,
|
||||
},
|
||||
LogEntry::LlmUsage {
|
||||
ts: 125,
|
||||
history_len: 2,
|
||||
input_total_tokens: 1,
|
||||
cache_read_tokens: 0,
|
||||
cache_write_tokens: 0,
|
||||
output_tokens: 1,
|
||||
},
|
||||
LogEntry::TurnEnd {
|
||||
ts: 126,
|
||||
turn_count: 1,
|
||||
},
|
||||
LogEntry::RunCompleted {
|
||||
entry_id: Some(session_store::LoggedSessionHistoryEntryId(
|
||||
"fixed-terminal-id".into(),
|
||||
)),
|
||||
ts: 126,
|
||||
interrupted: false,
|
||||
result: agen::EngineResult::Finished,
|
||||
active_run_turn_count: None,
|
||||
},
|
||||
],
|
||||
)
|
||||
.unwrap();
|
||||
fs::create_dir_all(worker.join("runs/old-run")).unwrap();
|
||||
fs::write(worker.join("runs/old-run/private.log"), "not exported").unwrap();
|
||||
drop(opened);
|
||||
let template = StoppedWorkerTemplate::extract(&worker).unwrap();
|
||||
(temp, worker, template)
|
||||
}
|
||||
|
||||
fn destination(template: &StoppedWorkerTemplate) -> (tempfile::TempDir, FsRuntimeStoreOptions) {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let root = temp.path().join("runtime");
|
||||
initialize(
|
||||
&root,
|
||||
template.runtime_id(),
|
||||
template.workspace_id(),
|
||||
NEW_URL,
|
||||
);
|
||||
(
|
||||
temp,
|
||||
FsRuntimeStoreOptions::new(root).with_runtime_id(template.runtime_id()),
|
||||
)
|
||||
}
|
||||
|
||||
fn assert_fresh(options: &FsRuntimeStoreOptions) {
|
||||
assert_eq!(fs::read_dir(&options.root).unwrap().count(), 1);
|
||||
assert!(options.root.join("auth.toml").is_file());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn extraction_install_roundtrip_preserves_bytes_and_is_inactive() {
|
||||
let (_source, worker, template) = source();
|
||||
let original_worker = fs::read(worker.join(WORKER_FILE)).unwrap();
|
||||
let original_metadata = fs::read(worker.join(WORKER_METADATA_FILE)).unwrap();
|
||||
let exported = serde_json::to_vec(&template).unwrap();
|
||||
let exported_text = String::from_utf8(exported.clone()).unwrap();
|
||||
for forbidden in [
|
||||
"old-config",
|
||||
"old-submission-operation",
|
||||
"restore_guard",
|
||||
"private_key",
|
||||
"http://",
|
||||
"/old/host",
|
||||
"private.log",
|
||||
"resolved_manifest_snapshot",
|
||||
] {
|
||||
assert!(!exported_text.contains(forbidden));
|
||||
}
|
||||
let template: StoppedWorkerTemplate = serde_json::from_slice(&exported).unwrap();
|
||||
let (_target, options) = destination(&template);
|
||||
let auth_before = fs::read(options.root.join("auth.toml")).unwrap();
|
||||
template.install(&options, request(), &resolved()).unwrap();
|
||||
assert_eq!(
|
||||
fs::read(options.root.join("auth.toml")).unwrap(),
|
||||
auth_before
|
||||
);
|
||||
assert_eq!(fs::read(worker.join(WORKER_FILE)).unwrap(), original_worker);
|
||||
assert_eq!(
|
||||
fs::read(worker.join(WORKER_METADATA_FILE)).unwrap(),
|
||||
original_metadata
|
||||
);
|
||||
let installed = options
|
||||
.root
|
||||
.join(WORKERS_DIR)
|
||||
.join(template.worker_id().to_string());
|
||||
assert_eq!(
|
||||
fs::read(installed.join("session/session.json")).unwrap(),
|
||||
template.session_manifest
|
||||
);
|
||||
assert_eq!(
|
||||
fs::read(installed.join(format!("session/segments/{}.jsonl", template.segment_id())))
|
||||
.unwrap(),
|
||||
template.segment_jsonl
|
||||
);
|
||||
assert!(!installed.join("runs").exists());
|
||||
let metadata: WorkerMetadata =
|
||||
decode(&fs::read(installed.join(WORKER_METADATA_FILE)).unwrap()).unwrap();
|
||||
assert_eq!(metadata.active.unwrap().session_id, template.session_id());
|
||||
assert!(metadata.workspace_root.is_none() && metadata.resolved_manifest_snapshot.is_some());
|
||||
|
||||
let opened = FsRuntimeStore::open_or_create(options.root.clone(), RUNTIME).unwrap();
|
||||
let state = opened.state.unwrap();
|
||||
assert!(state.diagnostics.is_empty());
|
||||
let saved = state.workers.get(&worker_id()).unwrap();
|
||||
assert_eq!(saved.status, WorkerStatus::Stopped);
|
||||
assert!(saved.last_finished_submission_request_id.is_none());
|
||||
let PersistedWorkerExecutionState::Available(execution) = &saved.execution_state else {
|
||||
panic!("not available")
|
||||
};
|
||||
assert_eq!(execution.restore_intent, WorkerRestoreIntent::Explicit);
|
||||
assert!(execution.binding.is_none());
|
||||
assert_eq!(execution.request, request());
|
||||
// Fresh restore guard, no operation owner/receipt history.
|
||||
assert_ne!(
|
||||
serde_json::to_vec(&saved.restore_guard).unwrap(),
|
||||
serde_json::to_vec(
|
||||
&decode::<WorkerAggregateRecord>(&original_worker)
|
||||
.unwrap()
|
||||
.restore_guard
|
||||
)
|
||||
.unwrap()
|
||||
);
|
||||
let guard: GuardState = decode(&serde_json::to_vec(&saved.restore_guard).unwrap()).unwrap();
|
||||
assert!(guard.active_request_id.is_none() && guard.owners.is_empty());
|
||||
drop(opened.store);
|
||||
|
||||
struct NoExecution;
|
||||
impl crate::execution::WorkerExecutionBackend for NoExecution {
|
||||
fn backend_id(&self) -> &str {
|
||||
"fixture-no-execution"
|
||||
}
|
||||
fn spawn_worker(
|
||||
&self,
|
||||
_: crate::execution::WorkerExecutionSpawnRequest,
|
||||
) -> crate::execution::WorkerExecutionSpawnResult {
|
||||
panic!("fixture executed")
|
||||
}
|
||||
fn dispatch_input(
|
||||
&self,
|
||||
_: &WorkerRef,
|
||||
_: crate::interaction::WorkerInput,
|
||||
) -> crate::execution::WorkerExecutionResult {
|
||||
panic!("fixture submitted")
|
||||
}
|
||||
fn restore_worker(
|
||||
&self,
|
||||
_: crate::execution::WorkerExecutionRestoreRequest,
|
||||
) -> crate::execution::WorkerExecutionSpawnResult {
|
||||
panic!("fixture auto-restored")
|
||||
}
|
||||
}
|
||||
let runtime =
|
||||
crate::Runtime::with_fs_store_and_execution_backend(options.clone(), Arc::new(NoExecution))
|
||||
.unwrap();
|
||||
assert_eq!(runtime.list_workers().unwrap().len(), 1);
|
||||
assert_eq!(runtime.summary().unwrap().active_worker_count, 0);
|
||||
assert_eq!(
|
||||
runtime
|
||||
.worker_detail(&WorkerRef::new(worker_id()))
|
||||
.unwrap()
|
||||
.status,
|
||||
WorkerStatus::Stopped
|
||||
);
|
||||
drop(runtime);
|
||||
let runtime = crate::Runtime::with_fs_store(options).unwrap();
|
||||
assert_eq!(
|
||||
runtime.list_workers().unwrap()[0].status,
|
||||
WorkerStatus::Stopped
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn install_rejects_schema_identity_input_and_overwrite() {
|
||||
let (_source, _, template) = source();
|
||||
for change in [
|
||||
"template",
|
||||
"runtime-schema",
|
||||
"session-schema",
|
||||
"session-id",
|
||||
"segment-id",
|
||||
"runtime-id",
|
||||
"workspace-id",
|
||||
"worker-id",
|
||||
"origin-id",
|
||||
"duplicate-id",
|
||||
"truncated",
|
||||
] {
|
||||
let (_target, options) = destination(&template);
|
||||
let mut invalid = template.clone();
|
||||
match change {
|
||||
"template" => invalid.template_version += 1,
|
||||
"runtime-schema" => invalid.runtime_schema_version += 1,
|
||||
"session-schema" => {
|
||||
invalid.session_manifest = format!(
|
||||
"{{\"schema_version\":999,\"session_id\":\"{}\"}}",
|
||||
invalid.session_id
|
||||
)
|
||||
.into_bytes()
|
||||
}
|
||||
"session-id" => invalid.session_id = uuid::Uuid::now_v7(),
|
||||
"segment-id" => invalid.segment_id = uuid::Uuid::nil(), // no external attestation: valid new Segment ID is otherwise portable
|
||||
"runtime-id" => invalid.runtime_id = "other-runtime".into(),
|
||||
"workspace-id" => invalid.workspace_id = "other-workspace".into(),
|
||||
"worker-id" => invalid.worker_id = WorkerId::now_v7(),
|
||||
"origin-id" => {
|
||||
invalid.segment_jsonl = String::from_utf8(invalid.segment_jsonl)
|
||||
.unwrap()
|
||||
.replace(RUNTIME, "other-runtime")
|
||||
.into_bytes()
|
||||
}
|
||||
"duplicate-id" => {
|
||||
let last = invalid
|
||||
.segment_jsonl
|
||||
.split_inclusive(|b| *b == b'\n')
|
||||
.last()
|
||||
.unwrap()
|
||||
.to_vec();
|
||||
invalid.segment_jsonl.extend(last);
|
||||
}
|
||||
"truncated" => {
|
||||
invalid.segment_jsonl.pop();
|
||||
}
|
||||
_ => unreachable!(),
|
||||
}
|
||||
assert!(
|
||||
invalid.install(&options, request(), &resolved()).is_err(),
|
||||
"accepted {change}"
|
||||
);
|
||||
assert_fresh(&options);
|
||||
}
|
||||
let (_target, options) = destination(&template);
|
||||
let mut launch = request();
|
||||
launch.initial_input = Some(crate::interaction::WorkerInput::user("do not execute"));
|
||||
assert!(template.install(&options, launch, &resolved()).is_err());
|
||||
assert_fresh(&options);
|
||||
template.install(&options, request(), &resolved()).unwrap();
|
||||
let worker_path = options
|
||||
.root
|
||||
.join(WORKERS_DIR)
|
||||
.join(worker_id().to_string())
|
||||
.join(WORKER_FILE);
|
||||
let before = fs::read(&worker_path).unwrap();
|
||||
assert!(template.install(&options, request(), &resolved()).is_err());
|
||||
assert_eq!(fs::read(worker_path).unwrap(), before);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_uninitialized_untrusted_live_and_partial_destination() {
|
||||
let (_source, _, template) = source();
|
||||
let (_target, options) = destination(&template);
|
||||
let held = acquire_runtime_store_owner_lock(&options.root).unwrap();
|
||||
assert!(template.install(&options, request(), &resolved()).is_err());
|
||||
assert_fresh(&options);
|
||||
drop(held);
|
||||
initialize(&options.root, "wrong-runtime", WORKSPACE, NEW_URL);
|
||||
assert!(template.install(&options, request(), &resolved()).is_err());
|
||||
initialize(&options.root, RUNTIME, WORKSPACE, "http://127.0.0.1:23456");
|
||||
assert!(template.install(&options, request(), &resolved()).is_err());
|
||||
fs::remove_file(options.root.join("auth.toml")).unwrap();
|
||||
assert!(template.install(&options, request(), &resolved()).is_err());
|
||||
initialize(&options.root, RUNTIME, WORKSPACE, NEW_URL);
|
||||
fs::create_dir(options.root.join(WORKERS_DIR)).unwrap();
|
||||
assert!(template.install(&options, request(), &resolved()).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn extraction_rejects_nonstopped_pending_schema_and_metadata_mismatch() {
|
||||
for change in [
|
||||
"running",
|
||||
"pending",
|
||||
"guard-pending",
|
||||
"attachment",
|
||||
"schema",
|
||||
"worker-id",
|
||||
"metadata-workspace",
|
||||
"metadata-session",
|
||||
"artifact",
|
||||
"secret",
|
||||
"url",
|
||||
"path",
|
||||
] {
|
||||
let (_source, worker, _) = source();
|
||||
let mut aggregate: WorkerAggregateRecord =
|
||||
decode(&fs::read(worker.join(WORKER_FILE)).unwrap()).unwrap();
|
||||
match change {
|
||||
"running" => aggregate.status = WorkerStatus::Running,
|
||||
"pending" => {
|
||||
aggregate.execution_state = WorkerExecutionStateRecord::ReconciliationRequired(
|
||||
PersistedWorkerLifecycleOperation::Restore(PersistedWorkerRestoreOperation {
|
||||
operation_id: WorkerLifecycleOperationId::new(),
|
||||
mode: PersistedWorkerRestoreMode::Explicit,
|
||||
commit: crate::runtime::RestoreCommitFact::NotCommitted,
|
||||
source_repair_possible: false,
|
||||
request: request(),
|
||||
binding: None,
|
||||
last_settled_status: WorkerStatus::Stopped,
|
||||
}),
|
||||
)
|
||||
}
|
||||
"guard-pending" => aggregate.restore_guard = serde_json::from_str(r#"{"observation_token":"old-token","active_request_id":"pending-restore","owners":{}}"#).unwrap(),
|
||||
"attachment" => aggregate.logical_workdir_attachments.push(LogicalWorkdirAttachment {
|
||||
alias: workdir::WorkdirAttachmentAlias::new("source").unwrap(),
|
||||
working_directory_id: "old-workdir".into(),
|
||||
capabilities: workdir::WorkdirSessionCapabilities::READ_ONLY,
|
||||
}),
|
||||
"schema" => aggregate.schema_version += 1,
|
||||
"worker-id" => aggregate.worker_id = WorkerId::now_v7(),
|
||||
"metadata-workspace" | "metadata-session" => {
|
||||
let mut metadata: WorkerMetadata =
|
||||
decode(&fs::read(worker.join(WORKER_METADATA_FILE)).unwrap()).unwrap();
|
||||
if change == "metadata-workspace" {
|
||||
metadata.workspace_id = Some("wrong-workspace".into());
|
||||
} else {
|
||||
metadata.active.as_mut().unwrap().session_id = uuid::Uuid::now_v7();
|
||||
}
|
||||
atomic_write_json(
|
||||
&worker.join(WORKER_METADATA_FILE),
|
||||
&metadata,
|
||||
"test mismatch",
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
"artifact" => {
|
||||
fs::create_dir_all(worker.join("session/artifacts/paste")).unwrap();
|
||||
fs::write(
|
||||
worker.join("session/artifacts/paste/payload"),
|
||||
"unsupported",
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
"secret" | "url" | "path" => {
|
||||
let segment = fs::read_dir(worker.join("session/segments"))
|
||||
.unwrap()
|
||||
.next()
|
||||
.unwrap()
|
||||
.unwrap()
|
||||
.path();
|
||||
let replacement = match change {
|
||||
"secret" => "Bearer secret",
|
||||
"url" => "https://old-host.invalid",
|
||||
_ => "/old/host/path",
|
||||
};
|
||||
let text = fs::read_to_string(&segment)
|
||||
.unwrap()
|
||||
.replace("artificial fixture reply", replacement);
|
||||
fs::write(segment, text).unwrap();
|
||||
}
|
||||
_ => unreachable!(),
|
||||
}
|
||||
atomic_write_json(&worker.join(WORKER_FILE), &aggregate, "test aggregate").unwrap();
|
||||
assert!(
|
||||
StoppedWorkerTemplate::extract(&worker).is_err(),
|
||||
"accepted {change}"
|
||||
);
|
||||
}
|
||||
let (_source, worker, _) = source();
|
||||
let held =
|
||||
acquire_runtime_store_owner_lock(worker.parent().unwrap().parent().unwrap()).unwrap();
|
||||
assert!(StoppedWorkerTemplate::extract(&worker).is_err());
|
||||
drop(held);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn schema_v12_settled_template_installs_current_aggregate_without_rewriting_session() {
|
||||
let (_source, _, mut template) = source();
|
||||
template.runtime_schema_version = RESTORE_ADMISSION_SCHEMA_VERSION;
|
||||
let bytes = serde_json::to_vec(&template).unwrap();
|
||||
let (_target, options) = destination(&template);
|
||||
template.install(&options, request(), &resolved()).unwrap();
|
||||
assert_eq!(serde_json::to_vec(&template).unwrap(), bytes);
|
||||
let worker = options.root.join(WORKERS_DIR).join(worker_id().to_string());
|
||||
assert_eq!(
|
||||
fs::read(worker.join("session/session.json")).unwrap(),
|
||||
template.session_manifest
|
||||
);
|
||||
assert_eq!(
|
||||
fs::read(worker.join(format!("session/segments/{}.jsonl", template.segment_id))).unwrap(),
|
||||
template.segment_jsonl
|
||||
);
|
||||
let aggregate: WorkerAggregateRecord =
|
||||
read_bounded_json(&worker.join(WORKER_FILE), "read installed fixture").unwrap();
|
||||
let saved = aggregate.validate(&worker.join(WORKER_FILE)).unwrap();
|
||||
assert_eq!(saved.status, WorkerStatus::Stopped);
|
||||
assert!(matches!(
|
||||
saved.execution_state,
|
||||
PersistedWorkerExecutionState::Available(_)
|
||||
));
|
||||
// Unsupported template schemas do not enter the versioned reader.
|
||||
template.runtime_schema_version = RESTORE_ADMISSION_SCHEMA_VERSION - 1;
|
||||
assert!(template.validate_shape().is_err());
|
||||
template.runtime_schema_version = SCHEMA_VERSION + 1;
|
||||
assert!(template.validate_shape().is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fresh_manifest_and_memory_settings_are_required() {
|
||||
let (_source, _, template) = source();
|
||||
let (_target, options) = destination(&template);
|
||||
let mut manifest = resolved();
|
||||
manifest.worker.name = "wrong-worker".into();
|
||||
assert!(template.install(&options, request(), &manifest).is_err());
|
||||
let mut binding = request();
|
||||
binding.memory_settings = None;
|
||||
assert!(template.install(&options, binding, &resolved()).is_err());
|
||||
let mut binding = request();
|
||||
binding.memory_settings.as_mut().unwrap().workspace_id = "other-workspace".into();
|
||||
assert!(template.install(&options, binding, &resolved()).is_err());
|
||||
assert_fresh(&options);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn settled_checkpoints_preserve_receipts_but_refuse_pending_work() {
|
||||
let mut checkpoint = settled_checkpoint();
|
||||
assert!(validate_settled_activation(&checkpoint_extension(&checkpoint)).is_ok());
|
||||
checkpoint.pending.push(1);
|
||||
assert!(validate_settled_activation(&checkpoint_extension(&checkpoint)).is_err());
|
||||
checkpoint.pending.clear();
|
||||
checkpoint.activating = Some(1);
|
||||
assert!(validate_settled_activation(&checkpoint_extension(&checkpoint)).is_err());
|
||||
checkpoint.activating = None;
|
||||
checkpoint.invocation_receipts.push(1);
|
||||
assert!(validate_settled_activation(&checkpoint_extension(&checkpoint)).is_err());
|
||||
checkpoint.invocation_receipts.clear();
|
||||
checkpoint.invocation_recovery_blocked = true;
|
||||
assert!(validate_settled_activation(&checkpoint_extension(&checkpoint)).is_err());
|
||||
}
|
||||
|
||||
// Deliberately corrupt only test-owned synthetic source files, never the lab.
|
||||
fn assert_rejected_session_bytes(worker: &Path, template: &StoppedWorkerTemplate, bytes: Vec<u8>) {
|
||||
let path = worker.join(format!("session/segments/{}.jsonl", template.segment_id()));
|
||||
fs::write(path, &bytes).unwrap();
|
||||
let extraction_error = StoppedWorkerTemplate::extract(worker)
|
||||
.err()
|
||||
.expect("invalid source accepted");
|
||||
assert!(
|
||||
!extraction_error
|
||||
.to_string()
|
||||
.contains("synthetic-discarded-material")
|
||||
);
|
||||
let (_target, options) = destination(template);
|
||||
let mut invalid = template.clone();
|
||||
invalid.segment_jsonl = bytes;
|
||||
let installation_error = invalid
|
||||
.install(&options, request(), &resolved())
|
||||
.unwrap_err();
|
||||
assert!(
|
||||
!installation_error
|
||||
.to_string()
|
||||
.contains("synthetic-discarded-material")
|
||||
);
|
||||
assert_fresh(&options);
|
||||
}
|
||||
|
||||
fn synthetic_logs(template: &StoppedWorkerTemplate) -> Vec<LogEntry> {
|
||||
template
|
||||
.segment_jsonl
|
||||
.split(|byte| *byte == b'\n')
|
||||
.filter(|line| !line.is_empty())
|
||||
.map(|line| serde_json::from_slice(line).unwrap())
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn encode_synthetic_logs(logs: &[LogEntry]) -> Vec<u8> {
|
||||
let mut bytes = Vec::new();
|
||||
for log in logs {
|
||||
serde_json::to_writer(&mut bytes, log).unwrap();
|
||||
bytes.push(b'\n');
|
||||
}
|
||||
bytes
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn missing_system_prompt_matches_worker_restore_rejection() {
|
||||
let (_source, worker_dir, template) = source();
|
||||
let mut logs = synthetic_logs(&template);
|
||||
assert!(session_store::collect_state(&logs).system_prompt.is_some());
|
||||
let LogEntry::AnnotatedSegmentStart { system_prompt, .. } = &mut logs[0] else {
|
||||
panic!("missing seed")
|
||||
};
|
||||
*system_prompt = None;
|
||||
assert_rejected_session_bytes(&worker_dir, &template, encode_synthetic_logs(&logs));
|
||||
// The owning Worker constructor rejects this before model-client creation,
|
||||
// allocation, or execution. Do not start a controller or submit any input.
|
||||
let store = session_store::CombinedStore::new(
|
||||
WorkerSessionStore::open_read_only(worker_dir.join("session")).unwrap(),
|
||||
session_store::WorkerAggregateStore::open_read_only(&worker_dir, template.worker_name())
|
||||
.unwrap(),
|
||||
);
|
||||
let restored = worker::Worker::restore_from_manifest(
|
||||
template.session_id(),
|
||||
template.segment_id(),
|
||||
resolved(),
|
||||
store,
|
||||
worker::PromptCatalogSource::builtins_only(),
|
||||
)
|
||||
.await;
|
||||
assert!(matches!(
|
||||
restored,
|
||||
Err(worker::WorkerError::SegmentSystemPromptMissing { .. })
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn malformed_submission_and_notification_receipts_fail_extraction_and_install() {
|
||||
let (_source, worker, template) = source();
|
||||
for notification in [false, true] {
|
||||
let mut checkpoint = settled_checkpoint();
|
||||
if notification {
|
||||
checkpoint.notification_receipts =
|
||||
vec![ReceiptFixture::Malformed(1), ReceiptFixture::Malformed(2)];
|
||||
} else {
|
||||
checkpoint.receipts = vec![ReceiptFixture::Malformed(1), ReceiptFixture::Malformed(2)];
|
||||
}
|
||||
let mut logs = synthetic_logs(&template);
|
||||
for log in &mut logs {
|
||||
if let LogEntry::AnnotatedUserInput { extensions, .. } = log {
|
||||
*extensions = vec![checkpoint_extension(&checkpoint)];
|
||||
}
|
||||
}
|
||||
assert_rejected_session_bytes(&worker, &template, encode_synthetic_logs(&logs));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unknown_fields_at_every_retained_record_depth_are_rejected_without_secret_diagnostics() {
|
||||
let (_source, worker, template) = source();
|
||||
let original = String::from_utf8(template.segment_jsonl.clone()).unwrap();
|
||||
// These values are harmless synthetic markers, not lab/private material.
|
||||
// The compatibility decoder normally discards many of these fields.
|
||||
let locations = [
|
||||
r#""kind":"annotated_segment_start""#,
|
||||
r#""kind":"invoke""#,
|
||||
r#""kind":"annotated_user_input""#,
|
||||
r#""kind":"annotated_assistant_item""#,
|
||||
r#""kind":"llm_usage""#,
|
||||
r#""kind":"turn_end""#,
|
||||
r#""kind":"run_completed""#,
|
||||
r#""max_tokens":null"#,
|
||||
r#""kind":"message","role":"assistant""#,
|
||||
r#""kind":"text","text":"artificial fixture reply""#,
|
||||
r#""kind":"text","content":"Harmless artificial fixture input.""#,
|
||||
r#""entry_id":"fixed-entry-id""#,
|
||||
r#""kind":"model_output""#,
|
||||
r#""worker_id":"01a1657d-b762-7791-87ae-a49f28cadbce""#,
|
||||
r#""domain":"worker.pending_activations.v1""#,
|
||||
r#""next_activation_sequence":2"#,
|
||||
r#""submission_request_id":"completed-submission-request""#,
|
||||
r#""notification_request_id":"completed-notification-request""#,
|
||||
];
|
||||
for location in locations {
|
||||
assert!(original.contains(location));
|
||||
for extra in [
|
||||
r#", "reasoning":{"text":"synthetic-discarded-material"}"#,
|
||||
r#", "encrypted_content":"synthetic-discarded-material""#,
|
||||
] {
|
||||
let altered = original.replacen(location, &format!("{location}{extra}"), 1);
|
||||
assert_rejected_session_bytes(&worker, &template, altered.into_bytes());
|
||||
}
|
||||
}
|
||||
let duplicate = original.replacen(
|
||||
r#""role":"assistant""#,
|
||||
r#""role":"assistant","role":"assistant""#,
|
||||
1,
|
||||
);
|
||||
assert_rejected_session_bytes(&worker, &template, duplicate.into_bytes());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn opaque_feature_payloads_cannot_bypass_strict_session_validation() {
|
||||
let (_source, worker, template) = source();
|
||||
let mut logs = synthetic_logs(&template);
|
||||
for log in &mut logs {
|
||||
if let LogEntry::AnnotatedUserInput { history, .. } = log {
|
||||
if let session_store::LoggedItem::Message { role, .. } = &mut history[0].item {
|
||||
*role = session_store::LoggedRole::System;
|
||||
}
|
||||
history[0].metadata.input_metadata = Some(
|
||||
session_store::HistoryInputMetadata::new(
|
||||
session_store::HistoryInputUsePolicy {
|
||||
recording_material: session_store::InputEvidenceUse::Exclude,
|
||||
memory_evidence: session_store::InputEvidenceUse::Exclude,
|
||||
observation: session_store::InputObservationUse::Hidden,
|
||||
compaction: session_store::InputCompactionUse::Retain,
|
||||
},
|
||||
Some(
|
||||
session_store::FeatureInputPayload::encode(
|
||||
"fixture.test",
|
||||
1,
|
||||
&BTreeMap::from([("reasoning", "harmless synthetic payload")]),
|
||||
)
|
||||
.unwrap(),
|
||||
),
|
||||
None,
|
||||
)
|
||||
.unwrap(),
|
||||
);
|
||||
}
|
||||
}
|
||||
assert_rejected_session_bytes(&worker, &template, encode_synthetic_logs(&logs));
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[test]
|
||||
fn refuses_source_and_destination_symlinks() {
|
||||
let (_source, worker, template) = source();
|
||||
let (_target, options) = destination(&template);
|
||||
std::os::unix::fs::symlink(worker.join(WORKER_FILE), options.root.join("linked")).unwrap();
|
||||
assert!(template.install(&options, request(), &resolved()).is_err());
|
||||
fs::remove_file(options.root.join("linked")).unwrap();
|
||||
std::os::unix::fs::symlink(worker.join(WORKER_FILE), worker.join("runs/link")).unwrap();
|
||||
assert!(StoppedWorkerTemplate::extract(&worker).is_err());
|
||||
}
|
||||
|
||||
/// Opt-in only: source path supplied explicitly; assertions never display content.
|
||||
#[test]
|
||||
#[ignore = "requires the approved artificial lab aggregate via YOI_STOPPED_FIXTURE_SOURCE"]
|
||||
fn approved_lab_extract_install() {
|
||||
let path = PathBuf::from(
|
||||
std::env::var_os("YOI_STOPPED_FIXTURE_SOURCE").expect("explicit source required"),
|
||||
);
|
||||
let template = StoppedWorkerTemplate::extract(&path).unwrap();
|
||||
let (_target, options) = destination(&template);
|
||||
let mut bind = request();
|
||||
bind.worker_id = template.worker_id();
|
||||
bind.workspace_api.as_mut().unwrap().workspace_id = template.workspace_id().into();
|
||||
bind.memory_settings.as_mut().unwrap().workspace_id = template.workspace_id().into();
|
||||
let mut manifest = resolved();
|
||||
manifest.worker.name = template.worker_name();
|
||||
template.install(&options, bind, &manifest).unwrap();
|
||||
let installed = options
|
||||
.root
|
||||
.join(WORKERS_DIR)
|
||||
.join(template.worker_id().to_string());
|
||||
for relative in [
|
||||
"session/session.json".to_string(),
|
||||
format!("session/segments/{}.jsonl", template.segment_id()),
|
||||
] {
|
||||
assert!(
|
||||
fs::read(path.join(&relative)).unwrap() == fs::read(installed.join(relative)).unwrap()
|
||||
);
|
||||
}
|
||||
let runtime = crate::Runtime::with_fs_store(options).unwrap();
|
||||
assert_eq!(runtime.list_workers().unwrap().len(), 1);
|
||||
assert_eq!(runtime.summary().unwrap().active_worker_count, 0);
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -193,6 +193,21 @@ mod tests {
|
||||
use super::*;
|
||||
use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
|
||||
fn short_tempdir() -> tempfile::TempDir {
|
||||
let target = std::path::Path::new(env!("CARGO_MANIFEST_DIR"))
|
||||
.parent()
|
||||
.unwrap()
|
||||
.parent()
|
||||
.unwrap()
|
||||
.join("target");
|
||||
std::fs::create_dir_all(&target).unwrap();
|
||||
tempfile::Builder::new()
|
||||
.prefix("d")
|
||||
.rand_bytes(3)
|
||||
.tempdir_in(target)
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
fn read_request() -> ReadRequest {
|
||||
ReadRequest {
|
||||
path: WorkdirPath::new("README.md").unwrap(),
|
||||
@@ -225,9 +240,60 @@ mod tests {
|
||||
assert_eq!(attempts.load(Ordering::SeqCst), 0);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn router_cleanup_of_unopened_session_never_connects_or_follows_reused_alias() {
|
||||
let attempts = Arc::new(AtomicUsize::new(0));
|
||||
let observed = attempts.clone();
|
||||
let session = DeferredWorkdirSession::handle(
|
||||
"old",
|
||||
WorkdirSessionCapabilities::READ_ONLY,
|
||||
move || {
|
||||
observed.fetch_add(1, Ordering::SeqCst);
|
||||
Err("offline".into())
|
||||
},
|
||||
);
|
||||
let router = WorkdirSessionRouter::new();
|
||||
let alias = WorkdirAttachmentAlias::new("w").unwrap();
|
||||
let routed = router.attach(alias.clone(), session.clone()).unwrap();
|
||||
assert!(routed.subscribe_command_events().is_some());
|
||||
assert!(routed.command_snapshot().is_empty());
|
||||
router.close_all().await.unwrap();
|
||||
router.close_all().await.unwrap();
|
||||
assert!(router.aliases().is_empty());
|
||||
assert!(matches!(
|
||||
session.read(read_request()).await,
|
||||
Err(WorkdirError::SessionClosed)
|
||||
));
|
||||
assert_eq!(attempts.load(Ordering::SeqCst), 0);
|
||||
|
||||
let new_attempts = Arc::new(AtomicUsize::new(0));
|
||||
let new_observed = new_attempts.clone();
|
||||
let replacement = DeferredWorkdirSession::handle(
|
||||
"new",
|
||||
WorkdirSessionCapabilities::READ_ONLY,
|
||||
move || {
|
||||
new_observed.fetch_add(1, Ordering::SeqCst);
|
||||
Err("offline".into())
|
||||
},
|
||||
);
|
||||
let new_routed = router.attach(alias, replacement).unwrap();
|
||||
// A retained old connection cannot open the replacement provider.
|
||||
assert!(routed.read(read_request()).await.is_err());
|
||||
assert_eq!(new_attempts.load(Ordering::SeqCst), 0);
|
||||
assert!(matches!(
|
||||
new_routed.read(read_request()).await,
|
||||
Err(ref error) if matches!(error.classification_source(), WorkdirError::Unavailable(_))
|
||||
));
|
||||
assert_eq!(new_attempts.load(Ordering::SeqCst), 1);
|
||||
// Even after a failed open, cleanup must not try connecting again.
|
||||
router.close_all().await.unwrap();
|
||||
assert_eq!(new_attempts.load(Ordering::SeqCst), 1);
|
||||
assert_eq!(attempts.load(Ordering::SeqCst), 0);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn failed_open_retries_on_use_and_keeps_read_only_capabilities() {
|
||||
let temp = tempfile::tempdir().unwrap();
|
||||
let temp = short_tempdir();
|
||||
std::fs::write(temp.path().join("README.md"), "restored").unwrap();
|
||||
let root = temp.path().to_path_buf();
|
||||
let attempts = Arc::new(AtomicUsize::new(0));
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
//! Explicit, operation-owned repair of saved Session material. This is not
|
||||
//! read-only candidate preparation: the canonical Session/attribution is the
|
||||
//! receipt for replay after a lost response, and effects are reported as such.
|
||||
|
||||
use super::*;
|
||||
|
||||
impl ProfileRuntimeWorkerFactory {
|
||||
pub(super) async fn recover_saved_restore_source(
|
||||
&self,
|
||||
request: &WorkerExecutionRestoreRequest,
|
||||
) -> Result<bool, RuntimeRestoreFailure> {
|
||||
let cause = |cause| RuntimeRestoreFailure::started(RuntimeRestoreStage::Recovery, cause);
|
||||
let name = Self::runtime_worker_name_for_ref(&request.worker_ref);
|
||||
let root = self
|
||||
.worker_aggregate_dir(&request.worker_ref)
|
||||
.map_err(|_| cause("restore recovery aggregate unavailable"))?;
|
||||
let metadata_store = WorkerAggregateStore::open_read_only(&root, &name)
|
||||
.map_err(|_| cause("restore recovery metadata open failed"))?;
|
||||
let metadata = metadata_store
|
||||
.read_by_name(&name)
|
||||
.map_err(|_| cause("restore recovery metadata read failed"))?
|
||||
.ok_or_else(|| cause("restore recovery metadata missing"))?;
|
||||
let active = metadata
|
||||
.active
|
||||
.as_ref()
|
||||
.ok_or_else(|| cause("restore recovery Session missing"))?;
|
||||
let pending = active.segment_id.is_none();
|
||||
let incomplete = request.request.subjektiv_attached
|
||||
&& !matches!(
|
||||
metadata.subjektiv_session_attribution,
|
||||
Some(session_store::SubjektivSessionAttributionState::Confirmed { .. })
|
||||
);
|
||||
if !pending && !incomplete {
|
||||
return Ok(false);
|
||||
}
|
||||
request
|
||||
.context
|
||||
.authorize_restore_source_repair(request.operation_id)
|
||||
.map_err(|_| cause("canonical source repair authority is not durably confirmed"))?;
|
||||
// The accepted Runtime operation already owns this Session. Its stable
|
||||
// Session identity and sole committed seed, not a new restore request,
|
||||
// are used to reconcile an interrupted first-head publication.
|
||||
let session_store = WorkerSessionStore::new_with_retention(
|
||||
root.join("session"),
|
||||
self.session_retention()
|
||||
.map_err(|_| cause("restore recovery retention unavailable"))?,
|
||||
)
|
||||
.map_err(|_| cause("restore recovery Session open failed"))?;
|
||||
let (manifest, mut loader) = Self::manifest_for_restore(&metadata, &request.request)
|
||||
.map_err(|_| cause("restore recovery saved manifest invalid"))?;
|
||||
let workspace = RuntimeWorkspaceBackendRef::from_worker_request(
|
||||
&request.request,
|
||||
self.runtime_id.as_deref(),
|
||||
)
|
||||
.map_err(cause)?;
|
||||
#[cfg(feature = "http-server")]
|
||||
let request_client = request
|
||||
.request
|
||||
.workspace_api
|
||||
.as_ref()
|
||||
.and_then(|api| self.workspace_request_clients.get(&api.workspace_id));
|
||||
let context = workspace.worker_context(
|
||||
#[cfg(feature = "http-server")]
|
||||
&request.worker_ref,
|
||||
#[cfg(feature = "http-server")]
|
||||
request.workspace_scope.as_ref(),
|
||||
#[cfg(feature = "http-server")]
|
||||
self.worker_mutation_identity.as_ref(),
|
||||
#[cfg(feature = "http-server")]
|
||||
request_client,
|
||||
#[cfg(feature = "http-server")]
|
||||
self.embedded_worker_mutation_dispatcher.as_ref(),
|
||||
#[cfg(feature = "http-server")]
|
||||
Some(self.prompt_projection_cache.clone()),
|
||||
);
|
||||
if pending && context.workspace_id().is_some() {
|
||||
let bundle = request
|
||||
.config_bundle
|
||||
.as_ref()
|
||||
.ok_or_else(|| cause("pending recovery launch material missing"))?;
|
||||
let projection = self
|
||||
.observe_bundle_prompt_projection(
|
||||
bundle,
|
||||
request
|
||||
.request
|
||||
.workspace_api
|
||||
.as_ref()
|
||||
.map(|api| api.workspace_id.as_str()),
|
||||
)
|
||||
.map_err(|_| cause("pending recovery Prompt projection invalid"))?
|
||||
.ok_or_else(|| cause("pending recovery Prompt projection missing"))?;
|
||||
loader = loader.with_effective_catalog(projection.projection.catalog.clone());
|
||||
}
|
||||
self.failed_restore_resources
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.insert(
|
||||
request.worker_ref.clone(),
|
||||
FailedRestoreResources {
|
||||
operation_id: request.operation_id,
|
||||
sessions: Vec::new(),
|
||||
candidate_workdir_cleanup: None,
|
||||
allocation: None,
|
||||
feature_tasks: None,
|
||||
},
|
||||
);
|
||||
let store = CombinedStore::new(session_store, metadata_store);
|
||||
let worker_result = Worker::restore_pending_from_worker_metadata_with_context(
|
||||
&name,
|
||||
manifest,
|
||||
store,
|
||||
loader,
|
||||
context,
|
||||
WorkerFilesystemAuthority::None,
|
||||
)
|
||||
.await;
|
||||
let mut worker = worker_result.map_err(|error| {
|
||||
self.retain_construction_failure(request, &error);
|
||||
cause(restore_worker_cause(&error))
|
||||
})?;
|
||||
let tasks = worker.retain_startup_feature_cleanup();
|
||||
let sessions = worker.workdir_sessions();
|
||||
let allocation = worker.restore_allocation_cleanup_handle();
|
||||
self.failed_restore_resources
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.insert(
|
||||
request.worker_ref.clone(),
|
||||
FailedRestoreResources {
|
||||
operation_id: request.operation_id,
|
||||
sessions: vec![sessions.clone()],
|
||||
candidate_workdir_cleanup: None,
|
||||
allocation: allocation.clone(),
|
||||
feature_tasks: Some(tasks.clone()),
|
||||
},
|
||||
);
|
||||
let result = async {
|
||||
if let Some(binding) = request.request.backend_job.as_ref() {
|
||||
worker
|
||||
.bind_backend_job(binding.clone())
|
||||
.map_err(|_| cause("recovery Backend Job binding failed"))?;
|
||||
}
|
||||
worker
|
||||
.finalize_subjektiv_session_attribution(
|
||||
request.request.subjektiv_attached,
|
||||
SubjektivSessionAttributionLifecycle::RestoredSession,
|
||||
)
|
||||
.map_err(|error| cause(restore_worker_cause(&error)))?;
|
||||
if pending {
|
||||
worker::WorkerController::recover_pending_head(&mut worker, &root)
|
||||
.await
|
||||
.map_err(|_| cause("pending Session head recovery failed"))?;
|
||||
}
|
||||
Ok::<_, RuntimeRestoreFailure>(())
|
||||
}
|
||||
.await;
|
||||
// Never release allocation while the Worker guard is attached. Retained
|
||||
// handles in the same operation remain available on every cleanup error.
|
||||
drop(worker);
|
||||
let cleanup = async {
|
||||
tasks
|
||||
.shutdown()
|
||||
.await
|
||||
.map_err(|_| cause("recovery Feature settlement failed"))?;
|
||||
sessions
|
||||
.close_all()
|
||||
.await
|
||||
.map_err(|_| cause("recovery Workdir close failed"))?;
|
||||
if let Some(allocation) = allocation {
|
||||
allocation
|
||||
.release()
|
||||
.map_err(|_| cause("recovery allocation release failed"))?;
|
||||
}
|
||||
Ok::<_, RuntimeRestoreFailure>(())
|
||||
}
|
||||
.await;
|
||||
result?;
|
||||
cleanup?;
|
||||
self.failed_restore_resources
|
||||
.lock()
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.remove(&request.worker_ref);
|
||||
Ok(true)
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,927 @@
|
||||
// Included by the production Profile publication suite. Each fault is at a
|
||||
// distinct partial-publication boundary; no factory result is substituted.
|
||||
|
||||
fn publication_gate<T>() -> (
|
||||
mpsc::Sender<()>,
|
||||
mpsc::Receiver<T>,
|
||||
crate::execution::RestoreTestGate<T>,
|
||||
) {
|
||||
let (release, receiver) = mpsc::channel();
|
||||
let (entered, evidence) = mpsc::channel();
|
||||
(
|
||||
release,
|
||||
evidence,
|
||||
crate::execution::RestoreTestGate {
|
||||
entered,
|
||||
release: receiver,
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
fn assert_single_publication(fixture: &Fixture, expected: usize) {
|
||||
let faults = &fixture.backend.factory.restore_faults;
|
||||
assert_eq!(
|
||||
faults.executions.load(Ordering::SeqCst),
|
||||
expected,
|
||||
"construction reran"
|
||||
);
|
||||
assert_eq!(
|
||||
faults.effects_calls.load(Ordering::SeqCst),
|
||||
expected,
|
||||
"restore effects reran"
|
||||
);
|
||||
assert_eq!(
|
||||
faults.publication_attempts.load(Ordering::SeqCst),
|
||||
expected,
|
||||
"Controller start reran"
|
||||
);
|
||||
fixture.assert_quiet();
|
||||
}
|
||||
|
||||
fn assert_real_attach(
|
||||
fixture: &Fixture,
|
||||
handle: &WorkerHandle,
|
||||
) -> crate::execution::WorkerProtocolTransport {
|
||||
let worker_ref = &fixture.request.worker_ref;
|
||||
assert!(matches!(
|
||||
fixture
|
||||
.backend
|
||||
.worker_session(crate::execution::WorkerSessionObservationRequest {
|
||||
worker_ref: worker_ref.clone(),
|
||||
}),
|
||||
runtime_api::WorkerSessionAvailability::LiveProtocol
|
||||
));
|
||||
let transport = fixture
|
||||
.backend
|
||||
.clone()
|
||||
.attach_worker_protocol(worker_ref)
|
||||
.unwrap();
|
||||
transport.validate().unwrap();
|
||||
let paired = subscribe_worker_protocol_session(handle);
|
||||
assert_eq!(
|
||||
serde_json::to_value(&transport.snapshot).unwrap(),
|
||||
serde_json::to_value(paired.snapshot_event).unwrap()
|
||||
);
|
||||
assert!(
|
||||
matches!(&transport.snapshot, Event::Snapshot { session, greeting, state, in_flight, .. }
|
||||
if session == &session_store::public_snapshot::project_current_session_snapshot(&fixture.entries())
|
||||
&& !greeting.tools.is_empty() && state.state == protocol::WorkerState::Idle && in_flight.is_empty())
|
||||
);
|
||||
transport
|
||||
}
|
||||
|
||||
fn observe_publication_markers(
|
||||
fixture: &Fixture,
|
||||
observations: tokio::sync::broadcast::Receiver<crate::observation::WorkerObservationEvent>,
|
||||
markers: &'static [&'static str],
|
||||
) -> tokio::sync::broadcast::Receiver<crate::observation::WorkerObservationEvent> {
|
||||
fixture
|
||||
.backend
|
||||
.run_on_adapter_runtime(async move {
|
||||
let mut observations = observations;
|
||||
let mut counts = vec![0; markers.len()];
|
||||
let mut snapshots = 0;
|
||||
tokio::time::timeout(WAIT, async {
|
||||
loop {
|
||||
let observed = observations.recv().await.unwrap();
|
||||
if matches!(observed.payload, Event::Snapshot { .. }) {
|
||||
snapshots += 1;
|
||||
}
|
||||
for (index, marker) in markers.iter().enumerate() {
|
||||
if is_marker(&observed.payload, marker) {
|
||||
counts[index] += 1;
|
||||
}
|
||||
}
|
||||
if counts.last() == Some(&1) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
})
|
||||
.await
|
||||
.map_err(|_| format!("lost original subscription markers: {counts:?}"))?;
|
||||
assert_eq!(
|
||||
counts,
|
||||
vec![1; markers.len()],
|
||||
"lost or duplicated prefix on original receiver"
|
||||
);
|
||||
assert_eq!(snapshots, 1, "initial full snapshot was lost or replayed");
|
||||
while let Ok(event) = observations.try_recv() {
|
||||
assert!(
|
||||
!markers
|
||||
.iter()
|
||||
.any(|marker| is_marker(&event.payload, marker)),
|
||||
"duplicate after tail fence"
|
||||
);
|
||||
}
|
||||
Ok(observations)
|
||||
})
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
fn partial_registration_recovers_same_controller(hub: bool) {
|
||||
let mut fixture = Fixture::no_model();
|
||||
Arc::get_mut(&mut fixture.backend)
|
||||
.unwrap()
|
||||
.spawn_restore_timeout = Duration::from_millis(250);
|
||||
let runtime = stopped_runtime(&fixture);
|
||||
let pause = runtime.test_pause_restore_continuation();
|
||||
let worker_ref = fixture.request.worker_ref.clone();
|
||||
let scope = fixture.request.workspace_scope.as_ref().unwrap();
|
||||
let request = runtime.test_restore_request(&worker_ref);
|
||||
let observations = runtime.subscribe_worker_observation().unwrap();
|
||||
let (started_release, started_entered) = activation_gate(&fixture, true);
|
||||
let first = runtime
|
||||
.restore_worker_operation_scoped(scope, &worker_ref, request.clone())
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
first.state,
|
||||
server_api::WorkerRestoreState::ReconciliationRequired
|
||||
);
|
||||
let started = started_entered.recv_timeout(WAIT).unwrap();
|
||||
let pending = assert_activation_owned(&fixture, &runtime, &request.request_id, &started);
|
||||
let old = started.handle.as_ref().unwrap().clone();
|
||||
let canonical = fixture.canonical_files();
|
||||
alert_marker(&old, "registration-original-paired-prefix");
|
||||
let (release, entered, fault) = publication_gate();
|
||||
if hub {
|
||||
*fixture
|
||||
.backend
|
||||
.factory
|
||||
.observation_hub
|
||||
.registration_gate
|
||||
.lock()
|
||||
.unwrap() = Some(RestoreHubRegistrationGate {
|
||||
gate: fault,
|
||||
fail: true,
|
||||
});
|
||||
} else {
|
||||
*fixture.backend.bridge_fault.lock().unwrap() = Some(fault);
|
||||
}
|
||||
started_release.release();
|
||||
drop(pause);
|
||||
let boundary_handle = entered.recv_timeout(WAIT).unwrap();
|
||||
assert!(boundary_handle.same_controller(&old));
|
||||
let durable = durable_operation(&runtime, &worker_ref).unwrap();
|
||||
assert_eq!(durable.operation_id, started.operation_id);
|
||||
assert_eq!(durable.commit, crate::runtime::RestoreCommitFact::Committed);
|
||||
assert!(restore_owner(&runtime, &worker_ref, &request.request_id)["receipt"].is_null());
|
||||
if hub {
|
||||
let (_, shared, _) = fixture
|
||||
.backend
|
||||
.factory
|
||||
.observation_hub
|
||||
.get(&worker_ref)
|
||||
.unwrap();
|
||||
assert!(
|
||||
Arc::ptr_eq(&shared, &old.shared_state),
|
||||
"real hub insert preceded fault"
|
||||
);
|
||||
} else {
|
||||
assert!(
|
||||
fixture
|
||||
.backend
|
||||
.factory
|
||||
.observation_hub
|
||||
.get(&worker_ref)
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
let (operation, context, uncertain) = runtime.test_pending_restore_context(&worker_ref);
|
||||
assert_eq!(operation, started.operation_id);
|
||||
assert!(!uncertain);
|
||||
assert!(
|
||||
context
|
||||
.test_buffered_events()
|
||||
.iter()
|
||||
.any(|event| matches!(event, Event::Snapshot { .. }))
|
||||
);
|
||||
alert_marker(&old, "registration-during-fault");
|
||||
let pause = runtime.test_pause_restore_continuation();
|
||||
release.send(()).unwrap();
|
||||
// This lock also observes the end of the actual registration attempt, not
|
||||
// merely the test's entered notification. Its retained scope owns cleanup.
|
||||
let execution = fixture.backend.workers.lock().unwrap()[&worker_ref].clone();
|
||||
assert!(old.same_controller(&execution.handle));
|
||||
assert_eq!(execution.restore_operation_id, Some(operation));
|
||||
assert_eq!(
|
||||
execution.tasks.tasks.lock().unwrap().len(),
|
||||
if hub { 2 } else { 1 }
|
||||
);
|
||||
let bridge = execution.tasks.protocol_bridge.lock().unwrap();
|
||||
if hub {
|
||||
assert!(bridge.is_none());
|
||||
} else {
|
||||
let bridge = bridge.as_ref().unwrap();
|
||||
assert!(
|
||||
!bridge.snapshot_pending,
|
||||
"snapshot already registered, do not replay it"
|
||||
);
|
||||
let streams = bridge.streams.lock().unwrap();
|
||||
assert!(
|
||||
streams.is_some(),
|
||||
"original paired streams survived failed task admission"
|
||||
);
|
||||
assert_eq!(
|
||||
serde_json::to_value(&streams.as_ref().unwrap().snapshot_event).unwrap(),
|
||||
serde_json::to_value(&started.snapshot).unwrap()
|
||||
);
|
||||
}
|
||||
drop(bridge);
|
||||
assert!(
|
||||
!old.protocol_is_closed(),
|
||||
"publication failure must not clean a committed Controller"
|
||||
);
|
||||
assert_single_publication(&fixture, 1);
|
||||
drop(pause);
|
||||
runtime.test_wait_restore_continuation(&worker_ref, operation, WAIT);
|
||||
assert!(durable_operation(&runtime, &worker_ref).is_none());
|
||||
let owner = restore_owner(&runtime, &worker_ref, &request.request_id);
|
||||
assert_eq!(
|
||||
owner["operation_id"],
|
||||
serde_json::to_value(operation).unwrap()
|
||||
);
|
||||
assert_eq!(owner["receipt"]["state"], "accepted");
|
||||
assert!(
|
||||
fixture
|
||||
.backend
|
||||
.pending_restore(&worker_ref)
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
assert!(!pending.task_in_flight.load(Ordering::Acquire));
|
||||
assert!(pending.activated.lock().unwrap().is_none());
|
||||
assert!(pending.result.lock().unwrap().is_none());
|
||||
assert!(pending.task.try_lock().unwrap().is_none());
|
||||
assert_eq!(fixture.backend.workers.lock().unwrap().len(), 1);
|
||||
assert!(old.same_controller(&fixture.backend.workers.lock().unwrap()[&worker_ref].handle));
|
||||
let transport = assert_real_attach(&fixture, &old);
|
||||
assert_eq!(fixture.canonical_files(), canonical);
|
||||
assert_single_publication(&fixture, 1);
|
||||
alert_marker(&old, "registration-tail-fence");
|
||||
let observations = observe_publication_markers(
|
||||
&fixture,
|
||||
observations,
|
||||
&[
|
||||
"registration-original-paired-prefix",
|
||||
"registration-during-fault",
|
||||
"registration-tail-fence",
|
||||
],
|
||||
);
|
||||
assert_eq!(
|
||||
runtime
|
||||
.restore_worker_operation_scoped(scope, &worker_ref, request.clone())
|
||||
.unwrap()
|
||||
.state,
|
||||
server_api::WorkerRestoreState::Accepted
|
||||
);
|
||||
assert_single_publication(&fixture, 1);
|
||||
// Retain the original transport/handle and scope through Stop/replacement.
|
||||
runtime
|
||||
.stop_worker_scoped(scope, &worker_ref, None)
|
||||
.unwrap();
|
||||
assert!(old.protocol_is_closed());
|
||||
assert!(transport.validate().is_err());
|
||||
let replacement_request = runtime.test_restore_request(&worker_ref);
|
||||
assert_eq!(
|
||||
runtime
|
||||
.restore_worker_operation_scoped(scope, &worker_ref, replacement_request.clone())
|
||||
.unwrap()
|
||||
.state,
|
||||
server_api::WorkerRestoreState::Accepted
|
||||
);
|
||||
let replacement = fixture.backend.workers.lock().unwrap()[&worker_ref].clone();
|
||||
assert!(!old.same_controller(&replacement.handle));
|
||||
assert_ne!(replacement.restore_operation_id, Some(operation));
|
||||
assert!(
|
||||
transport
|
||||
.dispatch(Method::Pause {
|
||||
command: protocol::WorkerCommandEnvelope::new(801)
|
||||
})
|
||||
.is_err()
|
||||
);
|
||||
assert!(
|
||||
transport
|
||||
.dispatch(Method::Shutdown {
|
||||
command: protocol::WorkerCommandEnvelope::new(802)
|
||||
})
|
||||
.is_err()
|
||||
);
|
||||
for method in [
|
||||
Method::Pause {
|
||||
command: protocol::WorkerCommandEnvelope::new(803),
|
||||
},
|
||||
Method::Shutdown {
|
||||
command: protocol::WorkerCommandEnvelope::new(804),
|
||||
},
|
||||
] {
|
||||
let old_command = old.clone();
|
||||
assert!(
|
||||
fixture
|
||||
.backend
|
||||
.run_on_adapter_runtime(async move {
|
||||
old_command
|
||||
.send(method)
|
||||
.await
|
||||
.map_err(|error| error.to_string())
|
||||
})
|
||||
.is_err(),
|
||||
"closed original Controller command must not route to replacement"
|
||||
);
|
||||
}
|
||||
fixture
|
||||
.backend
|
||||
.factory
|
||||
.deactivate_stopped_controller(&worker_ref, &old)
|
||||
.unwrap();
|
||||
let (_, shared, _) = fixture
|
||||
.backend
|
||||
.factory
|
||||
.observation_hub
|
||||
.get(&worker_ref)
|
||||
.unwrap();
|
||||
assert!(Arc::ptr_eq(&shared, &replacement.handle.shared_state));
|
||||
alert_marker(&old, "registration-stopped-event-must-not-leak");
|
||||
alert_marker(&replacement.handle, "registration-replacement-fence");
|
||||
fixture
|
||||
.backend
|
||||
.run_on_adapter_runtime(async move {
|
||||
let mut observations = observations;
|
||||
tokio::time::timeout(WAIT, async {
|
||||
loop {
|
||||
let event = observations.recv().await.unwrap();
|
||||
assert!(!is_marker(
|
||||
&event.payload,
|
||||
"registration-stopped-event-must-not-leak"
|
||||
));
|
||||
if is_marker(&event.payload, "registration-replacement-fence") {
|
||||
break;
|
||||
}
|
||||
}
|
||||
})
|
||||
.await
|
||||
.map_err(|_| "replacement observation fence missing".to_string())?;
|
||||
drop(transport);
|
||||
Ok(())
|
||||
})
|
||||
.unwrap();
|
||||
assert_single_publication(&fixture, 2);
|
||||
assert_eq!(
|
||||
runtime
|
||||
.restore_worker_operation_scoped(scope, &worker_ref, request.clone())
|
||||
.unwrap()
|
||||
.state,
|
||||
server_api::WorkerRestoreState::Accepted
|
||||
);
|
||||
assert_eq!(restore_owner_count(&runtime, &worker_ref), 2);
|
||||
let current = fixture.backend.workers.lock().unwrap()[&worker_ref].clone();
|
||||
assert!(current.handle.same_controller(&replacement.handle));
|
||||
assert_eq!(
|
||||
current.restore_operation_id,
|
||||
replacement.restore_operation_id
|
||||
);
|
||||
assert_eq!(
|
||||
replacement.handle.shared_state.snapshot().state,
|
||||
protocol::WorkerState::Idle
|
||||
);
|
||||
drop(current);
|
||||
drop(execution);
|
||||
drop(replacement);
|
||||
drop(pending);
|
||||
runtime
|
||||
.stop_worker_scoped(scope, &worker_ref, None)
|
||||
.unwrap();
|
||||
fixture.assert_released();
|
||||
}
|
||||
|
||||
fn restore_owner_count(runtime: &EmbeddedRuntime, worker_ref: &WorkerRef) -> usize {
|
||||
let saved = runtime
|
||||
.test_runtime_store()
|
||||
.read_worker_record(worker_ref)
|
||||
.unwrap();
|
||||
serde_json::to_value(saved.restore_guard).unwrap()["owners"]
|
||||
.as_object()
|
||||
.unwrap()
|
||||
.len()
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial(worker_allocation)]
|
||||
fn bridge_partial_snapshot_registration_retains_original_streams_and_autonomously_recovers() {
|
||||
partial_registration_recovers_same_controller(false);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial(worker_allocation)]
|
||||
fn hub_partial_insert_registration_retains_original_controller_and_autonomously_recovers() {
|
||||
partial_registration_recovers_same_controller(true);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial(worker_allocation)]
|
||||
fn context_partial_flush_retains_only_undelivered_queue_and_recovers_without_repost() {
|
||||
let mut fixture = Fixture::no_model();
|
||||
Arc::get_mut(&mut fixture.backend)
|
||||
.unwrap()
|
||||
.spawn_restore_timeout = Duration::from_millis(250);
|
||||
let runtime = stopped_runtime(&fixture);
|
||||
let pause = runtime.test_pause_restore_continuation();
|
||||
let worker_ref = fixture.request.worker_ref.clone();
|
||||
let scope = fixture.request.workspace_scope.as_ref().unwrap();
|
||||
let request = runtime.test_restore_request(&worker_ref);
|
||||
let observations = runtime.subscribe_worker_observation().unwrap();
|
||||
let (started_release, started_entered) = activation_gate(&fixture, true);
|
||||
assert_eq!(
|
||||
runtime
|
||||
.restore_worker_operation_scoped(scope, &worker_ref, request.clone())
|
||||
.unwrap()
|
||||
.state,
|
||||
server_api::WorkerRestoreState::ReconciliationRequired
|
||||
);
|
||||
let started = started_entered.recv_timeout(WAIT).unwrap();
|
||||
let pending = assert_activation_owned(&fixture, &runtime, &request.request_id, &started);
|
||||
let old = started.handle.unwrap();
|
||||
let (operation, context, _) = runtime.test_pending_restore_context(&worker_ref);
|
||||
// Stop after the actual bridge and hub registration, without failing that
|
||||
// boundary, so the flush begins with its real original full snapshot.
|
||||
let (hub_release, hub_entered, hub_gate) = publication_gate();
|
||||
*fixture
|
||||
.backend
|
||||
.factory
|
||||
.observation_hub
|
||||
.registration_gate
|
||||
.lock()
|
||||
.unwrap() = Some(RestoreHubRegistrationGate {
|
||||
gate: hub_gate,
|
||||
fail: false,
|
||||
});
|
||||
started_release.release();
|
||||
drop(pause);
|
||||
assert!(
|
||||
hub_entered
|
||||
.recv_timeout(WAIT)
|
||||
.unwrap()
|
||||
.same_controller(&old)
|
||||
);
|
||||
let buffered_prefix = context.test_buffered_events();
|
||||
assert!(matches!(
|
||||
buffered_prefix.first(),
|
||||
Some(Event::Snapshot { .. })
|
||||
));
|
||||
let prefix_length = buffered_prefix.len();
|
||||
let marker = |text: &str| Event::Error {
|
||||
code: protocol::ErrorCode::Internal,
|
||||
message: text.into(),
|
||||
};
|
||||
context
|
||||
.publish_protocol_event(marker("flush-delivered"))
|
||||
.unwrap();
|
||||
context
|
||||
.publish_protocol_event(marker("flush-failed"))
|
||||
.unwrap();
|
||||
context
|
||||
.publish_protocol_event(marker("flush-pending"))
|
||||
.unwrap();
|
||||
let (release, entered, gate) = publication_gate();
|
||||
let (restored, restored_queue) = mpsc::channel();
|
||||
context.test_fail_flush(crate::execution::RestoreFlushFault {
|
||||
remaining_successes: prefix_length + 1,
|
||||
gate,
|
||||
restored,
|
||||
});
|
||||
drop(hub_release);
|
||||
assert!(
|
||||
matches!(entered.recv_timeout(WAIT).unwrap(), Event::Error { message, .. } if message == "flush-failed")
|
||||
);
|
||||
let partial_execution = fixture.backend.workers.lock().unwrap()[&worker_ref].clone();
|
||||
assert!(old.same_controller(&partial_execution.handle));
|
||||
assert_eq!(partial_execution.restore_operation_id, Some(operation));
|
||||
assert_eq!(partial_execution.tasks.tasks.lock().unwrap().len(), 2);
|
||||
let (_, shared, _) = fixture
|
||||
.backend
|
||||
.factory
|
||||
.observation_hub
|
||||
.get(&worker_ref)
|
||||
.unwrap();
|
||||
assert!(Arc::ptr_eq(&shared, &old.shared_state));
|
||||
let pause = runtime.test_pause_restore_continuation();
|
||||
context
|
||||
.publish_protocol_event(marker("flush-arrived-during-error"))
|
||||
.unwrap();
|
||||
release.send(()).unwrap();
|
||||
let restored = restored_queue.recv_timeout(WAIT).unwrap();
|
||||
let undelivered: Vec<_> = restored
|
||||
.iter()
|
||||
.filter_map(|event| match event {
|
||||
Event::Error { message, .. } if message.starts_with("flush-") => Some(message.as_str()),
|
||||
_ => None,
|
||||
})
|
||||
.collect();
|
||||
assert_eq!(
|
||||
undelivered,
|
||||
[
|
||||
"flush-failed",
|
||||
"flush-pending",
|
||||
"flush-arrived-during-error"
|
||||
]
|
||||
);
|
||||
assert!(
|
||||
!restored
|
||||
.iter()
|
||||
.any(|event| matches!(event, Event::Snapshot { .. })),
|
||||
"delivered snapshot must not rejoin the failed suffix"
|
||||
);
|
||||
assert_eq!(
|
||||
serde_json::to_value(context.test_buffered_events()).unwrap(),
|
||||
serde_json::to_value(&restored).unwrap()
|
||||
);
|
||||
assert_eq!(
|
||||
durable_operation(&runtime, &worker_ref)
|
||||
.unwrap()
|
||||
.operation_id,
|
||||
operation
|
||||
);
|
||||
assert!(restore_owner(&runtime, &worker_ref, &request.request_id)["receipt"].is_null());
|
||||
// The next terminal gate observes successful retry only after the same
|
||||
// queued owner has repaired its failed callback and remaining suffix.
|
||||
let (terminal_release, terminal_entered, gate) = publication_gate();
|
||||
runtime.test_terminal_receipt_gate(gate);
|
||||
drop(pause);
|
||||
assert_eq!(terminal_entered.recv_timeout(WAIT).unwrap(), operation);
|
||||
assert!(context.test_buffered_events().is_empty());
|
||||
assert!(restore_owner(&runtime, &worker_ref, &request.request_id)["receipt"].is_null());
|
||||
assert_eq!(
|
||||
durable_operation(&runtime, &worker_ref).unwrap().commit,
|
||||
crate::runtime::RestoreCommitFact::Committed
|
||||
);
|
||||
let execution = fixture.backend.workers.lock().unwrap()[&worker_ref].clone();
|
||||
assert!(old.same_controller(&execution.handle));
|
||||
assert_eq!(execution.restore_operation_id, Some(operation));
|
||||
drop(terminal_release); // RAII disconnect, not a second request.
|
||||
runtime.test_wait_restore_continuation(&worker_ref, operation, WAIT);
|
||||
assert_eq!(
|
||||
restore_owner(&runtime, &worker_ref, &request.request_id)["receipt"]["state"],
|
||||
"accepted"
|
||||
);
|
||||
assert_real_attach(&fixture, &old);
|
||||
fixture
|
||||
.backend
|
||||
.run_on_adapter_runtime(async move {
|
||||
let mut observations = observations;
|
||||
let mut messages = Vec::new();
|
||||
let mut snapshots = 0;
|
||||
while let Ok(event) = observations.try_recv() {
|
||||
match event.payload {
|
||||
Event::Error { message, .. } if message.starts_with("flush-") => {
|
||||
messages.push(message)
|
||||
}
|
||||
Event::Snapshot { .. } => snapshots += 1,
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
assert_eq!(
|
||||
messages,
|
||||
[
|
||||
"flush-delivered",
|
||||
"flush-failed",
|
||||
"flush-pending",
|
||||
"flush-arrived-during-error"
|
||||
]
|
||||
);
|
||||
assert_eq!(snapshots, 1);
|
||||
Ok(())
|
||||
})
|
||||
.unwrap();
|
||||
assert_single_publication(&fixture, 1);
|
||||
drop(partial_execution);
|
||||
drop(execution);
|
||||
drop(pending);
|
||||
runtime
|
||||
.stop_worker_scoped(scope, &worker_ref, None)
|
||||
.unwrap();
|
||||
fixture.assert_released();
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug)]
|
||||
enum TerminalFault {
|
||||
BeforeRename,
|
||||
AfterRename,
|
||||
SyncUnknown,
|
||||
ReadUnknown,
|
||||
}
|
||||
|
||||
fn published_terminal_receipt_fault(fault: TerminalFault) {
|
||||
let mut fixture = Fixture::no_model();
|
||||
// Reuse the existing ordinary non-network model boundary, not the product
|
||||
// Codex profile. All fault/convergence assertions precede the sole input.
|
||||
let script = MockClient::sequential(vec![MockResponse::Complete(simple_text_events())]);
|
||||
let injected = script.clone();
|
||||
let factory = fixture
|
||||
.backend
|
||||
.factory
|
||||
.as_ref()
|
||||
.clone()
|
||||
.with_spawn_model_client_factory(move || Box::new(injected.clone()));
|
||||
fixture.backend = Arc::new(WorkerRuntimeExecutionBackend::new(factory).unwrap());
|
||||
let runtime = Arc::new(stopped_runtime(&fixture));
|
||||
let pause = runtime.test_pause_restore_continuation();
|
||||
let worker_ref = fixture.request.worker_ref.clone();
|
||||
let scope = fixture.request.workspace_scope.clone().unwrap();
|
||||
let request = runtime.test_restore_request(&worker_ref);
|
||||
let observations = runtime.subscribe_worker_observation().unwrap();
|
||||
let mut catalog = runtime
|
||||
.subscribe_event_selector_scoped(
|
||||
&scope,
|
||||
protocol::subscription::EventSubscriptionSelector::RuntimeWorkers,
|
||||
)
|
||||
.unwrap();
|
||||
let (release, entered, gate) = publication_gate();
|
||||
runtime.test_terminal_receipt_gate(gate);
|
||||
let restoring = {
|
||||
let runtime = runtime.clone();
|
||||
let worker_ref = worker_ref.clone();
|
||||
let scope = scope.clone();
|
||||
let request = request.clone();
|
||||
std::thread::spawn(move || {
|
||||
runtime.restore_worker_operation_scoped(&scope, &worker_ref, request)
|
||||
})
|
||||
};
|
||||
let operation = entered.recv_timeout(WAIT).unwrap();
|
||||
// Restore legitimately repairs derived metadata before publication. Receipt
|
||||
// failures must not rerun those effects; compare from this actual boundary.
|
||||
let canonical = fixture.canonical_files();
|
||||
let execution = fixture.backend.workers.lock().unwrap()[&worker_ref].clone();
|
||||
let old = execution.handle.clone();
|
||||
assert_eq!(execution.restore_operation_id, Some(operation));
|
||||
assert!(
|
||||
fixture
|
||||
.backend
|
||||
.pending_restore(&worker_ref)
|
||||
.unwrap()
|
||||
.is_none()
|
||||
);
|
||||
assert_eq!(execution.tasks.tasks.lock().unwrap().len(), 2);
|
||||
assert!(execution.tasks.protocol_bridge.lock().unwrap().is_none());
|
||||
assert_eq!(
|
||||
durable_operation(&runtime, &worker_ref)
|
||||
.unwrap()
|
||||
.operation_id,
|
||||
operation
|
||||
);
|
||||
assert_eq!(
|
||||
durable_operation(&runtime, &worker_ref).unwrap().commit,
|
||||
crate::runtime::RestoreCommitFact::Committed
|
||||
);
|
||||
let owner = restore_owner(&runtime, &worker_ref, &request.request_id);
|
||||
assert!(owner["receipt"].is_null());
|
||||
let (same_operation, context, uncertain) = runtime.test_pending_restore_context(&worker_ref);
|
||||
assert_eq!(same_operation, operation);
|
||||
assert!(!uncertain);
|
||||
assert!(context.test_buffered_events().is_empty());
|
||||
let transport = assert_real_attach(&fixture, &old);
|
||||
assert_single_publication(&fixture, 1);
|
||||
alert_marker(&old, "terminal-before-receipt");
|
||||
let store = runtime.test_runtime_store();
|
||||
let reconcile_before = store.test_worker_reconcile_counts();
|
||||
store.fail_next_worker_write(if matches!(fault, TerminalFault::BeforeRename) {
|
||||
AtomicWriteFault::BeforeRename
|
||||
} else {
|
||||
AtomicWriteFault::AfterRename
|
||||
});
|
||||
match fault {
|
||||
TerminalFault::SyncUnknown => store.fail_next_worker_reconcile_sync(),
|
||||
TerminalFault::ReadUnknown => store.fail_next_worker_reconcile_read(),
|
||||
_ => {}
|
||||
}
|
||||
release.send(()).unwrap();
|
||||
let first = restoring.join().unwrap().unwrap();
|
||||
let reconcile_after = store.test_worker_reconcile_counts();
|
||||
let reconciliation = (
|
||||
reconcile_after.0 - reconcile_before.0,
|
||||
reconcile_after.1 - reconcile_before.1,
|
||||
);
|
||||
assert_eq!(
|
||||
reconciliation,
|
||||
match fault {
|
||||
TerminalFault::BeforeRename => (0, 0),
|
||||
TerminalFault::AfterRename => (1, 4), // canonical file plus Worker/workers/root directories
|
||||
TerminalFault::SyncUnknown => (1, 1),
|
||||
TerminalFault::ReadUnknown => (1, 0),
|
||||
},
|
||||
"{fault:?}: actual canonical read/sync attempts"
|
||||
);
|
||||
assert_eq!(
|
||||
first.state,
|
||||
if matches!(fault, TerminalFault::AfterRename) {
|
||||
server_api::WorkerRestoreState::Accepted
|
||||
} else {
|
||||
server_api::WorkerRestoreState::ReconciliationRequired
|
||||
},
|
||||
"{fault:?}: {first:?}"
|
||||
);
|
||||
assert_eq!(fixture.canonical_files(), canonical);
|
||||
assert!(!old.protocol_is_closed());
|
||||
transport.validate().unwrap();
|
||||
assert_single_publication(&fixture, 1);
|
||||
let disk_owner = restore_owner(&runtime, &worker_ref, &request.request_id);
|
||||
assert_eq!(disk_owner["operation_id"], owner["operation_id"]);
|
||||
match fault {
|
||||
TerminalFault::BeforeRename => {
|
||||
assert!(disk_owner["receipt"].is_null());
|
||||
assert!(!runtime.test_pending_restore_context(&worker_ref).2);
|
||||
}
|
||||
TerminalFault::AfterRename => {
|
||||
assert_eq!(disk_owner["receipt"]["state"], "accepted");
|
||||
assert!(durable_operation(&runtime, &worker_ref).is_none());
|
||||
}
|
||||
TerminalFault::SyncUnknown | TerminalFault::ReadUnknown => {
|
||||
assert_eq!(
|
||||
disk_owner["receipt"]["state"], "accepted",
|
||||
"visible rename alone is not durable proof"
|
||||
);
|
||||
let (same_operation, _, fenced) = runtime.test_pending_restore_context(&worker_ref);
|
||||
assert_eq!(same_operation, operation);
|
||||
assert!(
|
||||
fenced,
|
||||
"unknown canonical outcome must retain the memory owner/write fence"
|
||||
);
|
||||
// Keep durability unproved for one competing Stop as well: it must
|
||||
// fail before any Shutdown, never manufacture a Stopped judgment.
|
||||
match fault {
|
||||
TerminalFault::SyncUnknown => store.fail_next_worker_reconcile_sync(),
|
||||
TerminalFault::ReadUnknown => store.fail_next_worker_reconcile_read(),
|
||||
_ => unreachable!(),
|
||||
}
|
||||
assert!(matches!(
|
||||
runtime.stop_worker_scoped(&scope, &worker_ref, None),
|
||||
Err(crate::error::RuntimeError::StoreCommitOutcomeUnknown { .. })
|
||||
));
|
||||
assert!(!old.protocol_is_closed());
|
||||
assert!(!execution.shutdown_requested.load(Ordering::Acquire));
|
||||
assert!(runtime.test_pending_restore_context(&worker_ref).2);
|
||||
assert_single_publication(&fixture, 1);
|
||||
}
|
||||
}
|
||||
// Recovery is autonomous after the failed write / confirmed unknown fence.
|
||||
// No POST, fresh request, or input is needed to make it converge.
|
||||
drop(pause);
|
||||
if !matches!(fault, TerminalFault::AfterRename) {
|
||||
runtime.test_wait_restore_continuation(&worker_ref, operation, WAIT);
|
||||
}
|
||||
assert!(durable_operation(&runtime, &worker_ref).is_none());
|
||||
assert_eq!(
|
||||
restore_owner(&runtime, &worker_ref, &request.request_id)["receipt"]["state"],
|
||||
"accepted"
|
||||
);
|
||||
let current = fixture.backend.workers.lock().unwrap()[&worker_ref].clone();
|
||||
assert!(old.same_controller(¤t.handle));
|
||||
assert_eq!(current.restore_operation_id, Some(operation));
|
||||
assert_eq!(fixture.backend.workers.lock().unwrap().len(), 1);
|
||||
assert_real_attach(&fixture, &old);
|
||||
assert_single_publication(&fixture, 1);
|
||||
// A canonical terminal reread may bypass the final redundant upsert. The
|
||||
// original subscriber must still have the current state and observation token.
|
||||
let fresh_catalog = runtime
|
||||
.subscribe_event_selector_scoped(
|
||||
&scope,
|
||||
protocol::subscription::EventSubscriptionSelector::RuntimeWorkers,
|
||||
)
|
||||
.unwrap();
|
||||
let protocol::subscription::SubscriptionSnapshot::Workers { workers } =
|
||||
fresh_catalog.snapshot()
|
||||
else {
|
||||
panic!("Runtime catalog must return Workers");
|
||||
};
|
||||
assert_eq!(workers.len(), 1);
|
||||
let mut last_applied = None;
|
||||
use futures::FutureExt;
|
||||
while let Some(update) = catalog.recv().now_or_never() {
|
||||
match update.unwrap().payload {
|
||||
protocol::subscription::SubscriptionEventPayload::WorkerUpserted { worker } => {
|
||||
last_applied = Some(worker);
|
||||
}
|
||||
payload => panic!("unexpected catalog update: {payload:?}"),
|
||||
}
|
||||
}
|
||||
assert!(
|
||||
serde_json::to_value(last_applied.expect("restore must publish its catalog state"))
|
||||
.unwrap()
|
||||
== serde_json::to_value(&workers[0]).unwrap(),
|
||||
"original catalog subscriber diverged after {fault:?}"
|
||||
);
|
||||
drop(fresh_catalog);
|
||||
drop(catalog);
|
||||
alert_marker(&old, "terminal-after-recovery");
|
||||
let _observations = observe_publication_markers(
|
||||
&fixture,
|
||||
observations,
|
||||
&["terminal-before-receipt", "terminal-after-recovery"],
|
||||
);
|
||||
let accepted = runtime
|
||||
.restore_worker_operation_scoped(&scope, &worker_ref, request.clone())
|
||||
.unwrap();
|
||||
assert_eq!(accepted.state, server_api::WorkerRestoreState::Accepted);
|
||||
assert_eq!(
|
||||
runtime
|
||||
.restore_worker_operation_scoped(&scope, &worker_ref, request.clone())
|
||||
.unwrap(),
|
||||
accepted
|
||||
);
|
||||
assert_eq!(restore_owner_count(&runtime, &worker_ref), 1);
|
||||
let mut stale = request;
|
||||
stale.request_id = "terminal-fresh-stale-intent".into();
|
||||
assert!(matches!(
|
||||
runtime.restore_worker_operation_scoped(&scope, &worker_ref, stale),
|
||||
Err(crate::error::RuntimeError::RestoreObservationConflict { .. })
|
||||
));
|
||||
assert_single_publication(&fixture, 1);
|
||||
assert_eq!(
|
||||
script.call_count.load(Ordering::SeqCst),
|
||||
0,
|
||||
"receipt recovery cannot trigger input or a model call"
|
||||
);
|
||||
// This transport was paired before the failed receipt write. Exercise its
|
||||
// normal log-entry receiver after autonomous settlement, not a new attach.
|
||||
transport
|
||||
.dispatch(Method::submit_text(
|
||||
"terminal-ordinary-input",
|
||||
"terminal-ordinary-input",
|
||||
))
|
||||
.unwrap();
|
||||
fixture
|
||||
.backend
|
||||
.run_on_adapter_runtime(async move {
|
||||
let mut transport = transport;
|
||||
let (mut admitted, mut committed, mut text, mut ended) = (false, false, false, false);
|
||||
tokio::time::timeout(WAIT, async {
|
||||
while !(admitted && committed && text && ended) {
|
||||
match transport.events.recv().await.unwrap() {
|
||||
Event::SubmissionAccepted {
|
||||
submission_request_id,
|
||||
..
|
||||
} if submission_request_id == "terminal-ordinary-input" => admitted = true,
|
||||
Event::UserMessage { segments, .. } => {
|
||||
assert_eq!(
|
||||
segments,
|
||||
vec![protocol::Segment::text("terminal-ordinary-input")]
|
||||
);
|
||||
assert!(!committed, "paired log receiver duplicated input");
|
||||
committed = true;
|
||||
}
|
||||
Event::TextDone { text: output } => {
|
||||
assert_eq!(output, "hello from worker");
|
||||
text = true;
|
||||
}
|
||||
Event::RunEnd { .. } => ended = true,
|
||||
Event::SubmissionRejected { .. } | Event::Error { .. } => {
|
||||
panic!("ordinary input failed after receipt recovery")
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
})
|
||||
.await
|
||||
.map_err(|_| {
|
||||
"ordinary input failed to settle on original paired transport".to_string()
|
||||
})?;
|
||||
drop(transport);
|
||||
Ok(())
|
||||
})
|
||||
.unwrap();
|
||||
assert_eq!(script.call_count.load(Ordering::SeqCst), 1);
|
||||
assert_eq!(fixture.entries().iter().filter(|entry| matches!(entry,
|
||||
LogEntry::AnnotatedUserInput { segments, .. } if segments == &vec![protocol::Segment::text("terminal-ordinary-input")]
|
||||
)).count(), 1);
|
||||
assert_single_publication(&fixture, 1);
|
||||
drop(execution);
|
||||
drop(current);
|
||||
runtime
|
||||
.stop_worker_scoped(&scope, &worker_ref, None)
|
||||
.unwrap();
|
||||
assert!(old.protocol_is_closed());
|
||||
fixture.assert_released();
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial(worker_allocation)]
|
||||
fn published_controller_terminal_receipt_before_rename_autonomously_settles_without_reconstruction()
|
||||
{
|
||||
published_terminal_receipt_fault(TerminalFault::BeforeRename);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial(worker_allocation)]
|
||||
fn published_controller_terminal_receipt_after_rename_is_canonically_reread_and_synced() {
|
||||
published_terminal_receipt_fault(TerminalFault::AfterRename);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial(worker_allocation)]
|
||||
fn published_controller_terminal_receipt_sync_unknown_retains_owner_and_write_fence() {
|
||||
published_terminal_receipt_fault(TerminalFault::SyncUnknown);
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial_test::serial(worker_allocation)]
|
||||
fn published_controller_terminal_receipt_read_unknown_retains_owner_and_write_fence() {
|
||||
published_terminal_receipt_fault(TerminalFault::ReadUnknown);
|
||||
}
|
||||
@@ -1,23 +1,32 @@
|
||||
use std::sync::Arc;
|
||||
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
||||
#[cfg(feature = "http-server")]
|
||||
use std::time::Duration;
|
||||
use std::time::{SystemTime, UNIX_EPOCH};
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
use crate::workspace_request::{
|
||||
RuntimeWorkspaceRequest, RuntimeWorkspaceRequestClient, RuntimeWorkspaceRequestError,
|
||||
};
|
||||
use worker::WorkspaceResponse;
|
||||
#[cfg(feature = "http-server")]
|
||||
use worker::{
|
||||
WorkspaceBinaryRequest, WorkspaceBinaryResponse, WorkspaceClient, WorkspaceClientError,
|
||||
WorkspacePromptCatalogResolution, WorkspacePromptProjection, WorkspaceRequest,
|
||||
WorkspaceRequestMethod, WorkspaceResponse, WorkspaceServerOperation,
|
||||
WorkspaceRequestMethod, WorkspaceServerOperation,
|
||||
};
|
||||
|
||||
use crate::auth::{
|
||||
RuntimeAuthError, RuntimeIdentityMaterial, RuntimeWorkerMutationSourceSigner,
|
||||
WORKER_REMOVE_PERMISSION, WORKSPACE_REQUEST_PERMISSION, WORKSPACE_WORKER_DISCOVERY_PERMISSION,
|
||||
WorkerMutationActorKind, WorkerMutationOperation, WorkerMutationSourceClaims, new_token_id,
|
||||
WORKER_REMOVE_PERMISSION, WorkerMutationActorKind, WorkerMutationOperation,
|
||||
WorkerMutationSourceClaims, new_token_id,
|
||||
};
|
||||
#[cfg(feature = "http-server")]
|
||||
use crate::auth::{WORKSPACE_REQUEST_PERMISSION, WORKSPACE_WORKER_DISCOVERY_PERMISSION};
|
||||
use crate::runtime::RuntimeWorkspaceScope;
|
||||
#[cfg(feature = "http-server")]
|
||||
use crate::worker_backend::WorkspacePromptProjectionCache;
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
fn ticket_assignment_role_segment(role: server_api::TicketAssignmentRole) -> &'static str {
|
||||
match role {
|
||||
server_api::TicketAssignmentRole::Orchestrator => "orchestrator",
|
||||
@@ -144,6 +153,7 @@ pub trait EmbeddedWorkerMutationDispatcher: Send + Sync {
|
||||
|
||||
#[derive(Clone)]
|
||||
enum RuntimeWorkerMutationTransport {
|
||||
#[cfg(feature = "http-server")]
|
||||
Remote {
|
||||
request_client: RuntimeWorkspaceRequestClient,
|
||||
},
|
||||
@@ -161,6 +171,7 @@ pub struct RuntimeWorkerMutationForwarder {
|
||||
}
|
||||
|
||||
impl RuntimeWorkerMutationForwarder {
|
||||
#[cfg(feature = "http-server")]
|
||||
pub fn remote(
|
||||
identity: &RuntimeIdentityMaterial,
|
||||
scope: RuntimeWorkspaceScope,
|
||||
@@ -206,6 +217,7 @@ impl RuntimeWorkerMutationForwarder {
|
||||
target_worker_id,
|
||||
)?;
|
||||
match (&self.transport, proof) {
|
||||
#[cfg(feature = "http-server")]
|
||||
(
|
||||
RuntimeWorkerMutationTransport::Remote { request_client },
|
||||
RuntimeOwnedWorkerMutationProof::Remote(token),
|
||||
@@ -231,6 +243,7 @@ impl RuntimeWorkerMutationForwarder {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
struct RemoteWorkerRemoveHttpRequest {
|
||||
request_client: RuntimeWorkspaceRequestClient,
|
||||
source_worker_id: String,
|
||||
@@ -240,6 +253,7 @@ struct RemoteWorkerRemoveHttpRequest {
|
||||
reason: String,
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
fn execute_remote_worker_remove_http(
|
||||
request: RemoteWorkerRemoveHttpRequest,
|
||||
) -> Result<WorkspaceResponse, RuntimeWorkerMutationForwardError> {
|
||||
@@ -263,6 +277,7 @@ fn execute_remote_worker_remove_http(
|
||||
execute_remote_worker_remove_http_blocking(request)
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
fn execute_remote_worker_remove_http_blocking(
|
||||
request: RemoteWorkerRemoveHttpRequest,
|
||||
) -> Result<WorkspaceResponse, RuntimeWorkerMutationForwardError> {
|
||||
@@ -315,6 +330,7 @@ fn execute_remote_worker_remove_http_blocking(
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
#[derive(Clone)]
|
||||
pub struct RuntimeOwnedWorkspaceClient {
|
||||
workspace_id: String,
|
||||
@@ -325,6 +341,7 @@ pub struct RuntimeOwnedWorkspaceClient {
|
||||
prompt_projection_cache: Option<Arc<WorkspacePromptProjectionCache>>,
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
impl RuntimeOwnedWorkspaceClient {
|
||||
pub fn new(
|
||||
workspace_id: impl Into<String>,
|
||||
@@ -495,6 +512,7 @@ impl RuntimeOwnedWorkspaceClient {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
impl std::fmt::Debug for RuntimeOwnedWorkspaceClient {
|
||||
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
formatter
|
||||
@@ -510,6 +528,7 @@ impl std::fmt::Debug for RuntimeOwnedWorkspaceClient {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
fn generated_workspace_response<T: serde::Serialize>(
|
||||
response: Result<T, server_api::client_support::ClientError<server_api::RepositoryApiError>>,
|
||||
) -> Result<WorkspaceResponse, WorkspaceClientError> {
|
||||
@@ -530,6 +549,7 @@ fn generated_workspace_response<T: serde::Serialize>(
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
impl WorkspaceClient for RuntimeOwnedWorkspaceClient {
|
||||
fn workspace_id(&self) -> Option<&str> {
|
||||
Some(self.request_client.workspace_id())
|
||||
@@ -929,12 +949,17 @@ fn unix_now_seconds() -> u64 {
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
#[cfg(feature = "http-server")]
|
||||
use crate::auth::{
|
||||
RUNTIME_REQUEST_SOURCE_PROOF_HEADER, WorkerMutationSourceExpectation,
|
||||
decode_runtime_request_source_claims, decode_worker_mutation_source_claims,
|
||||
request_body_digest, verify_worker_mutation_source_proof,
|
||||
RUNTIME_REQUEST_SOURCE_PROOF_HEADER, decode_runtime_request_source_claims,
|
||||
request_body_digest,
|
||||
};
|
||||
use crate::auth::{
|
||||
WorkerMutationSourceExpectation, decode_worker_mutation_source_claims,
|
||||
verify_worker_mutation_source_proof,
|
||||
};
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
fn binary_loopback(response: Vec<u8>) -> (String, std::thread::JoinHandle<(String, Vec<u8>)>) {
|
||||
use std::io::{Read, Write};
|
||||
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
|
||||
@@ -972,6 +997,7 @@ mod tests {
|
||||
(base_url, server)
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
#[test]
|
||||
fn generated_ticket_assignment_transport_signs_exact_authoring_and_bounded_read_subjects() {
|
||||
use crate::auth::{RuntimeRequestSourceExpectation, verify_runtime_request_source};
|
||||
@@ -1104,6 +1130,7 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
#[test]
|
||||
fn binary_roundtrip_signs_exact_query_method_worker_and_raw_body() {
|
||||
use crate::auth::{RuntimeRequestSourceExpectation, verify_runtime_request_source};
|
||||
@@ -1194,6 +1221,7 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
#[test]
|
||||
fn binary_responses_enforce_declared_and_streamed_limits_with_fixed_errors() {
|
||||
for response in [
|
||||
@@ -1213,6 +1241,7 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
#[test]
|
||||
fn binary_redirect_returns_status_without_forwarding_authentication() {
|
||||
let (base_url, server) = binary_loopback(b"HTTP/1.1 307 Temporary Redirect\r\nLocation: http://127.0.0.1:1/private\r\nContent-Length: 0\r\nConnection: close\r\n\r\n".to_vec());
|
||||
@@ -1231,6 +1260,7 @@ mod tests {
|
||||
assert!(result.body.is_empty());
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
#[test]
|
||||
fn binary_invalid_or_normalized_targets_fail_with_fixed_errors_before_http() {
|
||||
let client = RuntimeOwnedWorkspaceClient::new(
|
||||
@@ -1277,6 +1307,7 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
#[test]
|
||||
fn current_prompt_projection_uses_the_shared_runtime_cache_without_http() {
|
||||
let cache = Arc::new(WorkspacePromptProjectionCache::default());
|
||||
@@ -1312,6 +1343,7 @@ mod tests {
|
||||
assert!(Arc::ptr_eq(&projection.catalog, &second.catalog));
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
#[test]
|
||||
fn prompt_projection_cache_miss_fetches_content_identity() {
|
||||
use std::io::{Read, Write};
|
||||
@@ -1356,6 +1388,7 @@ mod tests {
|
||||
assert_eq!(fetched.projection.source_digest, "source-3");
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
#[test]
|
||||
fn concurrent_prompt_projection_miss_fetches_once_and_shares_catalog() {
|
||||
use std::io::{Read, Write};
|
||||
@@ -1427,6 +1460,7 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
#[test]
|
||||
fn current_prompt_projection_rejects_cross_workspace_response() {
|
||||
use std::io::{Read, Write};
|
||||
@@ -1473,6 +1507,7 @@ mod tests {
|
||||
assert!(error.to_string().contains("scope mismatch"));
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
#[test]
|
||||
fn ordinary_workspace_forwarding_stamps_runtime_identity_and_signs_path_and_query() {
|
||||
use std::io::{Read, Write};
|
||||
@@ -1531,6 +1566,7 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
#[test]
|
||||
fn workspace_worker_discovery_signs_dedicated_permission_and_encoded_query() {
|
||||
use std::io::{Read, Write};
|
||||
@@ -1652,6 +1688,7 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
async fn remote_forwarder_is_safe_in_async_runtime_and_stamps_signed_proof() {
|
||||
use std::io::{Read, Write};
|
||||
@@ -1815,6 +1852,7 @@ mod tests {
|
||||
assert_eq!(reason, "retire obsolete Worker");
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
#[test]
|
||||
fn runtime_owned_workspace_client_returns_successful_response_without_request_timeout() {
|
||||
// Effective deadline clearing is exercised with a short inherited deadline in
|
||||
@@ -1838,6 +1876,7 @@ mod tests {
|
||||
server.join().unwrap();
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
#[test]
|
||||
fn runtime_owned_workspace_client_honors_per_request_timeouts() {
|
||||
let (base_url, server) = delayed_workspace_response(Duration::from_millis(75));
|
||||
@@ -1857,6 +1896,7 @@ mod tests {
|
||||
server.join().unwrap();
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
#[test]
|
||||
fn runtime_owned_workspace_client_reports_request_timeouts() {
|
||||
let (base_url, server) = delayed_workspace_response(Duration::from_millis(75));
|
||||
@@ -1878,6 +1918,7 @@ mod tests {
|
||||
server.join().unwrap();
|
||||
}
|
||||
|
||||
#[cfg(feature = "http-server")]
|
||||
fn delayed_workspace_response(delay: Duration) -> (String, std::thread::JoinHandle<()>) {
|
||||
use std::io::{Read, Write};
|
||||
use std::net::TcpListener;
|
||||
|
||||
@@ -3676,6 +3676,7 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn bound_ssh_workdir_uses_attachment_scoped_agent_and_cleans_it_up() {
|
||||
crate::restore_test_environment::require_restore_test_environment();
|
||||
let repo = create_clean_repo();
|
||||
// ssh-agent and the broker receive relative socket paths in this owned
|
||||
// fixture tree, so the test does not depend on checkout/TMPDIR length.
|
||||
|
||||
Reference in New Issue
Block a user