Unify formatting on canonical syntax

doc/manual/souce/components.md

@@ -58,10 +58,11 @@ The default `decodal-lsp` binary reads Decodal imports from the filesystem.
 Embedded hosts can call its library entry point with a custom `LspEnvironment` to reuse structured imports and to make unsaved external documents, such as Markdown, visible to the loader.
 
 Source formatting lives in a separate Rust language tools crate.
-Keeping it separate prevents host-specific semantic services from inheriting the formatter's Tree-sitter and WebAssembly dependencies.
+It uses the canonical Decodal AST together with the runtime lexer's lossless syntax tokens, so native LSP and WebAssembly callers execute the same formatter implementation.
 
 This component is responsible for operations that must preserve source text details such as comments and whitespace.
 It is used by the CodeMirror package's bundled formatter WebAssembly and can also be used by an LSP adapter for formatting.
+It does not depend on Tree-sitter or Lezer.
 
 Important paths:
 
@@ -74,7 +75,7 @@ The current language tools crate exposes the formatter.
 ## Web editor components
 
 The Web playground editor uses CodeMirror 6 with a generated Lezer parser.
-This is the source of syntax highlighting, folding, indentation, editor syntax tree behavior, and the browser formatter command in the browser UI.
+Lezer provides syntax highlighting, folding, indentation, and editor syntax tree behavior. The browser formatter command calls the canonical Rust formatter compiled to WebAssembly.
 
 Important paths:
 
@@ -120,18 +121,20 @@ doc/manual/souce/language/grammar.md
 This EBNF is the language-level reference.
 The Rust parser, Lezer grammar, and Tree-sitter grammar should be kept aligned with it, but each implementation may encode precedence and recovery behavior in the form required by its parser generator or runtime.
 
-## What is not a public component
+## Syntax token API
 
-The Rust lexer is an implementation detail of the Rust parser.
-Decodal does not expose a standalone public tokenizer API for editor tooling.
-Consumers that need syntax information should use the component that matches their environment:
+The `decodal` crate exposes `tokenize_source`, `tokenize_source_with_source_id`, `SyntaxToken`, and `SyntaxTokenKind` for source-preserving tooling.
+Comments have explicit tokens, while whitespace is represented by gaps between token spans and can be recovered from the original source.
+The evaluator parser and formatter therefore share one lexical definition without making Tree-sitter an upstream dependency.
+
+Consumers should otherwise use the component matching their environment:
 
 - Rust execution and embedding: `decodal`
 - Browser execution: `decodal-wasm`
-- Web formatting and editor syntax: Lezer / CodeMirror
+- Web formatting and editor syntax: canonical formatter / CodeMirror / Lezer
 - Semantic editor analysis: `decodal-language-service`
 - Language Server Protocol integration: `decodal-lsp`
 - Rust formatting: `decodal-language-tools`
 - General editor syntax: Tree-sitter
 
-This avoids having a separate token stream API whose behavior would have to be kept compatible with both runtime parsing and editor grammars.
+Tree-sitter and Lezer remain downstream editor grammars and are not dependencies of the runtime formatter.