Unify formatting on canonical syntax

doc/manual/souce/components.md

@@ -58,10 +58,11 @@ The default `decodal-lsp` binary reads Decodal imports from the filesystem.
 Embedded hosts can call its library entry point with a custom `LspEnvironment` to reuse structured imports and to make unsaved external documents, such as Markdown, visible to the loader.
 
 Source formatting lives in a separate Rust language tools crate.
-Keeping it separate prevents host-specific semantic services from inheriting the formatter's Tree-sitter and WebAssembly dependencies.
+It uses the canonical Decodal AST together with the runtime lexer's lossless syntax tokens, so native LSP and WebAssembly callers execute the same formatter implementation.
 
 This component is responsible for operations that must preserve source text details such as comments and whitespace.
 It is used by the CodeMirror package's bundled formatter WebAssembly and can also be used by an LSP adapter for formatting.
+It does not depend on Tree-sitter or Lezer.
 
 Important paths:
 
@@ -74,7 +75,7 @@ The current language tools crate exposes the formatter.
 ## Web editor components
 
 The Web playground editor uses CodeMirror 6 with a generated Lezer parser.
-This is the source of syntax highlighting, folding, indentation, editor syntax tree behavior, and the browser formatter command in the browser UI.
+Lezer provides syntax highlighting, folding, indentation, and editor syntax tree behavior. The browser formatter command calls the canonical Rust formatter compiled to WebAssembly.
 
 Important paths:
 
@@ -120,18 +121,20 @@ doc/manual/souce/language/grammar.md
 This EBNF is the language-level reference.
 The Rust parser, Lezer grammar, and Tree-sitter grammar should be kept aligned with it, but each implementation may encode precedence and recovery behavior in the form required by its parser generator or runtime.
 
-## What is not a public component
+## Syntax token API
 
-The Rust lexer is an implementation detail of the Rust parser.
-Decodal does not expose a standalone public tokenizer API for editor tooling.
-Consumers that need syntax information should use the component that matches their environment:
+The `decodal` crate exposes `tokenize_source`, `tokenize_source_with_source_id`, `SyntaxToken`, and `SyntaxTokenKind` for source-preserving tooling.
+Comments have explicit tokens, while whitespace is represented by gaps between token spans and can be recovered from the original source.
+The evaluator parser and formatter therefore share one lexical definition without making Tree-sitter an upstream dependency.
+
+Consumers should otherwise use the component matching their environment:
 
 - Rust execution and embedding: `decodal`
 - Browser execution: `decodal-wasm`
-- Web formatting and editor syntax: Lezer / CodeMirror
+- Web formatting and editor syntax: canonical formatter / CodeMirror / Lezer
 - Semantic editor analysis: `decodal-language-service`
 - Language Server Protocol integration: `decodal-lsp`
 - Rust formatting: `decodal-language-tools`
 - General editor syntax: Tree-sitter
 
-This avoids having a separate token stream API whose behavior would have to be kept compatible with both runtime parsing and editor grammars.
+Tree-sitter and Lezer remain downstream editor grammars and are not dependencies of the runtime formatter.

doc/manual/souce/development.md

@@ -34,33 +34,13 @@ cargo run -q -p decodal-cli --features regex -- examples/regex/main.dcdl
 
 ## crates.io release
 
-The primary crates.io package is `decodal`, which contains the embeddable library.
-`decodal-derive` provides optional derive macros for Rust struct integration and is published only when the derive crate changes.
-Workspace support crates such as `decodal-cli`, the Rust source crate `decodal-wasm`, `decodal-language-tools`, and `decodal-lsp` are not published to crates.io.
+The crates.io release contains `decodal`, `decodal-derive`, `decodal-language-service`, `decodal-language-tools`, and `decodal-lsp`.
+`decodal-cli` and the Rust source crate `decodal-wasm` remain repository-only packages.
 The generated WebAssembly package under `packages/decodal-wasm/` is published to npm and JSR.
 The CodeMirror package bundles the generated formatter WebAssembly from `decodal-language-tools`.
 The project is dual licensed as `MIT OR Apache-2.0`.
 
-```sh
-cargo publish -p decodal
-```
-
-Publish `decodal-derive` first only when that crate has a new version:
-
-```sh
-cargo publish -p decodal-derive
-```
-
-Before publishing, run:
-
-```sh
-cargo fmt --check
-cargo test
-cargo check -p decodal --no-default-features
-cargo publish -p decodal --dry-run
-```
-
-If `decodal-derive` changed, also run `cargo publish -p decodal-derive --dry-run`.
+The authoritative validation commands, dependency order, and publish commands are maintained in `RELEASING.md` at the repository root.
 
 ## Web site and playground
 
@@ -125,7 +105,6 @@ npx jsr publish --dry-run
 
 The npm package name is `decodal-wasm`.
 The JSR package name is `@hare/decodal-wasm`.
-JSR currently expects a single SPDX license identifier in `jsr.json`; the WebAssembly package metadata uses `MIT` while the Rust workspace remains dual licensed as `MIT OR Apache-2.0`.
 
 The playground editor uses the `decodal-codemirror` package with the generated Lezer parser in `packages/decodal-codemirror/src/decodal-parser.js`.
 The canonical grammar is documented in `doc/manual/souce/language/grammar.md`; regenerate the Lezer parser when that grammar or `editors/lezer-decodal/decodal.grammar` changes.
@@ -149,7 +128,6 @@ npx jsr publish --dry-run
 
 The npm package name is `decodal-codemirror`.
 The JSR package name is `@hare/decodal-codemirror`.
-JSR currently expects a single SPDX license identifier in `jsr.json`; the CodeMirror package metadata uses `MIT` while the Rust workspace remains dual licensed as `MIT OR Apache-2.0`.
 
 The documentation build still uses the lightweight JavaScript fallback highlighter so Astro can render Markdown without initializing WASM at build time.