feat: expose typed Text View tokens and plain-text conversion
T-28: generate owned semantic tokens directly from Object/Interface structure, including all escaped data and layout. Preserve render APIs through the same token generation and delimiter-free concatenation; preserve input limits, validation order and errors. Document splitting, semantic roles and destination escaping boundaries. Validation: cargo fmt --all -- --check; cargo check --workspace --all-targets; cargo test --workspace (312 passing tests/doctests, including 16 Text View consumer tests and 4 Text View/README doctests); nix develop --command cargo clippy --workspace --all-targets -- -D warnings; RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps; RUSTDOCFLAGS="-D warnings" cargo test -p wip-text-view -p wip-protocol --doc; git diff --check. All passed. Existing golden files unchanged; every prior Text View case now verifies token/plain/direct equality or exact error equality. No existing crates depend on wip-text-view; full workspace validation covers adjacent consumers.
This commit is contained in:
@@ -42,6 +42,84 @@ requests a path and never infers one from name or ref. It requires no descriptor
|
||||
An Interface needs no Object, separate name/scope/context arguments or lookup.
|
||||
There is no document aggregator, configuration or details selector.
|
||||
|
||||
## Typed display tokens
|
||||
|
||||
```rust
|
||||
use wip_protocol::{Object, InterfaceReference, InterfaceDescriptor, INTERFACE_FORMAT_V1};
|
||||
use wip_text_view::{
|
||||
Interface, TokenKind, tokenize_object, tokenize_interface, to_plain_text,
|
||||
render_object, render_interface,
|
||||
};
|
||||
|
||||
let reference = InterfaceReference { scope: "/".into(), name: "example".into() };
|
||||
let object = Object {
|
||||
name: "item".into(), description: None, interfaces: vec![reference.clone()],
|
||||
r#ref: None, validator: None,
|
||||
};
|
||||
let object_tokens = tokenize_object(&object)?;
|
||||
assert_eq!(object_tokens[0].kind(), TokenKind::Keyword);
|
||||
assert_eq!(object_tokens[0].text(), "object");
|
||||
assert_eq!(to_plain_text(&object_tokens), render_object(&object)?);
|
||||
|
||||
let descriptor = InterfaceDescriptor {
|
||||
format: INTERFACE_FORMAT_V1.into(), documentation: None,
|
||||
types: vec![], operations: vec![],
|
||||
};
|
||||
let input = Interface { reference: &reference, descriptor: &descriptor };
|
||||
let tokens = tokenize_interface(&input)?;
|
||||
for token in &tokens {
|
||||
// Custom renderers can select presentation by role without reparsing text.
|
||||
if token.kind() == TokenKind::Reference {
|
||||
assert_eq!(token.text(), "\"/::example\"");
|
||||
}
|
||||
}
|
||||
let joined: String = tokens.iter().map(|token| token.text()).collect();
|
||||
assert_eq!(to_plain_text(&tokens), joined);
|
||||
assert_eq!(to_plain_text(&tokens), render_interface(&input)?);
|
||||
# Ok::<(), wip_text_view::RenderError>(())
|
||||
```
|
||||
|
||||
`tokenize_object` / `tokenize_interface` return `Result<Vec<Token>, RenderError>`.
|
||||
They generate **directly from the input structure**, not by lexing a completed
|
||||
Text View string. Each token owns its text (and can outlive the input) and exposes
|
||||
`kind()` / `text()`. Existing render APIs use the same token generation followed
|
||||
by `to_plain_text(&[Token])`; there is no second grammar/escape/layout engine.
|
||||
|
||||
### Kind and splitting contract
|
||||
|
||||
| Kind | One token contains |
|
||||
| --- | --- |
|
||||
| `Keyword` | One grammar keyword, Object property label, or primitive type keyword |
|
||||
| `TypeName` | One type declaration name or Named type reference |
|
||||
| `OperationName` | One Operation declaration name |
|
||||
| `ParameterName` / `FieldName` | One parameter / record field name |
|
||||
| `CaseName` | One enum or union case name |
|
||||
| `String` | One complete quoted Object name or description |
|
||||
| `Reference` | One complete quoted absolute Interface reference, with both quoting layers |
|
||||
| `Documentation` | Entire `/// "summary"` content, including marker and internal space, excluding indentation/LF |
|
||||
| `Symbol` | One punctuation mark, except the indivisible two-character `->` arrow |
|
||||
| `Whitespace` | One inter-token ASCII space |
|
||||
| `Newline` | One LF, including the final LF |
|
||||
| `Indentation` | All leading spaces of a non-root line, two spaces per depth level |
|
||||
|
||||
Quoted identifiers retain their semantic role even when named like keywords.
|
||||
Quotes, backticks and all WIP escapes are part of the data token, never separate
|
||||
symbols. Host spaces inside quoted data remain in that token. Adjacent symbols
|
||||
are separate; no empty tokens or zero-depth indentation tokens are emitted.
|
||||
`TokenKind` is non-exhaustive so consumers should handle future roles with a
|
||||
fallback. The API does not distinguish type declaration names from Named uses,
|
||||
or enum cases from union cases; both share the documented role.
|
||||
|
||||
All layout is included. Concatenate token text **without separators**, not with
|
||||
`join(" ")`. `to_plain_text` is only that concatenation: no interpretation,
|
||||
validation, quoting, escaping or layout completion. An empty slice yields `""`;
|
||||
a filtered/reordered slice is allowed but need not be a complete signature.
|
||||
|
||||
Token text is escaped for **WIP Text View**, not for HTML or other destinations.
|
||||
Custom renderers must escape for their destination and treat Host text as data,
|
||||
not markup or instructions. This crate supplies no HTML, ANSI, color or theme
|
||||
renderer/adapter and no external text lexer/parser or AST.
|
||||
|
||||
## Output and validation
|
||||
|
||||
- UTF-8 plain text with LF, two-space indentation and a final LF, without ANSI,
|
||||
@@ -63,8 +141,8 @@ There is no document aggregator, configuration or details selector.
|
||||
- Object ref/validator, children, domain data and Interface scope_ref/validator
|
||||
are not signature content. No annotations are inferred from operation names.
|
||||
|
||||
Both functions return `Result<String, RenderError>` and never return partial
|
||||
signatures. Resource checks happen first, followed by existing Protocol validators:
|
||||
The render functions return `Result<String, RenderError>`; token generation
|
||||
returns `Result<Vec<Token>, RenderError>`. Neither returns partial signatures. Resource checks happen first, followed by existing Protocol validators:
|
||||
Object shape and reference uniqueness, or Interface reference shape plus descriptor
|
||||
structure, duplicate names, unknown Named references and cycles. Descriptor format
|
||||
is checked after semantics: valid but unsupported formats return
|
||||
@@ -94,7 +172,13 @@ the spec's Object, issue.read, CloseResult, search/return documentation and
|
||||
identifier examples (with permitted multiline layout), plus nested documentation.
|
||||
The suite checks all TypeExpr variants/positions, ordering and empty distinctions,
|
||||
keyword and Unicode names, both reference quoting layers, all required control
|
||||
escapes, input immutability, metadata exclusion and typed rejection cases.
|
||||
escapes, input immutability, metadata exclusion and typed rejection cases. Every
|
||||
existing case compares the render result (including exact errors) with token
|
||||
text concatenation and `to_plain_text`. Additional tests assert complete Object
|
||||
token splitting, all semantic roles, keyword collisions, atomic escaped data,
|
||||
all documentation positions, layout tokens and ownership beyond input lifetime.
|
||||
Input budgets bound token counts per node, indentation depth and escape expansion;
|
||||
token generation checks them before any rendering/validation traversal.
|
||||
|
||||
```console
|
||||
cargo test -p wip-text-view
|
||||
|
||||
Reference in New Issue
Block a user