Add canonical grammar and CodeMirror playground
This commit is contained in:
@@ -58,7 +58,11 @@ Important files:
|
||||
```text
|
||||
site/decodal-site/src/pages/docs/[...slug].astro
|
||||
site/decodal-site/src/pages/playground.astro
|
||||
site/decodal-site/src/scripts/playground.js
|
||||
site/decodal-site/src/scripts/playground-examples.js
|
||||
site/decodal-site/src/layouts/ManualLayout.astro
|
||||
site/decodal-site/src/lib/codemirror/decodal.js
|
||||
site/decodal-site/src/lib/codemirror/decodal-parser.js
|
||||
site/decodal-site/src/lib/docs.js
|
||||
site/decodal-site/src/lib/highlight.js
|
||||
crates/decodal-wasm/src/lib.rs
|
||||
@@ -81,7 +85,8 @@ site/decodal-site/src/wasm/
|
||||
|
||||
These generated files are committed so the site can be built without requiring every consumer to regenerate the wasm package first.
|
||||
|
||||
The playground editor uses `tokenizeSource` from `decodal-wasm` for token spans and maps those tokens to HTML classes in `src/lib/highlight.js`.
|
||||
The playground editor uses CodeMirror 6 with the generated Lezer parser in `src/lib/codemirror/decodal-parser.js`.
|
||||
The canonical grammar is documented in `doc/manual/souce/language/grammar.md`; regenerate the Lezer parser when that grammar or `editors/lezer-decodal/decodal.grammar` changes.
|
||||
The documentation build still uses the lightweight JavaScript fallback highlighter so Astro can render Markdown without initializing WASM at build time.
|
||||
|
||||
To run the site locally:
|
||||
@@ -156,12 +161,18 @@ The generated parser files under `editors/tree-sitter-decodal/src/` are committe
|
||||
|
||||
When the Decodal syntax changes:
|
||||
|
||||
1. Update `grammar.js`.
|
||||
2. Run `npx tree-sitter generate`.
|
||||
3. Add or update corpus tests in `corpus/`.
|
||||
4. Update highlight queries in `queries/highlights.scm` if token names changed.
|
||||
5. Run `npx tree-sitter test`.
|
||||
6. Run Rust checks from the repository root if the language parser or examples also changed.
|
||||
1. Update the canonical EBNF in `doc/manual/souce/language/grammar.md`.
|
||||
2. Update the Rust parser/tokenizer as needed.
|
||||
3. Update `editors/tree-sitter-decodal/grammar.js` and run `npx tree-sitter generate` / `npx tree-sitter test`.
|
||||
4. Update `editors/lezer-decodal/decodal.grammar` and regenerate the CodeMirror parser:
|
||||
|
||||
```sh
|
||||
cd site/decodal-site
|
||||
npx lezer-generator ../../editors/lezer-decodal/decodal.grammar -o src/lib/codemirror/decodal-parser.js
|
||||
```
|
||||
|
||||
5. Add or update corpus/tests/examples.
|
||||
6. Run Rust and site checks from the repository root.
|
||||
|
||||
## Development shell
|
||||
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
# Grammar
|
||||
|
||||
This page is the canonical grammar reference for Decodal syntax.
|
||||
Parser implementations such as the Rust parser, Tree-sitter grammar, and Lezer grammar should follow this grammar and may add implementation-specific precedence annotations where needed.
|
||||
|
||||
## Lexical grammar
|
||||
|
||||
```ebnf
|
||||
source_character = ? any Unicode scalar value ? ;
|
||||
newline = "\n" | "\r\n" | "\r" ;
|
||||
space = " " | "\t" | newline ;
|
||||
comment = "#" , { ? any character except newline ? } ;
|
||||
|
||||
digit = "0" … "9" ;
|
||||
letter = "A" … "Z" | "a" … "z" ;
|
||||
identifier = letter , { letter | digit | "_" } ;
|
||||
|
||||
integer = digit , { digit } ;
|
||||
float = digit , { digit } , "." , digit , { digit } ;
|
||||
|
||||
string = '"' , { string_character | escape } , '"' ;
|
||||
string_character = ? any character except '"', "\\", or newline ? ;
|
||||
escape = "\\" , source_character ;
|
||||
|
||||
regex = "/" , regex_character , { regex_character } , "/" ;
|
||||
regex_character = escape | ? any character except "/", "\\", or newline ? ;
|
||||
```
|
||||
|
||||
Whitespace and comments separate tokens and are otherwise ignored by the parser.
|
||||
Comments are retained by public tokenizer APIs for editor tooling.
|
||||
|
||||
## Syntactic grammar
|
||||
|
||||
```ebnf
|
||||
module = { statement } ;
|
||||
statement = field_definition , [ ";" ]
|
||||
| expression , [ ";" ] ;
|
||||
|
||||
expression = default_expression ;
|
||||
|
||||
default_expression = patch_expression , [ "default" , default_expression ] ;
|
||||
patch_expression = compose_expression , { "//" , compose_expression } ;
|
||||
compose_expression = logical_or_expression , { "&" , logical_or_expression } ;
|
||||
|
||||
logical_or_expression = logical_and_expression , { "||" , logical_and_expression } ;
|
||||
logical_and_expression = comparison_expression , { "&&" , comparison_expression } ;
|
||||
comparison_expression = concat_expression , [ comparison_operator , concat_expression ] ;
|
||||
concat_expression = additive_expression , { "++" , additive_expression } ;
|
||||
additive_expression = multiplicative_expression , { ( "+" | "-" ) , multiplicative_expression } ;
|
||||
multiplicative_expression = unary_expression , { ( "*" | "/" ) , unary_expression } ;
|
||||
|
||||
unary_expression = [ "!" | "-" ] , postfix_expression ;
|
||||
postfix_expression = primary_expression , { call_suffix | path_suffix } ;
|
||||
call_suffix = "(" , [ argument_list ] , ")" ;
|
||||
path_suffix = "." , identifier ;
|
||||
|
||||
primary_expression = literal
|
||||
| identifier
|
||||
| comparison_constraint
|
||||
| object
|
||||
| array
|
||||
| let_expression
|
||||
| function_expression
|
||||
| match_expression
|
||||
| import_expression
|
||||
| "(" , expression , ")" ;
|
||||
|
||||
literal = string | integer | float | "true" | "false" | regex ;
|
||||
comparison_operator = "==" | "!=" | "<" | "<=" | ">" | ">=" ;
|
||||
comparison_constraint = ( "<" | "<=" | ">" | ">=" ) , expression ;
|
||||
|
||||
object = "{" , [ field_definition , { ";" , field_definition } , [ ";" ] ] , "}" ;
|
||||
field_definition = field_path , "=" , expression ;
|
||||
field_path = identifier , { "." , identifier } ;
|
||||
|
||||
array = "[" , [ expression , { "," , expression } , [ "," ] ] , "]" ;
|
||||
|
||||
let_expression = "let" , { field_definition , ";" } , "in" , expression ;
|
||||
function_expression = "(" , [ parameter_list ] , ")" , "=>" , expression ;
|
||||
parameter_list = parameter , { "," , parameter } , [ "," ] ;
|
||||
parameter = identifier , [ ":" , expression ] ;
|
||||
|
||||
match_expression = "match" , expression , "{" , [ match_arm , { ";" , match_arm } , [ ";" ] ] , "}" ;
|
||||
match_arm = pattern , ":" , expression ;
|
||||
pattern = "_" | expression ;
|
||||
|
||||
import_expression = "import" , string ;
|
||||
argument_list = expression , { "," , expression } , [ "," ] ;
|
||||
```
|
||||
|
||||
## Precedence
|
||||
|
||||
Precedence is highest first.
|
||||
|
||||
1. function call and field path reference
|
||||
2. unary `!` and `-`
|
||||
3. `*` and `/`
|
||||
4. `+` and `-`
|
||||
5. `++`
|
||||
6. `==`, `!=`, `<`, `<=`, `>`, `>=`
|
||||
7. `&&`
|
||||
8. `||`
|
||||
9. `&`
|
||||
10. `//`
|
||||
11. `default`
|
||||
|
||||
Binary operators are left-associative except `default`, which is right-associative.
|
||||
|
||||
## Tooling mapping
|
||||
|
||||
Syntax tooling should derive token categories from this grammar rather than making a tool-specific grammar canonical.
|
||||
Tree-sitter and Lezer grammars are implementation artifacts that follow this page.
|
||||
Reference in New Issue
Block a user