Remove public tokenizer API

2026-07-09 01:09:40 +09:00
parent c4332ffb9d
commit 26d8495631
8 changed files with 4 additions and 332 deletions
@@ -27,7 +27,6 @@ regex_character  = escape | ? any character except "/", "\\", or newline ? ;
 ```

 Whitespace and comments separate tokens and are otherwise ignored by the parser.
-Comments are retained by public tokenizer APIs for editor tooling.

 ## Syntactic grammar