Bundle language tools with CodeMirror package

This commit is contained in:
2026-07-09 21:25:04 +09:00
parent 59cb8f31e4
commit 8eac5c5cb4
24 changed files with 155 additions and 219 deletions
+8 -9
View File
@@ -41,26 +41,23 @@ The WebAssembly package is for execution, not syntax highlighting.
## Language tools
Source-level tooling lives in the Rust language tools crate and its generated JavaScript/WebAssembly package.
Source-level tooling lives in the Rust language tools crate.
This component is responsible for operations that must preserve source text details such as comments and whitespace.
It is an internal crate used by the future Rust LSP and by the CodeMirror package's bundled formatter WebAssembly.
Important paths:
```text
crates/decodal-language-tools/
packages/decodal-language-tools/
```
The npm package is `decodal-language-tools`.
The JSR package is `@hare/decodal-language-tools`.
The current language tools package exposes the formatter.
The current language tools crate exposes the formatter.
Future LSP functionality should build on the same source-level tooling layer rather than on the runtime core.
## Web editor components
The Web playground editor uses CodeMirror 6 with a generated Lezer parser.
This is the source of syntax highlighting, folding, indentation, and editor syntax tree behavior in the browser UI.
This is the source of syntax highlighting, folding, indentation, editor syntax tree behavior, and the browser formatter command in the browser UI.
Important paths:
@@ -69,6 +66,8 @@ editors/lezer-decodal/decodal.grammar
packages/decodal-codemirror/src/decodal.js
packages/decodal-codemirror/src/decodal-parser.js
packages/decodal-codemirror/src/decodal-parser.terms.js
packages/decodal-codemirror/src/format.js
packages/decodal-codemirror/wasm/
```
The npm package is `decodal-codemirror`.
@@ -112,8 +111,8 @@ Consumers that need syntax information should use the component that matches the
- Rust execution and embedding: `decodal`
- Browser execution: `decodal-wasm`
- Source formatting and future LSP tooling: `decodal-language-tools`
- Web editor syntax: Lezer / CodeMirror
- Web formatting and editor syntax: Lezer / CodeMirror
- Rust LSP internals: `decodal-language-tools`
- General editor syntax: Tree-sitter
This avoids having a separate token stream API whose behavior would have to be kept compatible with both runtime parsing and editor grammars.
+6 -23
View File
@@ -25,7 +25,8 @@ cargo run -q -p decodal-cli --features regex -- examples/regex/main.dcdl
The primary crates.io package is `decodal`, which contains the embeddable library.
`decodal-derive` provides optional derive macros for Rust struct integration and is published only when the derive crate changes.
Workspace support crates such as `decodal-cli`, the Rust source crate `decodal-wasm`, and `decodal-language-tools` are not published to crates.io.
The generated WebAssembly packages under `packages/decodal-wasm/` and `packages/decodal-language-tools/` are published to npm and JSR.
The generated WebAssembly package under `packages/decodal-wasm/` is published to npm and JSR.
The CodeMirror package bundles the generated formatter WebAssembly from `decodal-language-tools`.
The project is dual licensed as `MIT OR Apache-2.0`.
```sh
@@ -58,7 +59,7 @@ site/decodal-site/
```
The site imports Markdown files from `doc/manual/souce/` and renders them as mdBook-style pages.
The playground loads `decodal-wasm` for evaluation and `decodal-language-tools` for source formatting.
The playground loads `decodal-wasm` for evaluation and `decodal-codemirror` for editor features and source formatting.
Important files:
@@ -74,8 +75,8 @@ packages/decodal-codemirror/src/decodal.js
packages/decodal-codemirror/src/decodal-parser.js
packages/decodal-wasm/decodal_wasm.js
packages/decodal-wasm/decodal_wasm_bg.wasm
packages/decodal-language-tools/decodal_language_tools.js
packages/decodal-language-tools/decodal_language_tools_bg.wasm
packages/decodal-codemirror/wasm/decodal_language_tools.js
packages/decodal-codemirror/wasm/decodal_language_tools_bg.wasm
crates/decodal-wasm/src/lib.rs
crates/decodal-language-tools/src/lib.rs
```
@@ -91,7 +92,7 @@ npm run build
`npm run build:wasm` builds both generated WebAssembly packages.
`npm run build:runtime` writes generated runtime files into `packages/decodal-wasm/`.
`npm run build:tools` writes generated language tools files into `packages/decodal-language-tools/`.
`npm run build:tools` writes generated language tools files into `packages/decodal-codemirror/wasm/`.
These generated files are committed so the site can be built without requiring every consumer to regenerate the wasm packages first.
@@ -114,24 +115,6 @@ The npm package name is `decodal-wasm`.
The JSR package name is `@hare/decodal-wasm`.
JSR currently expects a single SPDX license identifier in `jsr.json`; the WebAssembly package metadata uses `MIT` while the Rust workspace remains dual licensed as `MIT OR Apache-2.0`.
Publish the generated language tools package from its package directory:
```sh
cd packages/decodal-language-tools
npm publish
npx jsr publish
```
Run dry-runs first when preparing a release:
```sh
npm pack --dry-run
npx jsr publish --dry-run
```
The npm package name is `decodal-language-tools`.
The JSR package name is `@hare/decodal-language-tools`.
The playground editor uses the `decodal-codemirror` package with the generated Lezer parser in `packages/decodal-codemirror/src/decodal-parser.js`.
The canonical grammar is documented in `doc/manual/souce/language/grammar.md`; regenerate the Lezer parser when that grammar or `editors/lezer-decodal/decodal.grammar` changes.
+22 -2
View File
@@ -2,8 +2,8 @@
CodeMirror 6 language support for Decodal.
This package provides the Lezer parser metadata, highlighting tags, indentation rules, and block folding used by the Decodal Web playground.
It does not include the Decodal runtime or WebAssembly evaluator.
This package provides the Lezer parser metadata, highlighting tags, indentation rules, block folding, and formatter command used by the Decodal Web playground.
It does not include the Decodal runtime evaluator.
Use `decodal-wasm` when you want to evaluate Decodal in a browser.
## Install
@@ -52,3 +52,23 @@ decodal({ highlight: false })
```
Use this when you want the language support without the bundled highlight style.
## Formatting
The formatter is exposed as a separate subpath so plain syntax highlighting does not force callers to initialize WebAssembly.
```js
import {
formatDecodal,
formatDecodalCommand,
initDecodalFormatter,
} from 'decodal-codemirror/format';
import formatterWasm from 'decodal-codemirror/wasm/decodal_language_tools_bg.wasm?url';
await initDecodalFormatter(formatterWasm);
const result = formatDecodal('Server={port=Int default 8080;};');
if (result.ok) console.log(result.source);
formatDecodalCommand(view);
```
+15 -2
View File
@@ -1,10 +1,16 @@
{
"name": "@hare/decodal-codemirror",
"version": "0.1.3",
"version": "0.1.4",
"license": "MIT",
"exports": "./src/mod.ts",
"exports": {
".": "./src/mod.ts",
"./format": "./src/format.js",
"./parser": "./src/decodal-parser.js",
"./terms": "./src/decodal-parser.terms.js"
},
"imports": {
"@codemirror/language": "npm:@codemirror/language@^6.12.4",
"@codemirror/view": "npm:@codemirror/view@^6.43.6",
"@lezer/highlight": "npm:@lezer/highlight@^1.2.3",
"@lezer/lr": "npm:@lezer/lr@^1.4.10"
},
@@ -15,9 +21,16 @@
"src/decodal.js",
"src/decodal.d.ts",
"src/decodal.js.d.ts",
"src/format.js",
"src/format.d.ts",
"src/format.js.d.ts",
"src/decodal-parser-jsr.js",
"src/decodal-parser.js",
"src/decodal-parser.terms.js",
"wasm/decodal_language_tools.js",
"wasm/decodal_language_tools.d.ts",
"wasm/decodal_language_tools_bg.wasm",
"wasm/decodal_language_tools_bg.wasm.d.ts",
"package.json"
]
}
+4 -2
View File
@@ -1,20 +1,22 @@
{
"name": "decodal-codemirror",
"version": "0.1.3",
"version": "0.1.4",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "decodal-codemirror",
"version": "0.1.3",
"version": "0.1.4",
"license": "MIT OR Apache-2.0",
"devDependencies": {
"@codemirror/language": "^6.12.4",
"@codemirror/view": "^6.43.6",
"@lezer/highlight": "^1.2.3",
"@lezer/lr": "^1.4.10"
},
"peerDependencies": {
"@codemirror/language": "^6.12.4",
"@codemirror/view": "^6.43.6",
"@lezer/highlight": "^1.2.3",
"@lezer/lr": "^1.4.10"
}
+14 -3
View File
@@ -1,6 +1,6 @@
{
"name": "decodal-codemirror",
"version": "0.1.3",
"version": "0.1.4",
"description": "CodeMirror 6 language support for Decodal.",
"type": "module",
"license": "MIT OR Apache-2.0",
@@ -14,13 +14,22 @@
"types": "./src/decodal.d.ts",
"import": "./src/decodal.js"
},
"./format": {
"types": "./src/format.d.ts",
"import": "./src/format.js"
},
"./parser": "./src/decodal-parser.js",
"./terms": "./src/decodal-parser.terms.js"
"./terms": "./src/decodal-parser.terms.js",
"./wasm/decodal_language_tools_bg.wasm": "./wasm/decodal_language_tools_bg.wasm"
},
"types": "./src/decodal.d.ts",
"files": [
"README.md",
"src/"
"src/",
"wasm/decodal_language_tools.js",
"wasm/decodal_language_tools.d.ts",
"wasm/decodal_language_tools_bg.wasm",
"wasm/decodal_language_tools_bg.wasm.d.ts"
],
"keywords": [
"decodal",
@@ -29,11 +38,13 @@
],
"peerDependencies": {
"@codemirror/language": "^6.12.4",
"@codemirror/view": "^6.43.6",
"@lezer/highlight": "^1.2.3",
"@lezer/lr": "^1.4.10"
},
"devDependencies": {
"@codemirror/language": "^6.12.4",
"@codemirror/view": "^6.43.6",
"@lezer/highlight": "^1.2.3",
"@lezer/lr": "^1.4.10"
}
+17
View File
@@ -0,0 +1,17 @@
import type { EditorView } from '@codemirror/view';
export interface FormatSuccess {
ok: true;
source: string;
}
export interface FormatFailure {
ok: false;
error: string;
}
export type FormatResult = FormatSuccess | FormatFailure;
export declare function initDecodalFormatter(moduleOrPath?: RequestInfo | URL | Response | BufferSource | WebAssembly.Module): Promise<void>;
export declare function formatDecodal(source: string): FormatResult;
export declare function formatDecodalCommand(view: EditorView): boolean;
+28
View File
@@ -0,0 +1,28 @@
import initLanguageTools, { formatSource } from '../wasm/decodal_language_tools.js';
let initialized = false;
export async function initDecodalFormatter(moduleOrPath) {
await initLanguageTools(moduleOrPath);
initialized = true;
}
export function formatDecodal(source) {
if (!initialized) {
return { ok: false, error: 'Decodal formatter is not initialized' };
}
return JSON.parse(formatSource(source));
}
export function formatDecodalCommand(view) {
const result = formatDecodal(view.state.doc.toString());
if (!result.ok) return false;
view.dispatch({
changes: {
from: 0,
to: view.state.doc.length,
insert: result.source,
},
});
return true;
}
+17
View File
@@ -0,0 +1,17 @@
import type { EditorView } from '@codemirror/view';
export interface FormatSuccess {
ok: true;
source: string;
}
export interface FormatFailure {
ok: false;
error: string;
}
export type FormatResult = FormatSuccess | FormatFailure;
export declare function initDecodalFormatter(moduleOrPath?: RequestInfo | URL | Response | BufferSource | WebAssembly.Module): Promise<void>;
export declare function formatDecodal(source: string): FormatResult;
export declare function formatDecodalCommand(view: EditorView): boolean;
@@ -0,0 +1 @@
# wasm-pack output is committed as part of decodal-codemirror.
@@ -1 +0,0 @@
# wasm-pack output is committed for the browser language tools package.
-39
View File
@@ -1,39 +0,0 @@
# decodal-language-tools
Source-level language tooling for Decodal.
This package is generated from the Rust crate in `crates/decodal-language-tools` with `wasm-pack`.
It currently exposes a comment-preserving formatter that is shared by the official playground and future editor/LSP tooling.
Use `decodal-codemirror` separately when you need CodeMirror 6 language support.
Use `decodal-wasm` separately when you need to evaluate Decodal in a browser.
## Install
```sh
npm install decodal-language-tools
```
JSR:
```sh
deno add jsr:@hare/decodal-language-tools
```
## Usage
```js
import init, { formatSource } from 'decodal-language-tools';
await init();
const result = JSON.parse(formatSource('Server={# host\nhost=String default "localhost";};'));
if (result.ok) {
console.log(result.source);
} else {
console.error(result.error);
}
```
The exported functions return JSON strings so callers can handle diagnostics without depending on Rust data structures.
-15
View File
@@ -1,15 +0,0 @@
{
"name": "@hare/decodal-language-tools",
"version": "0.1.2",
"license": "MIT",
"exports": "./decodal_language_tools.js",
"publish": {
"include": [
"README.md",
"decodal_language_tools.js",
"decodal_language_tools.d.ts",
"decodal_language_tools_bg.wasm",
"package.json"
]
}
}
@@ -1,27 +0,0 @@
{
"name": "decodal-language-tools",
"type": "module",
"description": "Source-level language tooling for Decodal.",
"version": "0.1.2",
"license": "MIT OR Apache-2.0",
"repository": {
"type": "git",
"url": "https://gitea.hareworks.net/Hare/Decodal"
},
"files": [
"decodal_language_tools_bg.wasm",
"decodal_language_tools.js",
"decodal_language_tools.d.ts"
],
"main": "decodal_language_tools.js",
"types": "decodal_language_tools.d.ts",
"sideEffects": [
"./snippets/*"
],
"keywords": [
"decodal",
"formatter",
"lsp",
"tree-sitter"
]
}
+3 -10
View File
@@ -16,7 +16,6 @@
"astro": "^4.16.18",
"codemirror": "^6.0.2",
"decodal-codemirror": "file:../../packages/decodal-codemirror",
"decodal-language-tools": "file:../../packages/decodal-language-tools",
"decodal-wasm": "file:../../packages/decodal-wasm",
"marked": "^12.0.2"
},
@@ -26,23 +25,21 @@
}
},
"../../packages/decodal-codemirror": {
"version": "0.1.3",
"version": "0.1.4",
"license": "MIT OR Apache-2.0",
"devDependencies": {
"@codemirror/language": "^6.12.4",
"@codemirror/view": "^6.43.6",
"@lezer/highlight": "^1.2.3",
"@lezer/lr": "^1.4.10"
},
"peerDependencies": {
"@codemirror/language": "^6.12.4",
"@codemirror/view": "^6.43.6",
"@lezer/highlight": "^1.2.3",
"@lezer/lr": "^1.4.10"
}
},
"../../packages/decodal-language-tools": {
"version": "0.1.2",
"license": "MIT OR Apache-2.0"
},
"../../packages/decodal-wasm": {
"version": "0.1.2",
"license": "MIT OR Apache-2.0"
@@ -3265,10 +3262,6 @@
"resolved": "../../packages/decodal-codemirror",
"link": true
},
"node_modules/decodal-language-tools": {
"resolved": "../../packages/decodal-language-tools",
"link": true
},
"node_modules/decodal-wasm": {
"resolved": "../../packages/decodal-wasm",
"link": true
+1 -2
View File
@@ -8,7 +8,7 @@
"build": "astro build",
"preview": "astro preview --host 0.0.0.0",
"build:runtime": "wasm-pack build ../../crates/decodal-wasm --target web --out-dir ../../packages/decodal-wasm --release && node scripts/prepare-wasm-package.mjs",
"build:tools": "wasm-pack build ../../crates/decodal-language-tools --target web --out-dir ../../packages/decodal-language-tools --release && node scripts/prepare-language-tools-package.mjs",
"build:tools": "wasm-pack build ../../crates/decodal-language-tools --target web --out-dir ../../packages/decodal-codemirror/wasm --release && node scripts/prepare-codemirror-wasm.mjs",
"build:wasm": "npm run build:runtime && npm run build:tools",
"build:packages": "npm run build:wasm",
"deploy": "npm run build && node scripts/deploy-pages.mjs",
@@ -23,7 +23,6 @@
"astro": "^4.16.18",
"codemirror": "^6.0.2",
"decodal-codemirror": "file:../../packages/decodal-codemirror",
"decodal-language-tools": "file:../../packages/decodal-language-tools",
"decodal-wasm": "file:../../packages/decodal-wasm",
"marked": "^12.0.2"
},
@@ -0,0 +1,12 @@
import { rmSync, writeFileSync } from 'node:fs';
import { resolve } from 'node:path';
const wasmDir = resolve(import.meta.dirname, '../../../packages/decodal-codemirror/wasm');
rmSync(resolve(wasmDir, 'README.md'), { force: true });
rmSync(resolve(wasmDir, 'package.json'), { force: true });
writeFileSync(
resolve(wasmDir, '.gitignore'),
'# wasm-pack output is committed as part of decodal-codemirror.\n',
);
@@ -1,76 +0,0 @@
import { writeFileSync } from 'node:fs';
import { resolve } from 'node:path';
const packageDir = resolve(import.meta.dirname, '../../../packages/decodal-language-tools');
writeFileSync(
resolve(packageDir, '.gitignore'),
'# wasm-pack output is committed for the browser language tools package.\n',
);
writeFileSync(
resolve(packageDir, 'README.md'),
`# decodal-language-tools
Source-level language tooling for Decodal.
This package is generated from the Rust crate in \`crates/decodal-language-tools\` with \`wasm-pack\`.
It currently exposes a comment-preserving formatter that is shared by the official playground and future editor/LSP tooling.
Use \`decodal-codemirror\` separately when you need CodeMirror 6 language support.
Use \`decodal-wasm\` separately when you need to evaluate Decodal in a browser.
## Install
\`\`\`sh
npm install decodal-language-tools
\`\`\`
JSR:
\`\`\`sh
deno add jsr:@hare/decodal-language-tools
\`\`\`
## Usage
\`\`\`js
import init, { formatSource } from 'decodal-language-tools';
await init();
const result = JSON.parse(formatSource('Server={# host\\nhost=String default "localhost";};'));
if (result.ok) {
console.log(result.source);
} else {
console.error(result.error);
}
\`\`\`
The exported functions return JSON strings so callers can handle diagnostics without depending on Rust data structures.
`,
);
writeFileSync(
resolve(packageDir, 'jsr.json'),
JSON.stringify(
{
name: '@hare/decodal-language-tools',
version: '0.1.2',
license: 'MIT',
exports: './decodal_language_tools.js',
publish: {
include: [
'README.md',
'decodal_language_tools.js',
'decodal_language_tools.d.ts',
'decodal_language_tools_bg.wasm',
'package.json',
],
},
},
null,
2,
) + '\n',
);
+3 -4
View File
@@ -33,8 +33,7 @@ Production = Server & {
</a>
<ul class="package-list">
<li>decodal-wasm on <a href="https://jsr.io/@hare/decodal-wasm@0.1.2">jsr</a> / <a href="https://www.npmjs.com/package/decodal-wasm">npm</a></li>
<li>decodal-language-tools on <a href="https://jsr.io/@hare/decodal-language-tools@0.1.2">jsr</a> / <a href="https://www.npmjs.com/package/decodal-language-tools">npm</a></li>
<li>decodal-codemirror on <a href="https://jsr.io/@hare/decodal-codemirror@0.1.3">jsr</a> / <a href="https://www.npmjs.com/package/decodal-codemirror">npm</a></li>
<li>decodal-codemirror on <a href="https://jsr.io/@hare/decodal-codemirror@0.1.4">jsr</a> / <a href="https://www.npmjs.com/package/decodal-codemirror">npm</a></li>
</ul>
</section>
@@ -49,8 +48,8 @@ Production = Server & {
<article class="home-card">
<h2>Where it runs</h2>
<p>
Use the Rust crate for runtime embedding, the WASM package in browsers, language-tools for formatting,
CodeMirror/Lezer for the Web editor, and Tree-sitter for general editor integration.
Use the Rust crate for runtime embedding, the WASM package in browsers, CodeMirror/Lezer for Web editing
and formatting, and Tree-sitter for general editor integration.
</p>
</article>
<article class="home-card">
+4 -4
View File
@@ -1,10 +1,10 @@
import initRuntime, { evaluateProject } from 'decodal-wasm';
import runtimeWasmUrl from 'decodal-wasm/decodal_wasm_bg.wasm?url';
import initTools, { formatSource } from 'decodal-language-tools';
import toolsWasmUrl from 'decodal-language-tools/decodal_language_tools_bg.wasm?url';
import { EditorView, basicSetup } from 'codemirror';
import { keymap } from '@codemirror/view';
import { decodal } from 'decodal-codemirror';
import { formatDecodal, initDecodalFormatter } from 'decodal-codemirror/format';
import toolsWasmUrl from 'decodal-codemirror/wasm/decodal_language_tools_bg.wasm?url';
import { playgroundExamples } from './playground-examples.js';
const STORAGE_KEY = 'decodal-playground-project-v2';
@@ -221,7 +221,7 @@ function execute() {
}
function formatActiveFile() {
const result = JSON.parse(formatSource(getEditorText()));
const result = formatDecodal(getEditorText());
if (!result.ok) {
output.textContent = result.error;
output.classList.add('error');
@@ -285,7 +285,7 @@ function compareNodes(a, b) {
}
try {
await Promise.all([initRuntime(runtimeWasmUrl), initTools(toolsWasmUrl)]);
await Promise.all([initRuntime(runtimeWasmUrl), initDecodalFormatter(toolsWasmUrl)]);
run.disabled = false;
formatButton.disabled = false;
status.textContent = '';