Bundle language tools with CodeMirror package

doc/manual/souce/components.md

@@ -41,26 +41,23 @@ The WebAssembly package is for execution, not syntax highlighting.
 
 ## Language tools
 
-Source-level tooling lives in the Rust language tools crate and its generated JavaScript/WebAssembly package.
+Source-level tooling lives in the Rust language tools crate.
 This component is responsible for operations that must preserve source text details such as comments and whitespace.
+It is an internal crate used by the future Rust LSP and by the CodeMirror package's bundled formatter WebAssembly.
 
 Important paths:
 
 ```text
 crates/decodal-language-tools/
-packages/decodal-language-tools/
 ```
 
-The npm package is `decodal-language-tools`.
+The current language tools crate exposes the formatter.
-The JSR package is `@hare/decodal-language-tools`.
-
-The current language tools package exposes the formatter.
 Future LSP functionality should build on the same source-level tooling layer rather than on the runtime core.
 
 ## Web editor components
 
 The Web playground editor uses CodeMirror 6 with a generated Lezer parser.
-This is the source of syntax highlighting, folding, indentation, and editor syntax tree behavior in the browser UI.
+This is the source of syntax highlighting, folding, indentation, editor syntax tree behavior, and the browser formatter command in the browser UI.
 
 Important paths:
 
@@ -69,6 +66,8 @@ editors/lezer-decodal/decodal.grammar
 packages/decodal-codemirror/src/decodal.js
 packages/decodal-codemirror/src/decodal-parser.js
 packages/decodal-codemirror/src/decodal-parser.terms.js
+packages/decodal-codemirror/src/format.js
+packages/decodal-codemirror/wasm/
 ```
 
 The npm package is `decodal-codemirror`.
@@ -112,8 +111,8 @@ Consumers that need syntax information should use the component that matches the
 
 - Rust execution and embedding: `decodal`
 - Browser execution: `decodal-wasm`
-- Source formatting and future LSP tooling: `decodal-language-tools`
+- Web formatting and editor syntax: Lezer / CodeMirror
-- Web editor syntax: Lezer / CodeMirror
+- Rust LSP internals: `decodal-language-tools`
 - General editor syntax: Tree-sitter
 
 This avoids having a separate token stream API whose behavior would have to be kept compatible with both runtime parsing and editor grammars.

doc/manual/souce/development.md

@@ -25,7 +25,8 @@ cargo run -q -p decodal-cli --features regex -- examples/regex/main.dcdl
 The primary crates.io package is `decodal`, which contains the embeddable library.
 `decodal-derive` provides optional derive macros for Rust struct integration and is published only when the derive crate changes.
 Workspace support crates such as `decodal-cli`, the Rust source crate `decodal-wasm`, and `decodal-language-tools` are not published to crates.io.
-The generated WebAssembly packages under `packages/decodal-wasm/` and `packages/decodal-language-tools/` are published to npm and JSR.
+The generated WebAssembly package under `packages/decodal-wasm/` is published to npm and JSR.
+The CodeMirror package bundles the generated formatter WebAssembly from `decodal-language-tools`.
 The project is dual licensed as `MIT OR Apache-2.0`.
 
 ```sh
@@ -58,7 +59,7 @@ site/decodal-site/
 ```
 
 The site imports Markdown files from `doc/manual/souce/` and renders them as mdBook-style pages.
-The playground loads `decodal-wasm` for evaluation and `decodal-language-tools` for source formatting.
+The playground loads `decodal-wasm` for evaluation and `decodal-codemirror` for editor features and source formatting.
 
 Important files:
 
@@ -74,8 +75,8 @@ packages/decodal-codemirror/src/decodal.js
 packages/decodal-codemirror/src/decodal-parser.js
 packages/decodal-wasm/decodal_wasm.js
 packages/decodal-wasm/decodal_wasm_bg.wasm
-packages/decodal-language-tools/decodal_language_tools.js
+packages/decodal-codemirror/wasm/decodal_language_tools.js
-packages/decodal-language-tools/decodal_language_tools_bg.wasm
+packages/decodal-codemirror/wasm/decodal_language_tools_bg.wasm
 crates/decodal-wasm/src/lib.rs
 crates/decodal-language-tools/src/lib.rs
 ```
@@ -91,7 +92,7 @@ npm run build
 
 `npm run build:wasm` builds both generated WebAssembly packages.
 `npm run build:runtime` writes generated runtime files into `packages/decodal-wasm/`.
-`npm run build:tools` writes generated language tools files into `packages/decodal-language-tools/`.
+`npm run build:tools` writes generated language tools files into `packages/decodal-codemirror/wasm/`.
 
 These generated files are committed so the site can be built without requiring every consumer to regenerate the wasm packages first.
 
@@ -114,24 +115,6 @@ The npm package name is `decodal-wasm`.
 The JSR package name is `@hare/decodal-wasm`.
 JSR currently expects a single SPDX license identifier in `jsr.json`; the WebAssembly package metadata uses `MIT` while the Rust workspace remains dual licensed as `MIT OR Apache-2.0`.
 
-Publish the generated language tools package from its package directory:
-
-```sh
-cd packages/decodal-language-tools
-npm publish
-npx jsr publish
-```
-
-Run dry-runs first when preparing a release:
-
-```sh
-npm pack --dry-run
-npx jsr publish --dry-run
-```
-
-The npm package name is `decodal-language-tools`.
-The JSR package name is `@hare/decodal-language-tools`.
-
 The playground editor uses the `decodal-codemirror` package with the generated Lezer parser in `packages/decodal-codemirror/src/decodal-parser.js`.
 The canonical grammar is documented in `doc/manual/souce/language/grammar.md`; regenerate the Lezer parser when that grammar or `editors/lezer-decodal/decodal.grammar` changes.
 

packages/decodal-codemirror/README.md

@@ -2,8 +2,8 @@
 
 CodeMirror 6 language support for Decodal.
 
-This package provides the Lezer parser metadata, highlighting tags, indentation rules, and block folding used by the Decodal Web playground.
+This package provides the Lezer parser metadata, highlighting tags, indentation rules, block folding, and formatter command used by the Decodal Web playground.
-It does not include the Decodal runtime or WebAssembly evaluator.
+It does not include the Decodal runtime evaluator.
 Use `decodal-wasm` when you want to evaluate Decodal in a browser.
 
 ## Install
@@ -52,3 +52,23 @@ decodal({ highlight: false })
 ```
 
 Use this when you want the language support without the bundled highlight style.
+
+## Formatting
+
+The formatter is exposed as a separate subpath so plain syntax highlighting does not force callers to initialize WebAssembly.
+
+```js
+import {
+  formatDecodal,
+  formatDecodalCommand,
+  initDecodalFormatter,
+} from 'decodal-codemirror/format';
+import formatterWasm from 'decodal-codemirror/wasm/decodal_language_tools_bg.wasm?url';
+
+await initDecodalFormatter(formatterWasm);
+
+const result = formatDecodal('Server={port=Int default 8080;};');
+if (result.ok) console.log(result.source);
+
+formatDecodalCommand(view);
+```

packages/decodal-codemirror/jsr.json

@@ -1,10 +1,16 @@
 {
   "name": "@hare/decodal-codemirror",
-  "version": "0.1.3",
+  "version": "0.1.4",
   "license": "MIT",
-  "exports": "./src/mod.ts",
+  "exports": {
+    ".": "./src/mod.ts",
+    "./format": "./src/format.js",
+    "./parser": "./src/decodal-parser.js",
+    "./terms": "./src/decodal-parser.terms.js"
+  },
   "imports": {
     "@codemirror/language": "npm:@codemirror/language@^6.12.4",
+    "@codemirror/view": "npm:@codemirror/view@^6.43.6",
     "@lezer/highlight": "npm:@lezer/highlight@^1.2.3",
     "@lezer/lr": "npm:@lezer/lr@^1.4.10"
   },
@@ -15,9 +21,16 @@
       "src/decodal.js",
       "src/decodal.d.ts",
       "src/decodal.js.d.ts",
+      "src/format.js",
+      "src/format.d.ts",
+      "src/format.js.d.ts",
       "src/decodal-parser-jsr.js",
       "src/decodal-parser.js",
       "src/decodal-parser.terms.js",
+      "wasm/decodal_language_tools.js",
+      "wasm/decodal_language_tools.d.ts",
+      "wasm/decodal_language_tools_bg.wasm",
+      "wasm/decodal_language_tools_bg.wasm.d.ts",
       "package.json"
     ]
   }

packages/decodal-codemirror/package-lock.json

@@ -1,20 +1,22 @@
 {
   "name": "decodal-codemirror",
-  "version": "0.1.3",
+  "version": "0.1.4",
   "lockfileVersion": 3,
   "requires": true,
   "packages": {
     "": {
       "name": "decodal-codemirror",
-      "version": "0.1.3",
+      "version": "0.1.4",
       "license": "MIT OR Apache-2.0",
       "devDependencies": {
         "@codemirror/language": "^6.12.4",
+        "@codemirror/view": "^6.43.6",
         "@lezer/highlight": "^1.2.3",
         "@lezer/lr": "^1.4.10"
       },
       "peerDependencies": {
         "@codemirror/language": "^6.12.4",
+        "@codemirror/view": "^6.43.6",
         "@lezer/highlight": "^1.2.3",
         "@lezer/lr": "^1.4.10"
       }

packages/decodal-codemirror/package.json

@@ -1,6 +1,6 @@
 {
   "name": "decodal-codemirror",
-  "version": "0.1.3",
+  "version": "0.1.4",
   "description": "CodeMirror 6 language support for Decodal.",
   "type": "module",
   "license": "MIT OR Apache-2.0",
@@ -14,13 +14,22 @@
       "types": "./src/decodal.d.ts",
       "import": "./src/decodal.js"
     },
+    "./format": {
+      "types": "./src/format.d.ts",
+      "import": "./src/format.js"
+    },
     "./parser": "./src/decodal-parser.js",
-    "./terms": "./src/decodal-parser.terms.js"
+    "./terms": "./src/decodal-parser.terms.js",
+    "./wasm/decodal_language_tools_bg.wasm": "./wasm/decodal_language_tools_bg.wasm"
   },
   "types": "./src/decodal.d.ts",
   "files": [
     "README.md",
-    "src/"
+    "src/",
+    "wasm/decodal_language_tools.js",
+    "wasm/decodal_language_tools.d.ts",
+    "wasm/decodal_language_tools_bg.wasm",
+    "wasm/decodal_language_tools_bg.wasm.d.ts"
   ],
   "keywords": [
     "decodal",
@@ -29,11 +38,13 @@
   ],
   "peerDependencies": {
     "@codemirror/language": "^6.12.4",
+    "@codemirror/view": "^6.43.6",
     "@lezer/highlight": "^1.2.3",
     "@lezer/lr": "^1.4.10"
   },
   "devDependencies": {
     "@codemirror/language": "^6.12.4",
+    "@codemirror/view": "^6.43.6",
     "@lezer/highlight": "^1.2.3",
     "@lezer/lr": "^1.4.10"
   }

packages/decodal-codemirror/src/format.d.ts

@@ -0,0 +1,17 @@
+import type { EditorView } from '@codemirror/view';
+
+export interface FormatSuccess {
+  ok: true;
+  source: string;
+}
+
+export interface FormatFailure {
+  ok: false;
+  error: string;
+}
+
+export type FormatResult = FormatSuccess | FormatFailure;
+
+export declare function initDecodalFormatter(moduleOrPath?: RequestInfo | URL | Response | BufferSource | WebAssembly.Module): Promise<void>;
+export declare function formatDecodal(source: string): FormatResult;
+export declare function formatDecodalCommand(view: EditorView): boolean;

packages/decodal-codemirror/src/format.js

@@ -0,0 +1,28 @@
+import initLanguageTools, { formatSource } from '../wasm/decodal_language_tools.js';
+
+let initialized = false;
+
+export async function initDecodalFormatter(moduleOrPath) {
+  await initLanguageTools(moduleOrPath);
+  initialized = true;
+}
+
+export function formatDecodal(source) {
+  if (!initialized) {
+    return { ok: false, error: 'Decodal formatter is not initialized' };
+  }
+  return JSON.parse(formatSource(source));
+}
+
+export function formatDecodalCommand(view) {
+  const result = formatDecodal(view.state.doc.toString());
+  if (!result.ok) return false;
+  view.dispatch({
+    changes: {
+      from: 0,
+      to: view.state.doc.length,
+      insert: result.source,
+    },
+  });
+  return true;
+}

packages/decodal-codemirror/src/format.js.d.ts

@@ -0,0 +1,17 @@
+import type { EditorView } from '@codemirror/view';
+
+export interface FormatSuccess {
+  ok: true;
+  source: string;
+}
+
+export interface FormatFailure {
+  ok: false;
+  error: string;
+}
+
+export type FormatResult = FormatSuccess | FormatFailure;
+
+export declare function initDecodalFormatter(moduleOrPath?: RequestInfo | URL | Response | BufferSource | WebAssembly.Module): Promise<void>;
+export declare function formatDecodal(source: string): FormatResult;
+export declare function formatDecodalCommand(view: EditorView): boolean;

packages/decodal-codemirror/wasm/.gitignore

@@ -0,0 +1 @@
+# wasm-pack output is committed as part of decodal-codemirror.

packages/decodal-language-tools/.gitignore

@@ -1 +0,0 @@
-# wasm-pack output is committed for the browser language tools package.

packages/decodal-language-tools/README.md

@@ -1,39 +0,0 @@
-# decodal-language-tools
-
-Source-level language tooling for Decodal.
-
-This package is generated from the Rust crate in `crates/decodal-language-tools` with `wasm-pack`.
-It currently exposes a comment-preserving formatter that is shared by the official playground and future editor/LSP tooling.
-
-Use `decodal-codemirror` separately when you need CodeMirror 6 language support.
-Use `decodal-wasm` separately when you need to evaluate Decodal in a browser.
-
-## Install
-
-```sh
-npm install decodal-language-tools
-```
-
-JSR:
-
-```sh
-deno add jsr:@hare/decodal-language-tools
-```
-
-## Usage
-
-```js
-import init, { formatSource } from 'decodal-language-tools';
-
-await init();
-
-const result = JSON.parse(formatSource('Server={# host\nhost=String default "localhost";};'));
-
-if (result.ok) {
-  console.log(result.source);
-} else {
-  console.error(result.error);
-}
-```
-
-The exported functions return JSON strings so callers can handle diagnostics without depending on Rust data structures.

packages/decodal-language-tools/jsr.json

@@ -1,15 +0,0 @@
-{
-  "name": "@hare/decodal-language-tools",
-  "version": "0.1.2",
-  "license": "MIT",
-  "exports": "./decodal_language_tools.js",
-  "publish": {
-    "include": [
-      "README.md",
-      "decodal_language_tools.js",
-      "decodal_language_tools.d.ts",
-      "decodal_language_tools_bg.wasm",
-      "package.json"
-    ]
-  }
-}

packages/decodal-language-tools/package.json

@@ -1,27 +0,0 @@
-{
-  "name": "decodal-language-tools",
-  "type": "module",
-  "description": "Source-level language tooling for Decodal.",
-  "version": "0.1.2",
-  "license": "MIT OR Apache-2.0",
-  "repository": {
-    "type": "git",
-    "url": "https://gitea.hareworks.net/Hare/Decodal"
-  },
-  "files": [
-    "decodal_language_tools_bg.wasm",
-    "decodal_language_tools.js",
-    "decodal_language_tools.d.ts"
-  ],
-  "main": "decodal_language_tools.js",
-  "types": "decodal_language_tools.d.ts",
-  "sideEffects": [
-    "./snippets/*"
-  ],
-  "keywords": [
-    "decodal",
-    "formatter",
-    "lsp",
-    "tree-sitter"
-  ]
-}

site/decodal-site/package-lock.json

@@ -16,7 +16,6 @@
         "astro": "^4.16.18",
         "codemirror": "^6.0.2",
         "decodal-codemirror": "file:../../packages/decodal-codemirror",
-        "decodal-language-tools": "file:../../packages/decodal-language-tools",
         "decodal-wasm": "file:../../packages/decodal-wasm",
         "marked": "^12.0.2"
       },
@@ -26,23 +25,21 @@
       }
     },
     "../../packages/decodal-codemirror": {
-      "version": "0.1.3",
+      "version": "0.1.4",
       "license": "MIT OR Apache-2.0",
       "devDependencies": {
         "@codemirror/language": "^6.12.4",
+        "@codemirror/view": "^6.43.6",
         "@lezer/highlight": "^1.2.3",
         "@lezer/lr": "^1.4.10"
       },
       "peerDependencies": {
         "@codemirror/language": "^6.12.4",
+        "@codemirror/view": "^6.43.6",
         "@lezer/highlight": "^1.2.3",
         "@lezer/lr": "^1.4.10"
       }
     },
-    "../../packages/decodal-language-tools": {
-      "version": "0.1.2",
-      "license": "MIT OR Apache-2.0"
-    },
     "../../packages/decodal-wasm": {
       "version": "0.1.2",
       "license": "MIT OR Apache-2.0"
@@ -3265,10 +3262,6 @@
       "resolved": "../../packages/decodal-codemirror",
       "link": true
     },
-    "node_modules/decodal-language-tools": {
-      "resolved": "../../packages/decodal-language-tools",
-      "link": true
-    },
     "node_modules/decodal-wasm": {
       "resolved": "../../packages/decodal-wasm",
       "link": true

site/decodal-site/package.json

@@ -8,7 +8,7 @@
     "build": "astro build",
     "preview": "astro preview --host 0.0.0.0",
     "build:runtime": "wasm-pack build ../../crates/decodal-wasm --target web --out-dir ../../packages/decodal-wasm --release && node scripts/prepare-wasm-package.mjs",
-    "build:tools": "wasm-pack build ../../crates/decodal-language-tools --target web --out-dir ../../packages/decodal-language-tools --release && node scripts/prepare-language-tools-package.mjs",
+    "build:tools": "wasm-pack build ../../crates/decodal-language-tools --target web --out-dir ../../packages/decodal-codemirror/wasm --release && node scripts/prepare-codemirror-wasm.mjs",
     "build:wasm": "npm run build:runtime && npm run build:tools",
     "build:packages": "npm run build:wasm",
     "deploy": "npm run build && node scripts/deploy-pages.mjs",
@@ -23,7 +23,6 @@
     "astro": "^4.16.18",
     "codemirror": "^6.0.2",
     "decodal-codemirror": "file:../../packages/decodal-codemirror",
-    "decodal-language-tools": "file:../../packages/decodal-language-tools",
     "decodal-wasm": "file:../../packages/decodal-wasm",
     "marked": "^12.0.2"
   },

site/decodal-site/scripts/prepare-codemirror-wasm.mjs

@@ -0,0 +1,12 @@
+import { rmSync, writeFileSync } from 'node:fs';
+import { resolve } from 'node:path';
+
+const wasmDir = resolve(import.meta.dirname, '../../../packages/decodal-codemirror/wasm');
+
+rmSync(resolve(wasmDir, 'README.md'), { force: true });
+rmSync(resolve(wasmDir, 'package.json'), { force: true });
+
+writeFileSync(
+  resolve(wasmDir, '.gitignore'),
+  '# wasm-pack output is committed as part of decodal-codemirror.\n',
+);

site/decodal-site/scripts/prepare-language-tools-package.mjs

@@ -1,76 +0,0 @@
-import { writeFileSync } from 'node:fs';
-import { resolve } from 'node:path';
-
-const packageDir = resolve(import.meta.dirname, '../../../packages/decodal-language-tools');
-
-writeFileSync(
-  resolve(packageDir, '.gitignore'),
-  '# wasm-pack output is committed for the browser language tools package.\n',
-);
-
-writeFileSync(
-  resolve(packageDir, 'README.md'),
-  `# decodal-language-tools
-
-Source-level language tooling for Decodal.
-
-This package is generated from the Rust crate in \`crates/decodal-language-tools\` with \`wasm-pack\`.
-It currently exposes a comment-preserving formatter that is shared by the official playground and future editor/LSP tooling.
-
-Use \`decodal-codemirror\` separately when you need CodeMirror 6 language support.
-Use \`decodal-wasm\` separately when you need to evaluate Decodal in a browser.
-
-## Install
-
-\`\`\`sh
-npm install decodal-language-tools
-\`\`\`
-
-JSR:
-
-\`\`\`sh
-deno add jsr:@hare/decodal-language-tools
-\`\`\`
-
-## Usage
-
-\`\`\`js
-import init, { formatSource } from 'decodal-language-tools';
-
-await init();
-
-const result = JSON.parse(formatSource('Server={# host\\nhost=String default "localhost";};'));
-
-if (result.ok) {
-  console.log(result.source);
-} else {
-  console.error(result.error);
-}
-\`\`\`
-
-The exported functions return JSON strings so callers can handle diagnostics without depending on Rust data structures.
-`,
-);
-
-writeFileSync(
-  resolve(packageDir, 'jsr.json'),
-  JSON.stringify(
-    {
-      name: '@hare/decodal-language-tools',
-      version: '0.1.2',
-      license: 'MIT',
-      exports: './decodal_language_tools.js',
-      publish: {
-        include: [
-          'README.md',
-          'decodal_language_tools.js',
-          'decodal_language_tools.d.ts',
-          'decodal_language_tools_bg.wasm',
-          'package.json',
-        ],
-      },
-    },
-    null,
-    2,
-  ) + '\n',
-);

site/decodal-site/src/pages/index.astro

@@ -33,8 +33,7 @@ Production = Server & {
     </a>
     <ul class="package-list">
       <li>decodal-wasm on <a href="https://jsr.io/@hare/decodal-wasm@0.1.2">jsr</a> / <a href="https://www.npmjs.com/package/decodal-wasm">npm</a></li>
-      <li>decodal-language-tools on <a href="https://jsr.io/@hare/decodal-language-tools@0.1.2">jsr</a> / <a href="https://www.npmjs.com/package/decodal-language-tools">npm</a></li>
+      <li>decodal-codemirror on <a href="https://jsr.io/@hare/decodal-codemirror@0.1.4">jsr</a> / <a href="https://www.npmjs.com/package/decodal-codemirror">npm</a></li>
-      <li>decodal-codemirror on <a href="https://jsr.io/@hare/decodal-codemirror@0.1.3">jsr</a> / <a href="https://www.npmjs.com/package/decodal-codemirror">npm</a></li>
     </ul>
   </section>
 
@@ -49,8 +48,8 @@ Production = Server & {
     <article class="home-card">
       <h2>Where it runs</h2>
       <p>
-        Use the Rust crate for runtime embedding, the WASM package in browsers, language-tools for formatting,
+        Use the Rust crate for runtime embedding, the WASM package in browsers, CodeMirror/Lezer for Web editing
-        CodeMirror/Lezer for the Web editor, and Tree-sitter for general editor integration.
+        and formatting, and Tree-sitter for general editor integration.
       </p>
     </article>
     <article class="home-card">

site/decodal-site/src/scripts/playground.js

@@ -1,10 +1,10 @@
 import initRuntime, { evaluateProject } from 'decodal-wasm';
 import runtimeWasmUrl from 'decodal-wasm/decodal_wasm_bg.wasm?url';
-import initTools, { formatSource } from 'decodal-language-tools';
-import toolsWasmUrl from 'decodal-language-tools/decodal_language_tools_bg.wasm?url';
 import { EditorView, basicSetup } from 'codemirror';
 import { keymap } from '@codemirror/view';
 import { decodal } from 'decodal-codemirror';
+import { formatDecodal, initDecodalFormatter } from 'decodal-codemirror/format';
+import toolsWasmUrl from 'decodal-codemirror/wasm/decodal_language_tools_bg.wasm?url';
 import { playgroundExamples } from './playground-examples.js';
 
 const STORAGE_KEY = 'decodal-playground-project-v2';
@@ -221,7 +221,7 @@ function execute() {
 }
 
 function formatActiveFile() {
-  const result = JSON.parse(formatSource(getEditorText()));
+  const result = formatDecodal(getEditorText());
   if (!result.ok) {
     output.textContent = result.error;
     output.classList.add('error');
@@ -285,7 +285,7 @@ function compareNodes(a, b) {
 }
 
 try {
-  await Promise.all([initRuntime(runtimeWasmUrl), initTools(toolsWasmUrl)]);
+  await Promise.all([initRuntime(runtimeWasmUrl), initDecodalFormatter(toolsWasmUrl)]);
   run.disabled = false;
   formatButton.disabled = false;
   status.textContent = '';