Decodal/doc/manual/souce/development.md

Development

This document describes the development workflow for Decodal itself.

Rust checks

Run the normal Rust checks from the repository root.

cargo fmt --check
cargo test
cargo check -p decodal --no-default-features
nix flake check

Run the default stdio language server with:

cargo run -q -p decodal-lsp

The protocol integration tests use an in-memory LSP connection and can be run independently:

cargo test -p decodal-lsp

Regex support is optional and should be tested explicitly when touched.

cargo test -p decodal --features regex
cargo run -q -p decodal-cli --features regex -- examples/regex/main.dcdl

crates.io release

The crates.io release contains decodal, decodal-derive, decodal-language-service, decodal-language-tools, and decodal-lsp. decodal-cli and the Rust source crate decodal-wasm remain repository-only packages. The generated WebAssembly package under packages/decodal-wasm/ is published to npm and JSR. The CodeMirror package bundles the generated formatter WebAssembly from decodal-language-tools. The project is dual licensed as MIT OR Apache-2.0.

The authoritative validation commands, dependency order, and publish commands are maintained in RELEASING.md at the repository root.

Web site and playground

The Astro documentation site and browser playground are kept in:

site/decodal-site/

The site imports Markdown files from doc/manual/souce/ and renders them as mdBook-style pages. The playground loads decodal-wasm for evaluation and decodal-codemirror for editor features and source formatting.

Important files:

site/decodal-site/src/pages/docs/[...slug].astro
site/decodal-site/src/pages/playground.astro
site/decodal-site/src/scripts/playground.js
site/decodal-site/src/scripts/playground-examples.js
site/decodal-site/src/layouts/ManualLayout.astro
site/decodal-site/src/lib/docs.js
site/decodal-site/src/lib/highlight.js
packages/decodal-codemirror/src/decodal.js
packages/decodal-codemirror/src/decodal-parser.js
packages/decodal-wasm/decodal_wasm.js
packages/decodal-wasm/decodal_wasm_bg.wasm
packages/decodal-codemirror/wasm/decodal_language_tools.js
packages/decodal-codemirror/wasm/decodal_language_tools_bg.wasm
crates/decodal-wasm/src/lib.rs
crates/decodal-language-tools/src/lib.rs

Build the WebAssembly packages before building the site:

cd site/decodal-site
npm install
npm run build:wasm
npm run build

npm run build:wasm builds both generated WebAssembly packages. npm run build:runtime writes generated runtime files into packages/decodal-wasm/. npm run build:tools writes generated language tools files into packages/decodal-codemirror/wasm/.

These generated files are committed so the site can be built without requiring every consumer to regenerate the wasm packages first.

Publish the generated WebAssembly package from its package directory:

cd packages/decodal-wasm
npm publish
npx jsr publish

Run dry-runs first when preparing a release:

npm pack --dry-run
npx jsr publish --dry-run

The npm package name is decodal-wasm. The JSR package name is @hare/decodal-wasm. The npm package advertises MIT OR Apache-2.0; JSR metadata uses MIT because its publisher requires a single recognized license identifier. Both license files are included in the package.

The playground editor uses the decodal-codemirror package with the generated Lezer parser in packages/decodal-codemirror/src/decodal-parser.js. The canonical grammar is documented in doc/manual/souce/language/grammar.md; regenerate the Lezer parser when that grammar or editors/lezer-decodal/decodal.grammar changes.

Publish the CodeMirror package from its package directory:

cd packages/decodal-codemirror
npm install
npm publish
npx jsr publish

Run dry-runs first when preparing a release:

npm install
npm pack --dry-run
npx jsr publish --dry-run

The npm package name is decodal-codemirror. The JSR package name is @hare/decodal-codemirror. As with decodal-wasm, JSR metadata uses MIT while both the MIT and Apache-2.0 license files remain included.

The documentation build still uses the lightweight JavaScript fallback highlighter so Astro can render Markdown without initializing WASM at build time.

To run the site locally:

cd site/decodal-site
npm run dev

Deploy the static site to Cloudflare Pages with Wrangler direct upload:

cd site/decodal-site
npm run deploy

The deploy script runs npm run build and then uploads dist/ to the Pages project named decodal-site on branch master. Cloudflare Pages treats this as production when the project production branch is master. Use CLOUDFLARE_PROJECT_NAME when deploying to a differently named Pages project.

CLOUDFLARE_PROJECT_NAME=my-pages-project npm run deploy

If the committed WASM package must be regenerated before deploy, use:

npm run deploy:wasm

Tree-sitter grammar

The Tree-sitter grammar is kept in:

editors/tree-sitter-decodal/

Important files:

editors/tree-sitter-decodal/grammar.js
editors/tree-sitter-decodal/queries/highlights.scm
editors/tree-sitter-decodal/queries/locals.scm
editors/tree-sitter-decodal/corpus/basic.txt

The grammar is intended to be portable across editors such as Zed, Neovim, Helix, and Emacs. Zed support should consume this grammar rather than relying on a TextMate grammar.

Tree-sitter commands

From the grammar directory:

cd editors/tree-sitter-decodal
npm install
npx tree-sitter generate
npx tree-sitter test

To inspect a parse tree:

npx tree-sitter parse ../../examples/advanced/main.dcdl

The generated parser files under editors/tree-sitter-decodal/src/ are committed so editor integrations can consume the grammar without regenerating it first. node_modules/ is ignored and must not be committed.

Updating the grammar

When the Decodal syntax changes:

1. Update the canonical EBNF in doc/manual/souce/language/grammar.md.
2. Update the Rust parser/lexer as needed.
3. Update editors/tree-sitter-decodal/grammar.js and run npx tree-sitter generate / npx tree-sitter test.
4. Update editors/lezer-decodal/decodal.grammar and regenerate the CodeMirror parser:

cd site/decodal-site
npx lezer-generator ../../editors/lezer-decodal/decodal.grammar -o ../../packages/decodal-codemirror/src/decodal-parser.js

5. Add or update corpus/tests/examples.
6. Run Rust and site checks from the repository root.

Development shell

The Nix development shell includes Rust tooling, Node.js, Tree-sitter CLI tooling, and wasm-pack tooling.

nix develop

The shell provides:

• cargo
• rustc
• rustfmt
• clippy
• node
• npm
• wasm-pack
• lld
• tree-sitter
• nixfmt