GLRM grammar format

GLRMask supports general context-free grammars, including ambiguous ones. It accepts JSON schemas, GLRM, Lark, and EBNF.

GLRM is GLRMask’s native grammar format. This complete v1 grammar accepts an integer or null, with optional whitespace:

glrm 1;
start value;

t WS = /[ \t\r\n]+/;
ignore WS;

t NUMBER = /-?(0|[1-9][0-9]*)/;
nt value = NUMBER | "null";

Declarations use assignment = and terminate with a semicolon ;. Whitespace and newlines separate tokens without automatic semicolon insertion. Comments use // for single-line comments and /* ... */ for block comments.

Grammar structure and rules

A GLRM document contains declarations evaluated within a closed scope:

• start <name>;: Declares the root rule of the scope. Exactly one start declaration is required per grammar or subgrammar.
• nt <name> = <expr>;: Declares a context-free nonterminal rule. Nonterminal rules may be mutually recursive.
• t <name> = <expr>;: Declares an emitting lexical terminal rule.
• internal t <name> = <expr>;: Declares a non-emitting helper terminal recognized only when referenced by other terminal expressions.
• ignore <terminal>;: Directs the parser to permit the named emitting terminal implicitly between lexical atoms and at grammar boundaries.

Definitions are scope-local. Use unique names within a scope; a child can reuse a parent’s names for its own rules. A subgrammar exposes its start language to its parent, rather than exporting individual rules.

Identifiers start with an ASCII letter or underscore and continue with ASCII letters, digits, or underscores. References can precede their definitions. Terminal expressions can reference other terminal rules, including internal t helpers, but cannot reference nonterminal rules or special tokens. Internal terminals cannot be referenced directly by parser rules.

For example, this left-recursive grammar accepts n+n+n with both possible groupings:

glrm 1;
start expr;
nt expr = expr "+" expr | "n";

GLRM’s expression precedence controls how the grammar description is read. Precedence or associativity in the language being described must be expressed through its rules.

Epsilon and non-empty constraints

Epsilon is written explicitly with the keyword eps. GLRM v1 rejects empty declaration bodies, empty alternatives, and empty groups:

nt optional_sign = "+" | "-" | eps;

Writing nt empty = ; or nt sign = "+" | ; triggers a parser error. Optional expressions such as sign? can also derive the empty string.

Operator precedence and expressions

GLRM evaluates expressions according to strict precedence tiers, from lowest to highest:

Subtraction restrictions

In byte-terminal expressions, - and & compute regular language difference and intersection. For example, t NON_KEYWORD = IDENT - ("if" | "else"); excludes those two complete words from IDENT.

When the left operand of - names a nonterminal, subtraction removes structurally matching alternatives from that rule. It does not compute arbitrary context-free language difference. Intersection of arbitrary nonterminal languages is unsupported.

The right-hand side of a sequence subtraction must be parenthesized:

nt A = "a" | "b" "c";
nt without_pair = A - ("b" "c");

Here without_pair accepts only "a". Writing A - B C causes a syntax error; use A - (B C) when subtracting a sequence.

Separated sequences

Separated sequences use the tilde operator ~ or ~+ followed by parenthesized items:

t COMMA = ",";
nt item = "a" | "b";
nt items = COMMA ~ (item*);
nt required_items = COMMA ~+ (item*);

For this non-nullable item, items accepts the empty string, a, or a,b,a; required_items requires at least one item. Neither adds a trailing comma.

• sep ~ (items...) permits the whole sequence to derive empty if its item expressions allow epsilon.
• sep ~+ (items...) enforces that at least one item must be structurally present. Even when individual items are nullable, their presence satisfies the non-empty requirement for separator placement.

Required nullable items still participate in separator placement. Consequently, structural presence is different from consuming non-empty text.

Only one postfix quantifier may attach to an item inside a separated sequence. sep ~ (expr+) separates repeated occurrences of expr; sep ~ ((expr+)? ) instead treats the inner repetition as one optional item. To quantify the entire separated sequence, put it in a named rule and quantify the reference.

Terminals and literals

GLRM distinguishes between byte-level lexical terminals and context-free nonterminals.

Byte literals and escapes

String literals can use either double quotes ("...") or single quotes ('...'). Both produce byte strings with identical semantics. Literal Unicode text contributes its UTF-8 bytes.

GLRM processes the following escape sequences:

• \n — line feed (0x0A)
• \r — carriage return (0x0D)
• \t — horizontal tab (0x09)
• \\ — backslash (0x5C)
• \" — double quote inside a double-quoted literal
• \' — single quote inside a single-quoted literal
• \xHH — arbitrary byte from two hexadecimal digits (0x00–0xFF)

Any unrecognized escape \c preserves both the backslash and the following character verbatim as two literal bytes.

Character classes and any byte

Character classes define sets of matching bytes:

• [a-z0-9] matches any byte in the specified ASCII ranges.
• [^a-z] inverts the set, matching any byte outside the class.
• . matches any single byte (0x00 through 0xFF).
• [é-ê]/utf8 matches one UTF-8 character in that range. The suffix distinguishes character matching from raw byte classes.

Regular expressions

Regular expressions are written between forward slashes /.../ and are permitted only inside terminal declarations (t or internal t rules):

t IDENT = /[A-Za-z_][A-Za-z0-9_]*/;

Regex terminals match a complete lexeme, not a substring search. Forward slashes inside the pattern are escaped as \/. Raw regexes use UTF-8 character semantics; standalone GLRM . remains an any-byte atom.

The supported regex dialect has these restrictions:

• Anchors ^ and $ are forbidden because every regex terminal already matches fully against the lexeme.
• Backreferences (\1 through \9) are rejected.
• Lookaround assertions ((?=...), (?!...), (?<=...), (?<!...)) and inline flags are rejected.
• Non-capturing groups (?:...) and named capture groups (?P<name>...) are permitted.
• Escape shorthands \d, \s, and \w (and uppercase negations \D, \S, \W) represent their standard ASCII character classes. Unicode codepoints use \uHHHH or \UHHHHHHHH.

Lexer mechanics and ignore

Per-terminal longest match

GLRMask uses longest-match semantics separately for each terminal: each candidate terminal takes its longest accepting run of input bytes. Live partial lexemes remain viable while more input can extend them.

Competing terminal alternatives are retained simultaneously across active parser states. The lexer does not choose a single global winner across distinct terminal names; ambiguous terminal interpretations fork and survive in the parser graph until subsequent input resolves them.

Ignore terminals

The ignore directive names an emitting terminal skipped between tokens:

t WS = /[ \t\r\n]+/;
ignore WS;

The ignored terminal must be a local, emitting terminal defined in the same scope. It must not be nullable, and it cannot be referenced explicitly within nonterminal productions. During grammar lowering, the ignore terminal is automatically interleaved around lexical atoms.

Special tokens

Model vocabulary items or synthetic control boundaries that should not be matched through byte scanning are declared as external tokens:

glrm 1;
start message;

extern token TOOL_CALL;

nt message = TOOL_CALL call;
nt call = "lookup()";

An extern token has no byte-level definition and is valid only in nonterminal expressions or nonterminal finite automata edges. It cannot be used inside terminal (t) rules or as an ignore terminal. Legacy numeric @token(id) syntax is rejected in GLRM v1.

Bind the external token by name to one or more exact token IDs before compilation in Python:

grammar = glrmask.Grammar.from_glrm(source)

constraint = grammar.bind("TOOL_CALL", vocab.token(tool_call_token_id)).compile(vocab)

Special tokens match exact integer token IDs directly during decoding. Committing the same decoded bytes cannot satisfy a special token. Multiple IDs bound with vocab.tokens([id_a, id_b]) are interchangeable alternatives, each matching one generated token. Missing or empty bindings and bindings for undeclared names are errors.

Subgrammars

GLRM supports modular grammar composition using inline subgrammars or externally bound child grammars. Subgrammars operate in closed scopes, meaning parent rules, externals, and ignore terminals do not leak into the child.

Inline subgrammars

Inline subgrammars are declared with g <name> = { ... };:

glrm 1;
start document;

g inner = {
    start value;
    nt value = "null";
};

nt document = "[" inner "]";

The child grammar defines its own start rule. Referencing the subgrammar name in the parent invokes the child’s start language.

External subgrammars

Declare a child supplied by the host with extern grammar:

glrm 1;
start document;

extern grammar payload;

nt document = "<" payload ">";

External subgrammars are bound to child grammars or compiled constraints in Python:

parent = glrmask.Grammar.from_glrm(parent_source)
payload = glrmask.Grammar.from_json_schema(payload_schema)

# Source-level binding
document = parent.bind("payload", payload).compile(vocab)

# Compiled late-binding for reusable parent templates
unlinked_parent = parent.compile_unlinked(vocab)
compiled_child = payload.compile(vocab)
document = unlinked_parent.bind("payload", compiled_child).link()

Source bindings accept Grammar objects, while compiled bindings accept runnable Constraint instances. Nested external names use qualified paths such as inner::CONTROL. For compilation and binding details, see the Python reference.

Direct finite automata

Rules may use an explicit state-machine definition by providing an fa { ... } block as the rule body:

glrm 1;
start member;
nt value = "null";
nt member = fa {
    start 0;
    accept 2;

    0 -> 1: "key";
    1 -> 2: ":" value;
};

States can be non-negative integers or identifiers. Transitions follow the pattern <source> -> <destination>: <expression>;. Epsilon transitions use : eps;.

Nonterminal automata (nt name = fa { ... };) can use nonterminal expressions, terminal references, special tokens, and compound sequences or choices on their edges. Terminal automata (t name = fa { ... };) use lexical expressions and compile directly into the lexer.

Automata must specify at least one start state and at least one accept state.

Optimization pragmas

Compiler hints that do not affect the recognized grammar language are placed inside pragma glrmask blocks:

pragma glrmask {
    lexer group strings = STRING, KEY, @literals;
}

Lexer groups are compiler partition hints. Members can be named emitting terminals, quoted literals, @literals for anonymous literals, or * for otherwise unassigned terminals. They do not introduce lexical priority or change the accepted language. The current parser rejects unknown pragma namespaces and directives.

Legacy compatibility

Unversioned grammars that omit glrm 1; are parsed under the legacy compatibility dialect.

Legacy grammars use ::= instead of =, allow implicit epsilon alternatives (such as empty alternatives in |), support top-level fa name ::= { ... };, and accept numeric @token(id) expressions. The modern versioned syntax requires glrm 1;.

Constraint constructors

GLRMask accepts multiple grammar definitions in Python:

These constructors return immutable Grammar instances. Call grammar.compile(vocab) to produce a runnable Constraint, or grammar.compile_unlinked(vocab) for an unlinked parent template.

JSON Schema

GLRMask supports a subset of JSON Schema. Unsupported features may be rejected.

End tokens

Supply exact end-token IDs when compiling the root grammar:

constraint = grammar.compile(vocab, end_tokens=[eos_id])

The mask allows these IDs only when the grammar body is accepting. Committing a configured end token lets the decoder stop based on its chosen ID. This policy belongs to the final root and is not inherited from compiled children. A special token named in the grammar is ordinary grammar data, even if the model uses that ID for EOS.

Sources

Authoritative compiler and parser implementation details are pinned in IsaacBreen/glrmask@07a8a0ba:

• Parser and AST dumper: crates/glrmask-grammar/src/grammar/glrm.rs
• Regular expression validator: crates/glrmask-lexer/src/automata/lexer/regex.rs
• Subgrammar and external token test suite: tests/glrm_v1.rs
• Structural subtraction tests: tests/parser_grammar_constructor.rs