mike-lischke / antlr4ng Goto Github PK

We found that after importing antlr4ng into our project the resulting bundle becomes invalid. The error is quite cryptic

TypeError: Cannot read properties of undefined (reading 'Symbol(immer-state)')

We use redux toolkit (and therefore - immerjs) on the project, but the stacktrace somehow points to

var Gt = class o {
  static DEFAULT = new o; // <--this line
  walk(e, t) {
    if (t instanceof Q)
      e.visitErrorNode(t);
    else if (t instanceof F)
      e.visitTerminal(t);
    else {
      let n = t;
      this.enterRule(e, n);
      for (let i = 0; i < t.getChildCount(); i++)
        this.walk(e, t.getChild(i));
      this.exitRule(e, n)
    }
  }

which is the ParseTreeWalker.

To fix it we had to disable "compress -> unused" option from Terser.

So clearly there's something special about the library it confuses the Terser. We tried to produce it on a clean repo, but with no luck.

I do believe this is a Terser bug, so nothing to do here. Perhaps the report would be helpful for someone with the same problem.

My grammar uses a semantic predicate

TEXT: ('{' { String.fromCharCode(this._input.LA(1)) !== "{" }? | ~[{] )+ ;

Moving from antlr4's "native" Javascript runtime to antlr4ng Typescript runtime I noticed that the generated Lexer is complaining because this._input does not exist.

By looking at Lexer.d.ts I figured out that the property is not included, was this intentional?

Moreover, I have noticed that _input is also stored in a _tokenFactorySourcePair tuple. Can i use _tokenFactorySourcePair[1] to obtain my _input ?

I can't find a way to use antlr4ng in an Angular app. I get the following:

✖ Compiling with Angular sources in Ivy partial compilation mode.
../node_modules/antlr4ng/dist/CharStreams.d.ts:1:23 - error TS1452: 'resolution-mode' assertions are only supported when `moduleResolution` is `node16` or `nodenext`.

1 /// <reference types="node" resolution-mode="require"/>
                        ~~~~

If I set "moduleResolution" to "node16" or "nodenext" everything breaks in the build (it looks like all files are interpreted as CommonJS modules, while they aren't).

I've verified I have no references to CharStreams in my code, but it's exported from the "antlr4ng" module (index.d.ts).
Would it be possible to make it optional (at least the parts that depend on Node APIs)?

While I was investingating problems with bundle compilation, I've noticed we have Xpath lexer/parser bundled with the library, including its huge serialized ATN.

Unfortunately this code is not removed during dead code elimination phase in webpack. Can we move it to a separate package/project, so only those who actually need it would install and import it?

I'm not sure if it's right place to ask, since the library is a port of https://github.com/antlr/antlr4. There was even a similar proposal there.

"antlr4": "4.13.1",
"antlr4-c3": "3.3.5",
"antlr4ng": "2.0.10",

The location of the error：

I guess the corresponding code position should be：

I see a similar code in antlr4-c3 as well.

It's mine tsconfig.json:

{
  "compileOnSave": false,
  "compilerOptions": {
    "baseUrl": "./",
    "outDir": "./dist/out-tsc",
    "incremental": true,
    "forceConsistentCasingInFileNames": true,
    "noFallthroughCasesInSwitch": true,
    "sourceMap": true,
    "declaration": false,
    "downlevelIteration": true,
    "experimentalDecorators": true,
    "moduleResolution": "node",
    "importHelpers": true,
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true,
    "target": "es2015",
    "module": "ESNext",
    "lib": [
      "DOM",
      "ESNext"
    ]
  },
  "angularCompilerOptions": {
    "enableI18nLegacyMessageIdFormat": false,
    "strictInjectionParameters": true,
    "strictInputAccessModifiers": true,
    "strictTemplates": true
  }
}

tsconfig.worker.json:

{
  "extends": "./tsconfig.json",
  "compilerOptions": {
    "outDir": "./out-tsc/worker",
    "lib": [
      "ES2021",
      "webworker"
    ],
    "types": ["node"]
  },
  "include": [
    "src/**/*.worker.ts"
  ]
}

I think it can be solved this way：

class Parent {
  constructor(arg = null) {
    // ...
  }
}

class Child extends Parent {
  constructor(e) {
    super(e || someDefaultValue); // e is used if it exists, otherwise the default value is used
  }
}

Can we have a documentation on how to use this package as well as code examples?

I have a generated lexer:

import * as antlr from "antlr4ng";
import { Token } from "antlr4ng";
export class MyLexer extends antlr.Lexer {
...
}

Exposed by the root index.ts:

export { MyLexer as Lexer } from "./generated/MyLexer";

I deploy it on npm thanks to tsc, with this tsconfig.json file:

{
	"compilerOptions": {
		"module": "CommonJS",
		"moduleResolution": "node",
		"target": "ES2020",
		"lib": ["es2022"],
		"declaration": true,
		"sourceMap": true,
		"outDir": "./dist",
		"newLine": "LF",
		"noUnusedLocals": false,
		"noUnusedParameters": false,
		"incremental": true,
		"strict": true,
		"downlevelIteration": true,
		"jsx": "react-jsx",
		"noFallthroughCasesInSwitch": true,
		"strictPropertyInitialization": false
	},
	"include": ["src"]
}

I consume this package in a simple CRA app and have this error : Class extends value undefined is not a constructor or null

Any idea?

Use case is reproductible:

• clone https://github.com/Making-Sense-Info/ANTLR-Editor
• cd ANTLR-Editor
• yarn
• run simultaneously yarn link-in-app and yarn start-test-app

Just trying to eliminate the various diffs I am seeing in parser trace between Antlr4ng, CSharp, Java, JavaScript, in order to see why codeql/examples/alias.qll does not parse for Antlr4ng but does for all other targets.

The DFAState.statenNumber field is not being set in Antlr4ng here, but it is in all the other targets (CSharp, Java, JavaScript).

When i used antlr4ng in Mocha's tests, it will run into errors:

[...]/src/parser/ExprParser.ts:28
    public static readonly Colon = 14;
                           ^
Error [ERR_REQUIRE_ESM]: require() of ES Module [...]/node_modules/.pnpm/[email protected][email protected]/node_modules/antlr4ng/dist/antlr4.mjs not supported.
Instead change the require of [...]/node_modules/.pnpm/[email protected][email protected]/node_modules/antlr4ng/dist/antlr4.mjs to a dynamic import() which is available in all CommonJS modules.
    at Object.<anonymous> ([...]/src/parser/ExprParser.ts:28:28)
    at m._compile ([...]/node_modules/.pnpm/[email protected][email protected]/node_modules/ts-node/dist/index.js:556:29)
    at require.extensions.<computed> [as .ts] ([...]/node_modules/.pnpm/[email protected][email protected]/node_modules/ts-node/dist/index.js:558:16)
    at Object.<anonymous> ([...]/packages/expr/src/core/parse.ts:27:24)
    at m._compile ([...]/node_modules/.pnpm/[email protected][email protected]/node_modules/ts-node/dist/index.js:556:29)
    at require.extensions.<computed> [as .ts] ([...]/node_modules/.pnpm/[email protected][email protected]/node_modules/ts-node/dist/index.js:558:16)
    at Object.<anonymous> ([...]/packages/expr/tests/core.test.ts:3:17)
    at m._compile ([...]/node_modules/.pnpm/[email protected][email protected]/node_modules/ts-node/dist/index.js:556:29)
    at require.extensions.<computed> [as .ts] ([...]/node_modules/.pnpm/[email protected][email protected]/node_modules/ts-node/dist/index.js:558:16)
    at [...]/node_modules/.pnpm/[email protected]/node_modules/mocha/lib/mocha.js:334:36
    at Array.forEach (<anonymous>)
    at Mocha.loadFiles ([...]/node_modules/.pnpm/[email protected]/node_modules/mocha/lib/mocha.js:331:14)
    ......

As I know, Mocha only support CommonJS, Does it means that we cannot use antlr4ng in UnitTests?

Thanks~

I'm new to antlr and trying to use antlr4ng in a Typescript project. I've been able to successfully generate the Expression grammar from the README, as well as a Kotlin grammar (related to antlr4-c3).

I'm now trying to leverage the Python or JavaScript grammars defined in the antlr/grammars-v4 repo (e.g., JavaScript). However, it seems like to make them work requires additional base components, like JavaScriptParserBase.

Are these base components compatible with antlr4ng, or do they need to be converted to work with antlr4ng? I'm seeing errors when I try, but I'm not sure if I'm doing something wrong, or if they're not intended to be compatible, or something else.

Just did a clone and trying to build antlr4ng. I'm getting a compilation error.

$ node --version
v21.7.1
03/15-08:23:49 ~/temp/antlr4ng
$ npm run build

> [email protected] build
> tsc && npm run build-cjs && npm run build-mjs

src/misc/ParseCancellationException.ts:17:15 - error TS2339: Property 'captureStackTrace' does not exist on type 'ErrorConstructor'.

17         Error.captureStackTrace(this, ParseCancellationException);
                 ~~~~~~~~~~~~~~~~~

src/RecognitionException.ts:53:19 - error TS2339: Property 'captureStackTrace' does not exist on type 'ErrorConstructor'.

53         if (Error.captureStackTrace) {
                     ~~~~~~~~~~~~~~~~~

src/RecognitionException.ts:54:19 - error TS2339: Property 'captureStackTrace' does not exist on type 'ErrorConstructor'.

54             Error.captureStackTrace(this, RecognitionException);
                     ~~~~~~~~~~~~~~~~~


Found 3 errors in 2 files.

Errors  Files
     1  src/misc/ParseCancellationException.ts:17
     2  src/RecognitionException.ts:53
03/15-08:23:57 ~/temp/antlr4ng
$ tsc --version
Version 5.4.2
03/15-08:29:29 ~/temp/antlr4ng

I have the latest of node and tsc for Windows 11.

When building a listener, antlr4ng creates the following code (here the language is called FormulaLang and the start node is "formula", so the visitor function is visitFormula):

export class FormulaLangVisitor<Result> extends AbstractParseTreeVisitor<Result> {
    /**
     * Visit a parse tree produced by `FormulaLangParser.formula`.
     * @param ctx the parse tree
     * @return the visitor result
     */
    visitFormula?: (ctx: FormulaContext) => Result;

Note that the visitFormula is a property holding a function, not a function itself. When I follow the instructions in the readme, it indicates I should be building a function body:

class FormulaEvalVisitor extends FormulaLangVisitor<dfd.Series> {
    visitFormula(ctx: FormulaContext) : dfd.Series {
        if (ctx.where_list() != null) this.visit(ctx.where_list());
        return this.visit(ctx.expr());
     };
}

However, typescript throws an error saying that FormulaLangVisitor defines a property and I'm trying to create a function. Instead it wants something like:

class FormulaEvalVisitor extends FormulaLangVisitor<dfd.Series> {
    declare visitFormula = (ctx: FormulaContext) => { <code> }

Which should I be doing? Is the documentation incorrect or the generator? Or (shudder) is it me?

Description

In Webpack's production mode, if the optimization.minimize option is turned on (Webpack has this option turned on by default), will get an error.

It looks like it might be an issue with the compression tool, but I feel the need to report this issue to you, because webpack is very widely used.

Minimal demo to reproducing the problem

https://github.com/HaydenOrz/antlr4ng-trino

The good news is that Antlr4ng performs well in all of these situations:

• NodeJS
• Vite
• Webpack development mode
• webpack production mode and turn the optimization.minimize option off

I noticed that antlr4ng uses BigInts. BigInts require ES2020.

BigInts are not lowered to older syntax in esbuild evanw/esbuild#732 . If I set esbuild's target option to be lower than es2020, then esbuild throws an exception about not being able to convert bigint. Babel also has a similar issue .

Whether to consider replacing Bigint and BigUint64Array with number and Uint16Array.

This has the potential to affect the performance of antlr4ng, and in general the long type in java should indeed correspond to the javascript Bigint and BigUint64Array (since both are 64-bit). But there is currently no good solution to the compatibility problem.

• Testing Antlr4ng with all of grammars-v4 (I am adding the target to testing, started with antlr/grammars-v4#4010).
• Antlr4ng fails on grammar https://github.com/antlr/grammars-v4/tree/master/codeql with alias.qll. Output:

$ bash run.sh ../examples/alias.qll -trace > trace.out
line 1:0 mismatched input 'module' expecting {'predicate', 'abstract', 'cached', 'external', 'extensible', 'final', 'transient', 'library', 'private', 'deprecated', 'override', 'additional', 'query', 'pragma', 'language', 'bindingset', QL_DOC}
TypeScript 0 ../examples/alias.qll fail 0.131
Total Time: 0.134
03/15-06:55:25 ~/temp/g4-antlr4ng/codeql/Generated-Antlr4ng

• All other targets parse this input file without error.
• Parser.setTrace(true) works, but I really need the trace_atn_sim?: boolean; flag and associated output statements in order to compare ATN sets. This output has been extremely valuable in fixing bugs in several targets.

Problem: Parser's getter syntaxErrorsCount at src/Parser.js:302 is inconsistent with type src/Parser.d.ts:53 syntaxErrorCount.
This results in typescript not building unless adding @ts-ignore + autocomplete is wrong

Solution: Change either of them, but I would change the syntaxErrorsCount to syntaxErrorCount to be consistent with antlr4ts

Thanks for your work :)

We encountered an issue when upgrading from antlr4ts and antlr4-c3 to their latest versions at a time (antlr4-c3 v3.3.5 and antlr4ng v2.0.4). While there were no issues in Angular (v17) development mode, we encountered a runtime error in a production mode with the scripts optimization flag set to true in angular.json. The error message was: "ReferenceError: u0 is not defined." The error points to the following part of the minified JavaScript code:

yv =
          (new u0(),
          class {
            name = "";
            index = 0;
            data;
            constructor(t) {
              let e = [];
              for (let n of t) e.push(n.codePointAt(0));
              this.data = new Uint32Array(e);
            }
            reset() {
              this.index = 0;
            }
            consume() {
              if (this.index >= this.data.length)
                throw new Error("cannot consume EOF");
              this.index += 1;
            }
            LA(t) {
              if (0 === t) return 0;
              t < 0 && (t += 1);
              let e = this.index + t - 1;
              return e < 0 || e >= this.data.length ? F.EOF : this.data[e];
            }
            mark() {
              return -1;
            }
            release(t) {}
            seek(t) {
              t <= this.index
                ? (this.index = t)
                : (this.index = Math.min(t, this.data.length));
            }
            getText(t, e) {
              let n, r;
              return (
                t instanceof Pe
                  ? ((n = t.start), (r = t.stop))
                  : ((n = t), (r = e ?? this.data.length - 1)),
                r >= this.data.length && (r = this.data.length - 1),
                n >= this.data.length ? "" : this.#e(n, r + 1)
              );
            }
            toString() {
              return this.#e(0);
            }
            get size() {
              return this.data.length;
            }
            getSourceName() {
              return this.name ? this.name : pc.UNKNOWN_SOURCE_NAME;
            }
            #e(t, e) {
              let n = this.data.slice(t, e),
                r = "";
              return (
                n.forEach((i) => {
                  r += String.fromCodePoint(i);
                }),
                r
              );
            }
          }),

In our code, we initialize the lexer and parser as follows:

const inputCharStream = CharStreams.fromString(input);
const lexer = new FilterQueryLexer(inputCharStream);
const parser = new FilterQueryParser(new CommonTokenStream(lexer));

The methods from the problematic section of the JavaScript code are defined in the antlr4ng CharStream.

We attempted to upgrade antlr4ng to version 2.0.8 to address the issue but encountered a compile error as described in #26
Then, we made a temporary workaround: Changed default[] to any[] as suggested in the mentioned issue, but the error persisted (slightly different javascript code: new u0(), class is now new u0, class

Instead BufferedTokenStream.d.ts declares functions hiddenTokensToLeft and hiddenTokensToRight which do not exist

When running in WebWorder, it takes about 4 seconds to parse the same SQL statement (Oracle) in version 2.0 and about 15 seconds in version 3.0

My code:

    this._charStream = CharStream.fromString(input);
    this._lexer = new PlSqlLexer(this._charStream);
    this._lexer.removeErrorListeners();
    this._lexer.addErrorListener(new ParseErrorListener(this._parseErrors));

    this._tokenStream = new CommonTokenStream(this._lexer);

    this._parser = new PlSqlParser(tokenStream);

    parser.removeErrorListeners();
    parser.addErrorListener(new ParseErrorListener(this._parseErrors));
    console.time('root');
    // @ts-ignore
    this._parseTree = this._parser.root();
    console.timeEnd('root');

SQL statement tested: ORACLE's package SYS. DIUTIL

If a larger SQL is larger, the webworker will be automatically stopped, and the webwoker will be automatically recycled when it resolves to about 18,000 tokens in 2.0, and can only be resolved to 5,000 tokens in 3.0

I feel that the 3.0 version of the token will take up more memory

SQL statement tested: ORACLE's package SYS. STANDARD

As seen in this screenshot:

The generated code is incompatible with the declared method "getRuleContext", e.g. Argument of type 'typeof IdentifierContext' is not assignable to parameter of type 'new (...args: unknown[]) => IdentifierContext'.ts(2345)

First of all, thank you for your excellent work. Both antlr4ng and antlr4-c3 are great.

Description

I'm trying to migrate from antlr4ts to antlr4ng. DTStack/dt-sql-parser#261
Then I had some problems. The parser I generated directly via antlr4ng-cli comes with some type errors.

Looks like the antlr. is missing in front of the ParserRuleContext.

Maybe it's because I'm having a problem with my grammar file, but I need help.

Minimal demo to reproducing the problem

https://github.com/HaydenOrz/antlr4ng-trino

Pretty much whats in the title.
I wanted to generate the parser/lexer and visitor for this language:

• https://github.com/antlr/grammars-v4/tree/master/kuka

And used this cli command to generate the code (as was shown in the README):

antlr4ng -Dlanguage=TypeScript -visitor -listener src/grammar/krl/krl.g4

Also a potentionally stupid question but the grammar and lexer/parser clearly specify the loops and for loop, but I don't see it in the visitor as an option to visit? Is that a bug or is there another way to do that?

Steps to Reproduce:

1. Create a grammar with parser rule getText
2. Run antlr4ng
3. Attempt to compile the resulting Parser.ts file
4. Observe many tsc compile errors in the Parser file, indicating incompatible symbols (at first I thought this was a typescript strict issue, as the incompatibilty is with a type union that includes null)

As an antlr4 newbie, I was bit by the above this morning. I am guessing that this is because of a naming conflict with the getText method that the parser class inherits. I am not sure if this is something that antlr4ng should handle differently - perhaps it could output a warning or an error if a parser rule conflicts with a native method? At any rate, I wanted to document the gotcha in case anyone else ran into it and became confused, as I was. Feel free to resolve if no changes to antlr4ng are prudent.

Thanks for the great tool!

I saw your announcement in antlr-discussions. I updated the Trash templates to use the new versions (runtime 3.0.3, tool 2.0.0), but couldn't reproduce the 2x speed up. By chance, I looked at your .github actions workflow and noticed that you use Nodejs version 20. I updated my installation of Nodejs on Windows 11 to 20.11.1--the only thing I changed--and re-tested. Only then I saw the 2x speed up. In fact, under Nodejs 16.something, antlr4ng was slower than the old antlr4 runtime.

You might want to reproduce the slow times under the old version. If you also find the issue, make sure to mention that in the readme. If possible, can you force a specific version of Nodejs and tsc engines in your package.json in order to get this speed up?

Again, great work you are doing. Ty.

I was looking at the available documentation offered by Antlr and came across Tree pattern matching which I find useful.

However, after trying to replicate the example I couldn't figure out how to do it. and the method compileParseTreePattern doesn't seem to be implemented in this package.

Can you provide us with a minimal example? assuming that is possible with the current version

I've tried with Mocha and Jest (ts-jest) but both fail loading antlr4ng-generated parsers with ESM-related issues. I'm not sure how to tackle the problem. E.g. with Jest I get

 FAIL  tests/strumenta-playground.test.ts
  ● Test suite failed to run

    Must use import to load ES Module: /Users/alessio/projects/at-strumenta-ast/node_modules/antlr4ng/dist/antlr4.mjs

       5 | import {ParserTraceLoader, saveForStrumentaPlayground} from "../src/interop/strumenta-playground";
       6 | import {NodeSubclass} from "./nodes";
    >  7 | import {CharStream, CommonToken, Lexer, TerminalNode, Token, TokenStream} from "antlr4ng";
         | ^

Is there some recommended setup for using antlr4ng in tests?
Note that the non-test build apparently succeeds (maybe it would later fail when linked, or at runtime).

In version 3, parent() is now typed as this (https://github.com/mike-lischke/antlr4ng/blob/3afee898cdb33ddac439a8d1ae78443c11eed62e/src/ParserRuleContext.ts#L231C26-L231C30), which I believe is incorrect. It should be set to ParserRuleContext.

When inherited by a specific parser context class in a generated parser, this results in the parent having the same type as the context itself.

E.g.

visitFooExpression = (ctx: FooExpressionContext) {
  const fooParent = ctx.parent(); // this will now be of type FooExpressionContext, but should it? 
}

The side-effect to this is that to cast the parent context you now have to use unknown

visitFooExpression = (ctx: FooExpressionContext) {
  const fooParent = ctx.parent() as unknown as SomeParentExpressionContext; 
}

recently, i porting my grammar to typescript runtime, antlr4ng missing ProfilingATNSimulator now,it is important analysis tool to improve parsing performence in my workflow. i had implement a beta version but missing test cases, The trouble is change some function signature in Parser.i'l make a pr soon if i'm ready.

After upgrading to v3.0.3, I'm getting a TS error in ParseTreePatternMatcher.d.ts:

[{
	"resource": "/project/node_modules/antlr4ng/dist/tree/pattern/ParseTreePatternMatcher.d.ts",
	"owner": "typescript",
	"code": "2304",
	"severity": 8,
	"message": "Cannot find name 'ErrorOptions'.",
	"source": "ts",
	"startLineNumber": 87,
	"startColumn": 54,
	"endLineNumber": 87,
	"endColumn": 66
}]

When including this library in my project that uses webpack I get errors. Webpack minification (done through terser.js) breaks the code resulting in invalid syntax.
Any plans to fix that?

Same with the other library - antlr4-c3.

Is it the same as the JAR file built from https://github.com/antlr/antlr4 or is there an ng version stored somewhere?

Going through the parser trace diffs.

ATNConfig.reachesIntoOuterContext should be an "int" (number), as it is defined in CSharp, Java, assigned as a number in JavaScript, but defined as a boolean in Antlr4ng. We can confirm that with a call to int Max(a,b) called in CSharp, Java, JavaScript, but not Antlr4ng.