Giter Club home page Giter Club logo

Comments (5)

kaidjohnson avatar kaidjohnson commented on June 16, 2024 1

I modified my parser manually and was able to prevent the errors from being thrown. It appears that the inputStream might be the tokenStream, so I attached that to the parser. The ATN states don't have a getTransitions() method on them, even though c3 expects to always retrieve transitions this way, so I mapped a getTransitions method onto each state which returns the state.transitions that are defined.

const suggestExpression = (input) => {
	const inputStream = new antlr4.InputStream(input);
	const lexer = new ExprLexer(inputStream);
	const tokenStream = new antlr4.CommonTokenStream(lexer);

	const parser = new ExprParser(tokenStream);
	const errorListener = new antlr4.error.ErrorListener();
	parser.addErrorListener(errorListener);
	parser.inputStream = tokenStream;
	parser.atn.states.forEach((state) => {
		state.getTransitions = () => state.transitions;
	});

	const tree = parser.expression();
	const core = new c3.CodeCompletionCore(parser);
	return core.collectCandidates(input - 1);
};

I think this is a bit of progress, however I'm still getting 0 candidates back. I assume this has to do with the lack of CodeCompletionCore.preferredRules defined, but the test suggests otherwise:

            let inputStream = new ANTLRInputStream("var c = a + b()");
            let lexer = new ExprLexer(inputStream);
            let tokenStream = new CommonTokenStream(lexer);

            let parser = new ExprParser(tokenStream);
            let errorListener = new ErrorListener();
            parser.addErrorListener(errorListener);
            let tree = parser.expression();
            expect(errorListener.errorCount, "Test 1").equals(0);

            let core = new c3.CodeCompletionCore(parser);

            // 1) At the input start.
            let candidates = core.collectCandidates(0);

            expect(candidates.tokens.size, "Test 2").to.equal(3);
            expect(candidates.tokens.has(ExprLexer.VAR), "Test 3").to.equal(true);
            expect(candidates.tokens.has(ExprLexer.LET), "Test 4").to.equal(true);
            expect(candidates.tokens.has(ExprLexer.ID), "Test 5").to.equal(true);

            expect(candidates.tokens.get(ExprLexer.VAR), "Test 6").to.eql([ExprLexer.ID, ExprLexer.EQUAL]);
            expect(candidates.tokens.get(ExprLexer.LET), "Test 7").to.eql([ExprLexer.ID, ExprLexer.EQUAL]);
            expect(candidates.tokens.get(ExprLexer.ID), "Test 8").to.eql([]);

Given the other errors I was seeing, I'm assuming there's something fundamentally different about the JavaScript parser/lexer than the ones generated by antlr4ts-cli and something is still missing. Any help or insight as to what might be different would be useful.

from antlr4-c3.

HelgeG avatar HelgeG commented on June 16, 2024

I created a "pure" TypeScript project and used antlr4ts-cli to create the parser and lexer classes, and I am not seeing this issue.

from antlr4-c3.

kaidjohnson avatar kaidjohnson commented on June 16, 2024

I am also seeing this same issue when using the antlr4 js package with a JavaScript antlr target.

Using:

  • antlr4 4.7.2
  • anltr4-cs 1.1.11

My grammar test (based on the docs):

grammar Expr;
expression: assignment | simpleExpression;

assignment: (VAR | LET) ID EQUAL simpleExpression;

simpleExpression
    : simpleExpression (PLUS | MINUS) simpleExpression
    | simpleExpression (MULTIPLY | DIVIDE) simpleExpression
    | variableRef
    | functionRef
;

variableRef: ID;
functionRef: ID OPEN_PAR CLOSE_PAR;

VAR: [vV] [aA] [rR];
LET: [lL] [eE] [tT];

PLUS: '+';
MINUS: '-';
MULTIPLY: '*';
DIVIDE: '/';
EQUAL: '=';
OPEN_PAR: '(';
CLOSE_PAR: ')';
ID: [a-zA-Z] [a-zA-Z0-9_]*;
WS: [ \n\r\t] -> channel(HIDDEN);

The grammar is compiled using antlr4 -Dlanguage=JavaScript Expr.g4 per https://github.com/antlr/antlr4/blob/master/doc/javascript-target.md

My code (based on the docs):

import antlr4 from 'antlr4';
import * as c3 from 'antlr4-c3';

const suggestExpression = (input) => {
	const inputStream = new antlr4.InputStream(input);
	const lexer = new ExprLexer(inputStream);
	const tokenStream = new antlr4.CommonTokenStream(lexer);
	const parser = new ExprParser(tokenStream);
	const errorListener = new antlr4.error.ErrorListener();
	parser.addErrorListener(errorListener);

	const tree = parser.expression();
	const core = new c3.CodeCompletionCore(parser);
	return core.collectCandidates(0);
};

The error:

CodeCompletionCore.js:67 Uncaught TypeError: Cannot read property 'index' of undefined
    at CodeCompletionCore.collectCandidates (CodeCompletionCore.js:67)

The lines:

screen shot 2019-02-27 at 3 21 46 pm

I've tried adding the inputStream to the parser via parser.inputStream, but this results in an infinite loop in the while (true) block just below. It seems that the tokenStream.LT(offset++) call is returning -1 without a .type so the loop just keeps going.

Is antlr4ts required? Seems like using the antlr4 lib should work, too.

from antlr4-c3.

mike-lischke avatar mike-lischke commented on June 16, 2024

@kaidjohnson You know you are writing to a closed issue? Please don't hijack already finished issues but create a new one, if required.

Indeed there are differences between the JS and TS runtime versions. The authors of @antlr4ts insist on implementing their runtime in a way that is optimal for the TS language, thereby sacrificing compatibility. Best would be you open an issue against @antlr4ts, asking for more compatibility. However, I remember a discussion where people spoke about making the TS target the main one and generating the JS target from it. The current handwritten JS target would then be obsolete.

from antlr4-c3.

kaidjohnson avatar kaidjohnson commented on June 16, 2024

@mike-lischke It wasn't my intention to "hijack" anything. The issue I was seeing was the same as the user that posted the original issue here, and I thought I might get some additional insight into what the options were to get around it. Every git repo owner seems to have a different preference for creating tickets -- some complain about opening duplicates, some complain about extending closed tickets. It's always a 50/50 gamble; didn't mean to ruffle any feathers.

My current JS target was not handwritten, but rather generated via the core antlr4 library.

FWIW, I did work around the issue after a bit more trial-and-error by generating my parser and lexer via antlr4ts, converting the resulting ts (es2015 target) to js (using babel) -- the typescript es5 target wasn't handling the class conversion well -- and then using the core antlr4 runtime (not the ts version) for use in the browser. The parser needed to patched slightly to be compatible with the core Parser class from antlr4ts, but after that small patch (see below), I am getting tokens back.

Here's the patch, which essentially just copies over the missing method from antlr4ts:

import BaseExprParser from './ExprParser.ts';

class ExprParser extends BaseExprParser {
	constructor(...args) {
		super(...args);

		// The antlr4ts Parser expects `tryLT` to be defined on the token stream
		// and `LT` to defer to the `tryLT`.
		// @see 'node_modules/antlr4ts/Parser.js
		// @see 'node_modules/antlr4ts/BufferedTokenStream.js:203
		this._input.tryLT = this._input.LT;
		this._input.LT = function (k) {
			const result = this.tryLT(k);
			if (result === undefined) {
				throw new RangeError('requested lookback index out of range');
			}
			return result;
		};
	}
}

For now, this resolves the issue for me and I am able to run the antlr4-c3 engine in the browser. Should I run into any other issues, I'll be sure to open a new ticket. Thanks!

from antlr4-c3.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.