lezer-parser / javascript Goto Github PK

Hello there,

My team's using this parser as a part of @codemirror/lang-javascript with CodeMirror 6 for syntax highlighting.

However, we've noticed that the keyword undefined doesn't get highlighted, and while poking around, noticed that this parser's highlight.js doesn't seem to associate it with any tag.

I don't have much context, but from my understanding, if the change isn't much more than adding:

undefined: t.atom,

I'd be happy to open a PR!

This probably doesn't affect anything this grammar is being used for, but FWIW...

The grammar defines precedence

  bitOr @left,
  bitXor @left,
  bitAnd @left,

But according to MDN the correct precedence is

TypeScript decorators are in heavy use in certain parts of the JavaScript community (for example Angular). It'd be great to have them supported in CodeMirror as well.

Downstream issue: https://crbug.com/1385293

const r = RegExp(`^-{${n},}$`, 'm')
does not get properly parsed.
I am using lezer 1.4.10.

I ran into this issue when trying to get some tests to run with jest:

Cannot find module 'lezer-javascript' from 'index.js'
at Resolver.resolveModule (node_modules/jest-resolve/build/index.js:265:17)
      at Object.<anonymous> (../../node_modules/@codemirror/next/lang-javascript/dist/index.js:10:24)

All of the other packages point to index.cjs in their main, and when built, dist does not contain an index.js in lezer-javascript.

Hi,
I have a strange error since v1.4.12: just importing lang-javascript fails with this:

Uncaught TypeError: Cannot read properties of undefined (reading 'deserialize')
    at setProp (index.ac9dc4ba.js:27819:22)
    at new LRParser (index.ac9dc4ba.js:27830:51)
    at LRParser.deserialize (index.ac9dc4ba.js:28028:16)
    at dv8xE.@lezer/lr (index.ac9dc4ba.js:26252:34)
    at newRequire (index.ac9dc4ba.js:71:24)
    at localRequire (index.ac9dc4ba.js:84:35)
    at bo80H.@lezer/javascript (index.ac9dc4ba.js:25447:19)
    at newRequire (index.ac9dc4ba.js:71:24)
    at localRequire (index.ac9dc4ba.js:84:35)
    at aR1JP../styles.css (index.ac9dc4ba.js:582:23)

Reverting the @lezer/javascript to v1.4.11 works with no error, that's why I guess is related to the lezer. I also have no problems with other languages (html, sass)

I put a minimal test here
if you comment the import { javascript } from "@codemirror/lang-javascript"; line, there is no error

Here is a screenshot since codesandbox sometimes doesn't display the error, it just fails silently

When I patch up lezer-generator/test/test-parse.ts to parse files with {strict:false}, then many of the javascript tests fail.
E.g. expression/Arrays yields:

  2) expression
       Arrays:
     Error: Expected ArrayExpression in ExpressionStatement, got AssignmentExpression at 0 
Script(ExpressionStatement(AssignmentExpression(ArrayPattern("[","]"),⚠),";"),ExpressionStatement(ArrayExpression("[",String,"]"),";"),ExpressionStatement(ArrayExpression("[",String,",","]"),";"),ExpressionStatement(ArrayExpression("[",String,",",VariableName,"]"),";"),ExpressionStatement(ArrayExpression("[",",",VariableName,"]"),";"),ExpressionStatement(ArrayExpression("[",AssignmentExpression(VariableName,Equals,Number),"]"),";"))
      at Iteration.enter (node_modules/lezer-generator/dist/test.js:123:23)
      at Iteration.doEnter (node_modules/lezer-tree/dist/tree.js:30:26)
      at TreeBuffer.iterChild (node_modules/lezer-tree/dist/tree.js:586:33)
      at TreeBuffer.iterChild (node_modules/lezer-tree/dist/tree.js:588:30)
      at TreeBuffer.iterInner (node_modules/lezer-tree/dist/tree.js:575:30)
      at Tree.iterInner (node_modules/lezer-tree/dist/tree.js:413:23)
      at Tree.iterInner (node_modules/lezer-tree/dist/tree.js:413:23)
      at Tree.iterate (node_modules/lezer-tree/dist/tree.js:399:14)
      at testTree (node_modules/lezer-generator/dist/test.js:108:10)
      at run (node_modules/lezer-generator/dist/test.js:152:17)
      at Context.it (test/test-javascript.js:13:22)

https://codemirror.net/try/?c=aW1wb3J0IHtiYXNpY1NldHVwLCBFZGl0b3JWaWV3fSBmcm9tICJjb2RlbWlycm9yIgppbXBvcnQge2phdmFzY3JpcHR9IGZyb20gIkBjb2RlbWlycm9yL2xhbmctamF2YXNjcmlwdCIKCm5ldyBFZGl0b3JWaWV3KHsKICBkb2M6ICIvLyB3b3JrcyBmaW5lXG5mdW5jdGlvbiogZ2VuKCkge1xuICB5aWVsZCBmb29cbn1cblxuZnVuY3Rpb24qIGdlbigpIHtcbiAgeWllbGQgW11cbn1cblxuZnVuY3Rpb24qIGdlbigpIHtcbiAgeWllbGQgeyBmb286IFtiYXJdIH1cbn1cblxuLy8gYnJlYWtzXG5mdW5jdGlvbiogZ2VuKCkge1xuICB5aWVsZCogZm9vXG59XG5cbmZ1bmN0aW9uKiBnZW4oKSB7XG4gIHlpZWxkIFt2YWxpZF9hcnJheV1cbn1cblxuZnVuY3Rpb24qIGdlbigpIHtcbiAgeWllbGQqIFt2YWxpZF9hcnJheV1cbn1cblxuLy8gd29ya3MgZm9yIHNvbWUgcmVhc29uXG5mdW5jdGlvbiogZ2VuKCkge1xuICB5aWVsZCBbaW52YWxpZF9hcnJheS5dXG59IiwKICBleHRlbnNpb25zOiBbYmFzaWNTZXR1cCwgamF2YXNjcmlwdCgpXSwKICBwYXJlbnQ6IGRvY3VtZW50LmJvZHkKfSkK

You can tell it's not being parsed correctly because the yield keyword isn't being highlighted as a keyword sometimes.

Looking at the grammar, it's definitely clear that yield* is simply not being handled. I'm not sure what's going on with the array case though, maybe just a dynamic precedence thing?

Tagged template literals are often used to embed another language in JavaScript. It'd be nice if this grammar had built-in support for delegating to another grammar based on the template tag, ie you can register grammars against template tag identifiers, like html, css, svg, json, etc. Some editors allow registering against regexes that match the tag.

One complication is the expressions in template literals. A nested grammar should be able to parse the joined template literal strings as one character stream.

A stack of nested grammars and documents will be necessary for complex nesting. It's not uncommon to see structures like this:

html`
  <style>
    p { color: red; }
  </style>
  <div>${x ? html`<p>X</p>` : html`<p>Y</p>`</div>
  <script>...</script>
`

-? and -readonly are valid TS, but do not appear to work with lezer:

type Test<T> = { -readonly [P in keyof T]-?: T[P] }

type Result = Test<{ readonly foo?: number }>

// Result = { foo: number }

Using a generic type as the return type of arrow functions seems to confuse the parser.

const a = (): Array<string> => {
  return []
}

is parsed to

[
  "Script",
  "VariableDeclaration",
  "const",
  "VariableDefinition",
  "Equals",
  "BinaryExpression", // <- ?
  "BinaryExpression",
  "ArrowFunction",
  "ParamList",
  "(",
  ")",
  "TypeAnnotation",
  ":",
  "TypeName",
  "⚠",
  "CompareOp",
  "VariableName",
  "CompareOp",
  "⚠",
  "Arrow",
  "ObjectExpression",
  "{",
  "Property",
  "PropertyDefinition",
  "⚠",
  "}"
]

It only happens if specify dialect: "tsx", not dialect: "ts".

I tested that with following code:

import { parser } from "@lezer/javascript";

const script = "...";
const list: string[] = [];
parser.configure({ dialect: "tsx" }).parse(script).iterate({
  enter: (node) => {
    list.push(node.name);
  },
});
console.log(list);

Side-note: I apologize for filing all these bugs, I'm currently exploring replacing Acorn with Lezer in DevTools for pretty-printing and I encounter some bugs when parsing minified code. Feel free to close as "Wont'fix".

The following snippet fails to parse:

(a=b(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41, 42))

Note that it's sensitive to the number of arguments. Removing the 42 results in a successful parse. Adding more arguments fails the parse.

Note that it's not necessarily the argument list. The following snippet also fails the parse:

(a = function(b,c,d) {
  const e = 1 + 1 + 1;
  const f = 2 + 2 + 2;
  const g = 3 + 3 + 3;
  const h = 4 + 4 + 4;
  const i = 5 + 5 + 5;
  const j = 6 + 6 + 6;
  const k = 7 + 7 + 7;
  const l = 8 + 8 + 8;
}(1,2,3))

Again it's sensitive to the number of AST nodes inside the function. Deleting one ore more lines makes the parse successful. Adding more const m = ... keeps it failing.

I enabled the debug logging and seems that in the failing case it gets stuck trying to reduce with a ParamList (instead of an ArgList) for the CallExpression, but that might just be a red herring.

Hi, I ran into some issues while trying out CodeMirror 6 for typescript and I think it's due to this package.

In CodeMirror (live, source). Also notice that with jsx enabled it gets confused.

In VSCode/Monaco with a similar theme (TS Playground). The pic is from a .tsx file.

I thought it could be the theme but the html also turns out different. For the first example notice that the Y is missing the <span>.

(() => T extends X ? 1 : 2) extends	(() => T extends Y ? 1 : 2) ? true : false

I tried adding a simple test case to this parser and it throws SyntaxError: No parse at 23. (TS Playground)
(ignore the expected value, it's from another existing test)

# SimpleConditionalType {"dialect": "ts"}

type IsLiteralA<T> = T extends "A" ? true : false

==>

Script(EnumDeclaration(const, enum, TypeDefinition, EnumBody(PropertyName, Equals, Number, PropertyName, PropertyName)))

If you agree these are worth fixing, let me know what's the best format to report any other error like this. I'm trying to load stuff from https://github.com/type-challenges/type-challenges into CodeMirror, so I can take notes if I run into issues with some more complicated types.

import { default as x } from 'x'

It has the same semantics as import x from 'x' but is an allowed way of writing. I saw this kind of code from some outputs of rollup:

import { default as default2 } from "eslint-plugin-antfu";

Current AST output:

Script
  ImportDeclaration
    import
    ImportGroup
      {
      String
        ⚠
      ⚠
    ⚠
  ExpressionStatement
    VariableName
      ⚠

Update: It seems export { default as name } from 'x' has this issue too.

Trying to parse a class declaration like this

class A extends (class B {}) {}

seems to confuse the parser quite a bit:

Although from looking at the code it appears that arbitrary expressions are supported after extends.

Hi, I was trying to parse some typescript code and came across an inconsistency when parsing unions.

Here is the setup I used.

import { parser as javascriptParser } from "@lezer/javascript"
import { highlightTree, classHighlighter } from "@lezer/highlight"
import type { Tree } from "@lezer/common"

function highlight(str: string, tree: Tree) {
  let result = "", pos = 0
  highlightTree(tree, classHighlighter, (from, to, cls) => {    
    if (from > pos) result += (str.slice(pos, from))
    result += `<span class=${cls}>${(str.slice(from, to))}</span>`
    pos = to
  });
  if (pos < str.length) result += str.slice(pos)
  return result
}

const source = `export type JSONValue =
| number
| null
| boolean
| string;`

const parser = javascriptParser.configure({ dialect: 'ts jsx' });
const tree = parser.parse(source)
const result = highlight(source, tree)
console.log(result);

Output:

<span class=tok-keyword>export</span> <span class=tok-keyword>type</span> <span class=tok-typeName>JSONValue</span> <span class=tok-operator>=</span>
| <span class=tok-typeName>number</span>
<span class=tok-operator>|</span> <span class=tok-keyword>null</span>
<span class=tok-operator>|</span> <span class=tok-typeName>boolean</span>
<span class=tok-operator>|</span> <span class=tok-typeName>string</span><span class=tok-punctuation>;</span>

Expected Output:

<span class=tok-keyword>export</span> <span class=tok-keyword>type</span> <span class=tok-typeName>JSONValue</span> <span class=tok-operator>=</span>
<span class=tok-operator>|</span> <span class=tok-typeName>number</span>
<span class=tok-operator>|</span> <span class=tok-typeName>null</span>
<span class=tok-operator>|</span> <span class=tok-typeName>boolean</span>
<span class=tok-operator>|</span> <span class=tok-typeName>string</span><span class=tok-punctuation>;</span>

As shown above, the first | is treated as text and what follows becomes a keyword.

PS: Although jsx dialect does not serve any purpose, the code I had JSX, so I kept it.

Thank you for this great library and all the work you did.

error TS7016: Could not find a declaration file for module '@lezer/javascript'.
 '/home/nullvoxpopuli/Development/NullVoxPopuli/limber/node_modules/
    .pnpm/@[email protected]/node_modules/@lezer/javascript/dist/index.es.js'

Full error:

 > tsc --noEmit

src/content.ts(2,44): error TS7016: 
  Could not find a declaration file for module '@lezer/javascript'.
   '<repo>/node_modules/.pnpm/@[email protected]/node_modules/@lezer/javascript/dist/index.es.js'
      implicitly has an 'any' type.

  There are types at '<repo>/packages/syntax/glimmer/lezer/node_modules/@lezer/javascript/dist/index.d.ts', 
    but this result could not be resolved when respecting package.json "exports". 
  The '@lezer/javascript' library may need to update its package.json or typings.

There are 4 global properties in JavaScript: undefined, NaN, Infinity & globalThis. All of those are unique in the language, they have special features and undefined is also a primitive value. Current highlight.js treats them as variableName, because of which they share color with any other variable. Would it be possible to put those 4 values under a unique tag, so they can have a different style than rest of the variables? Two of those: undefined & NaN could easily share tag with null.

Yarn in PnP mode complains when using this package (and I assume any of the other language packages) with this message:

@lezer/javascript tried to access @lezer/common, but it isn't declared in its dependencies; this makes the require call ambiguous and unsound.

I think this is because an import of @lezer/common exists in the generated code but its version is not specified in the package.json, meaning it is unclear which version to use if there happen to be multiple versions of @lezer/common in the dependency tree.

The current grammar doesn't support computed property names in object destructuring, for example

const {[Symbol.iterator]: iterator} = array;

is valid JavaScript (and TypeScript), but CodeMirror gets highly confused about what it's looking at.

cc @jaro-sevcik

Downstream: crbug/1436873

The following doesn't parse:

function foo(this: Type) {}

Not sure how relevant this is for authored code, but bundlers remove spaces and leading zeroes from numbers. The snippet

foo?.1:2

currently fails to parse. Since I'm only superficially familiar with how Lezer grammar works, I don't actually know how a fix would look like. I'd have expected the parser to backtrack from the failed MemberExpression and try the ConditionalExpression, but maybe the tokenizer eagerly combined the ?. so the ConditionalExpression can't be applied?

I can see that this support was added in 568d51c

However, it doesn't seem to trickle down through to an actual editor - using

└─┬ @codemirror/[email protected]
  └── @lezer/[email protected]

"let" and "const" are properly highlighted, but "using" is not:

I'm not sure why - possibly because this is a contextual keyword?