lezer-parser / javascript Goto Github PK
View Code? Open in Web Editor NEWA JavaScript lezer grammar
License: MIT License
A JavaScript lezer grammar
License: MIT License
Hello there,
My team's using this parser as a part of @codemirror/lang-javascript
with CodeMirror 6 for syntax highlighting.
However, we've noticed that the keyword undefined
doesn't get highlighted, and while poking around, noticed that this parser's highlight.js
doesn't seem to associate it with any tag.
I don't have much context, but from my understanding, if the change isn't much more than adding:
undefined: t.atom,
I'd be happy to open a PR!
This probably doesn't affect anything this grammar is being used for, but FWIW...
The grammar defines precedence
bitOr @left,
bitXor @left,
bitAnd @left,
But according to MDN the correct precedence is
7: bitwise AND | left-to-right | Bitwise ANDx & y |
---|---|---|
6: bitwise XOR | left-to-right | Bitwise XORx ^ y |
5: bitwise OR | left-to-right | Bitwise ORx | y |
TypeScript decorators are in heavy use in certain parts of the JavaScript community (for example Angular). It'd be great to have them supported in CodeMirror as well.
Downstream issue: https://crbug.com/1385293
const r = RegExp(`^-{${n},}$`, 'm')
does not get properly parsed.
I am using lezer 1.4.10.
I ran into this issue when trying to get some tests to run with jest:
Cannot find module 'lezer-javascript' from 'index.js'
at Resolver.resolveModule (node_modules/jest-resolve/build/index.js:265:17)
at Object.<anonymous> (../../node_modules/@codemirror/next/lang-javascript/dist/index.js:10:24)
All of the other packages point to index.cjs in their main, and when built, dist does not contain an index.js in lezer-javascript.
Hi,
I have a strange error since v1.4.12: just importing lang-javascript fails with this:
Uncaught TypeError: Cannot read properties of undefined (reading 'deserialize')
at setProp (index.ac9dc4ba.js:27819:22)
at new LRParser (index.ac9dc4ba.js:27830:51)
at LRParser.deserialize (index.ac9dc4ba.js:28028:16)
at dv8xE.@lezer/lr (index.ac9dc4ba.js:26252:34)
at newRequire (index.ac9dc4ba.js:71:24)
at localRequire (index.ac9dc4ba.js:84:35)
at bo80H.@lezer/javascript (index.ac9dc4ba.js:25447:19)
at newRequire (index.ac9dc4ba.js:71:24)
at localRequire (index.ac9dc4ba.js:84:35)
at aR1JP../styles.css (index.ac9dc4ba.js:582:23)
Reverting the @lezer/javascript to v1.4.11 works with no error, that's why I guess is related to the lezer. I also have no problems with other languages (html, sass)
I put a minimal test here
if you comment the import { javascript } from "@codemirror/lang-javascript";
line, there is no error
Here is a screenshot since codesandbox sometimes doesn't display the error, it just fails silently
When I patch up lezer-generator/test/test-parse.ts to parse files with {strict:false}
, then many of the javascript tests fail.
E.g. expression/Arrays yields:
2) expression
Arrays:
Error: Expected ArrayExpression in ExpressionStatement, got AssignmentExpression at 0
Script(ExpressionStatement(AssignmentExpression(ArrayPattern("[","]"),⚠),";"),ExpressionStatement(ArrayExpression("[",String,"]"),";"),ExpressionStatement(ArrayExpression("[",String,",","]"),";"),ExpressionStatement(ArrayExpression("[",String,",",VariableName,"]"),";"),ExpressionStatement(ArrayExpression("[",",",VariableName,"]"),";"),ExpressionStatement(ArrayExpression("[",AssignmentExpression(VariableName,Equals,Number),"]"),";"))
at Iteration.enter (node_modules/lezer-generator/dist/test.js:123:23)
at Iteration.doEnter (node_modules/lezer-tree/dist/tree.js:30:26)
at TreeBuffer.iterChild (node_modules/lezer-tree/dist/tree.js:586:33)
at TreeBuffer.iterChild (node_modules/lezer-tree/dist/tree.js:588:30)
at TreeBuffer.iterInner (node_modules/lezer-tree/dist/tree.js:575:30)
at Tree.iterInner (node_modules/lezer-tree/dist/tree.js:413:23)
at Tree.iterInner (node_modules/lezer-tree/dist/tree.js:413:23)
at Tree.iterate (node_modules/lezer-tree/dist/tree.js:399:14)
at testTree (node_modules/lezer-generator/dist/test.js:108:10)
at run (node_modules/lezer-generator/dist/test.js:152:17)
at Context.it (test/test-javascript.js:13:22)
You can tell it's not being parsed correctly because the yield
keyword isn't being highlighted as a keyword sometimes.
Looking at the grammar, it's definitely clear that yield*
is simply not being handled. I'm not sure what's going on with the array case though, maybe just a dynamic precedence thing?
Tagged template literals are often used to embed another language in JavaScript. It'd be nice if this grammar had built-in support for delegating to another grammar based on the template tag, ie you can register grammars against template tag identifiers, like html
, css
, svg
, json
, etc. Some editors allow registering against regexes that match the tag.
One complication is the expressions in template literals. A nested grammar should be able to parse the joined template literal strings as one character stream.
A stack of nested grammars and documents will be necessary for complex nesting. It's not uncommon to see structures like this:
html`
<style>
p { color: red; }
</style>
<div>${x ? html`<p>X</p>` : html`<p>Y</p>`</div>
<script>...</script>
`
-?
and -readonly
are valid TS, but do not appear to work with lezer:
type Test<T> = { -readonly [P in keyof T]-?: T[P] }
type Result = Test<{ readonly foo?: number }>
// Result = { foo: number }
Using a generic type as the return type of arrow functions seems to confuse the parser.
const a = (): Array<string> => {
return []
}
is parsed to
[
"Script",
"VariableDeclaration",
"const",
"VariableDefinition",
"Equals",
"BinaryExpression", // <- ?
"BinaryExpression",
"ArrowFunction",
"ParamList",
"(",
")",
"TypeAnnotation",
":",
"TypeName",
"⚠",
"CompareOp",
"VariableName",
"CompareOp",
"⚠",
"Arrow",
"ObjectExpression",
"{",
"Property",
"PropertyDefinition",
"⚠",
"}"
]
It only happens if specify dialect: "tsx"
, not dialect: "ts"
.
I tested that with following code:
import { parser } from "@lezer/javascript";
const script = "...";
const list: string[] = [];
parser.configure({ dialect: "tsx" }).parse(script).iterate({
enter: (node) => {
list.push(node.name);
},
});
console.log(list);
Side-note: I apologize for filing all these bugs, I'm currently exploring replacing Acorn with Lezer in DevTools for pretty-printing and I encounter some bugs when parsing minified code. Feel free to close as "Wont'fix".
The following snippet fails to parse:
(a=b(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41, 42))
Note that it's sensitive to the number of arguments. Removing the 42
results in a successful parse. Adding more arguments fails the parse.
Note that it's not necessarily the argument list. The following snippet also fails the parse:
(a = function(b,c,d) {
const e = 1 + 1 + 1;
const f = 2 + 2 + 2;
const g = 3 + 3 + 3;
const h = 4 + 4 + 4;
const i = 5 + 5 + 5;
const j = 6 + 6 + 6;
const k = 7 + 7 + 7;
const l = 8 + 8 + 8;
}(1,2,3))
Again it's sensitive to the number of AST nodes inside the function. Deleting one ore more lines makes the parse successful. Adding more const m = ...
keeps it failing.
I enabled the debug logging and seems that in the failing case it gets stuck trying to reduce with a ParamList
(instead of an ArgList
) for the CallExpression, but that might just be a red herring.
Hi, I ran into some issues while trying out CodeMirror 6 for typescript and I think it's due to this package.
In CodeMirror (live, source). Also notice that with jsx enabled it gets confused.
In VSCode/Monaco with a similar theme (TS Playground). The pic is from a .tsx file.
I thought it could be the theme but the html also turns out different. For the first example notice that the Y is missing the <span>.
(() => T extends X ? 1 : 2) extends | (() => T extends Y ? 1 : 2) ? true : false |
---|---|
![]() |
![]() |
I tried adding a simple test case to this parser and it throws SyntaxError: No parse at 23
. (TS Playground)
(ignore the expected value, it's from another existing test)
# SimpleConditionalType {"dialect": "ts"}
type IsLiteralA<T> = T extends "A" ? true : false
==>
Script(EnumDeclaration(const, enum, TypeDefinition, EnumBody(PropertyName, Equals, Number, PropertyName, PropertyName)))
If you agree these are worth fixing, let me know what's the best format to report any other error like this. I'm trying to load stuff from https://github.com/type-challenges/type-challenges into CodeMirror, so I can take notes if I run into issues with some more complicated types.
import { default as x } from 'x'
It has the same semantics as import x from 'x'
but is an allowed way of writing. I saw this kind of code from some outputs of rollup:
import { default as default2 } from "eslint-plugin-antfu";
Current AST output:
Script
ImportDeclaration
import
ImportGroup
{
String
⚠
⚠
⚠
ExpressionStatement
VariableName
⚠
Update: It seems export { default as name } from 'x'
has this issue too.
Hi, I was trying to parse some typescript code and came across an inconsistency when parsing unions.
Here is the setup I used.
import { parser as javascriptParser } from "@lezer/javascript"
import { highlightTree, classHighlighter } from "@lezer/highlight"
import type { Tree } from "@lezer/common"
function highlight(str: string, tree: Tree) {
let result = "", pos = 0
highlightTree(tree, classHighlighter, (from, to, cls) => {
if (from > pos) result += (str.slice(pos, from))
result += `<span class=${cls}>${(str.slice(from, to))}</span>`
pos = to
});
if (pos < str.length) result += str.slice(pos)
return result
}
const source = `export type JSONValue =
| number
| null
| boolean
| string;`
const parser = javascriptParser.configure({ dialect: 'ts jsx' });
const tree = parser.parse(source)
const result = highlight(source, tree)
console.log(result);
Output:
<span class=tok-keyword>export</span> <span class=tok-keyword>type</span> <span class=tok-typeName>JSONValue</span> <span class=tok-operator>=</span>
| <span class=tok-typeName>number</span>
<span class=tok-operator>|</span> <span class=tok-keyword>null</span>
<span class=tok-operator>|</span> <span class=tok-typeName>boolean</span>
<span class=tok-operator>|</span> <span class=tok-typeName>string</span><span class=tok-punctuation>;</span>
Expected Output:
<span class=tok-keyword>export</span> <span class=tok-keyword>type</span> <span class=tok-typeName>JSONValue</span> <span class=tok-operator>=</span>
<span class=tok-operator>|</span> <span class=tok-typeName>number</span>
<span class=tok-operator>|</span> <span class=tok-typeName>null</span>
<span class=tok-operator>|</span> <span class=tok-typeName>boolean</span>
<span class=tok-operator>|</span> <span class=tok-typeName>string</span><span class=tok-punctuation>;</span>
As shown above, the first |
is treated as text and what follows becomes a keyword.
PS: Although jsx dialect does not serve any purpose, the code I had JSX, so I kept it.
Thank you for this great library and all the work you did.
error TS7016: Could not find a declaration file for module '@lezer/javascript'.
'/home/nullvoxpopuli/Development/NullVoxPopuli/limber/node_modules/
.pnpm/@[email protected]/node_modules/@lezer/javascript/dist/index.es.js'
Full error:
> tsc --noEmit
src/content.ts(2,44): error TS7016:
Could not find a declaration file for module '@lezer/javascript'.
'<repo>/node_modules/.pnpm/@[email protected]/node_modules/@lezer/javascript/dist/index.es.js'
implicitly has an 'any' type.
There are types at '<repo>/packages/syntax/glimmer/lezer/node_modules/@lezer/javascript/dist/index.d.ts',
but this result could not be resolved when respecting package.json "exports".
The '@lezer/javascript' library may need to update its package.json or typings.
There are 4 global properties in JavaScript: undefined
, NaN
, Infinity
& globalThis
. All of those are unique in the language, they have special features and undefined is also a primitive value. Current highlight.js treats them as variableName
, because of which they share color with any other variable. Would it be possible to put those 4 values under a unique tag, so they can have a different style than rest of the variables? Two of those: undefined
& NaN
could easily share tag with null
.
Yarn in PnP mode complains when using this package (and I assume any of the other language packages) with this message:
@lezer/javascript tried to access @lezer/common, but it isn't declared in its dependencies; this makes the require call ambiguous and unsound.
I think this is because an import of @lezer/common
exists in the generated code but its version is not specified in the package.json, meaning it is unclear which version to use if there happen to be multiple versions of @lezer/common
in the dependency tree.
The current grammar doesn't support computed property names in object destructuring, for example
const {[Symbol.iterator]: iterator} = array;
is valid JavaScript (and TypeScript), but CodeMirror gets highly confused about what it's looking at.
cc @jaro-sevcik
Not sure how relevant this is for authored code, but bundlers remove spaces and leading zeroes from numbers. The snippet
foo?.1:2
currently fails to parse. Since I'm only superficially familiar with how Lezer grammar works, I don't actually know how a fix would look like. I'd have expected the parser to backtrack from the failed MemberExpression
and try the ConditionalExpression
, but maybe the tokenizer eagerly combined the ?.
so the ConditionalExpression
can't be applied?
I can see that this support was added in 568d51c
However, it doesn't seem to trickle down through to an actual editor - using
└─┬ @codemirror/[email protected]
└── @lezer/[email protected]
"let" and "const" are properly highlighted, but "using" is not:
I'm not sure why - possibly because this is a contextual keyword?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.