Current behavior:

Suppose that I had a class that looked like this:

public class Foo constructor () { 
   < stuff in here >
}

This is meant to be parsed in the grammar as a class declaration, with a primary_constructor. Indeed, we see it parsed as such with the current grammar:

(source_file [0, 0] - [1, 0]
  (class_declaration [0, 0] - [0, 35]
    (modifiers [0, 0] - [0, 6]
      (visibility_modifier [0, 0] - [0, 6]))
    (type_identifier [0, 13] - [0, 16])
    (primary_constructor [0, 17] - [0, 31])
    (class_body [0, 32] - [0, 35])))

If I add a newline, however:

public class Foo
constructor () {
  < stuff in here>
}

it parses as something different.

(source_file [0, 0] - [2, 0]
  (class_declaration [0, 0] - [0, 16]
    (modifiers [0, 0] - [0, 6]
      (visibility_modifier [0, 0] - [0, 6]))
    (type_identifier [0, 13] - [0, 16]))
  (call_expression [1, 0] - [1, 18]
    (call_expression [1, 0] - [1, 14]
      (simple_identifier [1, 0] - [1, 11])
      (call_suffix [1, 12] - [1, 14]
        (value_arguments [1, 12] - [1, 14])))
    (call_suffix [1, 15] - [1, 18]
      (annotated_lambda [1, 15] - [1, 18]
        (lambda_literal [1, 15] - [1, 18])))))

The reason for this ends up being that the newline induces an opportunity for an automatic semicolon, so the token stream looks like:

PUBLIC CLASS "Foo" SEMICOLON CONSTRUCTOR ( ) ...

This makes the parser think that, since a class_declaration can have many optional things, that the public class Foo is a standalone class (with not very many things in it), and the thing that follows it is a call expression of constructor on a lambda or something, I think.

Expected behavior:
The above example should correctly parse to a single class declaration, with a primary constructor, as opposed to two top-level entities. This could be done by suppressing the insertion of the automatic semicolon in such a spot (which may require refactoring and heavier state being carried in the external scanner), or some grammar-level hacking.

Let me know what you think! I am trying to add better support for Kotlin in an open-source static analysis tool, Semgrep (https://github.com/returntocorp/semgrep), and this is blocking my ability to do so.

value class CloudStorageUri(val uri: URI) doesn't seem to parse properly.

It produces something like

[source_file](https://fwcd.dev/tree-sitter-kotlin/#) [0, 0] - [1, 0]
  [infix_expression](https://fwcd.dev/tree-sitter-kotlin/#) [0, 0] - [0, 41]
    [simple_identifier](https://fwcd.dev/tree-sitter-kotlin/#) [0, 0] - [0, 5]
    [simple_identifier](https://fwcd.dev/tree-sitter-kotlin/#) [0, 6] - [0, 11]
    [call_expression](https://fwcd.dev/tree-sitter-kotlin/#) [0, 12] - [0, 41]
      [simple_identifier](https://fwcd.dev/tree-sitter-kotlin/#) [0, 12] - [0, 27]
      [call_suffix](https://fwcd.dev/tree-sitter-kotlin/#) [0, 27] - [0, 41]
        [value_arguments](https://fwcd.dev/tree-sitter-kotlin/#) [0, 27] - [0, 41]
          [value_argument](https://fwcd.dev/tree-sitter-kotlin/#) [0, 28] - [0, 40]
            [infix_expression](https://fwcd.dev/tree-sitter-kotlin/#) [0, 28] - [0, 40]
              [simple_identifier](https://fwcd.dev/tree-sitter-kotlin/#) [0, 28] - [0, 31]
              [simple_identifier](https://fwcd.dev/tree-sitter-kotlin/#) [0, 32] - [0, 35]
              [ERROR](https://fwcd.dev/tree-sitter-kotlin/#) [0, 35] - [0, 36]
              [simple_identifier](https://fwcd.dev/tree-sitter-kotlin/#) [0, 37] - [0, 40]

When I remove the value, I get

[source_file](https://fwcd.dev/tree-sitter-kotlin/#) [0, 0] - [1, 0]
  [class_declaration](https://fwcd.dev/tree-sitter-kotlin/#) [0, 0] - [0, 35]
    [type_identifier](https://fwcd.dev/tree-sitter-kotlin/#) [0, 6] - [0, 21]
    [primary_constructor](https://fwcd.dev/tree-sitter-kotlin/#) [0, 21] - [0, 35]
      [class_parameter](https://fwcd.dev/tree-sitter-kotlin/#) [0, 22] - [0, 34]
        [simple_identifier](https://fwcd.dev/tree-sitter-kotlin/#) [0, 26] - [0, 29]
        [user_type](https://fwcd.dev/tree-sitter-kotlin/#) [0, 31] - [0, 34]
          [type_identifier](https://fwcd.dev/tree-sitter-kotlin/#) [0, 31] - [0, 34]

Notice the "class declaration"

this line get() = "Hello world." shouldn't be red

but when i delete the opening bracket of the class it goes back to normal! instead of turning red.

Can we bundle current code into a new release?

And an update to the web sample https://fwcd.dev/tree-sitter-kotlin/ would also be very much appreciated.

Parsing this class definition results in an error

class Binary(@field:Deps val branch: Boolean)

(class_declaration
 (type_identifier)
 (primary_constructor
  (ERROR
   (class_parameter
    (modifiers (annotation (simple_identifier)))
    (simple_identifier (MISSING "_lexical_identifier_token1"))
    (user_type (type_identifier))))
  (class_parameter (simple_identifier) (user_type (type_identifier)))))

After a small modification

class Binary(@field:Deps() val branch: Boolean)

(class_declaration
 (type_identifier)
 (ERROR (modifiers (annotation (simple_identifier))))
 (delegation_specifier
  (constructor_invocation
   (user_type (type_identifier)) 
   (value_arguments))))

tree-sitter: 0.16.1
tree-sitter-kotlin: 0.2.3

"parsing" category on crates.io is for low-level/general-purpose tooling for making parsers. Language-specific parsers belong to parser-implementations category.

Did you check the tree-sitter docs?

• I have read all the tree-sitter docs if it relates to using the parser

Is your feature request related to a problem? Please describe.

ast-grep is using a lot of tree-sitter grammars and is going to upgrading the tree-sitter parser library.

However, tree-sitter-kotlin dependency range is blocking the upgrade.

e.g.
ast-grep/ast-grep#1057

Describe the solution you'd like

Release a new tree-sitter 0.21 compatible version

Describe alternatives you've considered

Make the dependency more compatible by relaxing the version range

Additional context

No response

The current parser has been generated with tree-sitter-cli 0.15.7 which had bugs in the generation of the node-types.json file.

In particular:

• many node types have wrong "multiple" and "required" attributes,
• aliased node types, such type-identifier are missing from the file

To fix those errors, the tree-sitter-cli package needs to be upgraded to latest version (0.16.9) and the parser needs to be regenerated.

Kotlin has special syntax defined for Java interoperability where e.g. function names can be defined to be more or less anything delimited by backticks, e.g. (source: https://stackoverflow.com/a/65923081/51634):

fun `adding 3 and 4 should be equal to 7`() {
    assertEquals(calculator.add(3, 4), 7)
}

The documentation for this feature seems to be here:

• https://kotlinlang.org/docs/java-interop.html#escaping-for-java-identifiers-that-are-keywords-in-kotlin
• https://kotlinlang.org/docs/coding-conventions.html#names-for-test-methods

Notice that this feature is common for test functions. As such, I hope it could be possible to add support for it in the tree-sitter grammar.

I think the parsing result in the return statement is incorrect in the following example.

It is expected that it can be parsed into an additive expression. Like this

Hey 👋 . Thanks for your nice work!

I am trying to get kotlin working with tree sitter and syntax highlighting, but when I open any kotlin file neovim crashes.
I use Windows 10, and neovim 0.7.2

Here is a minimal config:

call plug#begin("~/VimFiles/plugged")
    Plug 'neovim/nvim-lspconfig'
    Plug 'nvim-treesitter/nvim-treesitter'
call plug#end()

lua require'nvim-treesitter.configs'.setup { highlight = { enable = true } }

"Also installed kotlin via: `TSInstall kotlin`

Thank you for your support :)

Hi,

I've found the latest release version was created 9 months before, and 49 commits in the current main branch haven't been included in version 0.2.11. Could you please update the version in npm?

0.2.11...main
https://www.npmjs.com/package/tree-sitter-kotlin

Dear fwcd,

My name is Vladimir Reshetnikov, I'm a software engineer at Relyance AI, currently working on static analysis of Kotlin code. In the past, I worked at JetBrains as Language Specification Lead on the Kotlin team, and was actively involved in design and specification of Kotlin.

In our project, we are using fwcd / tree-sitter-kotlin to parse Kotlin code. I appreciate all the effort you put into creating and maintaining this project, it has been an invaluable tool that I rely on.

To facilitate navigation through the AST and extraction of certain nodes, I decided to add named fields by wrapping parts of grammar rules into field('name', …), similar to how it is done in other tree-sitter parsers, such as tree-sitter / tree-sitter-java. For that purpose, I created a fork of your project and made necessary changes in it: vladimir-reshetnikov / tree-sitter-kotlin. None of the changes modify the shape of the generated AST in any other way, except for adding named fields. A typical example of the changes I made is included below this message.

I would like to submit a pull request to your repository with the changes I've made. Given that Kotlin is still being actively developed, I anticipate possible changes in the grammar in the future, and I would like to avoid divergent histories between the main project and my fork, and reduce the risk of future merge conflicts. I believe that the changes I'm proposing could be beneficial for other users of your parser as well. Please let me know if you are open to this discussion, and if so, then what are the requirements regarding the coding style, stability of field names, consistency of field names with other tree-sitter parsers, etc. I will make all required corrections.

Kind regards,
Vladimir Reshetnikov
Email: [email protected], [email protected]

--- a/grammar.js
+++ b/grammar.js
@@ -171,16 +171,16 @@ module.exports = grammar({
       $._semi
     ),

-    import_alias: $ => seq("as", alias($.simple_identifier, $.type_identifier)),
+    import_alias: $ => seq("as", field('name', alias($.simple_identifier, $.type_identifier))),

-    top_level_object: $ => seq($._declaration, optional($._semi)),
+    top_level_object: $ => seq(field('declaration', $._declaration), optional($._semi)),

     type_alias: $ => seq(
       optional(field('modifiers', $.modifiers)),
       "typealias",
-      alias($.simple_identifier, $.type_identifier),
+      field('name', alias($.simple_identifier, $.type_identifier)),
       "=",
-      $._type
+      field('type', $._type)
     ),

Here is a patch containing only the changes in grammar.js file:
grammar.js.patch

The PR I'm proposing will contain all the changes, including those in the generated parser code.

Given the source code:

val foo: Int?

tree-sitter-kotlin produces:

source_file (0, 0) - (1, 0)
  property_declaration (0, 0) - (0, 13)
    val (0, 0) - (0, 3) "val"
    variable_declaration (0, 4) - (0, 13)
      simple_identifier (0, 4) - (0, 7) "foo"
      : (0, 7) - (0, 8) ":"
      nullable_type (0, 9) - (0, 13)
        user_type (0, 9) - (0, 12)
          type_identifier (0, 9) - (0, 12) "Int"

If the ? was included, I can diff it in difftastic, and it would help syntax highlighters too.

Hey @fwcd ,
Could you please cut a new release after the function_value_parameters commit c84d16e `?

Compiled this to WebAssembly with tree-sitter build-wasm.

The emscripten build used about 12 GB virtual and 6.5 GB physical memory.

(I am currently limited to 8GB RAM on this machine, so that slowed the build significantly because of swap usage; took nearly half an hour.)

The resulting .wasm file has a size of 4.4 MB. I guess that isn't even so bad. Though the next largest grammar that I have build is TypeScript with 720 KB (which just took a few seconds to build).

Not sure if there is even anything to do about this. But it certainly surprised me. Feel free to just close this if you not deem this relevant.

Tree-sitter CLI version: 0.16.4
Emscripten: 2.0.1

I have a variable called isEligible. When I write return !isEligible, it gets parsed as a call to !is with a class called Eligible instead of a variable.

return !isEligible

is parsed as:

(source_file [0, 0] - [1, 0]
  (check_expression [0, 0] - [0, 18]
    (jump_expression [0, 0] - [0, 6])
    (user_type [0, 10] - [0, 18]
      (type_identifier [0, 10] - [0, 18]))))

Compare with:

return !foo

which is parsed as:

(source_file [0, 0] - [1, 0]
  (jump_expression [0, 0] - [0, 11]
    (prefix_expression [0, 7] - [0, 11]
      (simple_identifier [0, 8] - [0, 11]))))

New PRs regularly fail the CI check that verifies that the generated parser matches the committed parser.c, presumably because their development machine didn't generate a byte-for-byte-exact copy of the parser as the GitHub Actions runner.

Perhaps we could add some notes to clarify the motivation behind this check and how to successfully generate a conforming parser.c. Something along the lines of:

• The check exists since it is infeasible to review a ~20 MB C source file. Suggestions for something less brittle than a byte-for-byte check are of course welcome, however.
• Unfortunately, the output of tree-sitter generate only seems to be deterministic when run on the same platform. Since the GitHub-hosted runners use x86_64 Linux, contributors have to generate the parser from that platform too (even aarch64 Linux/macOS will produce a different output).
• One option besides running Linux natively or e.g. through a Docker container would be to use the regenerate-parser.yml workflow in their own fork. This would potentially require customizing the branch it runs on.

I am working on an Emacs major mode for highlighting Kotlin based on this grammar, but I discovered that it is not identified inside of lambda expressions. Ideally there would be a query like (call_expression target: (it_variable) @variable.builtin), but as far as I can tell, there is no support for identifying it as the use of the builtin variable.

Is it possible to identify this already? If not, would it be possible to add this to the grammar?

Thank you!

On this simple example:

bar.foo()
     .show()

tree-sitter will parse as

(source_file [0, 0] - [2, 0]
  (call_expression [0, 0] - [0, 9]
    (navigation_expression [0, 0] - [0, 7]
      (simple_identifier [0, 0] - [0, 3])
      (navigation_suffix [0, 3] - [0, 7]
        (simple_identifier [0, 4] - [0, 7])))
    (call_suffix [0, 7] - [0, 9]
      (value_arguments [0, 7] - [0, 9])))
  (ERROR [1, 3] - [1, 4])
  (call_expression [1, 4] - [1, 9]
    (simple_identifier [1, 4] - [1, 7])
    (call_suffix [1, 7] - [1, 9]
      (value_arguments [1, 7] - [1, 9]))))
test.kt	0 ms	(ERROR [1, 3] - [1, 4])

This is because newlines (parsed as 'semi' in the grammar) are not allowed in navigation expressions.

Originally discussed here: #25

Error message:
node:internal/modules/cjs/loader:1452
return process.dlopen(module, path.toNamespacedPath(filename));
^

Error: dlopen(/Users/jenny.cai/Source/detect-bug-in-code/node_modules/tree-sitter-kotlin/build/Release/tree_sitter_kotlin_binding.node, 0x0001): symbol not found in flat namespace (_tree_sitter_kotlin_external_scanner_create)
at Module._extensions..node (node:internal/modules/cjs/loader:1452:18)
at Module.load (node:internal/modules/cjs/loader:1197:32)
at Module._load (node:internal/modules/cjs/loader:1013:12)
at Module.require (node:internal/modules/cjs/loader:1225:19)
at require (node:internal/modules/helpers:177:18)
at load (/Users/jenny.cai/Source/detect-bug-in-code/node_modules/node-gyp-build/node-gyp-build.js:22:10)
at Object. (/Users/jenny.cai/Source/detect-bug-in-code/node_modules/tree-sitter-kotlin/bindings/node/index.js:3:43)
at Module._compile (node:internal/modules/cjs/loader:1356:14)
at Module._extensions..js (node:internal/modules/cjs/loader:1414:10)
at Module.load (node:internal/modules/cjs/loader:1197:32) {
code: 'ERR_DLOPEN_FAILED'
}

Version:
"tree-sitter": "^0.21.1",
"tree-sitter-javascript": "^0.21.1",
"tree-sitter-typescript": "^0.21.1",
"tree-sitter-java": "^0.21.0",
"tree-sitter-kotlin": "^0.3.6",
"tree-sitter-swift": "^0.3.6",
"tree-sitter-cpp": "^0.22.0",
"tree-sitter-html": "0.20.3"

Hi,
I am trying to write a highlights.scm query file for this parser so that it works with nvim-treesitter in-place of the current kotlin parser that is shipped with it. The edits needed for this would be:

• Unhide the _line_str_escaped_char and _escape_seq, so that escape sequences can be queried
• !is and ?. are not possible to be queried since they are locked behind the _not_is and _safe_nav rules, unlike is, ?:, in,!in and so on.
• Separate the labels from labeled jump_expressions. Could be done by adding a simple_identifier child node to jump_expression
• anonymous function parameters and return type are marked as ERROR nodes in the syntax tree

There is an error parsing functional interfaces.

For example (from kotlinlang.org):

fun interface IntPredicate {
   fun accept(i: Int): Boolean
}

The node which should represent the body/accept method is an error, rather than correctly parsing out the abstract function.

As shown here, interfaces can be implemented by delegation to a constructor parameter. Currently the delegation is parsed as a call_expression.

class Derived(b: Base) : Base by b {
    override fun printMessage() { print("abc") }
}

When editing kotlin-language-server's CodeAction.kt I've noticed that if you have bad syntax, the parser will remove the highlighting from the next lines within the import block. See the screenshot for more context.

Support enum classes.

It'd be great if a new release was cut, so the nixpkgs auto-updater can pick up the recent changes.

Automatically publish tree-sitter-kotlin through crates.io whenever a new tag is pushed.

The problem is described in nvim-treesitter/nvim-treesitter#5657 along with videos demonstrating the issue.

Basically, with the new kotlin parser, there is a noticeable delay in the opening of neovim buffers. The size of the file does not matter. The delay is noticeable even in a small kotlin file with only two lines of code.

For some reason, the 0.3.0 WASM binary built for the web playground (2b27f29) seems to run into an error:

RuntimeError: abort(Assertion failed: undefined). Build with -s ASSERTIONS=1 for more info.
    re https://fwcd.dev/tree-sitter-kotlin/tree-sitter.js:1
    N https://fwcd.dev/tree-sitter-kotlin/tree-sitter.js:1
    pe https://fwcd.dev/tree-sitter-kotlin/tree-sitter.js:1
    Se https://fwcd.dev/tree-sitter-kotlin/tree-sitter.js:1
    load https://fwcd.dev/tree-sitter-kotlin/tree-sitter.js:1
    promise callback*load https://fwcd.dev/tree-sitter-kotlin/tree-sitter.js:1
    handleLanguageChange https://fwcd.dev/tree-sitter-kotlin/playground.js:89
    <anonymous> https://fwcd.dev/tree-sitter-kotlin/playground.js:79
    async* https://fwcd.dev/tree-sitter-kotlin/playground.js:462
[playground.js:91:17](https://fwcd.dev/tree-sitter-kotlin/playground.js)
    handleLanguageChange https://fwcd.dev/tree-sitter-kotlin/playground.js:91
    <anonymous> https://fwcd.dev/tree-sitter-kotlin/playground.js:79
    <anonymous> https://fwcd.dev/tree-sitter-kotlin/playground.js:462

Possible causes would be the introduction of the new scanner or an issue with Emscripten. There seem to be quite a few upstream reports of similar issues. Just to mention a few:

• tree-sitter/tree-sitter#1652
• tree-sitter/tree-sitter#398
• tree-sitter/tree-sitter#1560
• tree-sitter/tree-sitter#1593
• tree-sitter/tree-sitter#1300

Note: Browsers like Firefox seem to cache the WASM binary pretty aggressively, so disabling the HTTP Cache in the devtools is recommended for development.

As a temporary workaround, the playground will keep using 0.2.11 until we've fixed this (5af36d3).

I observed that in current main an expression like

!x || y

gets parsed as prefix_expression(disjunction_expression ... .
In the tree-sitter web playground hosted online it is correctly parsed as disjunction_expression (prefix_expression ...

The ; b.y = in the following expression gets parsed as ERROR , and the entirety of b.z = temp is ERROR.

val reorient = { b: Beacon ->
	when (dir) {
		1 -> { val temp = b.y; b.y = b.z; b.z = temp }
	}
}

property_declaration [125, 1] - [129, 2]
  variable_declaration [125, 5] - [125, 13]
    simple_identifier [125, 5] - [125, 13]
  lambda_literal [125, 16] - [129, 2]
    lambda_parameters [125, 18] - [125, 27]
      variable_declaration [125, 18] - [125, 27]
        simple_identifier [125, 18] - [125, 19]
        user_type [125, 21] - [125, 27]
          type_identifier [125, 21] - [125, 27]
    statements [126, 2] - [128, 3]
      when_expression [126, 2] - [128, 3]
        when_subject [126, 7] - [126, 12]
          simple_identifier [126, 8] - [126, 11]
        when_entry [127, 3] - [127, 49]
          when_condition [127, 3] - [127, 4]
            integer_literal [127, 3] - [127, 4]
          control_structure_body [127, 8] - [127, 49]
            statements [127, 10] - [127, 36]
              property_declaration [127, 10] - [127, 36]
                variable_declaration [127, 14] - [127, 18]
                  simple_identifier [127, 14] - [127, 18]
                navigation_expression [127, 21] - [127, 35]
                  navigation_expression [127, 21] - [127, 24]
                    simple_identifier [127, 21] - [127, 22]
                    navigation_suffix [127, 22] - [127, 24]
                      simple_identifier [127, 23] - [127, 24]
                  ERROR [127, 24] - [127, 33]
                  navigation_suffix [127, 33] - [127, 35]
                    simple_identifier [127, 34] - [127, 35]
            ERROR [127, 37] - [127, 47]

However, changing the expression inside the when branch to:

//...
1 -> { val temp = b.y;; b.y = b.z; b.z = temp }
//...

fixes the parse error.

Add a workflow that creates a GitHub release and attaches the built WASM binary, whenever a tag is pushed.

The official playground looks a bit nicer than ours (which is based on an earlier version of the official playground).

See also:

• https://github.com/sambacha/tree-sitter-playground (a standalone version of the newer playground)
• https://github.com/ikatyang/tree-sitter-yaml (uses a custom playground too)

Source:

infix fun Int.oppa(a: Int) {
}

fun main() {
  1 oppa 4
}

Parsing the examples/Logger.kt file with the WASM parser (which I painfully generated #11 ), yields some errors.

I only tested with the wasm build. No idea whether this is actually an error of the grammar or just caused by the wasm generation being faulty.

https://gist.github.com/t-gebauer/7e42e8f32c95755f15657534e6b3f6fe

Currently, the implementation has to be updated manually whenever the built language version of the parser is no longer compatible with the bundled tree-sitter version, otherwise the web app will fail to launch:

Uncaught (in promise) Error: Incompatible language version 14. Compatibility range 13 through 13.
    setLanguage https://fwcd.dev/tree-sitter-kotlin/tree-sitter.js:1
    handleLanguageChange https://fwcd.dev/tree-sitter-kotlin/playground.js:101

Hi @fwcd

I am the original author of semgrep and we are considering using your tree-sitter-kotlin for adding support for Kotlin in Semgrep.
We have a few extensions to tree-sitter-kotlin that improve the parsing rate from 70% to 90% on the top 20 github kotlin projects.
Looking at the recent activity on this repository, there are a few people like @gzsombor that made PR that didn't get review or merged. Would you mind give more commit rights to external contributors to accelerate the development of tree-sitter-kotlin?

the following code

class Class {

    fun method() {
        return listOf(
            A(),
            B()
        )
    }

}

during parsing becomes

    (source_file 
          (class_declaration (type_identifier) 
             (class_body 
               (function_declaration (simple_identifier) 
                 (function_body 
                   (statements 
                     (infix_expression (jump_expression) (simple_identifier) 
                      (parenthesized_expression (call_expression 
                        (call_expression (simple_identifier) (call_suffix (value_arguments))) 
                        (ERROR (UNEXPECTED 'B')(call_suffix (value_arguments))
)))))))))

Similar list initialization e.g. for a local variable works fine.

code like below will not parse correctly, due to the comment line inside the code block inside the multiline comment:

/**
 *
 * ```kotlin
 * object Foo {
 *     /** this comment breaks syntax parsing */
 *     const val VAR = "VAL"
 * }
 * ```
 */

 object Foo {
     const val VAR = "VAL"
 }

Aliases such as type_identifier (also used e.g. by the Java grammar) would provide a convenient way for syntax highlighting plugins to colorize types.

The latest crate published is 2 years old