can someone suggest how to make this all work for inactive regions (e.g. proprocessor inactive chunks)? I think it would rely on clangd features in https://reviews.llvm.org/D67536. Will these get mapped to a specific hl group by this plugin, or is something else required? Thanks.

If I understand correctly a variable with both readOnly and fileScope will be end up with up to 3 highlight groups applied to it: Variable, ReadOnlyVariable and FileScopeVariable. That seems to assume that all modifiers are independent, but I don't think that always applies. I may want to highlight ReadOnlyFileScopeVariable specially, or at least differently from both ReadOnlyFunctionScopeVariable and (mutable) FileScopeVariable. That doesn't seem to be possible with your current table-based approach.

One option would be to have a user-supplied function like highlightGroups(fileType, tokenKind, modifierList) -> optional<highlightGroupList> and then fall back to the simple table-based approach if that returns nil. That allows fine tuning for "interesting" cases while still using a simple declarative style for normal cases. It also allows useful but non-trivial logic like looking for variables tagged with fileScope without readOnly and highlighting them MutableGlobalVariable (which is really the case I'd want to highlight).

If there is already some way in (neo)vim to color based on multiple highlight groups rather than just one, that would also work, but as far as I know, they are independent.

Should on_invalidate_range/clear_highlights be called on every event?

neovim: 0.7.2

I installed the latest nightly neovim, put this plugin in my plugins/start directory, pasted the setup code from the README.md file, and tried the following in a C file with clangd running:

:hi link LspEnumMember Keyword
:hi link LspEnum String
etc., using suggestions from the doc file on highlighting.

Nothing changes in my file. No color highlighting changes.

lsp.log shows ASTWorker building file /.../t.c

output of :lua =vim.lsp.semantic_tokens shows things like

{
  end_col = 18,
  extmark_added = true,
  line = 25,
  modifiers = { "declaration", "readonly", "fileScope" },
  start_col = 4,
  type = "enumMember"
}

and

{
  end_col = 14,
  extmark_added = true,
  line = 33,
  modifiers = { "globalScope" },
  start_col = 5,
  type = "enum"
}

So it looks like the server is sending back the correct info, but the highlight links don't appear to do anything.

The log shows this on a :LspRestart:

--> workspace/semanticTokens/refresh(0)\n"
<-- $/cancelRequest
I[21:00:16.192] <-- textDocument/semanticTokens/full(3)\n"
<-- reply(0)\n"
--> textDocument/publishDiagnostics\n"
--> reply:textDocument/semanticTokens/full(2) 48 ms, error: Task was cancelled.
I[21:00:16.219] --> reply:textDocument/semanticTokens/full(3) 27 ms\n"

Edit: forgot to mention I'm using:

$ clangd --version
Ubuntu clangd version 14.0.0-1ubuntu1
Features: linux+grpc
Platform: x86_64-pc-linux-gnu

Often (not always) when I trigger the "add missing include" code action by clangd, the following error pops up:

E486: Pattern not found: error:                                                                                                                                                                
Error executing vim.schedule lua callback: .../start/nvim-semantic-tokens/lua/nvim-semantic-tokens.lua:35: col value outside range                                                             
stack traceback:                                                                                                                                                                               
        [C]: in function 'nvim_buf_set_extmark'                                                                                                                                                
        .../start/nvim-semantic-tokens/lua/nvim-semantic-tokens.lua:35: in function 'highlight'                                                                                                
        .../start/nvim-semantic-tokens/lua/nvim-semantic-tokens.lua:46: in function 'highlight'                                                                                                
        ...ic-tokens/lua/nvim-semantic-tokens/table-highlighter.lua:60: in function 'highlight_token'                                                                                          
        .../start/nvim-semantic-tokens/lua/nvim-semantic-tokens.lua:50: in function 'on_token'                                                                                                 
        ...ntic-tokens/lua/nvim-semantic-tokens/semantic_tokens.lua:103: in function 'handler'                                                                                                 
        /home/devel/local/nvim/share/nvim/runtime/lua/vim/lsp.lua:1390: in function ''                                                                                                         
        vim/_editor.lua: in function <vim/_editor.lua:0>   

Is there any way to at least suppress this? It can be quite annoying.

Hi,

Since neovim/neovim#15723 has completely stalled, would you be interested in including the actual LSP functionality into this plugin? I've been using that code for a year now by just copying it to my dotfiles, but your code looks way cleaner and I would rather use it as a plugin.

I think I can create a PR too if you want.

I think this is because of line 71 in table-highlighter.lua.

If i remove the first parameter, i.e. change it to highlight(hls, hl), everything works as expected.

There is no such token type in the spec

vscode:

Hi,
I really want to try the semantic highlighting in neovim and searched a lot, finally found this plugin. 😀
So I have

• upgraded my neovim from 0.7.2 to neovim 0.8
• installed this plugin, using the default setup
• using the neovim native lsp
• open a python file or c file, pyright or clangd (ver 15.0) are successfully attached
• call the function lua im.lsp.buf.semantic_tokens_full() after entering the buffer, but returns "method textDocument/semanticTokens/full is not supported by any of the servers registered for the current buffer"

How could I debug for this? Thank a lot

Some token types are missing for the python server. Everything else works.

{
    "tokenTypes": [
        "comment",
        "keyword",
        "string",
        "number",
        "regexp",
        "type",
        "class",
        "interface",
        "enum",
        "enumMember",
        "typeParameter",
        "function",
        "method",
        "property",
        "variable",
        "parameter",
        "module",
        "intrinsic",
        "selfParameter",
        "clsParameter",
        "magicFunction",
        "builtinConstant"
    ],
    "tokenModifiers": [
        "declaration",
        "static",
        "abstract",
        "async",
        "documentation",
        "typeHint",
        "typeHintComment",
        "readonly",
        "decorator",
        "builtin"
    ]
}

Maybe this inside the on_attach would be better than an if statement:

local on_attach = function(client, bufnr)
    if client.resolved_capabilities.semantic_tokens_full == true then
        vim.cmd [[autocmd BufEnter,CursorHold,InsertLeave <buffer> lua vim.lsp.buf.semantic_tokens_full()]]
    end
    --...
end

I wanted a command like the one treesitter has to debug semantic tokens. So, I came up with this:

local function get_bit(n, k)
    if _G.bit then
        return _G.bit.band(_G.bit.rshift(n, k), 1)
    else
        return math.floor((n / math.pow(2, k)) % 2)
    end
end

local function modifiers_from_number(x, modifiers_table)
    local modifiers = {}
    for i = 0, #modifiers_table - 1 do
        local bit = get_bit(x, i)
        if bit == 1 then
            table.insert(modifiers, 1, modifiers_table[i + 1])
        end
    end
    return modifiers
end

local function handler(err, response, ctx, config)
    local client = vim.lsp.get_client_by_id(ctx.client_id)
    if not client then
        return
    end
    local legend = client.server_capabilities.semanticTokensProvider.legend
    local token_types = legend.tokenTypes
    local token_modifiers = legend.tokenModifiers
    local data = response.data

    local line
    local start_char = 0
    for i = 1, #data, 5 do
        local delta_line = data[i]
        line = line and line + delta_line or delta_line
        local delta_start = data[i + 1]
        start_char = delta_line == 0 and start_char + delta_start or delta_start
        local token_type = token_types[data[i + 3] + 1]
        local modifiers = modifiers_from_number(data[i + 4], token_modifiers)

        local token = {
            line = line,
            start_char = start_char,
            length = data[i + 2],
            type = token_type,
            modifiers = modifiers,
            offset_encoding = client.offset_encoding,
        }

        local line, col = unpack(config.cursor)
        if token.line == line - 1 then
            if token.start_char <= col and col <= token.start_char + token.length then
                vim.pretty_print(token)
            end
        end
    end
end

local function semantic_token_under_cursor()
    local cursor = vim.api.nvim_win_get_cursor(0)
    local params = { textDocument = require("vim.lsp.util").make_text_document_params() }
    local h = vim.lsp.with(handler, { cursor = cursor })
    vim.lsp.buf_request(0, "textDocument/semanticTokens/full", params, h)
end

vim.api.nvim_create_user_command("SemanticTokenUnderCursor", semantic_token_under_cursor, {})

It would be better if I could reuse the on_full handler with a custom on_token, but it returns at