Need help to create embeddings about embeddings-for-trees HOT 9 CLOSED

Hi!
Thank you for the feedback!

It seems that there is a little out of sync between astminer and this lib. In astminer AST stores in JSON under ast key and this library tries to read by key tree. You can change this behavior by changing the value here. Just set AST = "ast".

from embeddings-for-trees.

Hi
Thank you for help, its resolved.
But another same error appeared.

I tried to change NODE = "nodeType" to NODE= "typeLabel" but it didn't work.
I guess it is due to compatibility issue between astminer output and this lib.
There are 2 to 3 ways to create AST using astminer, can you let me know which format this lib is using or help me with above error.
Thanks.

from embeddings-for-trees.

Hi!

Using typeLabel should be ok.
You should use json AST storage in astminer:

storage:
  name: json AST

Could you provide some astminer output examples?

from embeddings-for-trees.

I created ASTs using Antrl provided by astminer.
Here is output example.

{"label":"1542.java","ast":[{"token":"","typeLabel":"compilationUnit","children":[1,2]},{"token":"date|time|parser","typeLabel":"IDENTIFIER","children":[]},{"token":"parser","typeLabel":"IDENTIFIER","children":[]}]}

from embeddings-for-trees.

Oh, it seems I found a bug in vocabulary implementation. Instead of using NODE value from common.py, it uses internal value:
https://github.com/JetBrains-Research/embeddings-for-trees/blob/master/embeddings_for_trees/data/vocabulary.py#L14
Try to change this NODE to typeLabel

from embeddings-for-trees.

Ok thank you. I'll try updating it.

from embeddings-for-trees.

I updated it then I get error in jsonl_dataset.py. It seems NODE value needs to be changed at some other place as well.

from embeddings-for-trees.

I changed NODE to typeLabel in common.py as well. It worked now.

https://github.com/JetBrains-Research/embeddings-for-trees/blob/master/embeddings_for_trees/utils/common.py

Thank you for support.

from embeddings-for-trees.

Nice to hear that it works!
I will update the codebase to make it easier for future usage.

If you have any other questions please feel free to open new issues.

from embeddings-for-trees.