gesistsa / tokenvars Goto Github PK

attr(x, "docvars")$tokenvars_ (rather than attr(x, "tokenvars")) sounds like a good deal; although quanteda:::is_system() will not protect it. Also, docvars(x) will expose it.

But tokens_subset(), tokens_chunk() and maybe more will work.

• tokennames (like corpus(docnames))
• tokennames()
• tokennames()<- (going to be difficult)

Given gesistsa/quanteda.proximity#35 and quanteda::tokens_*() will not respect tokenvars, it would be better to make this a new class for now.

• Create a new class tokens_with_tokenvars
• tokens_with_tokenvars.as.tokens()
• docvars.tokens_with_tokenvars()
• meta.tokens_with_tokenvars()

The functionalities are not completed

• tokenvars(x, field) <- value (list of vectors)
• tokenvars(x, field, docnames) <- value (vector / list of vectors)
• tokenvars(x, field, docnames, tokennames) <- value (single value / vector / list of vectors)
• tokenvars(x) <- value (list of dataframes)

I think the parsed result from udpipe is a good starting point. See also statsmaths/cleanNLP

What can one do with tokenvars?

• tokens_with_tokenvars_swap_tokens(x, field) - useful for replacing tokens with lemma
• tokens_with_tokenvars_select_tokens(x, subset) - Similar to tokens_select, but using subset (NSE) like tokens_subset; useful for e.g. selecting tokens by POS

1. a data.frame?
2. a list of vectors?
3. Or even fancy thing like a fastmap?