Tokenzier not adding spaceAfter=no about nlp-cube HOT 4 CLOSED

Decision to be made: for languages with no spaces, for the last token of each sentence (e.g. punctuation) the SpaceAfter attribute cannot be determined in the _get_tokens function.
Two options:

1. Use a flag that is passed to the eval/tokenize function by hand: space_after_end_of_sentence = True by default for all languages like EN/DE/RO etc , and set to false for ZH/JA etc. I suggest we use this method and let the user decide to set this optional flag if they need the correct SpaceAfter attribute for the last token.
2. Auto determine from the eval/tokenize input string if the string contains spaces or not. This means that for each tokenize function call, before doing anything, count the number of whitespaces and then if the ratio of whitespace/(total number of characters) < 0.02 (for example), or if the input string does not contain spaces at all, then set space_after_end_of_sentence = False as we will assume that the language does not use whitespaces (ZH/JA/etc.). This adds some overhead for each tokenize call, and could also fail by assuming that one particular sentence does not have spaces (e.g. for sentence "(n.1948-d.2005)." <-yes, that happens).

I implemented option 1, but we can switch to option 2 easily.

from nlp-cube.

Can we just send the SpaceAfter information from the SentenceSplitter?

from nlp-cube.

Added automatic detection for last token of sentence.

from nlp-cube.

Thank you!

from nlp-cube.