BreakIteratorSegmenter: New parameter for punctuation marks about dkpro-core-asl HOT 9 CLOSED

A patch for this issue.

I have had a look at the patch and it seems to work for German or English, but 
I am not quite sure what the side effects will be for other languages.
Additionally, the opinions about what should be considered as punctuation might 
differ.

Thus, I would rather suggest to implement this functionality as a subsequent 
filtering step where all annotations (type of the annotation would be a 
parameter) are removed that correspond to some pattern (another parameter).

I find it a bit strange that the patch looks at the last character of a token 
to decide if a token should be removed or not. 

In terms of alternatives, we have a TokenFilter in the tokit module that 
currently filters out tokens that are too long. That, however, does not handle 
attached POS or Lemma annotations. 
There is also the StopWordRemover which is dictionary-based and is configurable 
with respect to types.

Maybe it would be useful to merge all of that into a single AnnotationFilter 
which can do regexes or dictionaries - or maybe even dictionaries of regexes? 
;) For sake of speed separate parameters for max length and min length might 
also be useful.

> I find it a bit strange that the patch looks at the last character of a token 
to
> decide if a token should be removed or not. 

Indeed.

> Maybe it would be useful to merge all of that into a single AnnotationFilter 
which
> can do regexes or dictionaries - or maybe even dictionaries of regexes? ;) 
For sake
> of speed separate parameters for max length and min length might also be 
useful

Sounds good. Do we aim at the whole thing (lists of dictionaries of regexes ;) 
or better start small?

Changing from the dictionary-based to dictionary-of-regexes should be little 
more a parameter (PARAM_REGEX = true) and switching from equals() to matches().

What's bugging me more is the question of how to deal with POS, Lemma, Stem and 
so on. Traditionally these are co-located and there was no link in Token to 
refer to them. But this is awkward for programming and bad for performance. So 
since recently, the Token has explicit fields to refer to POS and Lemma and I 
am not sure about Stem. It would be possible to override the removeFromIndex() 
method in Token to automatically also remove associated POS. Lemma, Stems - but 
this would only work for JCas. The other alternative is the mechanism used in 
the StopWordRemover - works but is much slower and needs more configuration 
effort. Maybe a combination of both would be good so that CAS-based AEs could 
use the configurable method (that could become a convenience method in uimaFIT 
CASUtil) and JCas-based AEs could call Token.removeFromIndexes() and Lemma, POS 
and Stem are automatically cascaded.

What do you think?

As far as I see, it would make the component quite dependent on our type 
system, right.
This is not a problem in general, but I would prefer a type system agnostic 
component, and maybe additionally a more specialized one.

What do you mean by "more specialized".

Looks like this issue has been superseded by issue 14 (rename and enhance 
tokenfilter). We won't change the BreakIteratorSegmenter because that would 
imply we also need to change all other tokenizers that we have and may have in 
the future.