Discrepancy between totaloccurrencesingroup and patterns in getreverseindex about colibri-core HOT 3 CLOSED

confirmed, reproduces on a smaller corpus, cause still unknown

from colibri-core.

Original issue:

$ colibri-patternmodeller -f europarl250000-en-train.colibri.dat -t 2 -W 3 -m 3 -l 4 -s -o model -u && test.py
...
totaloccurrencesingroup:  18501817
(helper structure has 14657 unigrams mapping to 42319 skipgrams total)
Iterations:  21284687

Simplifying, no -W no -m, issue still occurs:

$ colibri-patternmodeller -f europarl250000-en-train.colibri.dat -t 2 -l 4 -s -o modelnowm -u && test.py modelnowm
...
totaloccurrencesingroup:  31341064
(helper structure has 15057 unigrams mapping to 43274 skipgrams total)
Iterations:  34122432.

No skipgrams, issue does not occur:

$ colibri-patternmodeller -f europarl250000-en-train.colibri.dat -t 2 -W 3 -m 3 -l 4 -o modelnos -u && test.py
...
totaloccurrencesingroup:  6214396
Iterations:  6214396

from colibri-core.

Ok, as I suspected this is not a bug, but it is confusing nevertheless. The reason you get more occurrences when you loop over the corpus explicitly and use getreverseindex() is that Colibri Core will actively match the ngram under consideration against all skipgrams in the model. If the ngram is an instantiation of a skipgram, the skipgram will be returned, it doesn't even require the n-gram to be in the model (after all, this should work also for models that contain no ngrams at all). This ngram may therefore well have been below any threshold during computation and not considered a candidate for the skipgram at the time, which is why you end up with a higher occurrence count.

from colibri-core.