Problematic tokens from tokenizer? about sourcerercc HOT 4 CLOSED

ERROR: more than one doc found is a problem. This must be happening when SourcererCC searches for the tokens of a document (using document id as the query) in the forward index. Ideally we should never get more than one doc, as document id for each document should be unique. My guess is that this is happening because we might have assigned same id for more than one document in the parsing stage, or may be we indexed one document twice.

About the ArrayIndexOutOfBoundsException, let's keep a track of these characters. We should remove them during the tokenizing stage.

from sourcerercc.

I meet the similar problem with you. My error message is
EXCEPTION CAUGHT, invalid line: Bud% @� @ @ @ E%DSDB @ @ @
index size of GTPM: 66012
Directory: dataset
indexing file : .DS_Store
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
at noindex.CloneHelper.deserialise(Unknown Source)
at indexbased.SearchManager.doIndex(Unknown Source)
at indexbased.SearchManager.main(Unknown Source)`
Every step of the operation is performed in accordance with the steps in readme. What does it mean? My data is wrong?

from sourcerercc.

please delete the .DS_Store file from the dataset directory.

from sourcerercc.

I'm hoping this solved the problem. If not, please open another issue.

from sourcerercc.