bscong / text-indexing Goto Github PK

The programm should stay in search mode until a key combo like Cntrl-C is pressed. This facilitates debugging and improves the user experience.

Describe design, architecture

Todo list for implementing the merge based algorithm:

• Computing the IDF: IDF must be computed at the end (after all batches).tf_per_doc (mapping each word to a dictionary mapping docid to count) might be too large for a large set of files as a parameter to compute the IDF. We need a data structure like a map mapping each word to the number of documents it appears in (equivalent to a map mapping each word to len(tf_per_doc[word]))

• Storing of non-finalized PL: for the reason above, we should store each batched PL with their term frequency as score (non multiplied by the IDF, as we won't have its value yet).

• This means two things: To make each batch usable, we need to keep two separate files: a fully computed PL (with the current IDF) and a un-finalized PL (only with the frequency as a score). The un-finalized PL will be used with the next batch. To keep track of the current IDF, we also need to store the mapping to each word to the number of docs in which they appear.

• Merging process: for the merging, we will use the un-finalized PL (on disk) and the computed PL for this batch (in memory). We read line by line (word by word) the on-disk PL, if the same word appears in the current PL we merge both lines. We save two versions of the line: one with IDF multiplication, one without. We do that for each line

Separate functions into different classes

Maybe SortedContainer?