bscong / text-indexing Goto Github PK
View Code? Open in Web Editor NEWWord indexing for large sets of documents
Word indexing for large sets of documents
The programm should stay in search mode until a key combo like Cntrl-C is pressed. This facilitates debugging and improves the user experience.
Describe design, architecture
Todo list for implementing the merge based algorithm:
Computing the IDF: IDF must be computed at the end (after all batches).tf_per_doc
(mapping each word to a dictionary mapping docid
to count) might be too large for a large set of files as a parameter to compute the IDF. We need a data structure like a map mapping each word to the number of documents it appears in (equivalent to a map mapping each word to len(tf_per_doc[word])
)
Storing of non-finalized PL: for the reason above, we should store each batched PL with their term frequency as score (non multiplied by the IDF, as we won't have its value yet).
This means two things: To make each batch usable, we need to keep two separate files: a fully computed PL (with the current IDF) and a un-finalized PL (only with the frequency as a score). The un-finalized PL will be used with the next batch. To keep track of the current IDF, we also need to store the mapping to each word to the number of docs in which they appear.
Merging process: for the merging, we will use the un-finalized PL (on disk) and the computed PL for this batch (in memory). We read line by line (word by word) the on-disk PL, if the same word appears in the current PL we merge both lines. We save two versions of the line: one with IDF multiplication, one without. We do that for each line
Separate functions into different classes
Maybe SortedContainer?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.