-
Generate 5 grams from corpus
-
Generate long-log plot of frequency vs rank order
-
Show whether 5-grams follow Zipf's law and approximate value of alpha
-
Working of parse.py
- Uses parallel processing to digest the corpus and generate the 5-gram files
- Creates a master_dict file that contains an inverted frequency tuple list
- plots the freq vs rank order in log scale using the matplotlib library
-
To Do
- effectively use the multiprocessing functionality to build the master_dict file the reduce part
- try using cluster of computers, probably a cloud to digest data
fxdpntthm / info-ret Goto Github PK
View Code? Open in Web Editor NEWexploring n gram
License: GNU General Public License v3.0