This script processes wikipedia article dumps from https://dumps.wikimedia.org/enwiki/ and gathers the word frequency distribution data. The script uses wikiextractor to fetch raw text, then strips punctuation marks and normalizes unicode dashes and apostrophes. The script then disregards words that have a digit in them, and only takes words that were used in at least 3 different articles.
The script was inspired by this article which unfortunately provided very inaccurate data with punctuation marks and other sorts of inaccuracies.
The script needs Python 3. On macOS, there is a known bug with Python 3.8, so you will need to use Python 3.7 or lower.
Install requirements:
pip install -r requirements.txtDownload the current Wikipedia dumps for the desired language:
WIKI=enwiki
wget -np -r --accept-regex \
"https:\/\/dumps\.wikimedia\.org\/${WIKI}\/latest\/${WIKI}-latest-pages-articles[0-9]*\.xml.bz2" \
https://dumps.wikimedia.org/${WIKI}/latest/Note that for enwiki (as of April 2023) this will require about 19 Gb of free space.
Parse dumps and save results:
python ./gather_wordfreq.py dumps.wikimedia.org/${WIKI}/latest/*.bz2 > wordfreq.txtThe word frequency data for English, Spanish, French, German, Italian, Portuguese, Dutch, Arabic, Polish, Egyptian, Japanese, Russian, Cebuano, Swedish, Ukrainian, Vietnamese, Chinese, Waray, Afrikaans & Swahili are provided at results.
English results:
- Total unique words appearing at least in 3 articles: 2747823
- Top 20 most popular words: the, of, in, and, a, to, was, is, on, for, as, with, by, he, that, at, from, his, it, an.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent