Python package for Text Normalization. The purpose of this project is to build a standalone text normalizer for text pre-processing. Version 0.3 includes:

• Spell correct: Correct spelling based in Peter Norvig's algorithm.

• Text sanitization: Merge repeated punctuation or characters, handle contractions

• Text tagging: Place tag on patterns (URL, USER, ACRONYM, EMOJIS among other)

• Text tokenization: Tokenize text (match known patterns)

TextNorm Spell Corrector is based on Peter Norvig's algorithm[1] for word editing. Tagging and Tokenizer is based on ekphrasis[2]. Some features from textacy[3] are used in text pre-processing (only code snippets, no imports)

[1] http://norvig.com/spell-correct.html

[2] https://github.com/cbaziotis/ekphrasis

[3] https://github.com/chartbeat-labs/textacy

git clone https://github.com/kkorovesis/textnorm
cd textnorm
pip install -r requirements.txt
pip install .

from textnorm.components import text_normalizer
text_normalizer.test()