stopwords filtering about scattertext HOT 2 CLOSED

Thanks for the kind words.

There are a couple ways of removing stop words from your corpus, although function words can reveal interesting psychological traits. Moreover, modern lexicon mining techniques such as the Scaled F-Score and the Log-Odds-Ratio with a Dirichlet Prior can dampen the class-association score of highly frequent words.

If you'd still like to filter for stopwords, you can use the remove_terms function, as in term_doc_matrix.remove_terms(stop_words, ignore_absences=True).

Here's a complete, working example:

import scattertext as st
from sklearn.feature_extraction.stop_words import ENGLISH_STOP_WORDS
corpus = (st.CorpusFromPandas(st.SampleCorpora.ConventionData2012.get_data(), 
                              category_col='party', 
                              text_col='text',
                              nlp = st.whitespace_nlp_with_sentences)
          .build()
          .remove_terms(ENGLISH_STOP_WORDS, ignore_absences=True))

from scattertext.

Wonderful, thank you very much for this :) I will put it to test.

from scattertext.