martinthenext / criticalsyncing Goto Github PK

Shocking: correspondence between vectors in the pickled matrix and corresponding articles for a particular source in the database is by index. This means that when we decided to look for the most relevant article in Fox News, we search for the closest vector in tfidf_<foxnews_source_id>.pkl. Suppose we get that vector 234 is the closest. Then we look up Article.objects.filter(source=Fox News).all()[234]. It would be better to have a list of article database id's pickled along with every matrix, so that we could find an id of a closest vector in O(1) and retrieve if from the database in O(1).

The goal is to implement Vectorizer module that handles everything concerning tf-idf vector representations of article texts.

How stuff works now

Comand fetch_articles calls Article.objects.download_from(source) and puts articles into the database.

Comand cache_matrices:

1. Gets all articles from all sources and fits a vectorizer.
2. Pickles the vectorizer.
3. Vectorizes all the articles and pickles the huge matrix to tfidf_global.pkl.
4. For every source vectorizes the articles from this source separately and pickles vectors to tfidf_<source_id>.pkl

As new articles will always contain new terms we need to include into the TF-IDF representation, re-fitting the vectorizer and re-computing vectors for all sources is necessary (at this stage, tickets on sklearn-related optimizations will follow). Luckily:

1. We don't have to re-run the update too often.
2. The frequency of updates will not be influenced by the amount of users.

So before we start caring about the opportunity to optimize vectorization, it is important to isolate vectorization into a different module.

Plan

Vectorizer can be a module running on Goldshtein HQ megaframe. It has an access to the database through vpn and runs cache_matrices upon request. The resulting matrices are then available to the django application.

In the conversation with Kirill I thought something more beutiful would work but now I think it's only possible if we find a way to incorporate new terms without re-vectorizing all articles.

Please discuss.