martinthenext / criticalsyncing Goto Github PK
View Code? Open in Web Editor NEWGet the different prospective on news
License: MIT License
Get the different prospective on news
License: MIT License
Shocking: correspondence between vectors in the pickled matrix and corresponding articles for a particular source in the database is by index. This means that when we decided to look for the most relevant article in Fox News, we search for the closest vector in tfidf_<foxnews_source_id>.pkl
. Suppose we get that vector 234 is the closest. Then we look up Article.objects.filter(source=Fox News).all()[234]
. It would be better to have a list of article database id's pickled along with every matrix, so that we could find an id of a closest vector in O(1) and retrieve if from the database in O(1).
The goal is to implement Vectorizer module that handles everything concerning tf-idf vector representations of article texts.
How stuff works now
Comand fetch_articles
calls Article.objects.download_from(source)
and puts articles into the database.
Comand cache_matrices
:
tfidf_global.pkl
.tfidf_<source_id>.pkl
As new articles will always contain new terms we need to include into the TF-IDF representation, re-fitting the vectorizer and re-computing vectors for all sources is necessary (at this stage, tickets on sklearn-related optimizations will follow). Luckily:
So before we start caring about the opportunity to optimize vectorization, it is important to isolate vectorization into a different module.
Plan
Vectorizer can be a module running on Goldshtein HQ megaframe. It has an access to the database through vpn and runs cache_matrices
upon request. The resulting matrices are then available to the django application.
In the conversation with Kirill I thought something more beutiful would work but now I think it's only possible if we find a way to incorporate new terms without re-vectorizing all articles.
Please discuss.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.