mlopatka / canosp2020 Goto Github PK
View Code? Open in Web Editor NEW[staging] CANOSP project repo for January 2020 cohort
License: Mozilla Public License 2.0
[staging] CANOSP project repo for January 2020 cohort
License: Mozilla Public License 2.0
Parse the entire SUMO corpus via the Kitsune API and store a local copy so that we are not hitting the API constantly.
Count the word frequency across the entire corpus.
The CrowdTruth library uses the timing associated with tag attribution events as a factor for assessing tagger and tag reliability. We currently clobber this functionality by passing a uniform time value for every tag/item/tagger triplet.
It would be ideal to remove the timing factor from the CrowdTruth code or refactor the code to optionally omit its usage in a custom fork.
Right now our requirements.txt
is broken, it doesn't work on a fresh clone of the repo. We should be able to:
We'll switch from pip's requirements.txt
approach to using the Anaconda package manager
Right now it just crashes when it encounters non-english text.
I propose that we just remove non-english text from our dataset.
Ensure that the outputted CSV is valid and conforms to our standard
Ensure that the outputted Dataframe is valid and conforms to our standard
Add jsonschema validation to ensure that all parsed data adheres to the JSON schema defined in the repo wiki.
Validate all JSON files under data/
directory.
The functionality described in CrowdTruth pertaining to annotation quality metrics in the presence of ambiguity/annotator disagreement may be useful in defining a tag-set for SUMO tickets.
A first step towards integration is to evaluate our manually generate tag set against the metrics defined in compute_ann_quality_factors
Possible normalization can be performed using the entirety of the SUMO corpus or a subset. Some strategies may be to:
This issue should be closed by the production of a notebook comparing the impact of different strategies and the downstream impact on some primitive models. A decision about what normalization strategy is advocated for should be specified.
The CSV file from #92 has duplicates:
cd ~/GIT/CANOSP2020/data
uniq -D mturk_tickets.csv | wc -l
8
This should be 0 not 8 right?
Please:
Each of the different exploratory directions are using a lot of manual regex-based data cleaning.
We should consolidate that code into a utils file and use consistent prepossessing across all our different approaches.
We'll need to think about how we handle strange text when calculating statistics and training models.
Human annotator tags support ticket with free-form text. However, many of them share similarities (typo, variation of words, etc) such that we could cluster them into different groups.
Use T-SNE model from sklearn to explore the possibility of using it to implement a classification algorithm
word2vec
or other pre-trained word embedding model won't work for us since all we have is a bunch of tags that are hardly a word.char2vec
might be handy, https://github.com/Lettria/Char2VecWe need to associate a set of manually contributed tags to a sumo issue with a particular id.
The tags are currently spread across multiple google sheets and csv files.
At least some of the tickets we tagged in #48 are not included in our 200MB ticket JSON. We need to scrape these new tickets so that we have the title and body for them.
Define a data structure in Python.
Write a function in Python which takes a text
argument and does the pre-processing work, the function should also implement a "feature flag" which we would be able to turn stuff on and off.
Useful aggregates that provide meaningful and actionable insights include:
Get a working implementation that wraps a word2vec constructor at the support ticket level.
https://github.com/danielfrg/word2vec
We need the title and content of the tickets that we tagged, so we need to request that using the Kitsune API.
See WIKI page, https://github.com/mlopatka/CANOSP2020/wiki/Ticket-JSON-format
Now that we've each tagged those 100 tickets, we need to add them to our data (csv / json) in order to analyze them.
Based on my research, there are two approaches to apply the Naive Bayes model to train our model.
The first one is to embed the Naive Bayes model inside classifiers, such as ClassifierChain, MLkNN, RandomForestClassifier, etc.
Another way is pretty simple, we use Naive Bayes directly and fit over data into it.
Which approach is better to use for our project?
@konantian @ExiaSR Please coordinate to get this done and discuss the process here.
We are not really getting anything out of textpipe
beside then stripping out the HTML tags.
It is giving us pain to install on our workstation, and hard to process large amount of data at once.
I propose we consider replace textpipe with beautifulsoup
+ spacy
for preprocessing work.
For some reason results["units"]
only shows myself and Yuan in the worker column. Try to figure out why.
Because there are some tokens after our current preprocessing are numbers, it is good to try to convert these numbers into simple words. For example, for number 70, we convert it to seventy instead. After converting all the numbers to words, we don't have to treat them as different tokens, since they express the same meanings to the user.
I believe this issue makes issue #106 obsolete. I will close #106
From our 1500 support tickets, Mechanical Turk has created this CSV file:
13march2020-630pm-Batch_3952790_batch_results.csv
Please ingest into our pipeline:
Right now we're missing this portion of our data pipeline
We need to determine a specific subset of support tickets to redundantly manually tag in order to reduce the sparsity of our training data.
We should be able to quickly go between a schematic JSON file including meta data fields and a reduced/flattene subset of the features that is useful for pytorch/torchtext or textpipe type libraries
The last couple taggers in tickets.csv don't have the is_expert column filled in. It should be 1
Optional in the pre-processing workflow
Similar to is_expert
and is_sumo
, we should include an is_mturk
column to our CSV schema.
From #48 , Mozilla employees Evelyn and Brady have completed their 2nd round of tagging and I've placed it in this Google Drive Folder:
Please:
Explore different classifiers using a defined set of tags. Compare performance over a subset of tagged data and over a subset of untagged data. Report on results using a cloned version of notebook template
fetch_tickets.py
has functionality for merging scraped SUMO tickets (the 200mb ticket JSON) with our own annotated tickets (data/annotated_tickets.csv
) into one big JSON file. Once we have this master JSON we can convert it to CSV and rerun all of our analysis with the new data.
Requirements:
Let's keep an up to date environment definition in this repo to ensure compatibility.
There are some tickets in this csv file that have empty tags or spam tickets. Since these tickects will not be used in the training dataset. It is better to remove all of them from the CSV file.
Include more information about the project objectives and the CANOSP program in the project README.md
This process should be manually completed by EVERY contributor before this issue can be closed. Please use the comments section here to track progress of each participants' completion of the 100.
All pre-processing code should use https://github.com/mlopatka/CANOSP2020/blob/master/canosp2020/preprocessing.py to perform text cleaning
From our 500 support tickets, Mechanical Turk has created this CSV file:
11March2020-820am-Batch_3945892_batch_results
Please ingest into our pipeline:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.