andrewfm / nlpfinal Goto Github PK

It's in the same overleaf project as our proposal report.

@MoonWatcher582
@Scorpio750
@AndrewFM

I'm thinking we'll have three modes of operation. If the program is very confident it has the correct subdomain that matches the question, it'll only search within that subdomain. If it's only kind of confident, it'll search within the top 3 subdomains or something. If it doesn't have a clue which subdomain to use, it'll reject, and tell the user it doesn't know the answer to the question.

Use NLTK's Porter Stemmer to stem the words in the user's question before passing it to the classifier.
(See 23.1.3 in the book)

Augment relevant synonymous words onto the user's question before passing it to the classifier. For simplicity, we can use NLTK's implementation of the WordNet thesaurus. If we want to get fancy (probably not), we can try using clustering to build our own domain-specific thesaurus from the training data, instead.
(See 23.1.6 in the book)

See: http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html#parameter-tuning-using-grid-search

In addition to doing a classification task to figure out which stackexchange subdomain the question belongs to (ie: the category of the question), we also need to do a separate classification task to determine the answer type of the question (ie: person, place, thing).

Read: http://cogcomp.cs.illinois.edu/papers/LiRo05a.pdf

http://cogcomp.cs.illinois.edu/page/resource_view/49
http://cogcomp.cs.illinois.edu/page/tools_view/10

The API has the ability to provide a search query, and it'll return questions that match that query. I think I also saw that it has another function called 'similar', which I think returns questions that are similar to a query. Maybe we want to use that instead of search.

We'll probably grab the top 5 or 10 results to work with, I guess. Also will probably only hold on to the highest-voted/accepted answer from each question.

See 23.2.3 in the book. I think we should aim to do pattern extraction rather than N-gram tiling.

See 23.2.2 in the book.