Learn Python for Data science #2

Registered my twitter app and used environment to setup config and avoid storing credentials in public github.

What did I learn ?

• Textblob very easy to use, based on nltk, need to download data before using it with
• but not convinced by sentiment analysis with bag of words (see counter example)

To download nltk data:

import nltk
nltk.download()

1. Copy paste the demo.py file
2. Register twitter app
3. Configure credentials with environment variables
4. Use csv library, caution, need to use 'w' and not 'wb' for file modifier with python3

Siraj provide some documentation:

• Some details about Sentiment Analysis: https://www.quora.com/How-does-sentiment-analysis-work
• alchemylanguage which have been superseded by watson

Code:

• Github source code

Documentation:

• About TextBlob: https://textblob.readthedocs.io/en/dev/
• Awesome standford course: http://cs224d.stanford.edu/syllabus.html

Learn Python for Data science #1

Siraj's videos are awesome :)))), @Siraj, when you come to Paris ping me and let's have a drink, you rock !

What did I learn ?

• Thanks to some tipos in buggy.py, I discovered that when data are incoherent, DecisionTreeClassifier.predict is returning sometimes 'male', sometimes 'female'
  • caused by mismatch in both X and Y values
  • DecisionTreeClassifier is using numpy.random.RandomState according to doc
  • It's really important to always test the model and not starting to code without checking the data manually
• About testing
  • unittest is fine to start with
  • Python ecosystem have a lot of testing tools
  • I'll probably use behave, a BDD style library, later
• About Visual Studio Code
  • I used it for nodeJS development, but look good for python too
  • pylint enabled by default

1. Use any 3 SciKit-Learn models on this dataset
2. Compare results
3. Print the best one

First I dig through the scikit doc which looks very good. I find a classifier.py code from classifier comparison page.

Siraj provided a link to choose the right estimator. According to the link for classification with low number of sample:

• try with Linear SVC
• if not ok as not text data, try with KNeighbors
• if still not ok last chance are SVC or ensemble classifiers

Data set is very small, it seems that decision tree, KNeighbors or SVC are OK. Unfortunately LinearSVC is not OK.

Code:

• Github source code

Documentation:

• Scikit documentation to find alternate models
• Docstring conventions helpfull to fix pylint errors

According to stack overflow when a name is not inside a function/class it is considered as a constant.

Regex to check constant name is (([A-Z_][A-Z0-9_]*)|(__.*__))$. I found it in pylint documentation.

Assuming we cannot change the code, what is the alternate option ? 

First generate a .pylintrc file:

pylint --generate-rcfile > .pylintrc

and then add C0103 to disable:

disable=C0103,print-statement,parameter-unpacking,unpacking-in-except,old-raise-syntax,backtick,import-star-module-level,apply-builtin,basestring-builtin,buffer-builtin,cmp-builtin,coerce-builtin,execfile-builtin,file-builtin,long-builtin,raw_input-builtin,reduce-builtin,standarderror-builtin,unicode-builtin,xrange-builtin,coerce-method,delslice-method,getslice-method,setslice-method,no-absolute-import,old-division,dict-iter-method,dict-view-method,next-method-called,metaclass-assignment,indexing-exception,raising-string,reload-builtin,oct-method,hex-method,nonzero-method,cmp-method,input-builtin,round-builtin,intern-builtin,unichr-builtin,map-builtin-not-iterating,zip-builtin-not-iterating,range-builtin-not-iterating,filter-builtin-not-iterating,using-cmp-argument,long-suffix,old-ne-operator,old-octal-literal,suppressed-message,useless-suppression

et voila !