aashray / data_science Goto Github PK

Fall 2014 Data Science Course at Stony Brook with Prof. Skiena

TeX 82.41% Python 17.59%

data_science's Introduction

Data Science Project Fall 2014

Fall 2014 Data Science Course at Stony Brook with Prof. Steven Skiena. This was a completely project based course. My team and I worked on the the Ghool Pool Project.

Challenge:

Ghoul pool is a game of prediction which involves guessing when someone will die. Our challenge is to predict/model the risks of people dying . We are given 32 celebrities, from different professions and countries, at a high risk of dying, because of age or lifestyle. The goal is to predict the death (probability) for each of the personalities. Models that make such predictions are used heavily in the life insurance industry.

Various Machine learning regression models have been used.

You can find the complete report and all analysis on : http://www3.cs.stonybrook.edu/~skiena/591/final_projects/ghoul_pool/

data_science's People

Contributors

Watchers

Forkers

anujverma1710

data_science's Issues

Data Matrices: report on the size (number of rows) and composition (descriptions of the columns) of the data matrices you have assembled to build your models from. Where did the data come from, and how satisfied are you with what you have for your project?

Need to make Freebase scrapper better so that it does not get stuck

@pereddy98 Sravanthi, you can work on this. This is priority. I can help you as well. We need to report how much data we have and that is a section by itself in this report. So I think it is important to professor. Let's work on getting this write.

We both can own this.

Please give this priority. We need a large data set with many parameters also, both will important to show good progress in this section.

Comment here with any ideas or if you face any issues.

Look into Zscore for normalization

Prof. said it is a good normalization technique, let's try this out. @varshapaidi

Structure new data correctly

Once, we have a decent chuck of the new (freebase) data. We need to decide on a file structure of it. Basically a schema of the the column we need in our fields, some columns will need to be derived from other columns, like the isAlive and age fields.

Cemetary Scene 'REDO' : The cemetary scenes are disapointing: it is not a pretty old cemetary (try from right across the Emma Clark Library on the Setauket Green) and have a more interesting discussion there in the next video. Don't film with garbage cans in them. Maybe sitting down is better than walking so you stay in tight

Life Tables Evaluation Video

Show your initial attempt at evaluation.
Show me reacting badly.
Explain why I am right and make clear with examples (maybe me explain).
Show how to do it right.
Present the results assuming only dead people.
Recalculate using live people too.
Which does better?

Prediction video

Predictions

Make a film of just your animation of the people being tested, full screen video capture with your audio of comments over it. Perhaps do it twice, once at your current speed, once faster.

Need to submit this separately to prof.
@varshapaidi keeping you as owner, but we both need to do this one.

Make 'efficient' by using Scientific python and not loops

Evaluate Life Tables

Current Model and Baseline: describe your current (presumably machine learning-based) model, and present results from your evaluation environment showing the performance of it against your baseline models. What methods of fitting/data analysis are you using? (5 minutes, 3-5 pages)

Development/Evaluation Environment: report on the block structure of your development/evaluation environment, with what statistics you are reporting to demonstrate the performance of your various model. Show screenshots/output traces so we can see what you are getting from your environment. (5 minutes / 2-3 pages)

Halloween pumpkin video

Prof : Your group needs to make principled predictions on Halloween day, meaning that you have probability distributions which you trust for the death probabilities for all the figures. Film yourself on Halloween tight next some pumpkins or other halloweeen stuff making your predictions, going thrugh the top five highest probability figures. Then do it again making a count down from the top ten, ending on your highest probabilty figure.

Get new data from freebase

We have 90K + 50K + 50K = 180K rows now. Need to get more.

Use Machine Learning techniques in next Model

Next model needs to be an ML based one. Need to use nearest neighbor, decision trees, etc. kind of stuff. @pereddy98 can you look into this, being the ML pro among us :)

Make a square array of the people under evaluation and film over it

Current Prediction and Next steps: State your current forecast for your challenge. Present what you will do next to get a complete predictive model. Discuss any difficulties you will have to overcome in building a good model (4 min / 2-3 pages)

Bubble chart video

Profs task description : The chart you made with death probability with circule colors that picked Mugabi is very good, but the discussion. Produce a video where you are panned very tight over a large clear version of the graph (presuambly on a good screen) and explain it from the start and what it means and why you pick Mugabi

Clean 'normalized_actual_data.csv' to show only what we need for demo and video

Life Tables Explanation Video

Explain life tables. Describe what they are and where they came from.

Do a screen capture of your mouse explaining a web page -- show the life tables -- maybe look up Prof Skiena at 53 -- joke about me surviving the end of the course

Show WHO vs. SS tables and explain why one might be more appropriate than the other.