Open solution to the Avito Demand Prediction Challenge

Home Page: https://www.kaggle.com/c/avito-demand-prediction

License: MIT License

Python 42.68% Jupyter Notebook 57.32%

data-science machine-learning deep-learning kaggle python data-science-learning nlp neptune competition python3

open-solution-avito-demand-prediction's Introduction

Avito Demand Prediction Challenge: open solution

This is an open solution to the Avito Demand Prediction Challenge.

More competitions 🎇

Check collection of public projects 🎁, where you can find multiple Kaggle competitions with code, experiments and outputs.

The goal

Create (entirely) open solution to this competition. We are opening not only the code, but also the process of creating it. Rules are simple:

Clean code and extendable solution are - in the long run - much better than current public LB position
This solution should - by itself - establish solid benchmark, as well as provide good base for your custom ideas and experiments.

Disclaimer

In this open source solution you will find references to the neptune.ml. It is free platform for community Users, which we use daily to keep track of our experiments. Please note that using neptune.ml is not necessary to proceed with this solution. You may run it as plain Python script 😉.

Installation

clone this repository: git clone https://github.com/minerva-ml/open-solution-avito-demand-prediction.git
install requirements
register to Neptune (if you wish to use it)
update neptune.yaml configuration file with your data filepaths
run experiment

with neptune:

$ neptune login
$ neptune experiment run --config neptune.yaml main.py -- train_evaluate_predict --pipeline_name main

collect submit from /output/solution-1 directory.

with pure python:

$ python main.py -- train_evaluate_predict --pipeline_name main

collect submit from experiment_dir directory that was specified in neptune.yaml

Get involved

You are welcome to contribute your code and ideas to this open solution. To get started:

Check competition project here, on GitHub to see what we are working on right now.
Express your interest in particular task by writing comment in this task, or by creating new one with your fresh idea.
We will get back to you quickly in order to start working together.
Check CONTRIBUTING for some more information.

User support

There are several ways to seek help:

Kaggle discussion is our primary way of communication.
Read project's Wiki, where we publish descriptions about the code, pipelines and supporting tools such as neptune.ml.
Submit an issue directly in this repo.

open-solution-avito-demand-prediction's People

Stargazers

Watchers

Forkers

pknut gitter-badger kstrzala samratp-zz zouzias kant mahmud83 tiffen

open-solution-avito-demand-prediction's Issues

Extract meaningful features from images

translate (via some API) text and features to english

check if we can translate using Google translate or Bing.
translate all text and features into English.
make sure that you saved DataFrame with translated text.

add week of year

Add features for periods

periods_train.csv contains information about item_id being on auction. Could be useful

add simple image features

add simple image features based on intensity statistics and colours

Text features - hand crafted features from toxic competition

Check features from Toxic competition and add them to this challenge.

Image features

Based on: https://www.kaggle.com/shivamb/ideas-for-image-features-and-image-quality/code

join param_1/2/3 into new (text) feature

Text is in Russian

Add Stacking Transformer

Add stacking transformer abstraction (pipeline block)

Text features - count vectorizer

Add count vectorizer likely followed by svd

explore dataset in its entirety

images
text
csv
check discussion

train model on 'category name' and 'parent category name'

Train model on images, with targets (multi-output model):

category name
parent category name

Features are two vectors:

probability distribution over category name (softmax)
probability distribution over parent category name (softmax)

deal with NA explicitly

Validation RMSE does not seem to be a good representation of leaderboard score.

Val_loss	LB
0.2296	0.2352
0.2281	0.2351
0.2294	0.2367

requirements missing

Where can I find necessary requirements for the project?

Add simple label encoded features

I have column with text
execute LabelEncoder on this
load it to LGBM or other estimator

Timestamp Features

Explore/add timestamp features:

day
day of the week
month (two values only)

Then explore/add group-by features with times.

UPDATE (from @Leoniak713):

features are extracted
group-by accepts these features

    if train_mode:
        is_missing = Step(name='is_missing',
                          transformer=fe.IsMissing(**config.is_missing),
                          input_data=['input'],
                          adapter={'X': ([('input', 'X')])},
                          cache_dirpath=config.env.cache_dirpath, **kwargs)

        is_missing_valid = Step(name='is_missing_valid',
                                transformer=is_missing,
                                input_data=['input'],
                                adapter={'X': ([('input', 'X_valid')])},
                                cache_dirpath=config.env.cache_dirpath, **kwargs)

        return is_missing, is_missing_valid

    else:
        is_missing = Step(name='is_missing',
                          transformer=fe.IsMissing(**config.is_missing),
                          input_data=['input'],
                          adapter={'X': ([('input', 'X')])},
                          cache_dirpath=config.env.cache_dirpath, **kwargs)

        return is_missing

Text Features - tfidf vectorizer

Add tfidf vectorizer likely followed by svd

Build image category prediction model

Build model that is trained:

train on image and categories
use a feature based on top_1/top_5 predictins or probability for true label

make submit,
run code from branch master,
make submit calculated from this code,
review discussion, add some issues.

minerva-ml / open-solution-avito-demand-prediction Goto Github PK