Links to useful resources on the topic

Library for "minimal possible solution" using matrix factorization: https://benfred.github.io/implicit/

Agenda

• Module progress report
• Meeting with project coordinator: 21 - 23 - 24 of november (email the coordinator) - @Gromule

Agenda

• Introduction to github
• Introduction to zotero
• Initial literature review and project proposal

Agenda

• Project choice
• Using Git/Github as collaboration platform
• Initial data review
• Discussed potential downsampling
• List of useful notebook and other shared resources

• Scheduled meeting with H&M coordinator to discuss current state of the project

In order to keep the code well organized, it is useful to negotiate some kind of template structure. This will allow to make the code more easily manageable. Specifically by having a dedicated place where certain functional units of code (variables, functions and classes) are expected to be, it will hopefully reduce the time one needs to find and fix bugs.

Initial structure proposal will be placed into project_structure branch of the repository.

It would be nice to have the code to evaluate the performance of the mode. According to the rules, it should do it in the same way, as https://www.kaggle.com/competitions/h-and-m-personalized-fashion-recommendations/overview

Agenda

• Finalize presentation - @Gromule @Justs-Viduss
• Finalize implicit - @Gromule
• Finalize LightFM - @Justs-Viduss
• Finalize project and repository for the project portfolio - @omegatro

Agenda

• Finalize LightFM
• Finalize Implicit
• Finalize project portfolio

https://making.lyst.com/lightfm/docs/

https://github.com/recommenders-team/recommenders

Reminder for individual task

Ensure that team members can access github and use git to collaborate on the project

Scope

• Potential customer groups to target

Agenda

• Finalizing project proposal
• Finalizing literature review
• Selecting libraries to study

Library and course: https://www.tensorflow.org/recommenders

Agenda

• Task prioritization
• Progress report

Bibliography manager software to share common library of publications among group of people

• Please create an account (free): https://www.zotero.org/user/register/
• Send the username/email used to create the account to the chat, so you can be invited to Zotero group
• Library can be accessed online through https://www.zotero.org/groups/5231720/tsi_igp

In order to add resources to library and use Zotero to create citations, please follow the guide here:

• https://libguides.nps.edu/zotero/install-setup
• Zotero connectors (to add references directly from browser) can be obtained here: https://www.zotero.org/download/connectors

Agenda

• Progress update

Agenda

• Project proposal progress report
• Literature review progress report

Agenda

• Progress report
• Email coordinator about meeting time @Gromule
• Microsoft guidelines @omegatro @Gromule @Justs-Viduss

Agenda

• Presentation

18:00

Working with data this big requires a special approach to deal with limited memory of a local machine. One way is to use batch processing. This will imply loading a fraction of the full dataset, conducting the required operations and storing the results in a file, that can later be used by the downstream steps without the need to repeat upstream workflow. Different data types will require different methods.

Methods

• load_batch - load batch of <= N records/texts/images from the dataset into python object (e.g. dataframe, numpy array, lists etc.)
• serialize_batch - convert data from the format obtained after processing step into a format that can be saved into a file
• deserialize_batch - convert serialized data from file into python data structure, that can be passed to downstream task

Expected effect on the workflow

• Each step of the workflow - preprocessing, feature engineering, model training and evaluation will operate on batches
• Batches can be passed directly from previous step (e.g. load_batch > preprocess > extract_features > train_model)
• Batches can be passed from files, that were pre-generated by previous steps (e.g. run the entire dataset through preprocessing & feature extraction, save results to file, then train the model by passing batches of pre-processed data from files)

Pros (apart from RAM benefit)

• Modularity - each step of the workflow can be executed and tested without the need to rerun previous steps
• Batches of test data is easier to share in collaboration
• Easier to identify specific test cases, e.g. if a model has trouble with predicting for values of a specific batch
• Easier train-test split using different batches

Cons

• Takes time to develop
• Takes additional space to store intermediate processing results
• If pre-processing step changes, saved results from downstream steps need to be updated

There are a lot of useful discussions in the competition forum. Many of them are summarized here:

https://www.kaggle.com/c/h-and-m-personalized-fashionrecommendations/discussion/307001

Also, there are many other useful discussions:

1. 1st place solution:

1. https://www.kaggle.com/competitions/h-and-m-personalized-fashionrecommendations/discussion/324070

2. Exploratory Data Analysis (EDA):

1. https://www.kaggle.com/gpreda/h-m-eda-and-prediction
2. https://www.kaggle.com/vanguarde/h-m-eda-first-look

3. Baselines

1. https://www.kaggle.com/code/aerdem4/h-m-pure-pytorch-baseline/notebook
2. https://www.kaggle.com/julian3833/h-m-collaborative-filtering-user-user
3. https://www.kaggle.com/julian3833/h-m-implicit-als-model-0-014
4. https://www.kaggle.com/gpreda/h-m-eda-and-prediction
5. https://www.kaggle.com/titericz/h-m-ensembling-how-to
6. https://www.kaggle.com/lichtlab/0-0226-byfone-chris-combinationapproach/notebook?scriptVersionId=89289696
7. https://www.kaggle.com/lunapandachan/h-m-trending-products-weeklyadd-test/notebook
8. https://www.kaggle.com/tarique7/hnm-exponential-decay-with-alternateitems/notebook

4. Visualizations:

1. https://www.kaggle.com/cdeotte/customers-who-bought-this-frequently-buy-this

5. Processing images

1. https://www.kaggle.com/hamditarek/similar-image-cnn-cosine-similarity

6. The dataset is split into test and train:

1. https://www.kaggle.com/paweljankiewicz/hm-create-dataset-samples