Read Data

Currently, the data is stored on Labelstudio. We need to fetch the data. We already have a starter code, but we need to make it A) prettier and B) make sure that all passwords, tokens, etc are environment variables and not commited in code

Testing research

We need to figure out a testing plan for the following kinds of tests

• unit tests (what works well, is easy and integrates well with github actions)
• migration tests (same requirements
• E2E tests (same requirements)

Formatting with GH actions (bug)

Formatting with github actions does work, but doesn't get saved and merged with PR

Add a autoformatter

We should add an auto-formatter to our repo to improve code consistency and notation style

Documentation

We need to document how to set up the project locally with all the configuration

• linter so it matches github actions.
• Instructions on how to run linter such that if linting passes locally then it will pass on github
• Instructions on how to run test suites
• Instructions on how to set up poetry and install deps

Metrics Functions

We need a function that can evaluate a model and, given a test dataset, calculates the following aggregate metrics:

• f1 score
• precision
• recal

Setup

Based on the research in previous ticket #15, set up unit testing for the project. This includes the following acceptance criteria

• set up 1-2 unit tests for metrics file
• can run unit tests locally
• unit tests run on all commits and failed tests block merges

Training logic for powerset

We have already added a command line argument (main.py) where users can specify the domains or datasets. What we need to do is we need to have each domain load a different Dataset class. Then, during each iteration of for set of sources in the powerset, we need to use torch Dataset's concat method to concatinate the multiple domains to create a single Dataset. If we have N domains then we will end up creating 2^N - 1 dataset classes, one per iteration.

Where this could happen

It seems to me that this might happen inside the run training fuction. That is, around the for ... in range(epochs) there will be something like for domain_subset in powerset:

This means that we will need to pass the Dataset classes into this function, not the dataloaders.

We can schedule a meeting to discuss this in detail.