is there a better way to distinct labels from training data sources?

Data X [WIP]

Data X aims to be a swiss army knife for producing production ready models.

It works with 3 main 'modules':

DataSource (the sources of data to train models)
Model (the type of model, tasks & params eg distillbert, multilabel)
Pipeline (composed of datasources and a model)

You can see example configuration in the example directory.

Pip install

To install via pip you can clone the directory and run make -C data-x install TODO: make installable via git + pip

Poetry install

To install first install poetry, clone the repo and run poetry install TODO: pip package

Once installed you can use data-x --help to explore configured modules and do various tasks

To kick off the pipeline you can use data-x pipeline run-pipeline ./config.yml <PIPELINE_NAME>

TODO: Better README & documentation

NLP

Current version includes two types of classification model. Single and Multi label. These can be configured by config.yml by declaring it task: singlelabel or task: multilabel

	if ds.type == 'local-labels-txt': #TODO: is there a better way to distinct labels from training data sources?
	with open(ds.path, 'r') as f:
	content = f.readline()
	labels = content.split(',')
	return pd.DataFrame(labels, columns =['labels'])

	#TODO remove once transformation code is implemented expensive
	train_df['labels'] = train_df['labels'].apply(literal_eval)
	test_df['labels'] = test_df['labels'].apply(literal_eval)

	MODEL = m.variant
	MODEL_CASE = f'{MODEL}-base-uncased'

	To install via pip you can clone the directory and run `make -C data-x install` TODO: make installable via git + pip

	## Poetry install
	To install first install poetry, clone the repo and run `poetry install` TODO: pip package

	Once installed you can use `data-x --help` to explore configured modules and do various tasks

	TODO: Better README & documentation

	## NLP
	Current version includes two types of classification model. Single and Multi label. These can be configured by config.yml by declaring it `task: singlelabel` or `task: multilabel`

	## TODO: proper error, force quit pipeline
	print('task type not found')

	install: #TODO better way to install for local dev?
	pip install -r requirements.txt
	pip install .

pritesh-patel / data-x Goto Github PK

data-x's Introduction

Data X [WIP]

Pip install

Poetry install

NLP

data-x's People

Contributors

Stargazers

Watchers

data-x's Issues

This issue was generated by todo based on a TODO comment in 727a3ed. It's been assigned to @Pritesh-Patel because they committed the code.

This issue was generated by todo based on a TODO comment in 7fdec47. It's been assigned to @Pritesh-Patel because they committed the code.

This issue was generated by todo based on a TODO comment in f6f3808. It's been assigned to @Pritesh-Patel because they committed the code.

This issue was generated by todo based on a TODO comment in d8c15ed when #6 was merged. cc @Pritesh-Patel.

This issue was generated by todo based on a TODO comment in f2d987c. It's been assigned to @Pritesh-Patel because they committed the code.

This issue was generated by todo based on a TODO comment in f6f3808. It's been assigned to @Pritesh-Patel because they committed the code.

Recommend Projects

Recommend Topics

Recommend Org

This issue was generated by todo based on a `TODO` comment in `727a3ed`. It's been assigned to @Pritesh-Patel because they committed the code.

This issue was generated by todo based on a `TODO` comment in `7fdec47`. It's been assigned to @Pritesh-Patel because they committed the code.

This issue was generated by todo based on a `TODO` comment in `f6f3808`. It's been assigned to @Pritesh-Patel because they committed the code.

This issue was generated by todo based on a `TODO` comment in `d8c15ed` when #6 was merged. cc @Pritesh-Patel.

This issue was generated by todo based on a `TODO` comment in `f2d987c`. It's been assigned to @Pritesh-Patel because they committed the code.

This issue was generated by todo based on a `TODO` comment in `f6f3808`. It's been assigned to @Pritesh-Patel because they committed the code.