Is your feature request related to a problem? Please describe.

Right now there is no way to allow configs (other than hparams) to pass into build_model method.

Describe the solution you'd like

automodel = AutoImageClassification(datamodule=datamodule, build_model_conf = {"pretrained": True })
automodel.hp_tune()

Is your feature request related to a problem? Please describe.

Create Tasks directly from AutoClassifier instead of explicitly calling AutoImageClassification or AutoTextSummarization

Describe the solution you'd like

model=AutoClassification(datamodule, data_type="image")  # expected `data_type`-> image, text, infer
model.hp_tune()

Is your feature request related to a problem? Please describe.

Single epoch might be very large for few datasets. max_steps will reduce training time per epoch.

model = AutoImageClassifier(datamodule,
            suggested_conf=suggested_conf,
            max_epochs=10,
           max_steps=100,
            optimization_metric="val_accuracy",
            timeout=100)

Is your feature request related to a problem? Please describe.

Right now after hp_tune new model is instantiated with best-found hyper-parameters. We can load the already trained weights as well.

https://stackoverflow.com/questions/62144904/python-how-to-retrive-the-best-model-from-optuna-lightgbm-study

tuner.add(key, tune.func(...))

https://github.com/gradsflow/gradsflow/blob/main/gradsflow/tuner/tuner.py#L39

Why Ray over Optuna?

Initially, I started with Optuna for HPO because of its Pythonic APIs and powerful search algorithms. Later I realized that Gradsflow will have to incorporate logic for distributed training but Ray already provides this out of the box.

Ray provides a simple, universal API for building distributed applications and it already supports multiple hyperparameter tuning libraries including Optuna.

• With Ray, we get to leverage distributed training and HP search out of the box.
• We get to use search algorithms not only by ray but also optuna and some other cool libraries.
• Easy process and GPU management. Ray even supports gpu_fraction training.

NOTE: User API will remain the same and you won't feel any difference apart from Ray's cool distributed training features. 🔥

When I use "from gradsflow import AutoImageClassifier"
bug: cannot import name 'AutoImageClassifier'

Is your feature request related to a problem? Please describe.

optimization_metric should be flexible. RN only supports accuracy and loss (val_accuracy, val_loss, train_accuracy).

Describe the solution you'd like

It should also support other optimization metrics like - F1, Fbeta

Bug description

Error while running examples/nbs/01-ImageClassification.ipynb notebook

2022-01-09 14:26:54,162	WARNING experiment_analysis.py:677 -- Could not find best trial. Did you pass the correct `metric` parameter?

Expected result

Actual result

Steps to reproduce

Context

Your Environment

• Version used:
• Operating System and version:
• Link to your fork:

Bug description

model = ImageClassification(dm)
model.hp_tune()

GPU was not selected automatically. ray.tune need to pass resources_per_trial['gpu'] to enable GPU support.

Steps to reproduce

Run the attached code

• Version used: 0.0.3a2
• Operating System and version:
• Link to your fork:

🔥 Dependabot raised security vulnerability concern with mkdocs's version.

I would suggest you to only version-pin mkdocs-material (as every version of mkdocs-material installs mkdocs, and version-pins it as well). Otherwise, if you prefer to keep the mkdocs version-pinned, consider changing it to mkdocs>=1.2.3.

# file: docs/requirements.txt

git+https://github.com/gradsflow/gradsflow@main
mkdocs>=1.2.2
mkdocs-material>=7.2.4
mkdocs-material-extensions==1.0.1
mkdocs-git-revision-date-localized-plugin==0.9.2
mkdocs-macros-plugin==0.6.0
mkdocs-autorefs>=0.2.1
mkdocstrings>=0.15.2
tags-macros-plugin @ git+https://github.com/jldiaz/mkdocs-plugin-tags.git@d26e2f124e4f3471639d426459e281080988fe7a
mkdocs-jupyter>=0.18.0
mkdocs-meta-descriptions-plugin
jupyter_contrib_nbextensions
comet_ml
lightning-flash[image,text]>=0.5.1

The PyPI package of gradsflow has tests folder. Typically the tests folder should not be included in the package (one key reason is: this keeps the package leaner and the user has nothing to do with the tests either), unless they are kept their for some specific reason by design (this deviates from common-practice though).

This was discovered while working on this PR: conda-forge/staged-recipes#17215

Add meta_data in AutoDataset for storing classes, annotations info

Bug description

Load Project Description in __init__.py from mkdocs.yml maintaining one source of truth

Expected result

Actual result

Steps to reproduce

Context

Your Environment

• Version used:
• Operating System and version:
• Link to your fork:

• implement on_forward callback and modify HF model output to GF format.

Is your feature request related to a problem? Please describe.

We currently have support for image and text classification it would be good to have support for summarisation too

Is your feature request related to a problem? Please describe.

For image and text classification we have to do use ImageClassification or TextClassification.
This can be auto-inferred from the data type.

Describe the solution you'd like

automodel = AutoClassification(datamodule=dm, task='infer')  # other task could be 'image' or 'text'
automodel.hp_tune()

Bug description

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-26-61cc685a1634> in <module>
----> 1 model.fit(autodataset=autodata, max_epochs=10)

~/miniconda3/envs/aniket/lib/python3.7/site-packages/gradsflow/models/model.py in fit(self, autodataset, max_epochs, steps_per_epoch, callbacks, resume, show_progress, progress_kwargs)
    258 
    259         try:
--> 260             self.callback_runner.with_event("fit", self._fit_with_event, FitCancel)
    261         except KeyboardInterrupt:
    262             logger.error("Keyboard interruption detected")

~/miniconda3/envs/aniket/lib/python3.7/site-packages/gradsflow/callbacks/callbacks.py in with_event(self, event_type, func, exception, final_fn)
     41         try:
     42             getattr(self, start_event)()
---> 43             func()
     44         except exception:
     45             getattr(self, cancel_event)()

~/miniconda3/envs/aniket/lib/python3.7/site-packages/gradsflow/models/model.py in _fit_with_event(self)
    210     def _fit_with_event(self):
    211         self.callback_runner.on_fit_start()
--> 212         self.callback_runner.with_event("epoch", self.epoch, EpochCancel)
    213         self.callback_runner.on_fit_end()
    214 

~/miniconda3/envs/aniket/lib/python3.7/site-packages/gradsflow/callbacks/callbacks.py in with_event(self, event_type, func, exception, final_fn)
     41         try:
     42             getattr(self, start_event)()
---> 43             func()
     44         except exception:
     45             getattr(self, cancel_event)()

~/miniconda3/envs/aniket/lib/python3.7/site-packages/gradsflow/models/model.py in epoch(self)
    202             self.tracker.current_epoch = epoch
    203 
--> 204             self._train_epoch_with_event()
    205             self._val_epoch_with_event()
    206 

~/miniconda3/envs/aniket/lib/python3.7/site-packages/gradsflow/models/model.py in _train_epoch_with_event(self)
    182         train_dataloader = self.tracker.autodataset.get_train_dl(self.send_to_device)
    183         self.train()
--> 184         self.train_one_epoch(train_dataloader)
    185         self.callback_runner.on_train_epoch_end()
    186         self.metrics.reset()

~/miniconda3/envs/aniket/lib/python3.7/site-packages/gradsflow/models/model.py in train_one_epoch(self, train_dataloader)
    144             # ----- METRIC UPDATES -----
    145             self.tracker.train.step_loss = outputs["loss"].item()
--> 146             self.metrics.update(outputs.get("logits"), outputs.get("target"))
    147             self.tracker.track_metrics(self.metrics.compute(), mode="train", render=True)
    148 

~/miniconda3/envs/aniket/lib/python3.7/site-packages/torchmetrics/collections.py in update(self, *args, **kwargs)
    120         for _, m in self.items(keep_base=True):
    121             m_kwargs = m._filter_kwargs(**kwargs)
--> 122             m.update(*args, **m_kwargs)
    123 
    124     def compute(self) -> Dict[str, Any]:

~/miniconda3/envs/aniket/lib/python3.7/site-packages/torchmetrics/metric.py in wrapped_func(*args, **kwargs)
    247             self._computed = None
    248             self._update_called = True
--> 249             return update(*args, **kwargs)
    250 
    251         return wrapped_func

~/miniconda3/envs/aniket/lib/python3.7/site-packages/torchmetrics/classification/accuracy.py in update(self, preds, target)
    261             # Update states
    262             if self.reduce != "samples" and self.mdmc_reduce != "samplewise":
--> 263                 self.tp += tp
    264                 self.fp += fp
    265                 self.tn += tn

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

Solution

#104

Description

An unused variable take up space in the code, and can lead to confusions. It should be removed. If this variable is necessary, name the variable _ or start the name with unused or _unused.

Occurrences

There is 1 occurrence of this issue in the repository.

See all occurrences on DeepSource → deepsource.io/gh/gradsflow/gradsflow/issue/PYL-W0612/occurrences/

Bug description

Model is not loaded from the best checkpoint after HPO.

Expected result

After automodel.hp_tune(), automodel.model should give the model with best hparams and checkpoints.

Actual result

Steps to reproduce

Context

Your Environment

• Version used:
• Operating System and version:
• Link to your fork:

I believe, adding a conda installation option for gradsflow will be helpful for growth and adoption of the library. I have started the work on it already. Once the 💡 PR gets approved and merged, you will have gradsflow on conda-forge.

conda install -c conda-forge gradsflow

⛔ 🔥 Roadblock to conda-forge packaging

However, there seems to be a problem: it appears that this library is somewhat tightly coupled with comet_ml (which has a proprietary license -- NOT OpenSource). If you could work on making this a weak coupling, or better yet make comet_ml optional (even for tests), that would allow us to make gradsflow available on conda-forge.

support Optuna Easy parallelization

• Enable storage argument
• Automatically launch multiple instances

Is your feature request related to a problem? Please describe.

GitHub now supports (natively) defining citations to use software repositories on hosted on the platform. Alike the LICENSE file, in this case, you essentially add a CITATION.cff file (formatted as yaml) to the repository root. And that creates the citations feature for the repository: you can get it as bibtex or in APA style.

Describe the solution you'd like

I would like to add the citation file CITATION.cff.

Describe alternatives you've considered

Additionally, you could also consider adding the bibtex to the readme file (assuming this being a relatively new feature of GitHub, not a lot of users may be aware of it yet).

cff-version: 1.2.0
message: "If you use this software, please cite it as below."
authors:
- family-names: "Maurya"
  given-names: "Aniket"
  #orcid: "https://orcid.org/0000-0000-0000-0000"
- family-names: "Bhatia"
  given-names: "Gagan"
  #orcid: "https://orcid.org/0000-0000-0000-0000"
title: "gradsflow"
#version: 1.0.0
#doi: 10.5281/zenodo.1234
#date-released: 2021-08-29
url: "https://github.com/gradsflow/gradsflow"

Additional context 🔥

Useful Links:

• Enhanced support for citations on GitHub
• What is a CITATION.cff file?
• Demo gif - LinkedIn Share
• Create your own citation.cff

Preferably use OrcID for each author

It's also better to have an orcid for each author. So, author(s) who do not have any orcid, I would suggest them creating one.

Current way-

data = AutoData(train_dataloader, val_dataloader)


# this will give error
data = AutoData(train_dataset, val_dataset)

Need to support dataset as well.

Bug description

Coverage is being run for each pytest. It is redundant.

Implement tensorboard callback for logging metrics

Is your feature request related to a problem? Please describe.

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

hey - first of all, congrats on the fantastic work!

I'm curious if gradsflow supports distributed training? either using PyTorch's DDP or something like Horovod under the hood.

Thanks!

Implement EarlyStopCallback to terminate training based on condition like - accuracy or loss not increasing beyond a certain threshold.

Is your feature request related to a problem? Please describe.

Add open in colab button to notebooks.

Is your feature request related to a problem? Please describe.

Examples are only available in the Github repo, it can be rendered directly into the documentation page.

Describe the solution you'd like

integrate https://github.com/danielfrg/mkdocs-jupyter

🚘 Callback Roadmap

• #120 - @aniketmaurya #121
• #122 - @aniketmaurya #151
• Tensorboard #123
• Comet #125
• WandB Callback #124
• CSV Logger - @aniketmaurya #116

Is your feature request related to a problem? Please describe.

I think it would be better to implement black (https://github.com/psf/black) as a coding style for Gradsflow.

dataloader = _PatchDataLoader(dataloader)
TypeError: init() missing 1 required positional argument: 'stage' for the code:-
from gradsflow.autoclassifier import text

from flash.core.data.utils import download_data
from flash.text import TextClassificationData

1. Create the DataModule

download_data("https://pl-flash-data.s3.amazonaws.com/imdb.zip", "./data/")

datamodule = TextClassificationData.from_csv(
"review",
"sentiment",
train_file="data/imdb/train.csv",
val_file="data/imdb/valid.csv",
backbone="prajjwal1/bert-medium",

)

suggested_conf = dict(
optimizers=["adam"],
lr=(5e-4, 1e-3),
)

model = AutoTextClassifier(datamodule,
suggested_backbones=['sgugger/tiny-distilbert-classification'],
suggested_conf=suggested_conf,
max_epochs=1,
optimization_metric="val_accuracy",
timeout=30)

print("AutoTextClassifier initialised!")
model.hp_tune()

Is your feature request related to a problem? Please describe.

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe.

allow list of class names
https://github.com/gradsflow/gradsflow/blob/d5dbb05ae8c5640e23b2559486a80436ee99dd77/gradsflow/data/image.py#L38

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe.

add optuna visualization

https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/005_visualization.html

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Description

An unused argument can lead to confusions. It should be removed. If this variable is necessary, name the variable _ or start the name with unused or _unused.

Occurrences

There are 5 occurrences of this issue in the repository.

See all occurrences on DeepSource → deepsource.io/gh/gradsflow/gradsflow/issue/PYL-W0613/occurrences/