Dataset only available when the trainer is instantiated about lightning HOT 5 CLOSED

@LouisTrezzini it sounds like your experimentation framework is just the trainer... if we made this change we'd end back up with scattered non-standard code.

I'm not sure how portability would be affected? even if you did want to use your own trainer and feed your own data, then you'd probably just want to define a standard PyTorch module... CoolModel is just a nn.Module... nothing fancy about it.

But maybe I'm not understanding your use case.

from lightning.

Probably one of the first things that happens in trainer is to get the dataset. so, your proposal is equivalent...

Current:

Class Trainer(...):
     def fit():
          load_datasets # one of the first things

Proposed:

trainer(load_datasets())    

Which is the same thing...

from lightning.

Hello @williamFalcon, thanks for your prompt answer

even if you did want to use your own trainer and feed your own data, then you'd probably just want to define a standard PyTorch module...

To reformulate, my use case would be to use your trainer and CoolModel's step methods, BUT with our own data / on multiple datasets.

How would you suggest we do this?
Thanks!

from lightning.

@LouisTrezzini sure. just return your own dataloader instead of MNIST. If you need multiple aggregated datasets construct a joint dataloader (https://pytorch.org/docs/stable/data.html#torch.utils.data.ConcatDataset).

Define your own data in:

@pl.data_loader
def tng_dataloader(self):
    # return your own dataloader or dataConcat

Same for val and test data. See the LightningModule template

from lightning.

@LouisTrezzini did this answer your questions? if so, we can close this issue

from lightning.