boschresearch / aldm Goto Github PK

For definition of classes for a new dataset, do I need a dataset_info.json file as for cityscapes?
Should the new dataloader has the structure of cityscapes.py or ade20k.py? Note that my new dataset has some classes from ade20 and some new classes.

Can you also explain, why in the function conver_labels in the ade20k.py file, we subsstract q from the label: label = label -1
Why don't we keep the ids as they are?

Hi author, could you please provide the code for the evaluation calculation?

Can I do fine-tuning for my own dataset just like with ControlNet?

Thanks for publishing the code.

I was wondering if you could also release the generated data that was used to train the semantic segmentation models?

Hi,
could you please provide the code for training COCO (e.g., dataloader, segmenter config) or weight file of COCO?

Thank you.

Thanks for your contributions and open-source code.

When I try to reproduce training with the command:

bash bash_script/train_ade20k.sh

these errors happened:

FileNotFoundError: [Errno 2] No such file or directory: 
'/fs/scratch/rng_cr_bcai_dl/lyu7rng/0_project_large_models/code_repo/0_ControlNet/checkpoint/control_seg_enc_scratch.ckpt'

FileNotFoundError: [Errno 2] No such file or directory: 
'/fs/scratch/rng_cr_bcai_dl/lyu7rng/0_project_large_models/pretrained/ade20k_semseg/upernet101/encoder_epoch_50.pth'

Could you please provide these checkpoints and provide instructions of how to train ALDM?

When I remove is_inference from:

class NewSegmentationModule(SegmentationModule):
def forward(self, image, label, is_inference=None):
segSize = (label.shape[-2], label.shape[-1])
pred = self.decoder(self.encoder(image, return_feature_maps=True), segSize=segSize, is_inference=is_inference)
return pred

The error is gone but I get another error:

/ALDM/cldm_seg/util.py", line 41, in forward
seg_emb = torch.index_select(self.class_embeddings, 0, seg_map_.cpu())
IndexError: index out of range in self

I am using the structure of cityscapes.py for my new dataset, where I am using self.lb_map = {el['id']: el['trainId'] for el in labels_info}
I created a json file "newdataset_info.json" where I packed :
{
"name": "terrain",
"ignoreInEval": false,
"id": 30,
"color": [
112,
9,
255
],
"trainId": 30
}
for each class. Note that I put all my classes and I am not ignoring any of them (No trainId = 255).
When I run the code, I receive this error (See above).
However, I tried to add 3 fictive classes to the json file, named them randomly but put their trainId as 255, it worked. I dont understand what is the problem here? And I don't know if the way it made the program compile is true and does not affect the training?

I have a problem. Can you help me solve it？

I want to train the model on another dataset similar to cityscapes but with other labels.
How can I do this? Are there any challenges?

Hi! Great work!
I want to train the model on my dataset.
Can you provide instructions on how this could be done?
What should we change to change the number of classes?

Thank you!

Is it possible to generate out-of-distribution samples using custom colormaps without the need to finetune(similar to the FreestyleNet implementation here: https://github.com/essunny310/FreestyleNet)?

According to my understanding of the codebase, the inference seems to be limited to only the colormaps defined in the dataloader modules for the respective datasets (coco-stuff/ADE20K/Cityscapes).

If yes, could you provide an example, or outline the steps needed to implement this feature?

Thanks in advance!

Great work! I would like to train your model on my own data, but need a warm-up model for the segmentation model upper-net. Can you provide the code for training the segmentation model during the warm-up stage? Thanks

I want to train the model on another urban dataset. One folder should contain the segmentation masks and the other the images right?
Also, do I need the caption.json for my new dataset?
Thank you :)

Thank you for your incredible work！I would like to ask about the GPU training time required for the model. Additionally, could you please provide guidance on how to find the evaluation functions for mIOU and FID?