yangling0818 / sgdiff Goto Github PK

Dear authors,
Thanks for sharing this amazing work, are you planning to also share the training code for the Masked Contrastive Pre-Training part? This would be very helpful to correctly reproduce the full approach.
Thank you

Hi, thank you for your work. While testing your model, I recognized some VG links in scripts/download_vg.sh are not working. These are okay for me:

wget https://homes.cs.washington.edu/~ranjay/visualgenome/data/dataset/objects.json.zip -O $VG_DIR/objects.json.zip
wget https://homes.cs.washington.edu/~ranjay/visualgenome/data/dataset/attributes.json.zip -O $VG_DIR/attributes.json.zip
wget https://homes.cs.washington.edu/~ranjay/visualgenome/data/dataset/relationships.json.zip -O $VG_DIR/relationships.json.zip
wget https://homes.cs.washington.edu/~ranjay/visualgenome/data/dataset/object_alias.txt -O $VG_DIR/object_alias.txt
wget https://homes.cs.washington.edu/~ranjay/visualgenome/data/dataset/relationship_alias.txt -O $VG_DIR/relationship_alias.txt
wget https://homes.cs.washington.edu/~ranjay/visualgenome/data/dataset/image_data.json.zip -O $VG_DIR/image_data.json.zip

I tried with GPUs of 3GB, 16GB and 24GB memory. Only 24 GB could run the training. Maybe this information is worth to mention in the readme.

Dear authors,
Thanks for the excellent work! About VQVAE for embedding image to latent, you have provided the pre-trained model. Could you please share the training and testing code and instruction for the VQVAE part? In my understanding, the VQVAE part determines the upper bound of the image generation quality. It would be very helpful for me to train the whole model from scratch.
Thanks a lot!

When I run preprocess.py, I get the vocab.json file, which looks like this. It shows that the number of objects is 260, not 179 as stated in the paper, and the number of relations is 66, not 46. I think it may be because object_alias.txt and relationships_alias.txt are downloaded from other links, resulting in the wrong number of objects and relationships. We would appreciate it if you could provide a download of these two files, or provide the pre-processed.h5 files directly.

I am trying to run the inference code (sampler) and the loading of the pretrained models fails. I tried some of the suggestions from another submitted issue, but run into different error with the vq-f8 model:

LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 395.77 M params.
Keeping EMAs of 630.
making attention of type 'vanilla' with 512 in_channels
making attention of type 'vanilla' with 512 in_channels
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
making attention of type 'vanilla' with 512 in_channels
making attention of type 'vanilla' with 512 in_channels
making attention of type 'vanilla' with 512 in_channels
Restored from pretrained/vq-f8-model.ckpt with 0 missing and 49 unexpected keys
_pickle.UnpicklingError: invalid load key, '<'.

Can you suggest what to do? I am using the code available as of today.

From paper, it seems that scene graph is in form of text triplet and you encode the text triplet using Graph encoder. Is my assumption true or image features is also used for Scene graph encoding? if yes what kind of graph model you are using to encode textual scene graph information. From code it seems like BERT is used for processing of text in the scene graph.

I have been training the model for more than a day, and it has reached the 27th epoch without stopping. Does the training automatically stop when using trainer.py? If it doesn't stop automatically, how many epochs should I train it for? Thank you!

I was just wondering when the code for this project will be available?

When you train this network by itself, how do you know to stop training other than the training and validation error going to 0? Do you use a metric to measure the quality of image reconstruction?

Which version of Taming-transformers should be installed? Is model.ckpt with sip_vg enough for running the test code?

I set line.22 in test_set_ddim_sampler.py and line.44 in config_vg as pretrained/model.ckpt. But the error info shows that the model and the checkpoints do not match.

I wonder if you directly use the java tool of standford NLP to synthesize scene graph with image capation?

Any python tool can achieve similar result by replacing http://nlp.stanford.edu/software/scenegraphparser.shtml

thanks

Dear authors,

After training the model for 335 epochs with image size 256,256 and performing evaluation using the testset_ddim_sampler.py script, I managed to obtain an FID of 23.86 on the test set of Visual Genome. This is better than the 26 reported in the paper, could you please provide details on how you evaluate your work? I used the pytorch-fid project for evaluation.

Hi， Ling Yang
If I want to test generating an image from a custom scene graph, What data should I need to prepare and which part of the code should I change?