When I run the run_reconstruction.py for pascal cars, I notice the dimension of vertices is 962x3, but the dimension of UV coordinate is 1054x2. So the mapping between vertices and UV coordinate is not one-to-one.

1. Which formula do you use to calculate UV coordinates for each vertex so that the mapping is not one-to-one?
2. Which line of code can tell us the mapping between vertices and UV coordinate/coordinates?

When training GAN with text, an error happened in function "update_generator_running_avg".
My enviroment is

OS : Ubuntu 18.04.4 LTS
Python : 3.6.12
Pytorch : 1.6.0
CUDA : 10.1

The error detail is below,

Traceback (most recent call last):
File "run_generation.py", line 713, in
update_generator_running_avg(epoch)
File "run_generation.py", line 452, in update_generator_running_avg
param.mul_(alpha).add_(g_state_dict[k], alpha=1-alpha)
RuntimeError: result type Float can't be cast to the desired output type Long

And I found type of "param" is changed float32 to long(int64) by print debug in for loop.

type of param : torch.float32
type of alpha : <class 'float'>
type of param : torch.float32
type of alpha : <class 'float'>
type of param : torch.float32
type of alpha : <class 'float'>
type of param : torch.float32
type of alpha : <class 'float'>
type of param : torch.float32
type of alpha : <class 'float'>
type of param : torch.float32
type of alpha : <class 'float'>
type of param : torch.float32
type of alpha : <class 'float'>
type of param : torch.float32
type of alpha : <class 'float'>
type of param : torch.float32
type of alpha : <class 'float'>
type of param : torch.float32
type of alpha : <class 'float'>
type of param : torch.int64
type of alpha : <class 'float'>

How can I solve this?

Hello, I downloaded PASCAL3D+_release1.1, but I can't find about p3d _data,p3d_sfm and p3d_labels.csv. Maybe I omitted some details. Could you please give me some help?

Congratulation on your amazing work! I want to train your mesh reconstruction on aeroplane in Pascal 3D+ instead of car. Could you please share the code which preprocesses the car data in Pascal 3D+ to get all info such as mask, sfm required for your mesh reconstruction training so that I can get all required info of aeroplane?

Hi Dario,

This is really a good work. Thanks for sharing. But I met some problems when running your code.

First, I set up the full environment including the datasets as your guideline, and I did generate pseudo-ground-truth data before evaluating. But I got the same wrong results below when evaluating on cub dataset with Class or Text condition.

FID (training set): 310.52
Texture-only FID (training set): 310.52
Mesh-only FID (training set): 310.52
FID (validation set): 309.70
Texture-only FID (validation set): 309.70
Mesh-only FID (validation set): 309.70

Second, when I run the training, I found that all masks generated by the Discriminator contains all zero, which results in nan GAN loss.

Could you please give me some hints to solve these problems?

Thank you.

Best,
F

I just followed your training step(I didn't change the code at all) and I was training GAN with cub dataset.
But I got this result during training epoch 0.

[0] epoch 0, 0/186, g_loss nan d_fake_loss 0.00000 d_real_loss 0.00000 flat 0.24812
[10] epoch 0, 10/186, g_loss nan d_fake_loss nan d_real_loss nan flat nan

I am so confused now because I did change nothing and I just follow your instruction.
I only changed the number of gpus, 4 gpus to 3 gpus.

Before linear_rasterizer, I guess each vertex in 'vtx' variable is in the unit UV sphere coordinate system? I notice the value of x/y/z for each vertex in 'vtx' is about [-1,1]. For example, one vertex is (1.2, 1.1, 0.8). And the input image size is 256x256.

My question is which pixel this vertex will be projected to on the image plane? Or the question can be what is the corresponding vertex coordinate for the pixel (0,0) and pixel (256,256)?

I'm wondering which visualization tool/code do you use to show those 3D textured meshes in this repo and in the paper? Could you please provide some sources/code/link? Thanks!!!

@dariopavllo , congratulations on your presentation at NIPS 2020. Interesting work. Have a few quick questions.

1. What exactly would be involved in using custom images to generate 3D meshes and textures ? Would fine-tuning work ?
2. Looking to generate 3D meshes and then render views from different directions of custom images (i.e keeping geometry and texture constant) - would that be doable ? If so what would that pipeline look like for training and inference ?

Hi, I am confused about the sfm_pose in your code.
https://github.com/dariopavllo/convmesh/blob/master/cmr_data/base.py

sfm_pose = (sfm_c, sfm_t, sfm_r)

sfm_pose = [np.copy(data_sfm.scale), np.copy(data_sfm.trans), np.copy(data_sfm.rot)]

I don't know why the dimension of sfm_t is 2 and why sfm_pose is B X 7 (s, tr, q).
Looking forward to your reply