Question about Kuka Training and Kuka Dataset. about diffuser HOT 3 CLOSED

Hi sorry about the late reply -- the goal of the model is to learn pick and place one single block (given other already stacked blocks)

In the second setting -- the model predicts when to form contacts with individual blocks. When the model is not trained well, the contacts will be incorrectly added, which leads to very odd physics simulation.

Feel free to let me know if you have any other questions.

from diffuser.

@yilundu , Thank you very much for you reply. Unfortunately, I didn't understand.

Let me write my question in little bit detail.

The part before training

You have this kind of dataset , kuka_dataset/data_897.npy, kuka_dataset/data_898.npy .... and diffuser gets this dataset and trains the model.

When I check your dataset (e.g kuka_dataset/data_897.npy) , the dataset consists of 470x39 parameters.

(The first axis of the dataset) The 470 is timestep t
(The second axis of the dataset) 7 variables from the 39 dimension are the joint variables of the robot. The rest of the parameters (32 parameters) is the position as well as the orientation of the 4 cubes ( 3 pos and 4 rot for each cube)

Basically, the data has the trajectory points of each cube as well as the joint values for the robot in each timestep.

If I render the following datasets (without starting any training), I can already see that the robot picks objects and places them. ( In the following video)

tmp = np.load("kuka_dataset/data_897.npy")
renderer.renders(tmp)

Here is the rendering of the dataset data_897

tmp = np.load("kuka_dataset/data_898.npy")
renderer.renders(tmp)

Here is the rendering of the dataset data_898
https://user-images.githubusercontent.com/81103451/227969534-5ec34a13-8f73-4c9a-ae01-4c5e204db0ea.mp4

So, the data was literally trained, and the robot learned how to pick and place. You start with a learned policy.

The part after training

Then I start training using the pretrained model model-650.pt, then the error is around 0.020
Then diffision model samples the trajectories points from the below code line ( here https://github.com/jannerm/diffuser/blob/kuka/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py#L806 )

When I check the dimension of the variable unnormed, It has size

unnormed.shape
torch.Size([1, 128, 39])

It means that It has joint values as well as pos and rot for each cube for 128 timesteps. When I render this data ( the sampled data from the diffusion model), I see the following behaviour;

So, It is simply the similar behaviour before starting to the training.

You are giving some data from a learned policy(e.g robot pick and place the cube). The diffuser model gets trajectory points of the cube as well as joints of the robot, It trains it and obtains some trajectory points(still robot picks and place the cube). ( If I understand correctly)

So my question is;

The robot already picks and place the object before training. After the training, It has still the same behavior. What is the objective of the training?

from diffuser.

Ahh I see -- yes the objective of training is to learn how to pick / place blocks from a large dataset of existing demonstrations. The goal is to generalize to new configuration of blocks seen at test time, as well as pick and place blocks in a controllable manner.

from diffuser.