How to specify GPUs? about infinigen HOT 4 CLOSED

Hello,

local_256GB is intended to respect CUDA_VISIBLE_DEVICES, although the current implementation would expect a comma separated list of integers IE CUDA_VISIBLE_DEVICES=4,5,6,7. The code which parses this is here

The logs should also contain what setting of CUDA_VISIBLE_DEVICES each job actually observed, which gets printed by this line. Please check the results of this print statement in your coarse.out logs and let me know what it says so we can continue with debugging

from infinigen.

I just ran this command CUDA_VISIBLE_DEVICES=4,5,6,7 screen python -m infinigen.datagen.manage_jobs --output_folder outputs/videos_20240405 --num_scenes 100 \ --pipeline_config local_256GB monocular_video cuda_terrain blender_gt \ --cleanup big_files --warmup_sec 60000 --config video high_quality_terrain

But it still used an unspecified GPU, while the specified GPUs were idle:

coarse.out:

The following command is sent: "nice -n 20 /home/zitian_he/zitian_he/anaconda3/envs/infinigen/bin/python -m infinigen_examples.generate_nature -- --output_folder /mnt1/public/zitian_he/infinigen/outputs/videos_20240405/cdf5cfd/coarse --seed cdf5cfd --task coarse --task_uniqname coarse -g river video high_quality_terrain -p LOG_DIR='/mnt1/public/zitian_he/infinigen/outputs/videos_20240405/cdf5cfd/logs' execute_tasks.frame_range=[1, 192] execute_tasks.camera_id=[0, 0]"
keep_placeholder=False placeholder.name='TreeFlowerFactory(6065525).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='TreeFlowerFactory(6065525).spawn_asset(0)' list(obj.children)=[]
keep_placeholder=False placeholder.name='TreeFlowerFactory(6065525).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='TreeFlowerFactory(6065525).spawn_asset(1)' list(obj.children)=[]
keep_placeholder=False placeholder.name='TreeFlowerFactory(6065525).spawn_placeholder(2)' list(placeholder.children)=[] obj.name='TreeFlowerFactory(6065525).spawn_asset(2)' list(obj.children)=[]
keep_placeholder=False placeholder.name='TreeFlowerFactory(6065525).spawn_placeholder(3)' list(placeholder.children)=[] obj.name='TreeFlowerFactory(6065525).spawn_asset(3)' list(obj.children)=[]
keep_placeholder=False placeholder.name='TreeFlowerFactory(6065525).spawn_placeholder(4)' list(placeholder.children)=[] obj.name='TreeFlowerFactory(6065525).spawn_asset(4)' list(obj.children)=[]
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
keep_placeholder=False placeholder.name='GenericTreeFactory(6065525).spawn_placeholder(0)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(6065525).spawn_asset(0)']] obj.name='GenericTreeFactory(6065525).spawn_asset(0)' list(obj.children)=[]
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
keep_placeholder=False placeholder.name='GenericTreeFactory(6065525).spawn_placeholder(1)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(6065525).spawn_asset(1)']] obj.name='GenericTreeFactory(6065525).spawn_asset(1)' list(obj.children)=[]
keep_placeholder=False placeholder.name='FruitFactoryDurian(6065525).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='FruitFactoryDurian(6065525).spawn_asset(0)' list(obj.children)=[]
keep_placeholder=False placeholder.name='FruitFactoryDurian(6065525).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='FruitFactoryDurian(6065525).spawn_asset(1)' list(obj.children)=[]
keep_placeholder=False placeholder.name='FruitFactoryDurian(6065525).spawn_placeholder(2)' list(placeholder.children)=[] obj.name='FruitFactoryDurian(6065525).spawn_asset(2)' list(obj.children)=[]
keep_placeholder=False placeholder.name='FruitFactoryDurian(6065525).spawn_placeholder(3)' list(placeholder.children)=[] obj.name='FruitFactoryDurian(6065525).spawn_asset(3)' list(obj.children)=[]
keep_placeholder=False placeholder.name='FruitFactoryDurian(6065525).spawn_placeholder(4)' list(placeholder.children)=[] obj.name='FruitFactoryDurian(6065525).spawn_asset(4)' list(obj.children)=[]
keep_placeholder=False placeholder.name='BranchFactory(6065525).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='BranchFactory(6065525).spawn_asset(0)' list(obj.children)=[]
keep_placeholder=False placeholder.name='BranchFactory(6065525).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='BranchFactory(6065525).spawn_asset(1)' list(obj.children)=[]
keep_placeholder=False placeholder.name='LeafFactoryBroadleaf(2067777).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='LeafFactoryBroadleaf(2067777).spawn_asset(0)' list(obj.children)=[]
keep_placeholder=False placeholder.name='LeafFactoryBroadleaf(2067777).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='LeafFactoryBroadleaf(2067777).spawn_asset(1)' list(obj.children)=[]
keep_placeholder=False placeholder.name='LeafFactoryBroadleaf(2067777).spawn_placeholder(2)' list(placeholder.children)=[] obj.name='LeafFactoryBroadleaf(2067777).spawn_asset(2)' list(obj.children)=[]
keep_placeholder=False placeholder.name='LeafFactoryBroadleaf(2067777).spawn_placeholder(3)' list(placeholder.children)=[] obj.name='LeafFactoryBroadleaf(2067777).spawn_asset(3)' list(obj.children)=[]
keep_placeholder=False placeholder.name='LeafFactoryBroadleaf(2067777).spawn_placeholder(4)' list(placeholder.children)=[] obj.name='LeafFactoryBroadleaf(2067777).spawn_asset(4)' list(obj.children)=[]
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Error: Evaluated geometry from modifier does not contain a mesh
Warning: No mesh data to join
keep_placeholder=False placeholder.name='GenericTreeFactory(2067777).spawn_placeholder(0)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(2067777).spawn_asset(0)']] obj.name='GenericTreeFactory(2067777).spawn_asset(0)' list(obj.children)=[]
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Error: Evaluated geometry from modifier does not contain a mesh
Warning: No mesh data to join
keep_placeholder=False placeholder.name='GenericTreeFactory(2067777).spawn_placeholder(1)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(2067777).spawn_asset(1)']] obj.name='GenericTreeFactory(2067777).spawn_asset(1)' list(obj.children)=[]
keep_placeholder=False placeholder.name='FruitFactoryCoconutgreen(2067777).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='FruitFactoryCoconutgreen(2067777).spawn_asset(0)' list(obj.children)=[]
keep_placeholder=False placeholder.name='FruitFactoryCoconutgreen(2067777).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='FruitFactoryCoconutgreen(2067777).spawn_asset(1)' list(obj.children)=[]
keep_placeholder=False placeholder.name='FruitFactoryCoconutgreen(2067777).spawn_placeholder(2)' list(placeholder.children)=[] obj.name='FruitFactoryCoconutgreen(2067777).spawn_asset(2)' list(obj.children)=[]
keep_placeholder=False placeholder.name='FruitFactoryCoconutgreen(2067777).spawn_placeholder(3)' list(placeholder.children)=[] obj.name='FruitFactoryCoconutgreen(2067777).spawn_asset(3)' list(obj.children)=[]
keep_placeholder=False placeholder.name='FruitFactoryCoconutgreen(2067777).spawn_placeholder(4)' list(placeholder.children)=[] obj.name='FruitFactoryCoconutgreen(2067777).spawn_asset(4)' list(obj.children)=[]
keep_placeholder=False placeholder.name='BranchFactory(2067777).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='BranchFactory(2067777).spawn_asset(0)' list(obj.children)=[]
keep_placeholder=False placeholder.name='BranchFactory(2067777).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='BranchFactory(2067777).spawn_asset(1)' list(obj.children)=[]
keep_placeholder=False placeholder.name='LeafFactory(1751696839).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='LeafFactory(1751696839).spawn_asset(0)' list(obj.children)=[]
keep_placeholder=False placeholder.name='LeafFactory(1751696839).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='LeafFactory(1751696839).spawn_asset(1)' list(obj.children)=[]
keep_placeholder=False placeholder.name='LeafFactory(1751696839).spawn_placeholder(2)' list(placeholder.children)=[] obj.name='LeafFactory(1751696839).spawn_asset(2)' list(obj.children)=[]
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
keep_placeholder=False placeholder.name='GenericTreeFactory(1751696839).spawn_placeholder(0)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(1751696839).spawn_asset(0)']] obj.name='GenericTreeFactory(1751696839).spawn_asset(0)' list(obj.children)=[]
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
keep_placeholder=False placeholder.name='GenericTreeFactory(1751696839).spawn_placeholder(1)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(1751696839).spawn_asset(1)']] obj.name='GenericTreeFactory(1751696839).spawn_asset(1)' list(obj.children)=[]
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
keep_placeholder=False placeholder.name='GenericTreeFactory(1751696839).spawn_placeholder(2)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(1751696839).spawn_asset(2)']] obj.name='GenericTreeFactory(1751696839).spawn_asset(2)' list(obj.children)=[]
split obj.name='OpaqueTerrain' into inview area 1090.53 and outofview area 0.00
split obj.name='OpaqueTerrain' into inview area 697.52 and outofview area 0.00
keep_placeholder=False placeholder.name='BlenderRockFactory(64661).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='BlenderRockFactory(64661).spawn_asset(0)' list(obj.children)=[]
keep_placeholder=False placeholder.name='BlenderRockFactory(64661).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='BlenderRockFactory(64661).spawn_asset(1)' list(obj.children)=[]
keep_placeholder=False placeholder.name='BlenderRockFactory(64661).spawn_placeholder(2)' list(placeholder.children)=[] obj.name='BlenderRockFactory(64661).spawn_asset(2)' list(obj.children)=[]
keep_placeholder=False placeholder.name='BlenderRockFactory(64661).spawn_placeholder(3)' list(placeholder.children)=[] obj.name='BlenderRockFactory(64661).spawn_asset(3)' list(obj.children)=[]
keep_placeholder=False placeholder.name='BlenderRockFactory(64661).spawn_placeholder(4)' list(placeholder.children)=[] obj.name='BlenderRockFactory(64661).spawn_asset(4)' list(obj.children)=[]
50 scatter:BlenderRockFactory(64661) instances created.
keep_placeholder=False placeholder.name='LeafFactoryV2(25497).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='LeafFactoryV2(25497).spawn_asset(0)' list(obj.children)=[]
keep_placeholder=False placeholder.name='LeafFactoryV2(25497).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='LeafFactoryV2(25497).spawn_asset(1)' list(obj.children)=[]
keep_placeholder=False placeholder.name='LeafFactoryV2(25497).spawn_placeholder(2)' list(placeholder.children)=[] obj.name='LeafFactoryV2(25497).spawn_asset(2)' list(obj.children)=[]
keep_placeholder=False placeholder.name='LeafFactoryV2(25497).spawn_placeholder(3)' list(placeholder.children)=[] obj.name='LeafFactoryV2(25497).spawn_asset(3)' list(obj.children)=[]
keep_placeholder=False placeholder.name='LeafFactoryV2(25497).spawn_placeholder(4)' list(placeholder.children)=[] obj.name='LeafFactoryV2(25497).spawn_asset(4)' list(obj.children)=[]
2133 scatter:LeafFactoryV2(25497) instances created.
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(0)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(0)']] obj.name='GenericTreeFactory(23779).spawn_asset(0)' list(obj.children)=[]
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(1)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(1)']] obj.name='GenericTreeFactory(23779).spawn_asset(1)' list(obj.children)=[]
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(2)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(2)']] obj.name='GenericTreeFactory(23779).spawn_asset(2)' list(obj.children)=[]
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(3)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(3)']] obj.name='GenericTreeFactory(23779).spawn_asset(3)' list(obj.children)=[]
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(4)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(4)']] obj.name='GenericTreeFactory(23779).spawn_asset(4)' list(obj.children)=[]
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(5)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(5)']] obj.name='GenericTreeFactory(23779).spawn_asset(5)' list(obj.children)=[]
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(6)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(6)']] obj.name='GenericTreeFactory(23779).spawn_asset(6)' list(obj.children)=[]
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(7)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(7)']] obj.name='GenericTreeFactory(23779).spawn_asset(7)' list(obj.children)=[]
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(8)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(8)']] obj.name='GenericTreeFactory(23779).spawn_asset(8)' list(obj.children)=[]
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
Info: Applied modifier was not first, result may not be as expected
keep_placeholder=False placeholder.name='GenericTreeFactory(23779).spawn_placeholder(9)' list(placeholder.children)=[bpy.data.objects['GenericTreeFactory(23779).spawn_asset(9)']] obj.name='GenericTreeFactory(23779).spawn_asset(9)' list(obj.children)=[]
53 scatter:GenericTreeFactory(23779) instances created.
33 scatter:chopped_tree instances created.
keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(0)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(0)' list(obj.children)=[]
keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(1)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(1)' list(obj.children)=[]
keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(2)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(2)' list(obj.children)=[]
keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(3)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(3)' list(obj.children)=[]
keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(4)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(4)' list(obj.children)=[]
keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(5)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(5)' list(obj.children)=[]
keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(6)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(6)' list(obj.children)=[]
keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(7)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(7)' list(obj.children)=[]
keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(8)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(8)' list(obj.children)=[]
keep_placeholder=False placeholder.name='GrassTuftFactory(89086).spawn_placeholder(9)' list(placeholder.children)=[] obj.name='GrassTuftFactory(89086).spawn_asset(9)' list(obj.children)=[]
121090 scatter:GrassTuftFactory(89086) instances created.
Info: Saved "scene.blend"
Error: Not freed memory blocks: 30, total unfreed memory 0.000973 MB

from infinigen.

Hello, apologies for the delay, I have a patch for this and will release it shortly.

from infinigen.

should be fixed.

from infinigen.