Log file conflicts when running colcon with parallel and COLCON_LOG_PATH variable set about colcon-core HOT 5 CLOSED

using the paralell tool

What kind of tool is that?

This is because both instances of my colcon build command are trying to write to the same latest_build folder.

Each invocation will only write to the specific directory with the timestamp. So invoking the tool with the same verb at the same second is currently indeed a problem (both invocations using the same timestamp). I don't think that is your problem though.

The latest* symlinks are only created for the user for convenience. There could be a race condition if two invocations run through the same logic with checking and then creating the symlink.

Can you please provide a reproducible example.

from colcon-core.

Please try #130 which makes sure that each invocation gets a separate log directory and all FS calls which might be subject to a race are handling exceptions gracefully.

from colcon-core.

I'm using https://www.gnu.org/software/parallel/

The command is:

parallel < commands.txt

The contents of commands.txt is:

colcon build --base-paths ./workspace1 --build-base ./build/workspace1/build-output --install-base ./build/workspace1/install-output
colcon build --base-paths ./workspace2 --build-base ./build/workspace2/build-output --install-base ./build/workspace2/install-output

The environment has COLCON_LOG_PATH set to /home/ec2-user/build/logs. My current working directory is /home/ec2-user. Maybe it should be possible to set the log directory via a command line argument?

from colcon-core.

I can't make it fail with similar commands (e.g. building a ROS 2 workspace twice in parallel). Maybe it is related to your filesystem? Mine is on a SSD which will be less likely to run into a race condition due to its speed.

Please try #130 since it clearly separates your two log directories. Please comment here with your results.

from colcon-core.

I successfully ran the build in parallel with your changes. Thanks for the solution!

from colcon-core.