I was planning to use cqfd, to test if something can compile in different environment. So, "Dockerfile.envA, Dockerfile.envB", etc.

It looks like cqfd is not (yet) designed for multiple Dockerfiles, is this something interesting? If so, how would be the best way to implement it?

Something like:

[withfftw]
dockerfile='Dockerfile.withfftw'
command='cmake . && make'

[withffmpeg]
dockerfile='Dockerfile.withffmpeg'
command='cmake . && make'

[build]
command='cmake . && make'
flavors='withfftw withffmpeg'

If this is not what cqfd was meant for I can just write my own bash/Docker wrapper to accomplish this.

I've noticed that one of my collegue had v5.1.0 just like me, but he kinda had some new features (like the "-d" option. I believe such changes would require a bump to 5.2.0 so that our teammates don't get confused on which cqfd version to use.
Thanks!

Hi,

I think it could be interesting to support additional compression methods such as gzip or zip.

Maybe the make_archive() function could be improved with a simple if/elif/else statement such as:

local ext="${release_archive#*.}"                                          
                                                                           
if [ "$ext" == "tar.xz" ]; then                                      
        XZ_OPT=-9 tar -cJf \                                               
                "$release_archive" $release_files                          
elif [ "$ext" == "tar.gz" ]; then                                    
        tar -czf \                                                         
                "$release_archive" $release_files                          
elif [ "$ext" == "zip" ]; then                                       
        zip -9 -r \                                                        
                "$release_archive" $release_files                          
else                                                                       
        die "Unknown archive file extension: $extension"                   
fi                                                                         

Thanks for considering!

Prior to 5.1.0, we could prefix cqfd with the sudo command to step in as a superuser. As per 5.1.0, it is written in README.md since cqfd will use sudo(8) when it's available, the user is responsible for including sudo in the container if he needs to run commands at high priviledge.

The issue we have here is that whenever we step inside a container and then try to use sudo, a password is required which we don't have. Do you have any idea how we can work around this problem?

Original problem

When running a cqfd run with the username "test@test", I get this error :

useradd: invalid user name 'test@test': use --badname to ignore
error: useradd command failed.

.cqfdrc

[project]
org='orgt'
name='name'

[build]
command='echo command build'

.cqfd/docker/Dockerfile

FROM fedora:36
ENV DEBIAN_FRONTEND noninteractive

Investigation

By using special options when creating a user, it is technically possible to use '@' and '$' characters in Linux usernames.

When launching docker, cqfd use the useradd command to create a user with the same username as the user who launched the cqfd command.
If this username contains '@' or '$' the useradd command will fail, but only on some systems :

• Fedora and Arch Linux failed displaying the error above
• Ubuntu doesn't fail

One possible fix would be to add the --badnames option to useradd in the make_launcher function to make it accept bad usernames. But old Linux distributions (ex : Ubuntu 14, 16 and 18 ) doesn't recognize this option.

TL;DR : For now, only Ubuntu images works with bad usernames

When someone want to support multiple build environments (such as ubuntu, centos, etc...) there is a feature named "distro" which is used when defining a flavour to override the Dockerfile path to .cqfd/DISTRONAME/Dockerfile

However this feature is quite hidden inside cqfd and might be worthy to be documented

Hi guys,
I want to restrict 'builder' to an unprivileged user, no matter who's calling cqfd. How'd I do that?
The use case is: cqfd is run by root (can't change that), yocto is built inside. Yocto does not like root.
Cheers,

Consider the following file hierarchy:

philipp@snic1-ubuntu:~/proj$ tree -a
.
├── .cqfd
│   └── docker
│       └── Dockerfile
├── .cqfdrc
└── subproj
    ├── .cqfd
    │   └── docker
    │       └── Dockerfile
    ├── .cqfdrc
    └── scripts
        └── cqfd

6 directories, 5 files

If to run:

philipp@snic1-ubuntu:~/proj$ pwd
/home/philipp/proj
philipp@snic1-ubuntu:~/proj$ ./proj/scripts/cqfd init

it will take .cqfdrc from the ~/proj dir - which might be unwanted.

Would propose using dirname for locating .cqfdrc or make-style "-C" or anything else.
A warning of multiple configs along the run path would be helpful as well.

The command cqfd -b flavor1 -b flavor2 launch only flavor2 and doesn't trigger a warning

The command executed by cqfd is executed by a newly created, dedicate user : builder

However, the HOME environment variable for this user is never set.
That may poses a problem if the command try to use it, find it empty and falls back to using /root/.

I ran into this issue while executing a 'pip install' command via cqfd where pip uses $HOME to locate the local cache and thus end us trying to write to /root/.cache/pip/wheels/... and obviously fails with a PermissionError.

For the record, the issue remains the same with the proposed fix for issue #40. Only the name of the user is different.

Similar to issue #44 and somewhat related to issue #40 (at least to its proposed fix).

When needed to modify PATH for cqfd, I naturally added its modification in CQFD_EXTRA_ENV.
(Edit: change for CQFD_EXTRA_ENV as @d-k-c mentionned)

This won't work and PATH is not modified.

The culprit is 'su' from shadow-utils used in cqfd.
based on su's man:

The current environment is passed to the new shell. The value of
$PATH is reset to /bin:/usr/bin for normal users, or
/sbin:/bin:/usr/sbin:/usr/bin for the superuser. This may be changed
with the ENV_PATH and ENV_SUPATH definitions in /etc/login.defs.

So PATH will be changed for the default root user but not for builder.
A work around is to modify ENV_PATH in the Dockerfile using sed, for example:
sed 's#\(ENV_PATH[[:space:]]*\)PATH=#\1PATH=<modif>:#' -i /etc/login.defs

Should we fine a better way around this issue or just add this in the README.md?

We'd like to append one or many groups to the cqfd_user. A typical use case is when we want to run the docker command from inside a container.

See attached patch file. How would you feel if I create a pull request containing something like this?

patch_idea.txt

I tried using cqfd with docker rootless, but I get permission denied when creating directories.
Has cqfd ever been used with docker rootless ? Is there a specific configuration for using cqfd with docker rootless ?

If the user name running cqfd contained a @ cqfd will fail with the error: docker: invalid reference format.

Redefining the USER variable prevent the error.

I just ran into an issue with your tool trying to run it on OSX Mojave (10.14.4).

The mktemp version is generating a file by default which is located somewhere in /var/folders/...

Problem is, it seems impossible to mount those folders with Docker for Mac.

I tried setting TMPDIR to something else, but mktemp seemed to ignore it.

My solution was to create a new branch and manually changing make_launcher:

local tmpfile=$(mktemp /tmp/tmp.XXXXXX)

I'd be glad to hear if you had a more permanent solution.

Docker-compose offers the possibility to expose ports
ports: - “5000:5000”

Is there currently a way to do the same thing with CQFD that is already available?

Thanks

Prerequisites:

• Remove every docker images on your computer: docker rmi $(docker images -aq)

Steps to reproduce:

• Run the tests: make test

Expected results:

• Every tests are run successfully

Actual results:

[11:25:55|notice] preparing "symlink files are copied in the tar archive"
Unable to find image 'cqfd_cqfd_test:latest' locally
docker: Error response from daemon: pull access denied for cqfd_cqfd_test, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.
[11:25:56|error] error triggered by ./06-cqfd_release
cqfd: fatal: Cannot release: can't find link.txt
[11:25:57|error] error triggered by ./06-cqfd_release
tar: cqfd-test.tar.xz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
diff: /tmp/tmp.jT6ALCQ7t3/a/cqfd_a.txt: No such file or directory
diff: /tmp/tmp.jT6ALCQ7t3/link.txt: No such file or directory
[11:25:57|info] result: symlink files are copied in the tar archive: FAIL
Unable to find image 'cqfd_cqfd_test:latest' locally
docker: Error response from daemon: pull access denied for cqfd_cqfd_test, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.
[11:25:58|error] error triggered by ./06-cqfd_release

Hello,

As mentioned in the README.md, the invocation of su lacks from controlling its tty.

$ cqfd run bash
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

To workaround this limitation, cqfd will use sudo(8) when it is available in the container instead. The user is responsible for including it in the related Dockerfile.

su has two options to address that issue:

• --session-command instead of -c: this reuses the same tty

       --session-command=command
           Same as -c, but do not create a new session. (Discouraged.)

• -P or --pty in addiction: this creates a new pseudo tty

       -P, --pty
           Create a pseudo-terminal for the session. The independent
           terminal provides better security as the user does not share
           a terminal with the original session. This can be used to
           avoid TIOCSTI ioctl terminal injection and other security
           attacks against terminal file descriptors. The entire session
           can also be moved to the background (e.g., su --pty -
           username -c application &). If the pseudo-terminal is
           enabled, then su works as a proxy between the sessions (sync
           stdin and stdout).

           This feature is mostly designed for interactive sessions. If
           the standard input is not a terminal, but for example a pipe
           (e.g., echo "date" | su --pty), then the ECHO flag for the
           pseudo-terminal is disabled to avoid messy output.

My concern is that installing sudo is probably overkill to address the issue of controlling the tty for interactive commands. I have no idea why --session-command is discouraged.

Maybe, cqfd can add the -P to the su command

@@ -349,7 +349,7 @@ if [ -n "\$has_sudo" ]; then
        sudo -E -u $cqfd_user sh -c "\$@"
 else
        debug "Using \"su\" to execute command \"\$@\" as user \"$cqfd_user\""
-       su $cqfd_user -p -c "\$@"
+       su $cqfd_user -p -P -c "\$@"
 fi
 EOF
        echo $tmpfile

What is your opinion on that?

See #63 and #62

For development, it could be useful to have easily access to a shell in the containerized environment, to find out missing dependencies for instance. A devshell would invoke docker run with all volumes mapped etc. but instead of invoking the command specified in the config file, it would drop the user in a shell.

When a path to the project directory contains spaces, the current behavior will cause cqfd interpret the directory's second space-separated token, failing to run the command.

When inside a git repository, running git commit -m "some message" will create a new commit, but running cqfd run git commit -m "some message" send an error : error: pathspec 'message' did not match any file(s) known to git.
This error is the same when running git commit -m some message (without the double quotes) inside a terminal without cqfd.

cqfd seems to remove the quotes when running a custom command. It does it also with additional commands when running run -c.

.cqfdrc

[project]
org='default-org'
name='default-name'

[build]
command=''

.cqfd/docker/Dockerfile

FROM ubuntu:22.04

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update && apt-get install -y --no-install-recommends \
  git

Run a git init inside the current directory to have git support

In make_launcher the executed command is run using "su":
su $cqfd_user -p -c "$@"

When I want to run bash inside the docker container, I see the following error messages:
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell

The problem is the -c option:
-c, --command COMMAND
Specify a command that will be invoked by the shell using its -c.
The executed command will have no controlling terminal. This option cannot be used to execute interactive programs which need a controlling TTY.

For further explaination see:
https://unix.stackexchange.com/questions/286336/error-thrown-when-using-su-c-bash-no-job-control-in-this-shell

I found a solution for me:
sudo -u $cqfd_user sh -c "$@"

sudo does not have this security feature. However sudo is not installed out of the box (e.g. in the Ubuntu default image)

It would be great when the user could control somehow whether su or sudo is used.

In previous versions of cqfd, the default user was builder, and its home directory was hardcoded to /home/builder. It was thus simple to add lines like the following :

COPY a_file /home/builder/.my_user_folder/

Now that cqfd reflects the current user's name and home directory, how can we install a file inside this user's home? Docker offers the --build-arg option which, when building an image, we can provide and ARG specific to docker's internal user, but it cannot be used through a cqfd init command.

If a command in the .cqfdrc spans across several lines, cqfd will fail to parse it there is an option that looks like "--flag=option" on any line starting from the second one. My guess is that the equal sign is wronly interpreted by cqfd.

Excerpt from .cqfdrc

[build]
command='configure --enable-debug --extra-cflags=-I/usr/include && \
        --extra-ldflags=-L/usr/lib && \
        make'

Output of cqfd run:

bash: -c: line 0: syntax error near unexpected token `-L/usr/lib'
bash: -c: line 0: `cd ~builder/src/ && configure --enable-debug --extra-cflags=-I/usr/include && --extra-ldflags=( -L/usr/lib && make'

Note the parenthesis added after the '--extra-ldflags=' by cqfd.

Added in #49

Hello;

I'm using cqfd to build yocto project and simplify user's learning curve.

But to run a qemu machine, it requires user be sudo to run and ask for a password, that I don't have it. There is any workaround?

Best regards
Taka

When using cqfd in a development environment, one might needs to inspect source code and build artifacts easily.

The fact that cqfd uses a different $USER and map $PWD to /home/builder/src makes that difficult because basically all symlinks created in cqfd are invalid. Using the same username and mounting the working directory at the same location in the container could solve most of that problems.

Setup

• Apple Silicon (M1 Mac Mini)
• macOS Ventura 13.5 (22G74)
• cqfd v5.4.0
• Docker Desktop 4.22.0 (117440)

With this Dockerfile:

FROM ubuntu:20.04

RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    bash \
    git \
    sudo

I was able to create the image with cqfd init. However, I am not able to cqfd run bash for example.

Here's the error I get:

docker: Error response from daemon: error while creating mount source path '/host_mnt/private/tmp/com.apple.launchd.bMzWTSF9Hd/Listeners': mkdir /host_mnt/private/tmp/com.apple.launchd.bMzWTSF9Hd/Listeners: operation not supported.

I tried adding /host_mnt to the filesharingDirectories list in ~/Library/Group\ Containers/group.com.docker/settings.json, but it seems then I get:

ERROR: Cannot connect to the Docker daemon at unix:///Users/rbrule/.docker/run/docker.sock. Is the docker daemon running?

when doing cqfd init.

Update

I've found a workaround:

@@ -218,7 +218,6 @@ docker_run() {
               -v "$cqfd_project_dir":"$cqfd_project_dir" \
               ${home_env_var:+ -e "$home_env_var"} \
               $interactive_options \
-              ${SSH_AUTH_SOCK:+ -v $SSH_AUTH_SOCK:"$cqfd_user_home"/.sockets/ssh} \
               ${SSH_AUTH_SOCK:+ -e SSH_AUTH_SOCK="$cqfd_user_home"/.sockets/ssh} \
               $docker_img_name cqfd_launch "$@" 2>&1
 }

• Context: Using cqfd from a host user (jenkins) who has no user group. For example, on our server, jenkins belongs to the docker group only (which is its primary group).

• Expected: commands run by cqfd inherits the credentials/ownership of the host user. Especially files generated by cqfd encapsulated commands have the same user and group than the files generated without cqfd. In our case, generated files should be owned by jenkins:docker (user jenkings group docker).

• Observed: files generated within cqfd are owned by jenkins:jenkins instead of jenkins:docker

Correction proposal (TBC): In make_launch() bash function, useradd should be called with the --no-user-group option to make the docker user consistent with the host user (that is with the same primary group).

Hi,

I'd like to be able to archive files but also directories that were created as part of the build process while preserving the structure.

Currently testing cfqd with a node.js app and I would like to be able to archive the following files/folders:

• client/
• lib/
• node_modules/
• package.json
• index.js

Note: Removing the --transform "s/.*\///g" solves my issue but there might be missing test coverage to document the reason why it was put in the first place as the tests still pass without it.

The following code:

causes a potential glob of the cqfd release files to happen, in the config_load() function, i.e., before the build command has been started.

If a file has already been built previously, and contains a name that can be different from a build to another (for instance because it contains the build date, the git description, ...), it therefore may break on cqfd release calls as it tries to glob the filename with the previous name.

Here is an example environment that reproduces this bug:

.cqfdrc

[project]
org='org'
name='name'

[build]
command='rm -f file_* && touch file_$(cat /dev/urandom | tr -dc 'A-Za-z0-9' | fold -w 8 | head -n 1)'
files='file_*'

.cqfd/docker/Dockerfile

FROM ubuntu:20.04
ENV DEBIAN_FRONTEND noninteractive

Hi guys,

I encountered an issue because I wanted to keep using a versioned cqfd copy in my project instead of installing it on my system.

So I have a cqfd folder containing cqfd, .cqfdrc and a docker folder

Right now (HEAD I used is 2c80b18) these are the commands I have to type:

    # From the root project directory
    cd cqfd/
    ./cqfd -f .cqfdrc init
    cd ..
    cqfd/cqfd -f cqfd/.cqfdrc run

'init' can be done only in the cqfd folder since the path to the docker file is hardcoded to docker/Dockerfile.

But 'run' must be executed from the root directory since cqfd now mounts the current dir. If you run it inside the cqfd folder the project is not visible.

I was just wondering if I should update to use the install version of cqfd or if we could keep both behaviours (local / global). Personally I like the fact that after cloning you don't need to install any other tool on your system.

It is also easier to deliver it to professional clients/customers. I don't think they want to have to install cqfd on their system before being able to build a binary.

Thanks

This would be cool, to know where cqfd crash

We miss a way to differentiate between cqfd versions. A release workflow and an option to print the version should help.

Hello (again)! Hope everyone's doing fine.

There are many Docker containers in the world which are designed to be used as compile/build containers, e.g. https://github.com/clux/muslrust for Rust

Does it make sense to adapt cqfd to be able to use an upstream Docker container, e.g.:

$ cat .cqfdrc
[build]
dockerhub='clux/muslrust'
command='cargo build'

In this case instead of docker build it will run docker pull.

I'm not sure if this is a good idea or it deviates too much from the intention of cqfd. The reason I want to use cqfd is that I like the DSL (flavors, archive) and all the Docker permission stuff is taken care of (I often repeat && chown -R 1000:1000 $(PWD) in all of my Docker-based compile scripts).

Hello,

I have been playing around with gitlab-ci and got an unexpected behavior when trying to run a flavor:

the input device is not a TTY

After digging in cqfd I realized that we are adding '-ti' options on the docker run command if the env var JENKINS_URL is not set, causing this issue.

Are these options necessary? I could submit a fix removing them but maybe there is a better way.

Salut les gars! :)
When I have cqfd installed on the remote host where I login through ssh, it silently exits, doing nothing.
Thanks!

Philipp