Access host from CIBW_TEST_COMMAND about cibuildwheel HOT 16 CLOSED

The root of the system is available as / on Windows and OS X (since we're on the same system), and the root of the host running the Docker image as /host on Linux builds.

If this does not solve your problem, could you give an example of the use case?

from cibuildwheel.

In the README you mention:

Environment variable: CIBW_TEST_COMMAND

Shell command to run tests after the build. The wheel will be installed automatically and available for import from the tests. The project root should be included in the command as "{project}".

Example: nosetests {project}/tests

If we run cibuildwheels on a subfolder how can we access the root of our cloned repository (e.g. for travis $TRAVIS_BUILD_DIR) from the CIBW_TEST_COMMAND? Something like {home} or {host} ...

More specifically I want to access something during pytests that is downloaded from the travis script in the home folder and has to be used in all tests in a consistent way in mac,linux and appveyor ...
e.g.: https://travis-ci.org/kavvkon/gams-api/jobs/329083740

from cibuildwheel.

Sorry about the slow reply, but I half-forgot I still had to reply to this.

There's not really anything like that since (as far as I can see) cibuildwheel doest not really know about this. You only start cibuildwheel from a certain directory, and the git clone directory that Travis CI/AppVeyor downloads your project into is on a higher level than cibuildwheel.

Couldn't you move/copy/link your file before starting cibuildwheel?

If not, you could quickly write an if statement on the OS in the CI configurations: Windows/AppVeyor -> %APPVEYOR_BUILD_FOLDER%, OS X/Travis CI -> "$TRAVIS_BUILD_DIR", and Linux/Travis CI -> "/host$TRAVIS_BUILD_DIR" (since this one is running in a docker image!)

from cibuildwheel.

As @YannickJadoul mentioned, this is not possible in the Docker due to the way the filesystems are mounted.

The /host mountpoint should provide the escape hatch for you, though... something like CIBW_ENVIRONMENT="repo_root=/host/project" on Travis should allow you to get back to dry land.

from cibuildwheel.

Hey.. Sorry for reopening this but still did not manage to make it work

I try to define as you suggest but it seems that it is not parsed properly:

environment: ParsedEnvironment(['GAMS_DIR=/host$TRAVIS_BUILD_DIR/gams'])
so when I check from the container it has the value: 'GAMS_DIR': '/host/gams',

Here is the config file
https://travis-ci.org/kavvkon/gams-api/jobs/445038025/config

from cibuildwheel.

No problem. Thanks for reopening.

But why is that not parsed properly? If I see 'GAMS_DIR=/host${TRAVIS_BUILD_DIR}/gams' in your .travis.yml and you are showing GAMS_DIR=/host$TRAVIS_BUILD_DIR/gams, so something has happened to that environment.

Could it rather be that the problem is that TRAVIS_BUILD_DIR is not defined inside the manylinux1 container?

EDIT: After having a look through the actual code, in linux.py, I do believe this is the problem.

from cibuildwheel.

Wouldn't it be better to replace the '$TRAVIS_BUILD_DIR' with its value while parsing it ?
In that case you don't have to define it in the container

I havent tested yet but I guess the same should apply for mac and windows

from cibuildwheel.

In this case, that would work, but I believe there's other cases where this would be a problem (e.g. when the result of an environment variable comes from the execution of a command to get the architecture).

I do agree that accessing the host's environment variables seems like a logical thing to do.

from cibuildwheel.

At the same time, the directory where you execute your script gets copied to /project in the container. So just setting GAMS_DIR=/project (or GAMS_DIR=`pwd` ) would also do the job, if I'm not mistaken?

from cibuildwheel.

Well I am not sure as the file is downloaded at the $HOME of Travis.
I have 3 projects in the repository:

Directory structure

TRAVIS_BUILD_DIR
|-GAMS_DIR (big library that I need for the tests in each project)
|- proj1 (projects to build with cibuildwheel)
|- proj2
|- proj3

Travis script

language: python

matrix:
  include:
    - sudo: required
      services:
        - docker
      env: PIP=pip
    - os: osx
      language: generic
      env: PIP=pip2


env:
  global:
    - CIBW_SKIP='cp35-*'
    - CIBW_TEST_REQUIRES='pytest'
    - CIBW_TEST_COMMAND='pytest -v {project}/tests/'
    - CIBW_ENVIRONMENT='GAMS_DIR=/host$TRAVIS_BUILD_DIR/gams'
    - GAMS=24.9
    - GAMS_URL=https://d37drm4t2jghv5.cloudfront.net/xxx.zip

install:
  # Install GAMS
  - wget -N $GAMS_URL -O gams.zip
  - unzip -qu gams.zip -d $HOME
  - ln -s $HOME/gams${GAMS}_linux_x64_64_sfx ${TRAVIS_BUILD_DIR}/gams #ln to root
  - export PATH="$HOME/gams:$PATH"


script:
  - $PIP install cibuildwheel==0.10.0
  - cibuildwheel ./proj1--output-dir wheelhouse
  - cibuildwheel ./proj2--output-dir wheelhouse
  - cibuildwheel ./proj3 --output-dir wheelhouse

from cibuildwheel.

Aha, I see. That makes things more complicated, indeed :-(

from cibuildwheel.

Quick and dirty hack: before calling cibuildwheel, echo -n $TRAVIS_BUILD_DIR/gams > /GAMS_DIR
And CIBW_ENVIRONMENT='GAMS_DIR=/host`cat /host/GAMS_DIR`'.

But once again, I agree there should be a better way :-)

from cibuildwheel.

@joerick, any thoughts or alternatives to do this cleanly? :-D

from cibuildwheel.

That is why I proposed something like {host} or {home} in the first and third post of this thread...

I tried quickly with
$ sudo ln -s $HOME/gams${GAMS}_linux_x64_64_sfx /gams and CIBW_ENVIRONMENT='GAMS_DIR=/host/gams' but didn't work either. I will try again tomorrow ...

from cibuildwheel.

Gah, the separate container environment strikes again. I can suggest a cleaner method:

script:
- export CIBW_ENVIRONMENT="GAMS_DIR=/host$TRAVIS_BUILD_DIR/gams"
- [your existing script commands]

That way the shell will expand the env var before it goes into docker.

from cibuildwheel.

I tried quickly with
$ sudo ln -s $HOME/gams${GAMS}_linux_x64_64_sfx /gams and CIBW_ENVIRONMENT='GAMS_DIR=/host/gams' but didn't work either. I will try again tomorrow ...

This is probably because the soft link is basically only the name of the file it's linking to (and it won't be true anymore inside the container).

export CIBW_ENVIRONMENT="GAMS_DIR=/host$TRAVIS_BUILD_DIR/gams"

Ah yes, of course: setting the environment variable during the script instead of before. That should do the job, since $TRAVIS_BUILD_DIR gets replaced during the execution outside of the container.

from cibuildwheel.