astropy / ci-helpers Goto Github PK

Comment from @cdeil:
"Many packages need this. Move this to ci-helpers and make it opt-in, i.e. execute these commands if the user sets something like SETUP_DISPLAY_FOR_MATPLOTLIB?"

This refers to the setup needed for mpl:

# Make sure that interactive matplotlib backends work
export DISPLAY=:99.0;
sh -e /etc/init.d/xvfb start;

What do you think @astrofrog? Would this work for both the linux an osx builds? For the latter I saw packages prefer to use pyqt rather than pyside, but do we want hardwire such things?

In some cases, people might want to not have conda builds be quiet, so maybe we can add an env variable (or simply re-use $DEBUG) to indicate that we want verbose output from conda.

At the moment it needs to be set to a version, and dev or development or stable is not supported.

The conda-forge people use the bit of code below to cancel an appveyor build of a PR if there is a newer commit in the PR when the appveyor build starts. We haven't been really backed up since 1.2 came up but it might be helpful in the future: https://github.com/conda-forge/staged-recipes/blob/master/appveyor.yml#L56-L64

The issue leading to the mkl pin (https://github.com/astropy/ci-helpers/blob/master/appveyor/install-miniconda.ps1#L156) was fixed several months ago. Could this be unpinned?

They are not designed to be listed in CONDA_DEPENDENCIES and if one of the string versions were used in e.g. ASTROPY_VERSION, the version pinning fails the build.

This issue came up with Chandra-MARX/marxs#62, here is a travis log snippet:

++++CONDA_DEPENDENCIES='numpy scipy astropy cffi sphinx matplotlib'

++++[[ True == True ]]

++++cat /home/travis/miniconda/envs/test/conda-meta/pinned

numpy stable*

scipy

astropy stable*

cffi

sphinx

matplotlib

++++echo numpy scipy astropy cffi sphinx matplotlib

numpy scipy astropy cffi sphinx matplotlib

++++[[ stable == dev* ]]

++++[[ stable == stable ]]

++++conda install numpy

Fetching package metadata: ..........

Solving package specifications: .

Error:  Packages missing in current linux-64 channels: 

  - numpy stable*

  - astropy stable*

For example: https://travis-ci.org/astropy/astropy/jobs/106803692

usage: coveralls [-h] [--verbose] [--dryrun] [--gcov FILE]
                 [--gcov-options GCOV_OPTS] [-r DIR] [-b DIR] [-e DIR|FILE]
                 [-i DIR|FILE] [-E REGEXP] [-x EXT] [-y FILE] [-n] [-t TOKEN]
                 [--encodings ENCODINGS [ENCODINGS ...]] [--dump [FILE]]
                 [--follow-symlinks]

coveralls: error: unrecognized arguments: --merge=c-coveralls.json --rcfile=astropy/tests/coveragerc

It seems that the coveralls command here is the one from the cpp-coveralls package (ahah, good joke : https://github.com/eddyxu/cpp-coveralls/blob/master/setup.py#L47), not the one from coveralls.

And the reason why it breaks now is probably #60 ? Comparing with a succesful job, cpp-coveralls was installed before coverage/coveralls.

I'd like to be able to use ci-helpers run coverage tests under Python 3.5, but coverage 3.7.1 doesn't appear to work. Is there a reason that it is still pinned to 3.7.1 by default?

In the following example:

https://ci.appveyor.com/project/Eric89GXL/vispy/build/1.0.1568

There is an issue with a download, but the build proceeds anyway. This should not be the case, but not sure how to change that?

@bsipocz and @pllim
I still have a couple of tests failing due to conda ci issues:
https://travis-ci.org/LCOGT/banzai/jobs/139364978

https://travis-ci.org/LCOGT/banzai/jobs/139364982

Fetching package metadata .............
Solving package specifications: ....
The following specifications were found to be in conflict:

• astropy 1.0* -> numpy 1.10_|1.6_|1.7_|1.8_|1.9*
• astropy 1.0* -> python 2.6_|2.7_|3.3_|3.4_
• python 3.5*
  Use "conda info " to see the dependencies for each package.

The other builds pass: https://travis-ci.org/LCOGT/banzai/builds/139364970

Thanks for your help.

I was testing a setup using old numpy (1.7), and with the following dependencies. Since the latest conda version of scipy for this numpy is 0.13, it had been moved to be installed with pip to get a recent version. However as scikit-image was also a dependency, scipy was installed via the conda steps and PIP_DEPENDENCIES had no effect whatsoever. As a workaround I've moved scikit-image and cython to the PIP_DEPENDENCIES, too, but wonder whether ci-helpers could be cleaver.

       - PIP_DEPENDENCIES='scipy'
       - CONDA_DEPENDENCIES='Cython scikit-image matplotlib'

The pep8 package has been renamed to pycodestyle, so we should make sure we take that into account in the future (probably not urgent)

I realized that the python 3.5 appveyor build seems to fail with "Error code 1" (see for example here)

There was a new msvc_runtime build on conda (yesterday or tonight) and it seems likely that this is somehow connected. But I cannot reproduce it locally - maybe because I'm using mkl 11.3.3 and some other packages that aren't installed in AppVeyor - so I can't investigate it.

Just ran into this issue

git clone git://github.com/astropy/ci-helpers.git
Cloning into 'ci-helpers'...
powershell ci-helpers/appveyor/install-miniconda.ps1
Installing miniconda latest for -x64 bit architecture to C:\conda
Downloading Miniconda-latest-Windows-x86_64.exe from https://repo.continuum.io/miniconda/Miniconda-latest-Windows-x86_64.exe
Exception calling "DownloadFile" with "2" argument(s): "The request was 
aborted: Could not create SSL/TLS secure channel."
At C:\projects\qtawesome\ci-helpers\appveyor\install-miniconda.ps1:45 char:8
+        $webclient.DownloadFile($url, $filepath)
+        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : WebException

Installing C:\projects\qtawesome\Miniconda-latest-Windows-x86_64.exe to C:\conda
C:\projects\qtawesome\Miniconda-latest-Windows-x86_64.exe /InstallationType=AllUsers /S /AddToPath=1 /RegisterPython=1 /D=C:\conda
Start-Process : This command cannot be run due to the error: The system cannot 
find the file specified.
At C:\projects\qtawesome\ci-helpers\appveyor\install-miniconda.ps1:65 char:5
+     Start-Process -FilePath $filepath -ArgumentList $args -Wait -Pass ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Start-Process], InvalidOp 
   erationException
    + FullyQualifiedErrorId : InvalidOperationException,Microsoft.PowerShell.C 
   ommands.StartProcessCommand

Failed to install Python in C:\conda
Command exited with code 1

Is this an appveyor thing?

If I write e.g.

NUMPY_VERSION=stable

in the global environment variables, the pinned file will contain:

numpy stable*

The conda command then crashes later because it can't find that version. I think this is not really a big issue because really writing

NUMPY_VERSION=stable

is redundant as long as numpy is in the CONDA_DEPENDENCIES, but we should probably still fix this in the long run.

Astropy's appveyor has been failing for a few days. It seems that the miniconda file it tries to download doesn't exists any more and while the same url perfectly works for the other packages, it's consistently broken for astropy and when trying download manually. I'm not sure why it's still working for the other packages, we don't use the cache in the template, but I guess it will brake there, too sometimes soon.

Build started
git clone -q --branch=master https://github.com/astropy/astropy.git C:\projects\astropy
git checkout -qf 5d3b7103f5805a22befba6a7ff3cd52b178c9611
Running Install scripts
git clone git://github.com/astropy/ci-helpers.git
Cloning into 'ci-helpers'...
powershell ci-helpers/appveyor/install-miniconda.ps1
Installing miniconda latest for -x64 bit architecture to C:\conda
Downloading Miniconda-latest-Windows-x86_64.exe from https://repo.continuum.io/miniconda/Miniconda-latest-Windows-x86_64.exe
Exception calling "DownloadFile" with "2" argument(s): "The remote server 
returned an error: (404) Not Found."
At C:\projects\astropy\ci-helpers\appveyor\install-miniconda.ps1:47 char:8
+        $webclient.DownloadFile($url, $filepath)
+        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : WebException
 
Installing C:\projects\astropy\Miniconda-latest-Windows-x86_64.exe to C:\conda
C:\projects\astropy\Miniconda-latest-Windows-x86_64.exe /InstallationType=AllUsers /S /AddToPath=1 /RegisterPython=1 /D=C:\conda
Start-Process : This command cannot be run due to the error: The system cannot 
find the file specified.
At C:\projects\astropy\ci-helpers\appveyor\install-miniconda.ps1:67 char:5
+     Start-Process -FilePath $filepath -ArgumentList $args -Wait -Pass ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Start-Process], InvalidOp 
   erationException
    + FullyQualifiedErrorId : InvalidOperationException,Microsoft.PowerShell.C 
   ommands.StartProcessCommand
 
Failed to install Python in C:\conda
Command exited with code 1

Opening this as a reminder.

I had a start on this but ended up having weird errors in the powershell emulator I tried, and SO tips don't work there either.

We should make sure that the tests in this repository cover all possibilities, since this package is going to have to be very robust. I started adding more tests in #7 but we need to make sure all routes are covered, and that we do the same for AppVeyor.

Lately I have been seeing Travis builds failing because:

E   ImportError: libgfortran.so.1: cannot open shared object file: No such file or directory

For example: https://travis-ci.org/weaverba137/pydl/jobs/115556329

My .travis.yml file includes:

- CONDA_DEPENDENCIES='libgfortran scipy matplotlib'

and the output of source ci-helpers/travis/setup_conda_$TRAVIS_OS_NAME.sh appears to show that libgfortran is installed.

Any help would be appreciated.

First of all thanks for the work on this.

I managed to (in a kinda hacky way) use the ci-helpers within circle ci for different versions of python (up to 4 different, and one per container)

Since every command in circle ci is its own shell, the path needs to be updated on every command (a bit annoying...!).

Also I had to use Travis names for the env variables which is fine, but we could perhaps expand on something like this¿
https://github.com/spyder-ide/qtpy/blob/master/circle.yml

Cheers

The new system of specifying package versions via environment variables works nicely. However, there is a small difference in behavior between specifying them via environment variables and via the CONDA_DEPENDENCIES, i.e.

MATPLOTLIB_VERSION=1.3

vs

CONDA_DEPENDENCIES='matplotlib=1.3'

The first case results in the pinned file containing

matplotlib 1.3*

and the second:

matplotlib ==1.3

I think that the second case should also be changed to mean 1.3*, because that's normally how it works if we do conda install matplotlib=1.3 (1.3.1 could be installed)

This was first reported in astropy/astropy#5094 (comment), the install process erroneously picked up the wrong numpy version (1.9.0b1) from the astropy-ci-extras conda channel rather than a stable released version (e.g. 1.9.3) from the default conda channel.

There are several issues with this, first we should never pick up a beta release unless asked for it with NUMPY_VERSION=prerelease. Also we should always pick up the latest available version of a given package.

As a side feature request: would be nice to find a way to specify the priority order of conda channels for cases when a package is available in several ones.

I needed to run a test against Numpy 1.12.0rc2. I tried running a Travis test with NUMPY_VERSION=prerelease, but the test exited prematurely (resulting in a Travis error rather than a failure). When I ran the test by hand, I found that the prerelease version was trying to load MKL libraries and failing. When I look at the setup_dependencies_common.sh file, I can see that nomkl is preinstalled for NUMPY_VERSION=dev* but not NUMPY_VERSION=pre*.

If this is intentional, how can I work around this problem?

Currently, if e.g:

      MINICONDA_VERSION: "3.5.5"

is not given in appveyor.yml, the installer crashes.

I got nagged by both Travis CI and Appveyor to upgrade to pip 8.1.2. Here is an example message from Appveyor:

You are using pip version 8.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

On Travis, I am able to get rid of that message by doing:

install:
    - git clone git://github.com/astropy/ci-helpers.git
    - source ci-helpers/travis/setup_conda_$TRAVIS_OS_NAME.sh

    - pip install --upgrade pip

But this does not work on Appveyor:

install:
    - "git clone git://github.com/astropy/ci-helpers.git"
    - "powershell ci-helpers/appveyor/install-miniconda.ps1"
    - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
    - "activate test"

    - "python -m pip install --upgrade pip"

I see that it updates pip but then it still complains about the old version:

Successfully installed astropy-1.2.dev15418
You are using pip version 8.1.1, however version 8.1.2 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

SET PATH=%PYTHON%;%PYTHON%\Scripts;%PATH%
activate test
Deactivating environment "C:\conda"...
Activating environment "C:\conda\envs\test"...
python -m pip install --upgrade pip
Collecting pip
  Downloading pip-8.1.2-py2.py3-none-any.whl (1.2MB)
Installing collected packages: pip
  Found existing installation: pip 8.1.1
    Uninstalling pip-8.1.1:
      Successfully uninstalled pip-8.1.1
Successfully installed pip-8.1.2

Moving the upgrade command to before git clone git://github.com/astropy/ci-helpers.git also does not work.

Why am I getting these messages? Does ci-helpers set the pip version to 8.1.1 for some reason? Should there be extra command in ci-helpers to automatically upgrade pip?

For sphinx builds either:

• a) use older np version
• b) pip install rather than conda install mpl
• c) build (or find) a conda package of the combination of mpl 1.5.0 and np 1.11.0

OpenAstronomy/conda-channel#9 approaches the problem through option c).

Based on recent discussion in #97 #98 and on gitter, we will need an option to fall back on altrnative ways of installing a package is the version available on conda is too old.

E.g. if a build asks for 'stable' astropy version (currently we have v1.2rc1), then it's not OK to install v0.3 or v0.4 or even v1.0.3 from conda, but should go instead pip install v1.1.2

On Travis, doing something like

PIP_DEPENDENCIES="pydicom https://github.com/glue-viz/glue/archive/master.zip"

is fine, but on AppVeyor, it results in the following error:

https://ci.appveyor.com/project/astrofrog/glue-medical/build/1.0.12/job/sr8nrcrybw7e47d5

This is not super high priority, but it would be good to investigate it at some point. I wonder if it could potentially be a pip bug.

We should recognize build_docs as an alternative to build_sphinx

Is it possible to change the setup so that the executed commands appear in the travis-ci log?

Currently only

$ source ci-helpers/travis/setup_conda_$TRAVIS_OS_NAME.sh

is displayed and then the output from the commands in that script, but not the commands themselves.
Example: https://travis-ci.org/astropy/ci-helpers/jobs/92443363#L268

This makes the logs very hard to read and makes it difficult to debug issues if one of those commands fails or doesn't do what one expects. It's cost me a few hours over the past year and it's always frustrating when issues happen at that stage.

On the other hand, the travis-ci part of the build log is usually very easy to read because the commands are printed and even highlighted / grouped in the build log.

I'm not familiar much with shell scripting, but there should be a flag / option to run these scripts so that commands are echoed before they are executed, no?

I have the following setup in my appveyor.yml:

  global:
      PYTHON: "C:\\conda"
      MINICONDA_VERSION: "latest"
      CMD_IN_ENV: "cmd /E:ON /V:ON /C .\\ci-helpers\\appveyor\\windows_sdk.cmd"
      PYTHON_ARCH: "64" # needs to be set for CMD_IN_ENV to succeed. If a mix
                        # of 32 bit and 64 bit builds are needed, move this
                        # to the matrix section.
      CONDA_DEPENDENCIES: "scipy matplotlib"
      NUMPY_VERSION: "1.9.1"
      ASTROPY_VERSION: "stable"

AppVeyor exits with

(...)
Fetching package metadata: ...... 
Error: No packages found in current win-64 channels matching: astropy stable*

You can search for this package on anaconda.org with

    anaconda search -t conda astropy stable*

You may need to install the anaconda-client command line client with

    conda install anaconda-client


SET PATH=%PYTHON%;%PYTHON%\Scripts;%PATH%
(...)

Since today, there's conda install issues in the Gammapy builds:
https://travis-ci.org/gammapy/gammapy/jobs/103558898

This error mentioning gawk is present in Astropy core and Gammapy builds:
https://travis-ci.org/astropy/astropy/jobs/103553616#L181

cc @bsipocz

Low priority, but would be nice in future to avoid having two sets of instructions.

If using ASTROPY_VERSION=stable, I think we should uninstall astropy from conda in the following block:

if [[ $ASTROPY_VERSION == stable ]]; then
    old_astropy=$(python -c "from distutils.version import LooseVersion;\
                  import astropy; import os;\
                  print(LooseVersion(astropy.__version__) <\
                  LooseVersion(os.environ['LATEST_ASTROPY_STABLE']))")

    if [[ $old_astropy == True ]]; then
        $PIP_INSTALL --upgrade --no-deps --ignore-installed astropy==$LATEST_ASTROPY_STABLE
    fi
fi

Otherwise, under some circumstances, the conda version takes precedence over pip.

Most of the packages catch the docs build warnings and get failing the build status on travis if there are any of them. We need to reproduce that here, and see whether any WARNINGs can come out of the install process.
(See this astropy build on the v1.0.x branch https://travis-ci.org/astropy/astropy/jobs/100148639#L362)

This may be useful as easy to add (at least to travis).

At the moment four channels are used unless the user sets CONDA_CHANNELS. These are defaults, astropy-ci-extras, astropy, and openastronomy.

Recently we saw a few problems when a package is available in both astropy and openastronomy and still somehow the build ends in conflicts, e.g. https://travis-ci.org/radio-astro-tools/spectral-cube/jobs/147998479#L526

So I propose to remove openastronomy from the default channels (as we tend to host all of dependencies of the affiliates in astropy). Should any package needs something from that channel they can specify it in CONDA_CHANNELS.

Thoughs @astrofrog, @mwcraig?

I think that we might want to have a flag that indicates whether to install Cython and jinja2. Jinja2 is only required by the astropy core package, and Cython only by a small fraction of affiliated packages (+core).

We should check that PYTHON_VERSION or TRAVIS_PYTHON_VERSION is taken into account

Individual packages may want to have their own conda setups, e.g. for channel priorities, etc.

Ideally we can introduce an env variable for this.

This would allow to better integrate with other CI services like circle ci where the limit of containers is 4, hence a matrix consisting of more options would need to run several test on the same machine (create several different environments).

source ci-helpers/travis/setup_conda.sh "ENV-NAME"

We had the issue that we coundn't fix the mpl version number to 1.5.0 in the photutils docs builds (astropy/photutils#318) as it always got upgraded.

The upgrade is due to https://github.com/astropy/ci-helpers/blob/master/travis/setup_dependencies_common.sh#L106 where mpl isn't inheriting the fixed version number.

We can have temporary workarounds like #41, or introduce version number enviroment variables (like we have for astropy and numpy alread) for all the packages that get installed as an instrastructural dependency (like sphinx and matplotlib).

However it may be the best to pin the version numbers of all the packages that are listed with a required version number in CONDA_DEPENDENCIES. I have a half baked hack for this, and will open a PR once I manage to get it work.

cc: @larrybradley @astrofrog

This issue was originally reported on astropy/package-template#165, but it was recommended to move it here.

I'd like to be able to run Travis tests in Python 3.5, but currently there is a package compatibility issue in anaconda that prevents the installation of scipy, which my package requires.

Below is a snippet of the Travis log for a recent build, showing the incompatibility report. You can also view the full log.

The following specifications were found to be in conflict:
  - matplotlib -> numpy 1.9*
  - matplotlib -> python 2.6*|2.7*|3.3*|3.4*
  - numpy 1.9* -> python 2.6*|2.7*|3.3*|3.4*
  - pip (target=pip-8.1.1-py35_1.tar.bz2) -> python 2.6*|2.7*|3.3*|3.4*
  - py (target=py-1.4.31-py35_0.tar.bz2) -> python 2.6*|2.7*|3.2*|3.3*|3.4*|3.5*
  - pytest (target=pytest-2.9.1-py35_0.tar.bz2) -> python 2.6*|2.7*|3.2*|3.3*|3.4*|3.5*
  - python 3.5*
  - scipy 0.14* -> numpy 1.9*
  - scipy 0.14* -> python 2.6*|2.7*|3.3*|3.4*
  - setuptools (target=setuptools-20.7.0-py35_0.tar.bz2) -> python 2.6*|2.7*|3.2*|3.3*|3.4*|3.5*
  - wheel (target=wheel-0.29.0-py35_0.tar.bz2) -> python 2.7*|3.3*|3.4*|3.5*
Use "conda info <package>" to see the dependencies for each package.

Ideally we want to install packages in a strict order.

1. numpy
2. astropy
3. dependencies

However it seems that due to the existence of the version pin file, all the dependencies are picked up during the first step. This is suboptimal for several of our typical builds (e.g. when older astropy is required, or when the build is aborted as no numpy prerelease is available, etc.).

E.g. here is a build were only numpy needed to be installed, and then the build aborted as there is no prerelease. Then in reality everything gets installed before exiting.
https://travis-ci.org/astropy/astroquery/jobs/150718956#L685

This is an LTS astropy build, yet 1.2.1 is installed first and then downgraded:
https://travis-ci.org/astropy/astroquery/jobs/150718941#L544

It's not yet clear what is an optimal workaround. To ignore the pin file, we could use e.g. --no-pin. Maybe adding it to the first, the numpy builds would be enough. Then most of the astropy version checking can be added to the pin file, too.

@astrofrog - Do you have a better idea or preference?

@bsipocz - I've started to add instructions to the README for how to use this module, but this issue is a reminder that we need to add the rules about which environment variables need to/can be set for the dependencies.

Hi, I don't know if I have missed something, but this is rather annoying:

https://travis-ci.org/sunpy/sunpy/jobs/139200223#L606

# All requested packages already installed.
# packages in environment at /home/travis/miniconda/envs/test:
#
astropy                   1.2                 np111py27_0  
numpy                     1.11.0          py27_blas_openblas_200  [blas_openblas]  conda-forge
python                    2.7.11                        0  
Due to a matplotlib issue (#5836), the version for the
            sphinx builds needs to be !=1.5.1. This may override the version
            number specified in 
Fetching package metadata ............
Solving package specifications: ....
The following specifications were found to be in conflict:
  - matplotlib !=1.5.1
  - python 2.7*
Use "conda info <package>" to see the dependencies for each package.

it seemed to be working fine before...

It seems there is a bug with PIP_DEPENDENCIES on appveyor.

Invalid requirement: 'keyring aplpy pyregion'
Traceback (most recent call last):
  File "C:\conda\envs\test\lib\site-packages\pip\req\req_install.py", line 78, in __init__
    req = Requirement(req)
  File "C:\conda\envs\test\lib\site-packages\pip\_vendor\packaging\requirements.py", line 96, in __init__
    requirement_string[e.loc:e.loc + 8]))
InvalidRequirement: Invalid requirement, parse error at "'aplpy py'"
SET PATH=%PYTHON%;%PYTHON%\Scripts;%PATH%
activate test
prepending C:\\conda\\envs\\test and C:\\conda\\envs\\test\\Library\\

https://ci.appveyor.com/project/Astropy/astroquery/build/1.0.24/job/to6reykyqnk5esxa#L142

Based on this conversation astropy/astropy#4556 (comment) (copied below), we may add an option of either skip (or run) a subset of the travis builds when there is a string pattern in the commit message.

I'm not sure which approach is better, given that many times even [ci skip] is not used optimally. However before implementing anything, we need to decide whether this is needed, and which approach (run or skip) to go with (or can do both).

mhvk commented 12 hours ago
Agreed that we should not build both! Indeed, in principle, I think it would be nice to reduce the tests run for PRs to a smaller subset, and only run a full suite before actual merging. But not sure that is possible (and easy).

bsipocz commented 11 hours ago
I think it shouldn't be too difficult to add an env variable to ci-helpers that makes sure that the tests are only run when the commit message contains a certain keyword (e.g. Merge pull request (as that is included in all PR merge, or something like [full tests] or [full suite], etc).

I'm cc-ing for brainstorming whoever may be interested in this for astropy and other packages.
@astrofrog @mhvk @eteq @cdeil @Cadair