conda / constructor Goto Github PK

View Code? Open in Web Editor NEW

450.0 33.0 164.0 2.02 MB

tool for creating installers from conda packages

Home Page: https://conda.github.io/constructor/

License: Other

Python 85.43% Shell 8.00% NSIS 5.36% Makefile 0.01% Rich Text Format 0.64% Xonsh 0.43% PowerShell 0.12%

constructor's Introduction

(conda) Constructor

Constructor is a tool which allows constructing an installer for a collection of conda packages.

It solves needed packages using user-provided specifications, and bundles those packages. It can currently create 3 kinds of installers, which are best thought of as delivery vehicles for the bundled packages. There are shell .sh installers, macOS .pkg installers, and Windows .exe installers.

Each of these will create an environment on the end user's system that contains the specs you provided, along with any necessary dependencies. These installers are similar to the Anaconda and Miniconda installers, and indeed constructor is used to create those installers.

Installation

constructor can be installed into any environment using:

$ conda install constructor

Once installed, the constructor command will be available:

$ constructor -h

Usage

The constructor command takes an installer specification directory as its argument. This directory needs to contain a file construct.yaml, which specifies the name of the installer, the conda channels to pull packages from, the conda packages included in the installer, etc. The complete list of keys in this file can be found in CONSTRUCT.md. Also, the directory may contain some additional optional files (such as a license file, and image files for the Windows installer). Examples for various installations are located in the examples directory.

More details can be found in the documentation.

Development

To build or update CONSTRUCT.md at the root of the repo you'll need to install the jinja2 package (conda install jinja2) and then run make doc, or invoke the the script directly with python scripts/make_docs.py.

Build status


`conda install defaults::constructor`
`conda install conda-forge::constructor`
`conda install conda-canary/label/dev::constructor`

constructor's People

Contributors

Stargazers

Watchers

Forkers

gmarkall bheklilr willingc petdoan insertinterestingnamehere msarahan croth1 aseemdrockstar dedop dsludwig gpfreitas vidartf treetoc gabm waltonseymour jjhelmus wulmer karthikeyashastry david-gibbs mbargull cryoem looooo pombredanne megies darkdreamingdan kalefranz thomasantony andyli nehaljwani sylvaincorlay hokreb h0r5e schrodinger jakirkham martindurant jdetle andbmme alep sci-bots astrofrog www3838438 gabrielclow jschueller jaimergp y2kbugger austinsmom radiusoss harperjohnarc wenjinlv jdbcode ericpre dakcarto neviko soapy1 kwells-anaconda living1069 osgeo-forge david-baddeley sscherfke huamichaelchen axgkl algerjiang k-dominik programfan b2220333 axiros bbbbb38438 felixs90 patrickchirdon spara jlstevens astaric tberry9 forrestwaters davidlowryduda rrigdon carterbox mohneeshjhapuresoftware ossdev07 ross1503 appliedbinf cmckeague isuruf yarikoptic sed-inria-sophia angloyna bbbbbgit fbriol scopatz edennuriel abhisheknishantpuresoftware ydmytryk pradghos nerohmot pelson fagan2888 chrisburr hmaarrfk risyomei marimeireles

constructor's Issues

Existing file causes error

I have a constructor file like the following

name: Rattlesnake
version: 1.0.0.0

channels:
  - https://repo.continuum.io/pkgs/r/
  - https://repo.continuum.io/pkgs/free/

specs:
  - python 3.5*
  - anaconda
  - r-essentials

Running constructor more than once results in the following error on a Windows 10 system:

platform: win-64
Fetching package metadata .....
Solving package specifications: ..........

fetching: https://repo.continuum.io/pkgs/r::m2w64-bwidget-1.9.10-1.tar.bz2
Traceback (most recent call last):
  File "C:\Anaconda3\lib\site-packages\conda\fetch.py", line 441, in download
    os.rename(pp, dst_path)
FileExistsError: [WinError 183] Eine Datei kann nicht erstellt werden, wenn sie bereits vorhanden ist: 'C:\\Users\\<USERNAME>\\.conda\\constructor\\win-64\\m2w64-bwidget-1.9.10-1.tar.bz2.part' -> 'C:\\Users\\<USERNAME>\\.conda\\constructor\\win-64\\m2w64-bwidget-1.9.10-1.tar.bz2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Anaconda3\Scripts\constructor-script.py", line 5, in <module>
    sys.exit(main())
  File "C:\Anaconda3\lib\site-packages\constructor\main.py", line 146, in main
    verbose=opts.verbose)
  File "C:\Anaconda3\lib\site-packages\constructor\main.py", line 81, in main_build
    fcp.main(info, verbose=verbose)
  File "C:\Anaconda3\lib\site-packages\constructor\fcp.py", line 185, in main
    fetch(info)
  File "C:\Anaconda3\lib\site-packages\constructor\fcp.py", line 162, in fetch
    fetch_pkg(pkginfo, download_dir)
  File "C:\Anaconda3\lib\site-packages\conda\fetch.py", line 326, in fetch_pkg
    download(url, path, session=session, md5=info['md5'], urlstxt=True)
  File "C:\Anaconda3\lib\site-packages\conda\fetch.py", line 444, in download
    (pp, dst_path, e))
RuntimeError: Could not rename 'C:\\Users\\<USERNAME>\\.conda\\constructor\\win-64\\m2w64-bwidget-1.9.10-1.tar.bz2.part' to 'C:\\Users\\<USERNAME>\\.conda\\constructor\\win-64\\m2w64-bwidget-1.9.10-1.tar.bz2': FileExistsError(17, 'Eine Datei kann nicht erstellt werden, wenn sie bereits vorhanden ist')

The german parts in this message mean "A file cannot be created if it already exists". Deleting the .conda/constructor directory solves this issue but causes a lengthy re-download of all packages.

Anaconda references in `OptionsDialog.nsh`

In constructor/nsis/OptionsDialog.nsh the installation is still referred to as "Anaconda" which should be changed at some point.
Furthermore, and more importantly, the AddToPath-option's text says:

         $\nIf unchecked, then you must use the Anaconda Command Prompt \
         $\n(located in the Start Menu under $\"Anaconda (${ARCH})$\")."

Which might be misleading for end users if the corresponding menu entries are not provided by every installer.

Version numbers for Windows must be in X.X.X.X format for NSIS

When trying out constructor today I stumbled across an undocumented behavior with version numbers on Windows platforms. The NSIS system expects the version number to be in the format X.X.X.X, so version numbers given in the construct.yaml file must also be in that format.

I would be happy with a warning from constructor that NSIS is going to reject the version number, or if a semantic version number has already be specified in the construct.yaml file for .0 to be appended to the version number string until it's in the appropriate format.

It would also be nice if the version field was automatically interpreted as a string rather than an int or a float, although this is simple and fast to fix after seeing the error message.

prepend path in .bashrc file

Hi all,

At the end of an installation (in Linux), the installer asks whether to prepend the install location to the PATH in the .bashrc file. For an update (with -u), and if the user always selects 'yes', multiple entry will be added to the .bashrc file. Will it be possible to ignore this when it is an update? Or better, will it be possible to add a little script to find out whether the same path has previously been added to .bashrc. If it has, do nothing.

Thanks,
Hans

feature: index-url + pip in spec

Any plans to support specifying an index-url and pip only packages in the spec like conda env export produces?

Feature request: MSI installers on Windows

Not sure whether NSIS supports making MSI installers.

Constructor Name and versions are too strict

constructor conda.recipe --output-dir=$CIRCLE_ARTIFACTS
platform: linux-64
Error: invalid name 'Anaconda Repository'

constructor conda.recipe --output-dir=$CIRCLE_ARTIFACTS returned exit code 1

Action failed: constructor conda.recipe --output-dir=$CIRCLE_ARTIFACTS

platform: linux-64
Error: invalid version '2.21.0-180'

Conda clean --all should also clean the cached constructor packages

When using constructor the packages are cached in $HOME/.conda/constructor

This folder is not cleaned out by the conda clean command build logically it perhaps should be.

Also, when updating a local conda channel using conda index if the updated package has the same name as a previous package then conda constructor does not allow it to be overwritten

channels in constructor should behave the same as in condarc

It is confusing and inconvenient that .condarc acceptss defaults or sean but constructor requires a full url.

This also makes it harder do create installers with private packages because it does not respect anaconda login.

Feature request: implement pre_install scripts on Windows

Hi @ilanschnell

I am making a check for windows users installing Anaconda Fusion via constructor installer and I was using but it seems this can't be done :-| (yet?)

Checking for 'C:\Users\builder\Anaconda3\NSIS\makensis.exe'
NSIS version: v3.0
Error: Cannot run pre install on Windows, sorry

Missing files in NSIS 3.01 (with constructor 1.5.2)

When I try to build a constructor package, a few NSIS plugin DLLs are missing:

Plugin not found, cannot call UAC::_

When I copy the UAC.dll from NSIS/Plugins/x86-unicode/ to NSIS/Plugins/x86-ansi/, the next error message appears:

Plugin not found, cannot call untgz::extract

So I copied the untgz.dll and then the installer build worked. But it seems the DLLs are not compatible and when I execute the installer, it fails extracting the zip files.

Is it possible to provide a new nsis package with all the necessary plugins?

Add SSL CA verify option to file

Add a way to specify the SSL CA root when running against a Repo using SSL.

Post-install fails to execute correctly

Hi, I have distributed a software for Windows using Constructor which has a post-install.bat script. The script is used to write a registry key which other installers can inspect to find the chosen installation path.

On some systems (none I have access to) it seems like the script is not being run or is not running correctly and the registry key is not being written (rather than being written incorrectly). I have attached the script in question (renamed to *.txt).

post_install.txt

I was wondering if there is any logging that can be viewed to see the result of running the script? My only other ideas were that there was some permissions or anti-virus blocking occurring.

I have the corresponding issue here: DTOcean/dtocean#3

Many thanks,

Mat

Feature request: ability to specify root .condarc file values

Currently when an installer is run there is no .condarc file in the root env. Often it is desirable to setup specific channels, default env packages, proxies, env paths, etc in the root .condarc especially within enterprises.

Add a section to the construct.yaml where the root .condarc values can be specified then have the installer create the root .condarc file when the installer is run.

R-Channel causes wrong filenames in generated main.nsi

I have the following constructor file:

name: Rattlesnake
version: 1.0.0.0

channels:
  - https://repo.continuum.io/pkgs/r/
  - https://repo.continuum.io/pkgs/free/

specs:
  - python 3.5*
  - anaconda
  - r-essentials

Running constructor on a Windows 10 machine results in the following error message:

platform: win-64
Fetching package metadata .....
Solving package specifications: ..........

fetching: python-3.5.1-4.tar.bz2
<SNIP>
fetching: heapdict-1.0.0-py35_0.tar.bz2
fetching: https://repo.continuum.io/pkgs/r::m2w64-bwidget-1.9.10-1.tar.bz2
<SNIP>
fetching: https://repo.continuum.io/pkgs/r::r-zoo-1.7_13-r3.3.0_0.tar.bz2
fetching: idna-2.0-py35_0.tar.bz2
<SNIP>
fetching: zlib-1.2.8-vc14_2.tar.bz2
Reading: C:\Anaconda3\lib\site-packages\constructor\nsis\main.nsi.tmpl
MSVC runtimes found: ['vs2015_runtime-14.00.23026.0-0.tar.bz2']
Created C:\Users\<USERNAME>\AppData\Local\Temp\tmppnnabn7b\main.nsi file
Calling: ['C:\\Anaconda3\\NSIS\\makensis.exe', '/V2', 'C:\\Users\\<USERNAME>\\AppData\\Local\\Temp\\tmppnnabn7b\\main.nsi']
File: "C:\Users\<USERNAME>\.conda\constructor\win-64\https://repo.continuum.io/pkgs/r::m2w64-bwidget-1.9.10-1.tar.bz2" -> no files found.
Usage: File [/nonfatal] [/a] ([/r] [/x filespec [...]] filespec [...] |
   /oname=outfile one_file_only)
Error in script "C:\Users\<USERNAME>\AppData\Local\Temp\tmppnnabn7b\main.nsi" on line 735 -- aborting creation process
Traceback (most recent call last):
  File "C:\Anaconda3\Scripts\constructor-script.py", line 5, in <module>
    sys.exit(main())
  File "C:\Anaconda3\lib\site-packages\constructor\main.py", line 146, in main
    verbose=opts.verbose)
  File "C:\Anaconda3\lib\site-packages\constructor\main.py", line 84, in main_build
    create(info)
  File "C:\Anaconda3\lib\site-packages\constructor\winexe.py", line 156, in create
    check_call(args)
  File "C:\Anaconda3\lib\subprocess.py", line 584, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['C:\\Anaconda3\\NSIS\\makensis.exe', '/V2', 'C:\\Users\\<USERNAME>\\AppData\\Local\\Temp\\tmppnnabn7b\\main.nsi']' returned non-zero exit status 1

Opening the file C:\Users\<USERNAME>\AppData\Local\Temp\tmppnnabn7b\main.nsi, removing all instances of https://repo.continuum.io/pkgs/r:: and running the last command manually creates a functioning installer.

Example entry in main.nsi before manual edit

    # --> https://repo.continuum.io/pkgs/r::m2w64-bwidget-1.9.10-1.tar.bz2 <--
    File "C:\Users\<USERNAME>\.conda\constructor\win-64\https://repo.continuum.io/pkgs/r::m2w64-bwidget-1.9.10-1.tar.bz2"
    untgz::extract "-d" "$INSTDIR" "-zbz2" "$INSTDIR\pkgs\https://repo.continuum.io/pkgs/r::m2w64-bwidget-1.9.10-1.tar.bz2"
    ExecWait '"$INSTDIR\pythonw.exe" "$INSTDIR\Lib\_nsis.py" postpkg'
    Delete "$INSTDIR\pkgs\https://repo.continuum.io/pkgs/r::m2w64-bwidget-1.9.10-1.tar.bz2"

Spurious(?) Post-Install Error Message on Windows

Adding wxpython to the grin demo package (see attached .yaml) causes the windows self-installer to display an error message "Failed to run post install script". There is no evidence that wxpython has a post-install script.

construct.yaml.txt

Uninstall information in HKCU when running silent installer as admin

Hi, I tried running an installer (created by constructor) with the /S option from an admin console. While the installation went well, I noticed that the uninstaller registry entry is located under HKCU\Software\Microsoft\Windows\CurrentVersion\Uninstall.

When I run the installer without /S, the uninstall information is written under HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall.

In my opinion, the location of the uninstaller entry should not change depending on the mode, so I consider this a bug.
I have checked the NSI source code and I think it has to do with the automatic detection of the install mode. While $InstMode might be correctly guessed as ${ALL_USERS} (didn't verify this), the uninstaller entry is written to WriteRegStr SHCTX "${UNINSTREG}" .... I am guessing that using SHCTX here might not always reflect the choice taken by the automatic $InstMode guessing mechanism.

Error if version is not a string in construct.yaml

On OS X, an error is thrown when executing constructor on construct.yaml if the version is entered as a number. constructor is expecting a string. Works fine if version is provided as a string.

Either documentation addition or changing the example 'maxiconda' would help. Overall, thanks for releasing this project. Very cool @ilanschnell.

post-link.bat not run for constructor-built installations

When I create a package and include it into a windows dist with constructor, it does not appear that the post-link.bat file is run (as opposed to if I use conda install). This has been acknowledged as a bug by Ilan.

bug: Excluding anaconda-client breaks constructor build

The following constructor.yaml will fail. It doesn't seem to like the hyphen in anaconda-client.
constructor 1.3.1 py27_0

name: test
version: 1.0.0

channels:
  - http://repo.continuum.io/pkgs/free
  - http://repo.continuum.io/pkgs/pro

exclude:
  - anaconda-client

specs:
  - python 2.7.12
  - anaconda 4.1.1
  - conda

Bus Error (core dumped) with anaconda.org/anaconda and anaconda.org/conda-forge

Hi constructor devs,

I am getting a bus error when using conda.anaconda.org/anaconda and conda.anaconda.org/conda-forge as my channels in a construct.yaml. I have pared down the construct.yaml as far as it can go

name: python
version: 3.5 
channels:
    - https://conda.anaconda.org/conda-forge
    - https://conda.anaconda.org/anaconda
specs:
    - python 3.5*

Running the constructor that is produced by this construct.yaml produces the following error:

$ bash ./python-3.5-Linux-x86_64.sh -b -p /tmp/foo
PREFIX=/tmp/foo
installing: python-3.5.2-4 ...
Python 3.5.2
./python-3.5-Linux-x86_64.sh: line 161: 20122 Bus error               (core dumped) $PYTHON -E -s $PREFIX/pkgs/.install.py $INST_OPT

I tried to do a diff of the installer scripts produced when I remove conda-forge as a channel, but, as expected, that diff is pretty much inscrutable and filled with binary diffs 😞

cc @msarahan

[edit] Running constructor without conda-forge as a channel does work

Feature request: implement post_install scripts on Windows

At present, only Mac/Linux installers can run post_install scripts. Adding the ability to launch post_install .bat files (or a fix for #34 would be very helpful.)

Ignore excluded packages that do not exist

When the exclude list includes a package that doesn't exist the build will fail. Instead of failing just ignore the package and continue. This shouldn't be an error case since the package is suppose to be excluded. At a minimum only output a warning but don't fail.

Example constructor.yaml that will fail.

name: test
version: 1.0.0

channels:
  - http://repo.continuum.io/pkgs/free
  - http://repo.continuum.io/pkgs/pro

exclude:
  - abc123

specs:
  - python 2.7.12
  - anaconda 4.1.1
  - conda

More test cases around constructor

Let's gather some more test cases around constructor from some of the recent issues.

#28 wrong filename
#29 existing file
#30 wrong filename

Feature request: support making ``pkg`` installers on MacOS X

Constructor is great, and works very nicely to make .sh installers on MacOS X. It would be really nice to be able to also make .pkg installers, similar to the graphical Anaconda installer. Would there be any way to do this?

Issue if version number is of the form major.minor

If I use a version number such as:

version: 1.0

then constructor crashes with the error:

Error: key 'version' points to <class 'float'>,
       expected <class 'str'>

But using:

version: 1.0.0

works fine.

channel priority for constructor

Feature request to add channel priority for constructor

bash installer wants ruamel_yaml

I built a fairly plain installer (no custom channels or packages section) with conda 4.1.3 and constructor 1.2.0 or 1.2.1. It builds fine. But when I run the installer (in either a conda-free or conda-and-ruamel_yaml-available shell), I get the below. An installer built with constructor 1.2.0 and conda 4.0.9 builds and installs fine.

>>> bash shortened/psi4conda-0.9.0-Linux-x86_64.sh 

Welcome to psi4conda 0.9.0

psi4conda will now be installed into this location:
/shortened/psi4conda

  - Press ENTER to confirm the location
  - Press CTRL-C to abort the installation
  - Or specify a different location below

[/shortened/psi4conda] >>> /shortened/psi4conda90b
PREFIX=/shortened/psi4conda90b
installing: python-2.7.11-0 ...
Python 2.7.11 :: Continuum Analytics, Inc.
installing: alabaster-0.7.8-py27_0 ...
installing: babel-2.3.3-py27_0 ...
installing: cmake-3.3.1-0 ...
installing: conda-4.1.3-py27_0 ...
ERROR:root:code for hash md5 was not found.
Traceback (most recent call last):
  File "/shortened/psi4conda90b/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/shortened/psi4conda90b/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type md5
ERROR:root:code for hash sha1 was not found.
Traceback (most recent call last):
  File "/shortened/psi4conda90b/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/shortened/psi4conda90b/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha1
ERROR:root:code for hash sha224 was not found.
Traceback (most recent call last):
  File "/shortened/psi4conda90b/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/shortened/psi4conda90b/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha224
ERROR:root:code for hash sha256 was not found.
Traceback (most recent call last):
  File "/shortened/psi4conda90b/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/shortened/psi4conda90b/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha256
ERROR:root:code for hash sha384 was not found.
Traceback (most recent call last):
  File "/shortened/psi4conda90b/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/shortened/psi4conda90b/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha384
ERROR:root:code for hash sha512 was not found.
Traceback (most recent call last):
  File "/shortened/psi4conda90b/lib/python2.7/hashlib.py", line 147, in <module>
    globals()[__func_name] = __get_hash(__func_name)
  File "/shortened/psi4conda90b/lib/python2.7/hashlib.py", line 97, in __get_builtin_constructor
    raise ValueError('unsupported hash type ' + name)
ValueError: unsupported hash type sha512
No yaml library available.
To proceed, please conda install ruamel_yaml

At installer build time, I tried explicitly adding ruamel_yaml to the specs list along with install_in_dependency_order: true, but that only yielded:

>>> constructor installer/
platform: linux-64
Fetching package metadata .....
Solving package specifications: ..........
Traceback (most recent call last):
  File "/shortened/miniconda11/bin/constructor", line 6, in <module>
    sys.exit(main())
  File "/shortened/miniconda11/lib/python2.7/site-packages/constructor/main.py", line 146, in main
    verbose=opts.verbose)
  File "/shortened/miniconda11/lib/python2.7/site-packages/constructor/main.py", line 81, in main_build
    fcp.main(info, verbose=verbose)
  File "/shortened/miniconda11/lib/python2.7/site-packages/constructor/fcp.py", line 172, in main
    resolve(info)
  File "/shortened/miniconda11/lib/python2.7/site-packages/constructor/fcp.py", line 41, in resolve
    dists.extend(d + '.tar.bz2' for d in r.graph_sort(sort_info))
AttributeError: 'Resolve' object has no attribute 'graph_sort'

Feature request: add option to not delete the package files

Add option to save the package files (.bz2) on Linux after the install.

In some standalone/offline installation situations it is desirable to retain the package files and use them to create a local package channel so conda can be configured to point to the local channel.

feature: pass options to pre and post install steps

This is a hard requirement for the AEN installers. I had created the PR in cas-installer for this feature. It was probably taken out because it isn't the cleanest implementation (ie: how to tell the difference between options sent to constructor vs. the pre/post scripts?)

My implementation ended up passing all options onto the pre/post install scripts, so if your pre/post script depending on an option and constructor adds a new option, you could shadow the option.

One option would be to define a namespace that is reserved for constructor and pass everything else through. You'd need to switch to long options though. Something like:

./my-installer.sh --constructor-batch --constructor-prefix /opt/install_path --pre-server-name 192.168.1.1 --post-do-cool-stuff=1

Feature request: option to include conda-bld channel via command line

Is it possible to add a --use-local flag that grants access to the local repository of conda-bld?

Local channels can be included by adding

channels:
   - file:///path/to/miniconda/conda-bld

to construct.yaml

This however is not portable as it hardcodes the path of my miniconda installation.

It would be cool to have a simple way to get access to the local repository. This is especially useful for not yet released packages.

Constructor should use package cache

A great optimization that would work well with continuous integration would be to use the local package cache if available

resulting Installer needs an --update-existing option

We need a robust way to update existing installs with a new installer file.

Bug: the cx_oracle package does not install correctly on Windows 64

The cx_oracle package gets included in the installer correctly but it does get fully installed so it cannot be imported.

This sample construct.yaml will reproduce the issue. Build and installer and install it on Windows.

name: test
version: 1.0.0

channels:

specs:

python
cx_oracle

Cannot access the listed channels in construct.yaml

Hi all,

We have used this library to build an installer for DeDop project (https://github.com/DeDop/dedop-conda) and it works fine when it refers to a local package. However, now is the time to upload the package to Anaconda cloud and we did (https://anaconda.org/pans/dedop). This link is placed under channels in the yaml file. Unfortunately, it always returns with the following error:

> constructor installer                                                                                                      
platform: win-64                                                                                                             
Fetching package metadata: ...Traceback (most recent call last):                                                             
  File "C:\Miniconda3\lib\site-packages\libconda\fetch.py", line 98, in fetch_repodata                                       
    resp.raise_for_status()                                                                                                  
  File "C:\Miniconda3\lib\site-packages\requests\models.py", line 884, in raise_for_status                                   
    raise HTTPError(http_error_msg, response=self)                                                                           
requests.exceptions.HTTPError: 404 Client Error: NOT FOUND for url: https://anaconda.org/pans/dedop/win-64/repodata.json.bz2 
                                                                                                                             
During handling of the above exception, another exception occurred:                                                          
                                                                                                                             
Traceback (most recent call last):                                                                                           
  File "C:\Miniconda3\Scripts\constructor-script.py", line 5, in <module>                                                    
    sys.exit(constructor.main.main())                                                                                        
  File "C:\Miniconda3\lib\site-packages\constructor\main.py", line 149, in main                                              
    verbose=opts.verbose)                                                                                                    
  File "C:\Miniconda3\lib\site-packages\constructor\main.py", line 84, in main_build                                         
    fcp.main(info, verbose=verbose)                                                                                          
  File "C:\Miniconda3\lib\site-packages\constructor\fcp.py", line 179, in main                                               
    for url in info['channels']                                                                                              
  File "C:\Miniconda3\lib\site-packages\libconda\utils.py", line 68, in __call__                                             
    value = self.func(*args, **kw)                                                                                           
  File "C:\Miniconda3\lib\site-packages\libconda\fetch.py", line 218, in fetch_index                                         
    repodatas.append((url, future.result()))                                                                                 
  File "C:\Miniconda3\lib\concurrent\futures\_base.py", line 398, in result                                                  
    return self.__get_result()                                                                                               
  File "C:\Miniconda3\lib\concurrent\futures\_base.py", line 357, in __get_result                                            
    raise self._exception                                                                                                    
  File "C:\Miniconda3\lib\concurrent\futures\thread.py", line 55, in run                                                     
    result = self.fn(*self.args, **self.kwargs)                                                                              
  File "C:\Miniconda3\lib\site-packages\libconda\fetch.py", line 62, in func                                                 
    res = f(*args, **kwargs)                                                                                                 
  File "C:\Miniconda3\lib\site-packages\libconda\fetch.py", line 141, in fetch_repodata                                      
    raise RuntimeError(msg)                                                                                                  
RuntimeError: Could not find URL: https://anaconda.org/pans/dedop/win-64/

It seems that it tries to find this file repodata.json.bz2, which is simply not there. Why is it looking for this file? I have had a look at other Anaconda packages and only a small number of them are browseable with /win-64 (or osx-64 etc.). Can you give me an enlightenment?

Here are some information about my environment:
conda-env : 2.6.0
conda-build : 2.0.10
constructor : 1.5.0
conda : 4.2.13
python : 3.5.2.final.0
platform : win-64

Thank you,

Hans

Feature request: Startmenu and Desktop shortcuts

I'm trying to use constructor to bundle and distribute my Python application.

On Windows, it seems that constructor unconditionally installs menu items found as JSON files in the Menu folder of the Miniconda3 root installation. Could you please make this an optional behaviour?

On the other hand it would be great if constructor users could provide their own JSON files which would be turned into Windows Startmenu and Desktop shortcuts, e.g. for launching their applications with a preconfigured Python environment.

The same applies of course to OS X Dock entries and Ubuntu Desktop shortcuts. However, from looking at the code it seems easy to enable this for Windows by conditionally enhancing the NSIS script.

Thanks
-- Norman

Named default channels

Currently one can provide the option conda_default_channels in the construct.yaml file to specify what channels the installer will call defaults and pre-include. However, it would be nice if the default channels were not lumped into one channel called defaults. For instance, it would be nice to create an installer where the canonical channels in defaults would be included as defaults then conda-forge would be included as a default channel, but still listed as conda-forge. This is essential for debugging of installs to not get too hairy. If this is already provided that is awesome, please help me find the relevant info to get started using this.

If this is not already provided, I would like to propose the follow syntax. To name a channel or group of channels that is preincluded a dictionary should be used. This doesn't break the normal listing usage shown below.

conda_default_channels:
   - http://repo.continuum.io/pkgs/free

Simply one adds the dictionary key to the channel they want to name. With the example below, there is a pre-included conda-forge channel that pulls from http://conda.anaconda.org/conda-forge and a defaults channel that pulls from http://repo.continuum.io/pkgs/free.

conda_default_channels:
   - conda-forge: http://conda.anaconda.org/conda-forge
   - http://repo.continuum.io/pkgs/free

If there are multiple channels that share a name in that dictionary, a list should be used. In this case, the conda-forge channel pulls from http://conda.anaconda.org/conda-forge and the fictitious http://conda.anaconda.org/conda-forge-test.

conda_default_channels:
   - conda-forge:
      - http://conda.anaconda.org/conda-forge
      - http://conda.anaconda.org/conda-forge-test
   - http://repo.continuum.io/pkgs/free

Providing this functionality of named default channels allows one to ensure the user still gets packages from the expected locations, but avoids potential issues where the definition of what is pre-included in those channels (e.g. defaults) breaks with existing user expectation. It also helps one to debug when packages coming from to channels that conflict with each other.

cc @loriab @msarahan

constructor requires bzip2

On a fresh install of CentOS 7

When running the bash installer created by constructor, the following error occurs:

tar (child): bzip2: Cannot exec: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now

Workaround is to run:
yum install bzip2

Can we make constructor more self sufficient and either provide its own bzip2 so its able to extract its own packages, or can it at least do a check if bzip2 is installed and throw a warning that you must install bzip2 before using this installer?

build a recipe(metapackage) instead of an installer?

there are many cases where metapackages are better than installers, is there a way to build a metapackage with conda constructor?

Bug: using a packages file and the conda package in specs fails

The attached example construct.yaml file will fail to build.
If the conda entry under specs is commented out then the build will work.

If the user builds the specs file from a conda env using conda list --explicit it cannot include conda since conda isn't installed in envs. Adding conda to the specs section causes the constructor build to fail. This issue prevents a user from adding to conda to an installer when using a packages files.

Also, adding conda-build to the specs section will cause that same issue.

This failed with constructor version 1.5.1

construct.yaml.txt
specs.txt

Clarify installation source

So from reading the docs, I suspect that a constructor-based installer will pre-bundle all dependencies into the installer script / exe. This means that the installer is therefore:

Deterministic
Does not require internet connectivity during installation

Am I correct in this interpretation?

Don't unset LD_LIBRARY_PATH

https://github.com/conda/constructor/blob/master/constructor/header.sh#L13

unset LD_LIBRARY_PATH

Or at least store it as a new name so pre/post install scripts can use binaries.

Add back Post-install and pre-install scripts

I notice that there is no way to run post/pre install scripts which are often necessary.

Install multiple environments

It would be great if one could create an installer that installs multiple conda envs.

Custom channel causes file not found

When using a custom conda channel constructor gets confused about the filenames.

constructor file:

name: adam
version: 1.3.0

channels:
  - http://repo.continuum.io/pkgs/free/
  - https://conda.anaconda.org/t/XXXXX/anaconda-adam

specs:
  - python
  - adam
  ...

Generates:

platform: osx-64
Fetching package metadata .....
Solving package specifications: ..........

fetching: python-2.7.11-0.tar.bz2
fetching: anaconda-adam::adam-1.3.0-py27_0.tar.bz2
... 
fetching: zlib-1.2.8-3.tar.bz2
Traceback (most recent call last):
  File "/Users/danielfrg/anaconda/bin/constructor", line 6, in <module>
    sys.exit(main())
  File "/Users/danielfrg/anaconda/lib/python3.5/site-packages/constructor/main.py", line 146, in main
    verbose=opts.verbose)
  File "/Users/danielfrg/anaconda/lib/python3.5/site-packages/constructor/main.py", line 84, in main_build
    create(info)
  File "/Users/danielfrg/anaconda/lib/python3.5/site-packages/constructor/shar.py", line 97, in create
    t.add(join(info['_download_dir'], fn), 'pkgs/' + fn)
  File "/Users/danielfrg/anaconda/lib/python3.5/tarfile.py", line 1925, in add
    tarinfo = self.gettarinfo(name, arcname)
  File "/Users/danielfrg/anaconda/lib/python3.5/tarfile.py", line 1794, in gettarinfo
    statres = os.lstat(name)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/danielfrg/.conda/constructor/osx-64/anaconda-adam::adam-1.3.0-py27_0.tar.bz2'

cd /Users/danielfrg/.conda/constructor/osx-64/
➜  osx-64 ls
total 48808
-rw-r--r--  1 danielfrg  staff    80K 20 Jun 12:09 _license-1.1-py27_1.tar.bz2
-rw-r--r--  1 danielfrg  staff    83K 20 Jun 12:09 adam-1.3.0-py27_0.tar.bz2
...

Constructor downlaods the file as adam-1.3.0-py27_0.tar.bz2 but then tries to use anaconda-adam::adam-1.3.0-py27_0.tar.bz2.

How to specify pip dependencies in construct.yml?

Using the following syntax (like in environment.yml):

  - pip:
      - somepackage

results in:

platform: linux-64
Traceback (most recent call last):
  File "/home/cyrille/miniconda3/bin/constructor", line 6, in <module>
    sys.exit(main())
  File "/home/cyrille/miniconda3/lib/python3.5/site-packages/constructor/main.py", line 145, in main
    verbose=opts.verbose)
  File "/home/cyrille/miniconda3/lib/python3.5/site-packages/constructor/main.py", line 78, in main_build
    info[key] = [line.strip() for line in info[key]]
  File "/home/cyrille/miniconda3/lib/python3.5/site-packages/constructor/main.py", line 78, in <listcomp>
    info[key] = [line.strip() for line in info[key]]
AttributeError: 'dict' object has no attribute 'strip'

No conda recipe or packages?

The instructions suggest installation with conda, but there appear to be no packages in the default channels or on anaconda.org, and there is no conda recipe in this repo. Are packages/recipes kept somewhere else?

Conda wrapper script gets unusable if path to python interpreter gets too long

From @rammig on April 12, 2017 11:27

We first encountered the problem using Jenkins CI for automated testing and deployment of our projects. We were able to generalize this issue as it is a problem with the conda wrapper script.

Bug Description:
If the full path to the python interpreter installed in a conda environment exceeds 126 characters in Linux, the conda wrapper script gets unusable.

This effect is due to the she-bang line in the conda wrapper script exceeding the setting of BINPRM_BUF_SIZE, a Linux kernel setting found in include/linux/binfmts.h.

Expectation: While this problem is not a bug in Anaconda itself, the conda wrapper script still should detect this issue and work around it. Possible workarounds can be found here:
http://stackoverflow.com/questions/10813538/shebang-line-limit-in-bash-and-linux-kernel

Test system

tester@bugreport:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"

steps to reproduce the error

the full path to the python binary in the conda environment is 127 characters:

tester@bugreport:~$ mkdir -p this/is/a/subdirectory/with/127/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
tester@bugreport:~$ cd this/is/a/subdirectory/with/127/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
tester@bugreport:~/this/is/a/subdirectory/with/127/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX$ echo $PWD
/home/tester/this/is/a/subdirectory/with/127/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
tester@bugreport:~/this/is/a/subdirectory/with/127/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX$ echo -en "$PWD" | wc -c
106
tester@bugreport:~/this/is/a/subdirectory/with/127/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX$ /bin/bash /opt/Miniconda3-4.1.11-Linux-x86_64.sh -b -p $PWD/condaenv
PREFIX=/home/tester/this/is/a/subdirectory/with/127/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/condaenv
installing: python-3.5.2-0 ...
installing: conda-env-2.5.2-py35_0 ...
installing: openssl-1.0.2h-1 ...
installing: pycosat-0.6.1-py35_1 ...
installing: pyyaml-3.11-py35_4 ...
installing: readline-6.2-2 ...
installing: requests-2.10.0-py35_0 ...
installing: ruamel_yaml-0.11.14-py35_0 ...
installing: sqlite-3.13.0-0 ...
installing: tk-8.5.18-0 ...
installing: xz-5.2.2-0 ...
installing: yaml-0.1.6-0 ...
installing: zlib-1.2.8-3 ...
installing: conda-4.1.11-py35_0 ...
installing: pycrypto-2.6.1-py35_4 ...
installing: pip-8.1.2-py35_0 ...
installing: wheel-0.29.0-py35_0 ...
installing: setuptools-23.0.0-py35_0 ...
Python 3.5.2 :: Continuum Analytics, Inc.
creating default environment...
installation finished.
tester@bugreport:~/this/is/a/subdirectory/with/127/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX$ head -n1 condaenv/bin/conda
#!/home/tester/this/is/a/subdirectory/with/127/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/condaenv/bin/python3
tester@bugreport:~/this/is/a/subdirectory/with/127/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX$ echo -en "/home/tester/this/is/a/subdirectory/with/127/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/condaenv/bin/python3" | wc -c
127
tester@bugreport:~/this/is/a/subdirectory/with/127/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
tester@bugreport:~/this/is/a/subdirectory/with/127/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX$ export PATH=${PWD}/condaenv/bin:${PATH}
tester@bugreport:~/this/is/a/subdirectory/with/127/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX$ conda --version
bash: /home/tester/this/is/a/subdirectory/with/127/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/condaenv/bin/conda: /home/tester/this/is/a/subdirectory/with/127/characters/full/path/XXXXXXXXXXX: bad interpreter: No such file or directory

with less characters, however, it is working fine:

tester@bugreport:~$ mkdir -p this/is/a/subdirectory/with/126/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
tester@bugreport:~$ cd this/is/a/subdirectory/with/126/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
ter@bugreport:~/this/is/a/subdirectory/with/126/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX$ echo $PWD
/home/tester/this/is/a/subdirectory/with/126/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
tester@bugreport:~/this/is/a/subdirectory/with/126/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX$ echo -en "$PWD" | wc -c
105
tester@bugreport:~/this/is/a/subdirectory/with/126/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX$ /bin/bash /opt/Miniconda3-4.1.11-Linux-x86_64.sh -b -p $PWD/condaenv
PREFIX=/home/tester/this/is/a/subdirectory/with/126/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/condaenv
installing: python-3.5.2-0 ...
installing: conda-env-2.5.2-py35_0 ...
installing: openssl-1.0.2h-1 ...
installing: pycosat-0.6.1-py35_1 ...
installing: pyyaml-3.11-py35_4 ...
installing: readline-6.2-2 ...
installing: requests-2.10.0-py35_0 ...
installing: ruamel_yaml-0.11.14-py35_0 ...
installing: sqlite-3.13.0-0 ...
installing: tk-8.5.18-0 ...
installing: xz-5.2.2-0 ...
installing: yaml-0.1.6-0 ...
installing: zlib-1.2.8-3 ...
installing: conda-4.1.11-py35_0 ...
installing: pycrypto-2.6.1-py35_4 ...
installing: pip-8.1.2-py35_0 ...
installing: wheel-0.29.0-py35_0 ...
installing: setuptools-23.0.0-py35_0 ...
Python 3.5.2 :: Continuum Analytics, Inc.
creating default environment...
installation finished.
tester@bugreport:~/this/is/a/subdirectory/with/126/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX$ head -n1 condaenv/bin/conda
#!/home/tester/this/is/a/subdirectory/with/126/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/condaenv/bin/python3
tester@bugreport:~/this/is/a/subdirectory/with/126/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
tester@bugreport:~/this/is/a/subdirectory/with/126/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX$ export PATH=${PWD}/condaenv/bin:${PATH}
tester@bugreport:~/this/is/a/subdirectory/with/126/characters/full/path/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX$ conda --version
conda 4.1.11

Copied from original issue: conda/conda#5041

Platform key?

https://github.com/conda/constructor/blob/master/examples/maxiconda/README.md

However, it is also possible to build installers for other platforms, see the platform key.

I don't see a platform key. Am i missing something?

Include urls.txt in package set

It would be good for conda to know the channels from which the packages were taken when performing the initial installation. The easiest way to accomplish this is to include a urls.txt file into the package.