If we want to have such a file, it would be better to obtain it automatically from Git, or even point directly to https://github.com/ESSS/conda-devenv/graphs/contributors.

Hi Devs,

I would like to suggest a new feature for your great tool!

Conda latest version allows using URL for the environment yaml file:

conda env create -f https://domain.com/my-environment.yml

See conda/conda#9835

Having this ability in conda-devenv would be good, as well.
Regards

Though elegantly, but suppose some reasonably good library are not following the convention. It leave us no work around for now except for replacing the entire function as a hack.

Say httpspypi.org/project/ConfigArgParse

File"C:\Users\dummy\anaconda\Anaconda_py3.6.4.4.0.\\ py36\lib\site
 packages conda deveny devenv py". line 221, in merge_dependencies_version_specifications
 expected format.format(dep))
 RuntimeError: The package version specification "ConfigAng Farse==0.15.2" do not follow the
 expected format. 

sorry for the spelling, caused by ocr.

- channel::package is being exported as - channel ::package and this extra space makes conda think that the channel name is channel with the extra space in the end

in the docs:
"
Environment variables defined as a single string (like DB_LOCATION above) will overwrite an existing variable with the same name.
"

in code:

   "Can't merge the key: '{key}' because it will override the previous value.",
                            "Only lists and dicts can be merged. The type obtained was: {type}",

gabrielcnr/pytest-datadir#10

At the moment everything is sorted alphanumerically in the rendered environment. I think it is not correct:

• channel priorities should be respected
• pip dependencies order should maybe be somewhat respected (since pip is, at the moment and until it gets the promised resolver, sensitive to the order in which we ask for dependencies)

I guess the way to go for includes, at least for channels, would be to have a simple consistency check and fail if two channels are ordered differently in different files.

Thanks for conda-devenv!

The following pip dependency section gets mangled:

# environment.devenv.yml
name: test
dependencies:
  - pip
  - pip:
    - hg+ssh://[email protected]/mforbes/[email protected]
    - hg+ssh://[email protected]/mforbes/[email protected]

conda devenv --print
# generated by conda-devenv, do not modify and do not commit to VCS
dependencies:
- pip
- pip:
  - hg +ssh://[email protected]/mforbes/[email protected],+ssh://[email protected]/mforbes/[email protected]
name: test

Note that a space gets inserted in "hg +ssh" and the two lines are concatenated. Both effects break the subsequent conda install with a message:

Invalid requirement: 'hg +ssh://[email protected]/mforbes/[email protected],+ssh://[email protected]/mforbes/[email protected]'
It looks like a path. File 'hg +ssh://[email protected]/mforbes/[email protected],+ssh://[email protected]/mforbes/[email protected]' does not exist.

The cool kids are adding type hints.

• Add type annotations;
• Check type annotation in actions (#136);

Might be time to think of conda/mumba as 'low level'. I think it's best to pass on env vars to conda env config vars set.

Hi folks,

It would be really nice to support mamba out of box with conda-devenv
Because, mamba has a solver which is faster than the solver used by conda, mamba uses libsolv which is the same used by some package managers on some Linux distributions such as OpenSuse, Red Hat, and Fedora.

I was wondering if we can use mamba directly when conda-devenv detects if mamba is installed in the base environment and we may add a flag to be able to disable this behaviour as well.

What do you think?

I am a bit busy these days but I may work on this if you agree with that. But if someone wants to tackle it, please go ahead! :)

Conda-devenv has become the very basis of our projects as Grade does. As our intentional common build mechanism, we currently are encountering the case that not each of our agent will be well prepared with dev-env, some might even be missing the Anaconda. As the policy goes the only sure-to-be-there thing is a JRE with not controllable version.
I know I can be achieving that by pulling our Anaconda installer from our internal repo and inject site wise configurations and then install dev-env accordingly. And that is only the first step, then everything just gets started handled by dev-env.

Is it possible to have a minimal packed distributable archive for dev-env? It is expected to do nothing other than providing working dev-env executables.

Say Gradle itself is definitely huge and with complicates setup procedures. So they created grade wrapper which is nice and neat and tiny and enough for setting Gradle up. Perhaps we could achieve the same? Was there any thought once brain-stormed on this? Grade wrapper works with a tiny Jar and one bat script and one shell script and one configuration file. That's all. (But of course it requires jre to be existing, we could also be expecting that there is an living Anaconda/Python outside) . What I want is simply that we don't have to install dev-env properly before using it. It has become to like a cornerstone now for us that we don't want too many preconditions before making use of it.

Problem

To ensure all projects use the same "blessed" versions, we use a strategy where we include other devenv.yml files which exist solely to specify package versions, for example:

name: app
includes:
   - {{ eden }}/environment.devenv.yml
   - {{ eden }}/pytest.devenv.yml
dependencies:
   - cogapp ==0.3

With the other files being:

# eden/environment.devenv.yml
name: eden
dependencies:
  - artifacts_cache ==3.0.0
  - attrs ==21.2.0
  - black ==21.12b0
environment:
  - PYTHONPATH: {{ root }}/source/python

# eden/pytest.devenv.yml
dependencies:
  - pytest ==6.2.5
  - pytest-mock ==3.0.0
  - pytest-lazy-fixture ==0.6.0

The purpose here is to make sure we are using the same versions, without putting all dependencies in eden/environment.devenv.yml, as that would bloat all environments that include it.

This works, but it has some problems:

• Versions are scattered in many devenv.yml files, which makes it difficult to update versions as often there are conflicts, and we need to hunt down the offending pins.
• Often, for brevity and avoiding having an explosion of devenv.yml files, we put a lot of packages in the same file because they are often related, but that's not always true or convenient. For example, when we include pytest.devenv.yml above, we will get pytest-lazy-fixture whether we use it or not.

Proposal

We add a new section in the devenv.yml files, named pins. This contains a number of packages and pins (mandatory), which will be used when processing the dependencies of the chain of devenv.yml files. Any dependency in devenv files which has a pin in the pins section, will be pinned accordingly in the final environment.yml file generated.

For example, we can have pins directly into eden/environment.devenv.yml, which is included by all other projects:

name: eden
pins:
  - artifacts_cache ==3.0.0
  - attrs ==21.2.0
  - black ==21.12b0
  - cogapp ==0.3
  - pytest ==6.2.5
  - pytest-mock ==3.0.0
  - pytest-lazy-fixture ==0.6.0
dependencies:
  - artifacts_cache
  - attrs
  - black

pins doesn't mean that dependency will be included, but if any of the dependencies include a package in pins which is not itself pinned, then it will be pinned accordingly.

name: app
includes:
   - {{ eden }}/environment.devenv.yml   
dependencies:
   - cogapp
   - pytest

Having a pin file, all dependencies sections in devenv.yml files can now leave things unpinned, with the pin being managed by the pins section.

Notes

Dependencies with pins

If a dependencies section specifies a pinned package, then the generated file will contain both pins:

name: eden
pins:
  - artifacts_cache ==3.0.0

name: app
includes:
   - {{ eden }}/environment.devenv.yml   
dependencies:
   - artifacts_cache >=2.4.0

In this example the generated environment.yml will contain the entry artifacts_cache: >=2.4.0, ==3.0.0, which conda will be able to solve. If they conflict for, conda will fail as expected.

Multiple pins

It is possible to have multiple pins sections in a chain of devenv.yml files, we should error out if different pin sections try to pin the same library (this from the POV of conda-devenv, we probably should have a single pins section in one file).

Usage in eden

We probably will have a separate pins.devenv.yml file, to avoid polluting eden/environment.devenv:

# eden/pins.devenv.yml
pins:
  - artifacts_cache ==3.0.0
  - attrs ==21.2.0
  - black ==21.12b0
  - cogapp ==0.3
  - pytest ==6.2.5
  - pytest-mock ==3.0.0
  - pytest-lazy-fixture ==0.6.0

name: eden
includes:
  - {{ root }}/pins.devenv.yml
dependencies:
  - artifacts_cache
  - attrs
  - black

What about "lock files"

Locking is a different use case/concern, and does not conflict with this proposal.

(note: much of the ideas and solutions here were result from discussions with @tadeu)

Hi,

How can I update just the environment variables contained in my devenv.yml file?
Manual or automatic are suitable for me.
I'm mostly editing the environment section and not the packages themselves and would rather just edit a file manually if needed instead of waiting for the process to finish,

environment.devenv.yml:

includes:

Results in:

  File "W:\Miniconda\lib\site-packages\conda_devenv\devenv.py", line 41, in handle_includes
    for included_filename in yaml_dict.get("includes", []):
TypeError: 'NoneType' object is not iterable

That line in handle_includes should be changed to:

for included_filename in yaml_dict.get("includes") or []:

I often find myself specifying flags in the devenv file via environment variables, e.g. I'd have a section like:

# development
{% if "PKG_DEV" in os.environ %}
- pytest
- bump2version
{% endif %}

for development dependencies that are separate from those which a normal user of the environment needs.

That means that I have to set this environment variable for the dev setup, but I'd much prefer something like this:

$ conda devenv -a dev

and a function like:

{% if "dev" in additional_args() %}
- ...
{% endif %}

I'd be open to work on a PR for this.

I was just updating an env in a linux machine with /full/path/to/conda devenv -f environment.devenv.yml and got this traceback:

Traceback (most recent call last):
  File "/home/company/Work/miniconda/bin/conda-devenv", line 11, in <module>
    load_entry_point('conda-devenv==0.9.0', 'console_scripts', 'conda-devenv')()
  File "/home/company/Work/miniconda/lib/python2.7/site-packages/conda_devenv/devenv.py", line 356, in main
    retcode = __call_conda_env_update(args, output_filename)
  File "/home/company/Work/miniconda/lib/python2.7/site-packages/conda_devenv/devenv.py", line 268, in __call_conda_env_update
    return subprocess.call(command)
  File "/home/company/Work/miniconda/lib/python2.7/subprocess.py", line 168, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/home/company/Work/miniconda/lib/python2.7/subprocess.py", line 390, in _init_
    errread, errwrite)
  File "/home/company/Work/miniconda/lib/python2.7/subprocess.py", line 1024, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

After some time investigating I found out it was because I didn't have conda on PATH. I think it should either not require it to be on path (using absolute path instead) or give a better error message.

This causes problems like the one described in conda/conda#4957.
We should "and" the multiple version specifications with , (as described in Conda package specification - Package match specifications).

Hi --
I currently use anaconda-project to maintain repeatable cross platform environments, but it has complexity that isn't really needed for my use case and I would like to consider alternatives such as conda-devenv -- in particular conditioning packages based on platform would be extremely useful.

The one really useful thing about anaconda-project is that it knows how to solve environments for multiple platforms at once, independent of the host platform.

If would be useful to be able to run conda-devenv in a mode in which is renders fully solved environment files for a given platform (which then would be checked into version control or just used to verify that the environments are actually solvable)

best, nehal

Replace virtualenv instructions by using conda instead.

Example environment.devenv.yml:

dependencies:
  - pip:
    - "-e ./pado"
    - "-e ./pado_internal"
    - "-e ./pado_tggates"

Renders this environment.yml:

dependencies:
  - pip:
    - -e ./pado,./pado_internal,./pado_tggates

Which is incorrect and when run fails with an error like:
ERROR: ./pado,./pado_internal,./pado_tggates is not a valid editable requirement.

Described in: conda/conda#7279

Possibly workaround is to remove the history file, as suggested in conda/conda#7279 (comment).

includes:
- {{root}}/../somedir/environment.devenv.yml  # [False]

This still gets included. It makes sense to enable this syntax here as well since it's a sort of dependency just like the dependency section.

I rely heavily on not is_included to process my devenvs. I'm assuming that this is the way it's expressed in use cases (to be useful). But thinking about the negation of inclusion is too much to think about. I find myself always having to think about it. I think it's just simpler to think about the intent of its use.

The activate/deactivate scripts are currently being tested comparing its contents.
This is ok, but we should have some simple integration tests for each supported shell to at least make sure the syntax is right.

#46 introduced a bug: conda-devenv now calls conda_env.main directly, which probably calls sys.exit(), which skips activate/deactivate scripts generation.

The method get_env_directory on line 457 should be calling the conda directly instead of by calling subprocess.

Maybe add a method _call_conda_info similar to the already created _call_conda

It would be nice if in addition to jinja support to execute python commands, we could also have these preprocessing selectors operating like in conda-build. The idea is to allow us to write:

name: mylib

dependencies:
  - cmake
  - eigen
  - pip:
    - sphinx
  - gxx_linux-64=7.3.0      # [linux]
  - ccache                  # [unix]
  - clcache                 # [win]

instead of:

name: mylib

dependencies:
  - cmake
  - eigen
  - pip:
    - sphinx
  {% if linux %}
  - gxx_linux-64=7.3.0
  {% endif %}  
  {% if unix %}
  - ccache
  {% endif %}  
  {% if win %}
  - clcache
  {% endif %}

Would this feature be acceptable in conda-devenv?

Defining multiple dependencies in the pip section with flags such as --editable/-e:

dependencies:
- pip:
  - --editable path/to/first/package
  - --editable path/to/second/package
name: pip_editable_test

generates this:

dependencies:
- pip:
  - --editable path/to/first/package,path/to/second/package
name: pip_editable_test

which causes pip to fail because of the comma. And even replacing the comma with a space produces an incorrect result because only the first dependency is installed as editable.

Originally the recipe used to create this repository used bumpversion, but that project doesn't seem to be maintained any longer.

Follow up to #71

Guys, once this tool is mature I suggest adding to Conda wiki. It may help getting noticed and helping others ;)

Currently if the includes sections contain circular references (non-dag graphs) conda-devenv works fine, however, this is a sign of bad design of the environment.devenv.yml files.

An error should be generated in this case.

When I run mamba devenv, it doesn't use mamba, but conda.

This is the output that I see:

 mamba devenv
> Executing: /miniconda3/bin/conda-devenv env update --file /home/eric/environment.yml --prune

Simply executing this manually afterward works:

mamba env update --file /home/eric/environment.yml --prune

sys.argv is ['/home/eric/miniconda3/bin/conda-devenv']

So I'm not sure where to go next.

I spent quite a bit of time trying to figure out why the includes: section of my environment file was not being processed until stumbling on the apparent requirement that my environment file needs the .devenv.yml extension, even when explicitly specifying the --file option.

My preference would be to allow users to name the file whatever they like when explicitly checking, but if this is not possible, then perhaps at least emit a useful warning message if the file includes devenv features which will be ignored. This requirement was also not clear from the documentation or the emitted help message.

Thanks for a very useful tool.

Add triggers section to execute arbitrary code.
Use case: To trigger setup codes.

I saw some of the issues in #84 . I really like the limited scope of conda-devenv. At the same time, I want to be able to 'build'/setup a project made of many devenvs with dependencies between them without having to go full on conda build.

The includes already track dependencies, so it would be nice to make use of them to automate setups.

This is just a suggestion to define (boolean) python variables linux, macOS (or osx), and win32 such that the following:

{% if sys.platform.startswith('linux') or sys.platform.startswith('osx') %}

could be replaced with:

{% if linux or osx %}

Are there any limitations in the tools used by conda-devenv that prevent this simpler usage?

devenv can simply be for env vars while conda takes care of the rest. default passed arg would be --prune.

(maybe this will help with mamba integration if the conda folks do it...i expect this is the best path)

According to the discussion held in #74 and the PR #75, it remains to implement the variables armv6l, armv7l, ppc64le. These are supported by conda-build preprocessing selectors.

Problem

Programming in Jinja is not much fun, so if we need to do more complicated functions, the code can be quite complicated (or impossible).

Take this example (which doesn't work and just frustrates the user):

{% set versionformat = lambda ver: '.'.join(ver) %}
version = {{ versionformat(31) }}

Proposal

It would be great if I could have files (*.conda-devenv.py) with:

def versionformat():
    """Convert string `ver` to a semantic version formatted string if not already"""
    # Case 1: if ver is '372' (string without dots), return '3.7.2'
    # Case 2: if ver is '3.7.2' (string with dots), return ver (in this case, '3.7.2')
    # Case 3: if ver is None, return None
    return '.'.join(ver) if ver and '.' not in ver else ver

Then in my environment.devenv.yml:

version = {{ versionformat(31) }}

Implementation details

• We should export all symbols found in __all__.
• I guess we should be using importlib.import_module to load the file(s).

Questions

1. Do we want 1 single file (conda-devenv.py) or all files in the current directory (*.conda-devenv.py) ?
2. Should all files be loaded automatically (implicit) or should them be loaded explicitly through an exposed function (load_functions("custom-functions-conda-devenv.py")) ?
3. What about dependencies between these files? Do we care about that? Can one file import another?

Why???

This feature request is based on the discussion held in #83. If conda-devenv had this capability, then the user would never feel the need to open that PR, he would just write his custom functions.

Currently the user sees a traceback:

λ conda devenv
Traceback (most recent call last):
  File "E:\Miniconda\Scripts\conda-devenv-script.py", line 9, in <module>
    load_entry_point('conda-devenv==0.9.0', 'console_scripts', 'conda-devenv')()
  File "E:\Miniconda\lib\site-packages\conda_devenv\devenv.py", line 336, in main
    conda_yaml_dict, environment = load_yaml_dict(filename)
  File "E:\Miniconda\lib\site-packages\conda_devenv\devenv.py", line 101, in load_yaml_dict
    with open(filename, "r") as f:
IOError: [Errno 2] No such file or directory: 'e:\\ws\\SimBR\\Projects\\simbr\\environment.devenv.yml'

Having some like above on devenv.yml:

dependencies:
- wheel=0.29.0=py36_0
- pip:
  - fdb==1.6.1

throws the following error when calling conda devenv:

  File "C:\Miniconda3-x86\lib\re.py", line 163, in match
    return _compile(pattern, flags).match(string)
TypeError: expected string or bytes-like object

if you put this in say _.yml

channels:
- conda-forge
name: _
dependencies: ['curl']
environment:
  VAR: val

conda dev env --file _.yml doesn't put the activation script; the env var is not set. Furthermore, conda complains about the environment section.

I have a use case where I want to (programmatically) refer to envs that had included the subject environment (inverse of deps/'is_included' ).

Example: create one 'jupyter notebook' environment for all envs that include the subject env.

A problem was discovered when passing a include without the jinja rendered root ({{root}}).
the problem would appear in cases like this:

• projects
• • project A
• • • A.devenv.yml
• • project B
• • • B.devenv.yml
• • C.devenv.yml

if C include B, and B include A, without jinja root, conda-devenv would wrongly use C path to resolve the path from B to A.

Traceback:

09:23:35 -> call conda devenv --quiet -n _database10-win64-py36 -f "K:\etk\database10\\environment.devenv.yml" 
09:23:36 Traceback (most recent call last):
09:23:36   File "W:\Miniconda\Scripts\conda-devenv-script.py", line 5, in <module>
09:23:36     exit(main())
09:23:36   File "W:\Miniconda\lib\site-packages\conda_devenv\devenv.py", line 528, in main
09:23:36     truncate_history_file(env_directory)
09:23:36   File "W:\Miniconda\lib\site-packages\conda_devenv\devenv.py", line 354, in truncate_history_file
09:23:36     copyfile(history_filename, history_backup_filename)
09:23:36   File "W:\Miniconda\lib\shutil.py", line 120, in copyfile
09:23:36     with open(src, 'rb') as fsrc:
09:23:36 FileNotFoundError: [Errno 2] No such file or directory: 'W:\\Miniconda\\envs\\_database10-win64-py36\\conda-meta\\history'
09:23:36 Call failed with return code 1