ansible / ansible-dev-environment Goto Github PK

View Code? Open in Web Editor NEW

22.0 22.0 10.0 371 KB

Build and maintain a development environment including ansible collections and their python dependencies

Home Page: https://ansible.readthedocs.io/projects/dev-environment/

License: GNU General Public License v3.0

Python 100.00%

ansible-dev-tools

ansible-dev-environment's Introduction

Ansible

Ansible is a radically simple IT automation system. It handles configuration management, application deployment, cloud provisioning, ad-hoc task execution, network automation, and multi-node orchestration. Ansible makes complex changes like zero-downtime rolling updates with load balancers easy. More information on the Ansible website.

Design Principles

Have an extremely simple setup process with a minimal learning curve.
Manage machines quickly and in parallel.
Avoid custom-agents and additional open ports, be agentless by leveraging the existing SSH daemon.
Describe infrastructure in a language that is both machine and human friendly.
Focus on security and easy auditability/review/rewriting of content.
Manage new remote machines instantly, without bootstrapping any software.
Allow module development in any dynamic language, not just Python.
Be usable as non-root.
Be the easiest IT automation system to use, ever.

Use Ansible

You can install a released version of Ansible with pip or a package manager. See our installation guide for details on installing Ansible on a variety of platforms.

Power users and developers can run the devel branch, which has the latest features and fixes, directly. Although it is reasonably stable, you are more likely to encounter breaking changes when running the devel branch. We recommend getting involved in the Ansible community if you want to run the devel branch.

Get Involved

Read Community Information for all kinds of ways to contribute to and interact with the project, including mailing list information and how to submit bug reports and code to Ansible.
Join a Working Group, an organized community devoted to a specific technology domain or platform.
Submit a proposed code update through a pull request to the devel branch.
Talk to us before making larger changes to avoid duplicate efforts. This not only helps everyone know what is going on, but it also helps save time and effort if we decide some changes are needed.
For a list of email lists, IRC channels and Working Groups, see the Communication page

Coding Guidelines

We document our Coding Guidelines in the Developer Guide. We particularly suggest you review:

Branch Info

The devel branch corresponds to the release actively under development.
The stable-2.X branches correspond to stable releases.
Create a branch based on devel and set up a dev environment if you want to open a PR.
See the Ansible release and maintenance page for information about active branches.

Roadmap

Based on team and community feedback, an initial roadmap will be published for a major or minor version (ex: 2.7, 2.8). The Ansible Roadmap page details what is planned and how to influence the roadmap.

Authors

Ansible was created by Michael DeHaan and has contributions from over 5000 users (and growing). Thanks everyone!

Ansible is sponsored by Red Hat, Inc.

License

GNU General Public License v3.0 or later

See COPYING to see the full text.

ansible-dev-environment's People

Contributors

Stargazers

Watchers

Forkers

cristianonicolai raoulaime kb-perbyte saito-hideki viehoang68 techn0mad roverflow audgirka priyamsahoo cidrblock

ansible-dev-environment's Issues

RFE: Add ability to use `--system-site-packages`

This will allow for use of RPM based installs

Add ability to pre-install collections from source prior to collection build

Read source collection URIs from a requirements-source.yml file in .config directory

Enable this functionality with a --cpi --collection-pre-install

Assume ^^^ that filename for now is enabled

Allow user to specify core version or URL

Disable colored logging with `--no-ansi`

Support install `ansible.utils==a.b.c`

Support installing a specific version of a collection

Generate some tree view of collections and their deps

Clean up verbose

-v, --verbose Give more output. Option is additive, and can be used up to 3 times.

Build a dataclass for manifest information

This will elimin all the dict type checking in the caller

Support install from git repo

pip4a install git+https://github.com/organization/repo_name.git,devel

Refactor tests

So they are idempotent and smaller

Maybe an initial setup of a shared venv.

ade tree fails when collection contains a git repo as dependency

Issue

When I try to install my current collection that has a dependency to a git repository using:

ade install -e . --venv .venv

galaxy.yml

dependencies: {'[email protected]:organization/repo_name.git': 'devel'}

I get this error:

Traceback (most recent call last):
  File ".venv/bin/ade", line 8, in <module>
    sys.exit(main())/
  File ".venv/lib64/python3.10/site-packages/ansible_dev_environment/cli.py", line 177, in main
    cli.run()
  File ".venv/lib64/python3.10/site-packages/ansible_dev_environment/cli.py", line 158, in run
    subcommand.run()
  File ".venv/lib64/python3.10/site-packages/ansible_dev_environment/subcommands/treemaker.py", line 70, in run
    target[dep] = tree_dict[dep]
KeyError: '[email protected]:organization/repo_name.git'

Analyzation
It looks ADE cannot use the git repo path as key. Is it possible to collect the real collection name out of the installed collection? the repo name is mostly not the same as the collection name.

Sanitise the path input

Handle cases where the user might use ENV variables such as $HOME or short-hands such as ~ in the path.
Currently, the CLI tool is unable to recognize them.

Disable ansi output if no tty

Allow uninstallation of multiple collections

ADE complains about the default .ansible/collections directory

Maybe I don't fully understand what I'm doing but it seems weird that I can scaffold a collection with ansible-creator outside of the ~/.ansible/collections path and install it with ansible-galaxy but not with ade.

I encounter Critical: The development environment is not isolated, please resolve the above errors. but I don't understand why I get this because the collection is located outside the default collections path and I'm using a virtual environment.

Is it really expected that users have no other collections installed locally in order to use ADE? I feel like this should be addressed in the documentation because it seems like users might be likely to have collections already installed that they don't want to remove. Perhaps this is a lack of understanding on my part though.

Here are my steps:

$ adt --version
ansible-builder                          3.0.1
ansible-core                             2.17.0
ansible-creator                          24.6.0
ansible-dev-environment                  24.4.3
ansible-dev-tools                        24.5.0
ansible-lint                             24.6.0
ansible-navigator                        24.3.2
ansible-sign                             0.1.1
molecule                                 24.6.0
pytest-ansible                           24.1.3
tox-ansible                              24.6.0

$ which adt
~/workspace/dev-tools-venv/bin/adt

$ which ade
~/workspace/dev-tools-venv/bin/ade

$ which ansible-galaxy
~/workspace/dev-tools-venv/bin/ansible-galaxy

$ ansible-creator init collection=mynamespace.donstuff --init-path ~/workspace/dev-tools-workshop --project collection
    Note: collection collection=mynamespace.donstuff created at
          /home/dnaro/workspace/dev-tools-workshop

$ ls -la /home/dnaro/workspace/dev-tools-workshop/
total 92
drwxr-xr-x. 1 dnaro dnaro   576 Jun  5 19:53 .
drwxr-xr-x. 1 dnaro dnaro  2196 Jun  5 19:34 ..
-rw-r--r--. 1 dnaro dnaro   336 Jun  5 19:53 CHANGELOG.rst
drwxr-xr-x. 1 dnaro dnaro    22 Jun  5 19:53 changelogs
-rw-r--r--. 1 dnaro dnaro   162 Jun  5 19:53 CODE_OF_CONDUCT.md
-rw-r--r--. 1 dnaro dnaro   117 Jun  5 19:53 CONTRIBUTING
drwxr-xr-x. 1 dnaro dnaro    58 Jun  5 19:53 .devcontainer
-rw-r--r--. 1 dnaro dnaro   377 Jun  5 19:53 devfile.yaml
drwxr-xr-x. 1 dnaro dnaro    24 Jun  5 19:53 docs
drwxr-xr-x. 1 dnaro dnaro    22 Jun  5 19:53 extensions
-rw-r--r--. 1 dnaro dnaro  1509 Jun  5 19:53 galaxy.yml
drwxr-xr-x. 1 dnaro dnaro    18 Jun  5 19:53 .github
-rw-r--r--. 1 dnaro dnaro  3153 Jun  5 19:53 .gitignore
-rw-r--r--. 1 dnaro dnaro   155 Jun  5 19:53 .isort.cfg
-rw-r--r--. 1 dnaro dnaro 35149 Jun  5 19:53 LICENSE
-rw-r--r--. 1 dnaro dnaro     0 Jun  5 19:53 MAINTAINERS
drwxr-xr-x. 1 dnaro dnaro    22 Jun  5 19:53 meta
drwxr-xr-x. 1 dnaro dnaro   144 Jun  5 19:53 plugins
-rw-r--r--. 1 dnaro dnaro  1243 Jun  5 19:53 .pre-commit-config.yaml
-rw-r--r--. 1 dnaro dnaro   232 Jun  5 19:53 .prettierignore
-rw-r--r--. 1 dnaro dnaro   234 Jun  5 19:53 pyproject.toml
-rw-r--r--. 1 dnaro dnaro  3010 Jun  5 19:53 README.md
-rw-r--r--. 1 dnaro dnaro    67 Jun  5 19:53 requirements.txt
drwxr-xr-x. 1 dnaro dnaro     6 Jun  5 19:53 roles
-rw-r--r--. 1 dnaro dnaro   112 Jun  5 19:53 test-requirements.txt
drwxr-xr-x. 1 dnaro dnaro    50 Jun  5 19:53 tests
-rw-r--r--. 1 dnaro dnaro    82 Jun  5 19:53 tox-ansible.ini
drwxr-xr-x. 1 dnaro dnaro    30 Jun  5 19:53 .vscode

$ ansible-galaxy collection install ~/workspace/dev-tools-workshop/
Starting galaxy collection install process
Process install dependency map
Starting collection install process
Installing 'collection=mynamespace.donstuff:1.0.0' to '/home/dnaro/.ansible/collections/ansible_collections/collection=mynamespace/donstuff'
Created collection for collection=mynamespace.donstuff:1.0.0 at /home/dnaro/.ansible/collections/ansible_collections/collection=mynamespace/donstuff
collection=mynamespace.donstuff:1.0.0 was installed successfully

$ ansible-galaxy collection list

# /home/dnaro/.ansible/collections/ansible_collections
Collection                      Version
------------------------------- -------
ansible.eda                     1.4.7  
ansible.posix                   1.5.4  
collection=mynamespace.donstuff 1.0.0  
community.clickhouse            0.4.0  
community.general               7.0.1  

$ cd ~/workspace/dev-tools-workshop/

$ ade install -e .
   Error: Collections found in
          /home/dnaro/.ansible/collections/ansible_collections
    Hint: Run `rm -rf ~/.ansible/collections` to remove them.
Critical: The development environment is not isolated, please resolve the above errors.

$ mv ~/.ansible/collections/ ~/.ansible/backup-collections

$ ade install -e .
    Note: Installed collections include: mynamespace.donstuff
    Note: All python requirements are installed.
    Note: All required system packages are installed.

Drop dependency on gitpython

Add `--user` and `--system` flag to allow for use of the user/system sitepkgs directory instead of venv

This will help with use of the dev container, the alternate user/system should still be checked to ensure python deps are resolved correctly.

In the case of an EE, we may allow for installation of collection into user site packages and assume collections install in the system ansible path have already had thei python deps resolved and just ignor them...

Related #27

Facilitate installation of ADT

Once the venv is built provide a way to install ADT within the venv or do it by default.

Allow isolation check to be disabled

Installing local collection in editable mode fails

The installation of a local collection in editorial mode fails in macOS because of the following command where it tries to copy files tracked by git:

git ls-files 2> /dev/null || ls

The expectation is that if the first command (git ls-files) returns no files, the second command (ls) should be executed to return a list of all the files in the current directory. However, the issue with this chained command is that the first command exits with a success code even when it finds no files tracked using git, and thus, the second command is not executed.

This results in failing to install a local collection when the collection files are not tracked using git.

Installation in editable mode fails when collection is not specified under namespace/collection directory

While installing the collection in editable mode, the code looks for the collection files under the namespace/collection directory to copy them to the build folder.

This might not always be true, as users can also put collection files in other directories. When this happens, the CLI tool fails to install them in editable mode.

Allow for building an venv without an exisiting collection

This would allow ADE to be used prior to creator

Allow installation of multiple collections

pi4a install amazon.aws anaible.utils

RENAME: ansible-dev-environment

To be consistent with ADT.

Installing collection dependencies from git repositories fails

Issue

When I try to install my current collection that has a dependency to a git repository using:

ade install -e . --venv .venv

galaxy.yml

dependencies: {'[email protected]:organization/repo_name.git': 'devel'}

I get this error:
Could not find git executable to extract the collection from the Git repository '[email protected]:organization/repo_name.git'.

Analyzation

I debugged it a bit and I found out that the PATH env variable is empty after ADE is calling ansible-galaxy. ansible-galaxy tries to find the git executable here:
https://github.com/ansible/ansible/blob/1b209d742e39900e676e6a43f900801e67cc9154/lib/ansible/galaxy/collection/concrete_artifact_manager.py#L419

What I also tried:

install via requirements.yml works 👍
install via ansible-galaxy directly 👍

ansible-galaxy collection install "[email protected]:organization/repo_name.git,devel"

The only thing I can see that PATH is empty when ansible-galaxy is called via ADE and that get_bin_path is looking for git in PATH and some other default locations:
https://github.com/ansible/ansible/blob/1b209d742e39900e676e6a43f900801e67cc9154/lib/ansible/module_utils/common/process.py#L32

Allow for `-r requirements.yml`

Specify a requirements file from which collections will be installed.

Include `ade` at the command line

https://github.com/ansible/ansible-development-environment/blob/de3b51ba53e41c4f24d863ea7a7ab06c7fe91fce/pyproject.toml#L35

No module named `pkg_resources`

Description

When running poetry run ade install -r requirements.yml, the command fails with a ModuleNotFoundError: No module named 'pkg_resources'.

$ poetry run ade install -r requirements.yml
Failed to discover requirements: Command 'ansible-builder introspect /home/runner/.cache/pypoetry/virtualenvs/deadnews-template-ansible-kPNM4ADQ-py3.12/lib/python3.12/site-packages --write-pip /home/runner/.cache/pypoetry/virtualenvs/deadnews-template-ansible-kPNM4ADQ-py3.12/.ansible-dev-environment/discovered_requirements.txt --write-bindep /home/runner/.cache/pypoetry/virtualenvs/deadnews-template-ansible-kPNM4ADQ-py3.12/.ansible-dev-environment/discovered_bindep.txt --sanitize' returned non-zero exit status 1. ModuleNotFoundError: No module named 'pkg_resources'

Possible solution

The pkg_resources module is part of setuptools.

According to the Poetry documentation:

If your project relies on setuptools, you should declare it as a dependency.

Context

GitHub workflow log.

$ ade --version
24.1.0

RFE: Propose a EE manifest based on venv contents

Run bindep check before python check

Pip may try to build some deps in the dry run that require system packages, we should alert on those first

Generate a requirements.yml file with `pip4a freeze`

Bootstrapping project

renamed python package
make py310 minimum version required
add py312 support
setup GHA pipelinelines
setup mkdocs and RTD building under https://ansible.readthedocs.io/
include the project into ansible-dev-tools
fix macos bug and enable macos testing
make a pre-release

Installation fails with py312

I am getting the following error while using pip4a to install collection in editable mode, I am getting the following error. For those of you who have already used this tool, any clue why this is happening?

(venv) ➜  testname git:(main) ✗ pip4a install -e .
Critical: Failed to copy collection to build directory: Command 'cp -r --parents
          $(git ls-files 2> /dev/null || ls)
          /Users/prsahoo/.virtualenvs/creator/.pip4a/testns.testname/build'
          returned non-zero exit status 64. cp: illegal option -- -
          usage: cp [-R [-H | -L | -P]] [-fi | -n] [-aclpSsvXx] source_file
          target_file
                 cp [-R [-H | -L | -P]] [-fi | -n] [-aclpSsvXx] source_file ...
          target_directory

Bug: ADE fails if running from system site packages but pointed to a venv

If a venv is built with system-site-packages the install of collections fails.

(venv) media-rv ➜  new_project ade install -r requirements.yml --venv venv
Traceback (most recent call last):
  File "/home/bthornto/.local/bin/ade", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/bthornto/.local/lib/python3.12/site-packages/ansible_dev_environment/cli.py", line 177, in main
    cli.run()
  File "/home/bthornto/.local/lib/python3.12/site-packages/ansible_dev_environment/cli.py", line 158, in run
    subcommand.run()
  File "/home/bthornto/.local/lib/python3.12/site-packages/ansible_dev_environment/subcommands/installer.py", line 56, in run
    self._install_galaxy_requirements()
  File "/home/bthornto/.local/lib/python3.12/site-packages/ansible_dev_environment/subcommands/installer.py", line 184, in _install_galaxy_requirements
    cpath = self._config.site_pkg_collections_path / cnamespace / cname
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/bthornto/.local/lib/python3.12/site-packages/ansible_dev_environment/config.py", line 91, in site_pkg_collections_path
    site_pkg_collections_path.mkdir()
  File "/usr/lib64/python3.12/pathlib.py", line 1311, in mkdir
    os.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/local/lib64/python3.12/site-packages/ansible_collections'