ansible-community / collection_migration Goto Github PK

DONE AND DONE: Was scripts and scenarios for migration from core code to collections

Python 99.45% Shell 0.55%

collection_migration's Introduction

NO LONGER NEEDED

The migrate.py process was ran for the last time on Friday 6th March 2020.

This repo is left as a record of what was done.

If you need to move plugins between collections please see https://github.com/ansible-collections/overview/blob/master/README.rst#q-what-should-i-do-to-move-plugins-across-collections-during-migration

Old readme

migrate.py

The migration script (migrate.py) has incorporated help, so using --help you can see all the options.

Runtime pre-requisites

The main requisite is Python 3.7. You'll also need to install the dependency packages:

$ python3.7 -m venv .venv
$ . .venv/bin/activate
(.venv) $ python3.7 -m pip install -r requirements.in

Pro tip: If you wish to install exactly the same dep versions as the CI uses, make sure to add -c requirements.txt in the end of that last command.

Contributing

When making PRs, check for the CI green status. Occasionally, its red status has nothing to do with your contribution but most of the time it does. Ask questions if something looks weird.

There's a separate linting job in the CI now. It essencially runs the pre-commit tool. Besides checks, it is also able to do minor code formatting. If you see any related failures, pip install pre-commit locally and run pre-commit run --all-files. In case this generates file changes, you can safely commit those. But if after that there's still linter offences present, fix them manually.

Note: Some of the CI steps are set up to ignore failures, like running ansible-test [sanity|units] against the migrated collection artifacts. This is because the migration script is not ideal but we still want to have a log of how it's going.

Migration scenario

The script takes a scenario as a mandatory argument (-s path/to/dir), a scenario is a directory with one or more YAML files that describe the collection layout post migration.

Each file name is the namespace of the included collections, inside you can have a collection name, followed by the plugin types and actual plugin files (with extensions) in that collection as they appear in the ansible repo, including subdirectories from their expected locations. For example:

# test_scenario/microsoft.yml
azure:
  module_utils:
  - azure.py
  - azure_rm.py
  modules:
  - cloud/azure/azure_rm_instance.py
windows:
  lookups:
  - win_registry.py

Some existing scenarios are already provided in the repo, stdlib and mintest being the most useful ones as they can generate (with -m option) an ansible repo w/o most of the plugins (stdlib has none, mintest has the ones we have considered Ansible requires to be minimally testable).

Per collection options

A new _options key was introduced for per colleciotn settings, subkeys are:

  mycol:
	_options:
		flatmap: False
		# Preserves subdirs for modules and makes it a 'flatmap type collection'

		version: '0.1.0'
		# Semantic version string for collection version

		licence: GPL-3.0-or-later
		# SPEX license string

		license_file: COPYING
		# file containing license in repo, this will be a GPLv3 copy from migration change as needed.

		tags:
		# list of galaxy tags desired for this collection

Performing the migration

In order to run the migration based on the existing scenario, you'd need to execute somehting like this:

(.venv) $ python3.7 -m migrate -s scenarios/minimal

The migrate script has a --help for other options.

Generating a bare scenario

Another useful script is generate_glob_collection.sh which outputs a YAML structure to stdout that lists ALL the plugins from an Ansible checkout (which is the only required parameter), useful to regenerate the bare scenario or as a starting point for other scenarios.

Note: scenarios support 'informative' collections, that start with _ as a means to let collections know dependencies but not actually migrate, also the special _core collection is used to indicate plugins that would stay in core and not require rewrites for those referencing them.

Things to be aware of

If the scenario doesn't contain an explicit enumeration of artifacts related to the given resource, it may result in an incomplete migration. One example of such case it including an action plugin and omitting the module with the same name, or any other related files. This may result in various sanity and/or other tests failures. E.g. action plugin has no matching module to provide documentation (action-plugin-docs).

Definitions for the 2.10 Ansible Release

There are a few terms that are important to the understanding of the Ansible 2.10 release, that impact and indicate how ansible will be structured, and distributed.

base : This includes a small number of plugins and modules that roughly track the 2.9 definition of "core" supported plugins and modules. This will provide a limited functionality to support a standard use case that may involve bootstrapping a host, to a point where additionall collections can then be used.

collection_migration's People

Contributors

Stargazers

Watchers

collection_migration's Issues

fix dependency namespaces

currently dependencies from collections in other namespaces get the namespace from the current collection

Do not inject init.py into fixtures directories

The unit tests from test/units/modules/system/interfaces_file/test_interfaces_file.py seem to use files from test/units/modules/system/interfaces_file/fixtures/input/ and test/units/modules/system/interfaces_file/fixtures/golden_output/.

We inject __init__.py into test/units/modules/system/interfaces_file/fixtures/input (fixtures and all subdirs) and then the tests think it's a valid input file for a test.

As a result we get a permission denied error on ansible_collections/ansible/misc/tests/unit/modules/system/interfaces_file/fixtures/golden_output/__init__.py_revert.exceptions.txt.

Complete example (one of them):

___________________ TestInterfacesFileModule.test_no_changes ___________________
[gw2] linux -- Python 3.7.4 /tmp/python-rndnyy_e-ansible/python
self = <ansible_collections.ansible.misc.tests.unit.modules.system.interfaces_file.test_interfaces_file.TestInterfacesFileModule testMethod=test_no_changes>

    def test_no_changes(self):
        for testfile in self.getTestFiles():
            path = os.path.join(fixture_path, testfile)
            lines, ifaces = interfaces_file.read_interfaces_file(module, path)
>           self.compareInterfacesLinesToFile(lines, testfile)

tests/unit/modules/system/interfaces_file/test_interfaces_file.py:102: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/unit/modules/system/interfaces_file/test_interfaces_file.py:77: in compareInterfacesLinesToFile
    self.compareStringWithFile("".join([d['line'] for d in interfaces_lines if 'line' in d]), testname)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <ansible_collections.ansible.misc.tests.unit.modules.system.interfaces_file.test_interfaces_file.TestInterfacesFileModule testMethod=test_no_changes>
string = '', path = '__init__.py.test_no_changes'

    def compareStringWithFile(self, string, path):
        # self.assertEqual("","_",msg=path)
        testfilepath = os.path.join(golden_output_path, path)
        goldenstring = string
        if not os.path.isfile(testfilepath):
>           f = io.open(testfilepath, 'wb')
E           PermissionError: [Errno 13] Permission denied: '/root/ansible/ansible_collections/ansible/misc/tests/unit/modules/system/interfaces_file/fixtures/golden_output/__init__.py.test_no_changes'

tests/unit/modules/system/interfaces_file/test_interfaces_file.py:89: PermissionError

Needs further investigation.

Update: The command used for unit test run was ansible-test units --requirements --python "3.7" -vvv --docker

unit tests still don't correctly rewrite imports

example:

.cache/collections/ansible_collections/ansible/amazon/tests/unit/modules/cloud/amazon/test_aws_netapp_cvs_FileSystems.py

after running the bcs scenario has:

from ansible.modules.cloud.amazon.aws_netapp_cvs_FileSystems \
    import AwsCvsNetappFileSystem as fileSystem_module

though it should be
from ansible_collections.ansible.amazon.plugins.modules.aws_netapp_cvs_FilesyStems

Module imports contains subpackages even if not --preserve-module-subdirs

With #39 we rewrite module imports like:

from ansible_collections.ansible.amazon.plugins.modules.cloud.amazon.aws_netapp_cvs_snapshots \                                                                                                                                                                                          
    import AwsCvsNetappSnapshot as snapshot_module

Which is correct only if --preserve-module-subdirs, otherwise the import should look like:

from ansible_collections.ansible.amazon.plugins.modules.aws_netapp_cvs_snapshots \                                                                                                                                                                                          
    import AwsCvsNetappSnapshot as snapshot_module

So cloud.amazon which is module subdir/subpackage needs to be remove.

diff

-from ansible_collections.ansible.amazon.plugins.modules.cloud.amazon.aws_netapp_cvs_snapshots \                                                                                                                                                                                          
+from ansible_collections.ansible.amazon.plugins.modules.aws_netapp_cvs_snapshots \                                                                                                                                                                                          
     import AwsCvsNetappSnapshot as snapshot_module

mode incorrectly rewritten in integration tests

0644 becomes 420 during rewrite.

allow spec to indicate subdirectories for the plugins

Example:
lib/ansible/modules/cloud/vmware/vmware_guest.py --> collection:plugins/modules/cloud/vmware/vmware_guest.py

Rewritten modules should respect 160 chars line length limit

When running sanity tests against migrated collections, I've hit line length errors on lines like

from ansible_collections.new_azure_order.azure.plugins.module_utils.azure_rm_common import AzureRMModuleBase, azure_id_to_dict, normalize_location_name, format_resource_id

Instead, we should emit

from ansible_collections.new_azure_order.azure.plugins.module_utils.azure_rm_common \
    import (
        AzureRMModuleBase,
        azure_id_to_dict,
        normalize_location_name,
        format_resource_id,
    )

from ansible_collections.new_azure_order.azure.plugins.module_utils.azure_rm_common import AzureRMModuleBase
from ansible_collections.new_azure_order.azure.plugins.module_utils.azure_rm_common import azure_id_to_dict
from ansible_collections.new_azure_order.azure.plugins.module_utils.azure_rm_common import normalize_location_name
from ansible_collections.new_azure_order.azure.plugins.module_utils.azure_rm_common import format_resource_id

Add a CLI toggle for turning on pushing to Git remote

build and push generated collections to galaxy

ansible-galaxy collection build && ansible-galaxy publish <tarball>

probably a 2ndry script

ansible namespace should have repo per collection
other namespaces can host all collections in same repo (most of these will probably be informational only in the end)

Rewrite plugins refenrences in @patch in unit tests

e.g.: .cache/collections/ansible_collections/ansible/amazon/tests/unit/modules/cloud/amazon/test_aws_netapp_cvs_snapshots.py:

@patch('ansible.modules.cloud.amazon.aws_netapp_cvs_snapshots.AwsCvsNetappSnapshot.getSnapshotId')

Improve ignore.txt mappings

$ ansible-test sanity --python 3.7
...
Running sanity test 'ignores'
ERROR: Found 2 ignores issue(s) which need to be resolved:
ERROR: tests/sanity/ignore-2.10.txt:1:1: File 'plugins/module_utils/aci.py' does not exist
ERROR: tests/sanity/ignore-2.10.txt:2:1: File 'plugins/module_utils/aci.py' does not exist
...

integration tests improperly rewritten

see last line of excerpt of
ansible_collections/microsoft/azure/tests/integration/targets/azure_rm_devtestlab/tasks/main.yml

  microsoft.azure.azure_rm_devtestlabartifactsource:
    resource_group: '{{ resource_group }}'
    lab_name: '{{ lab_name }}'
    name: '{{ artifacts_name }}'
    source_type: github
    folder_path: /tasks
    security_token: '{{ github_token }}'
    ansible._core.uri: https://github.com/Azure/azure_preview_modules.git

2 issues:

uri: option was being detected as a module (wrong)
~~a module from _core was rewritten, it is supposed to skip those~~ FIXED in #47

Greedy unit tests discovery

Unit tests discovery is greedy and matches tests in parent dirs that are not relevant.

units: copy all needed `conftest.py`'s from parent dirs into collections

The problem

migrate.py doesn't copy the conftest modules from upper-level directories as it should.

Complete example (one of them):

______ ERROR at setup of test_xenserver_guest_powerstate_wait[wait-other] ______
[gw2] linux -- Python 3.7.4 /tmp/python-rndnyy_e-ansible/python
file /root/ansible/ansible_collections/ansible/misc/tests/unit/modules/cloud/xenserver/test_xenserver_guest_powerstate.py, line 250
  @pytest.mark.parametrize('patch_ansible_module',
                           testcase_module_params_wait['params'],
                           ids=testcase_module_params_wait['ids'],
                           indirect=True)
  def test_xenserver_guest_powerstate_wait(mocker, patch_ansible_module, capfd, XenAPI, xenserver_guest_powerstate):
E       fixture 'patch_ansible_module' not found
>       available fixtures: XenAPI, bigip, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, doctest_namespace, fake_ansible_module, mgmt_root, mock, mocker, monkeypatch, opt_bigip, opt_password, opt_peer, opt_port, opt_release, opt_token, opt_username, opt_vcmp_host, peer, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory, vcmp_host, worker_id, xenserver_guest_info, xenserver_guest_powerstate
>       use 'pytest --fixtures [testpath]' for help on them.

/root/ansible/ansible_collections/ansible/misc/tests/unit/modules/cloud/xenserver/test_xenserver_guest_powerstate.py:250

Easy solution

Walk the fs dir layout up the tree and copy all conftest.py modules from parent dirs.

Advanced solution

Analyze FST of test modules and record all the fixture names. Then, scan conftest.py files and copy ones declaring the corresponding fixtures.

Bonus complexity

We could also analyze imports in conftest.py files in order to identify helper utils modules imports. But that's a problem described in #80.

Rewrite ALL integration tests

Rewrite even those that stay in the core since those can use migrated plugins too.

push 'updated core' to migrated repo

https://github.com/ansible/migratedcore

Inject basic CI configs into migrated collections

correct integration tests path

currently they are copied to /test/ and it needs to be /tests/

Incorrectly rewritten string in DOCUMENTATION

DOCUMENTATION in ansible_collections/ansible/misc/plugins/modules/netscaler_cs_policy.py is incorrectly rewritten:

DOCUMENTATION = '''author: George Nikolopoulos (@giorgos-nikolopoulos)
description:
- Manage content switching policy.
- This module is intended to run either on the ansible  control node or a bastion
  (jumpserver) with access to the actual netscaler instance.
extends_documentation_fragment:
- ansible.misc.netscaler
module: netscaler_cs_policy
options:
  action:
    description:
    - Content switching action that names the target load balancing virtual server
      to which the traffic is switched.
  domain:
    description:
    - The domain name. The string value can range to 63 characters.
    - Minimum length = 1
  policyname:
    description:
    - Name for the content switching policy. Must begin with an ASCII alphanumeric
      or underscore C(_) character, and must contain only ASCII alphanumeric, underscore,
      hash C(#), period C(.), space C( ), colon C(:), at sign C(@), equal sign C(=),
      and hyphen C(-) characters. Cannot be changed after a policy is created.
    - 'The following requirement applies only to the NetScaler CLI:'
    - If the name includes one or more spaces, enclose the name in double or single
      quotation marks (for example, my policy or my policy).
    - Minimum length = 1
  rule:
    description:
    - Expression, or name of a named expression, against which traffic is evaluated.
      Written in the classic or default syntax.
    - 'Note:'
    - 'Maximum length of a string literal in the expression is 255 characters. A longer
      string can be split into smaller strings of up to 255 characters each, and the
      smaller strings concatenated with the + operator. For example, you can create
      a 500-character string as follows: ''"<string of 255 characters>" + "<string
      of 245 characters>"'''
  url:
    description:
    - 'URL string that is matched with the URL of a request. Can contain a wildcard
      character. Specify the string value in the following format: C([[prefix] [*]]
      [.suffix]).'
    - Minimum length = 1
    - Maximum length = 208
requirements:
- nitro python sdk
short_description: Manage content switching policy
version_added: '2.4'
'''

The issue is in the description of the rule option, quotes are incorrectly handled closing the DOCUMENTATION string too early.

remove version_added from migrated plugin docs

Currently version_added refers to ansible, but once in collections, it should refer to the collection itself.

Common unit tests code should not be moved with `-m` option

[I 191024 09:17:56 migrate:865] Copying .cache/releases/devel.git/test/units/mock -> .cache/collections/ansible_collections/huawei/cloudengine/tests/unit/mock
Traceback (most recent call last):
  File "./migrate.py", line 1686, in <module>
    main()
  File "./migrate.py", line 1664, in main
    assemble_collections(spec, args, args.target_github_org)
  File "./migrate.py", line 1045, in assemble_collections
    plugin_type, plugin, spec,
  File "./migrate.py", line 866, in copy_unit_tests
    shutil.copy(src, dest)
  File "/home/mkrizek/.pyenv/versions/3.7.3/lib/python3.7/shutil.py", line 245, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/home/mkrizek/.pyenv/versions/3.7.3/lib/python3.7/shutil.py", line 120, in copyfile
    with open(src, 'rb') as fsrc:
FileNotFoundError: [Errno 2] No such file or directory: '.cache/releases/devel.git/test/units/mock'

We already have one exception in the code:
https://github.com/ansible/collection_migration/blob/1e19e50167b84722aca3eabcc36fe678a9c332cc/migrate.py#L868-L870

Skip `pycache` and `*.py{c,d,o}` files during migration

It may make sense to filter files tracked by Git, ignoring others.

Copying .cache/releases/devel.git/test/units/plugins/strategy/__pycache__/test_strategy_linear.cpython-37-pytest-5.1.1.pyc -> .cache/collections/ansible_collections/yum_unit_test_migration_spec/yum_collection/tests/unit/plugins/strategy/__pycache__/test_strategy_linear.cpython-37-pytest-5.1.1.pyc

Need to copy & rewrite ignore.txt

test/sanity/ignore.txt -> tests/sanity/ignore-2.{9,10}.txt

Create git remotes for collections, if they don't exist

push created collections to ansible-collections

The created collections should be pushed to ansible-collections github org

Need to collect all third-party deps from tests

https://github.com/ansible/collection_migration/runs/221518664#step:9:183:

ImportError while loading conftest '/home/runner/work/collection_migration/collection_migration/.cache/collections/ansible_collections/ansible/amazon/tests/unit/module_utils/xenserver/conftest.py'.
/opt/hostedtoolcache/Python/3.7.4/x64/lib/python3.7/site-packages/ansible/utils/collection_loader.py:228: in load_module
    exec(code_object, newmod.__dict__)
tests/unit/module_utils/xenserver/conftest.py:18: in <module>
    from mock import MagicMock
E   ModuleNotFoundError: No module named 'mock'
ERROR: Command "pytest ..." returned exit status 4.

Slack comment about installing test requirements by @mattclay:

@webknjaz Testing with --docker, --venv or --requirements will make sure all the requirements are installed. If you're using the latest devel version of ansible-test then you can have collections specific test requirements as well (for unit tests they go in tests/unit/requirements.txt).

Support tests in spec

In addition to test discovery, provide a means to specify tests (integration/unit) in spec. Note that test key is for lib/ansible/plugins/test currently.

need to use the ansible yaml constructor for parsing integration tests

In a scenario in which i tried to migrate template out of core:

yaml.constructor.ConstructorError: could not determine a constructor for the tag '!unsafe'
  in ".cache/releases/devel.git/test/integration/targets/template/tasks/main.yml", line 601, column 14

I will guess that things like !vault will also fail to be parsed by migration script when dealing with integration tests

Rewrite only extends_documentation_fragment in DOCUMENTATION

Current state, e.g.:
.cache/collections/ansible_collections/ansible/amazon/plugins/modules/ec2.py:

...
aliases: [ 'ansible.amazon.aws_zone', 'ec2_zone' ]
...

Should be:

...
aliases: [ 'aws_zone', 'ec2_zone' ]
...

The reason why it's happening is because we do replace on the whole doc string and not just the specific item in the extends_documentation_fragment list in:
https://github.com/ansible/collection_migration/blob/ab5b1cd7fdc5f0bb402a88cbbf1cc28355fc14ad/migrate.py#L276

So we effectively rewrite all occurences of aws to ansible.amazon.aws in DOCUMENTATION, including things like aws_zone to ansible.amazon.aws_zone since aws is a substring of aws_zone and matches the replace.

Minimal scenario should copy related files/modules along with action plugins

This is from our CI: https://github.com/ansible/collection_migration/runs/280929046#step:10:127

The migrated collection has plugins/action/assert.py but not modules/assert.py which should contain the documentation string.

ERROR: Found 7 action-plugin-docs issue(s) which need to be resolved:
See documentation for help: https://docs.ansible.com/ansible/devel/dev_guide/testing/sanity/action-plugin-docs.html
ERROR: plugins/action/assert.py:0:0: action plugin has no matching module to provide documentation
ERROR: plugins/action/async_status.py:0:0: action plugin has no matching module to provide documentation
ERROR: plugins/action/fail.py:0:0: action plugin has no matching module to provide documentation
ERROR: plugins/action/fetch.py:0:0: action plugin has no matching module to provide documentation
ERROR: plugins/action/include_vars.py:0:0: action plugin has no matching module to provide documentation
ERROR: plugins/action/patch.py:0:0: action plugin has no matching module to provide documentation
ERROR: plugins/action/wait_for_connection.py:0:0: action plugin has no matching module to provide documentation

So whenever we do lib/ansible/plugins/action/assert.py -> plugins/action/assert.py, we should also include lib/ansible/modules/utilities/logic/assert.py -> modules/assert.py.

Do not create init.py in collection dirs when it's not neccessary

Providing an empty list (or wildcard that has no match see #129) for a plugin type in spec results in __init__.py being created while the dir is empty.

generate community 'meta' collection with migrated collections

example https://github.com/ansible-collections/community

Warn or fail when wildcard doesn't match

In spec:

plugin_type:
  - path/that/does/not/exist/*

We need to warn or fail in this case as it's incorrect spec, not silently ignore.

Add 'move' option

The current migration leaves the original in the ansible repo and just adds data to mark as migrated.

A scenario in which the files do not exist is required, not only for proper testing as currently we get false success due to the core files being accessed instead of collection ones.

Invalid value of migrated_to in BOTMETA after migration

As per
https://github.com/ansible/ansible/blob/98b6b98287436535f0dfc99ffe3333e5a196b9a3/test/sanity/code-smell/botmeta.py#L41-L43

the migrated_to must contain a link to the collection in galaxy. We currently put namespace.collection:
https://github.com/ansible/collection_migration/blob/305082c16b0d65a56e6cc6dc28918af6a9de345a/migrate.py#L1278

init.py is always removed in module subdirs

__init__.py is removed from module sub directories even if there are module(s) left. The file needs to be removed only if all modules are moved out of the directory.

Inject proper gitignore files

One case I'd like to solve is committing pyc files.
I think that it's a good idea to base it on gitignore.io Python templates.

Figure out version for created collections

Currently we just assume 1.0.0

handle 2nd party Base classes

from ansible.module_utils.azure_rm_common import AzureRMModuleBase - is skipped as Base class but should not be, we currently avoid migrating Ansible base classes by matching 'Base' in the name.

Handle integration tests dependencies (meta/main.yml)

Integration tests may depend on other targets, see example below. Figure out how to handle them.

test/integration/targets/yum/meta/main.yml:

dependencies:
  - prepare_tests
  - setup_rpm_repo
  - setup_remote_tmp_dir

Duplicate entries in BOTMETA after migration

The original BOTMETA contains:

  $plugins/callback/default:
    support: core

after migration where we migrate lib/ansible/plugins/callback/default.py we don't amend the existing entry but add a duplicate:

  lib/ansible/plugins/callback/default.py:
    migrated_to: yum_ns.yum_collection

Note that the original does not contain .py file extension which might be the reason we add a duplicate.

rewrite name properties in migrated plugins

Some plugin types have a NAME or <plugin type>_NAME property (callbacks)

These need to be rewritten to the FQCN as plugins can compare internally to determine some actions, mostly due to aliases, deprecation and subclassing.

Migrating `conftest.py` from parent test dirs includes ALL unit tests

When #88 is run on the following spec:

modules:
  - packaging/os/yum.py

It correctly includes conftest.py's from parent dirs:

'test/units/modules/conftest.py': 'tests/unit/modules/packaging/os/conftest.py',
'test/units/modules/packaging/os/conftest.py': 'tests/unit/modules/packaging/os/conftest.py',

However, in process of doing so it also includes ALL module unit tests with them. I believe this is because of the following code adding conftest.py's into matching_test_modules:
https://github.com/ansible/collection_migration/blob/b18c4a2e3b7fc08f4285ab0b4bad50e40238a919/migrate.py#L699-L703

which is then processed to find related test files:
https://github.com/ansible/collection_migration/blob/b18c4a2e3b7fc08f4285ab0b4bad50e40238a919/migrate.py#L803-L813

The problem is that in case of test/units/modules/conftest.py (not limited to) it means that it goes through test/units/modules and adds all modules unit tests.

bad test rewriting

when running 'bare' scenario got hte following output:

Traceback (most recent call last):
  File "./migrate.py", line 1826, in <module>
    main()
  File "./migrate.py", line 1802, in main
    assemble_collections(devel_path, spec, args, args.target_github_org)
  File "./migrate.py", line 1143, in assemble_collections
    unit_deps += rewrite_unit_tests_patch(unit_test_module_fst, collection, spec, namespace, args, file_path)
  File "./migrate.py", line 360, in rewrite_unit_tests_patch
    plugin_name = val[3]
IndexError: list index out of range```

Broken symlinks in core after migration when the move option is used

ERROR: Found 2 symlinks issue(s) which need to be resolved:
ERROR: lib/ansible/modules/notification/_osx_say.py:0:0: broken symlinks are not allowed
ERROR: lib/ansible/modules/system/_python_requirements_facts.py:0:0: broken symlinks are not allowed

e.g. lib/ansible/modules/notification/_osx_say.py -> lib/ansible/modules/notification/say.py, the symlink is not moved, the target file is.

Rewrite implicit relative Python 2 imports to be explicit

Basically, we need to mutate code like this:

-from oneview_module_loader import SOME_CONSTANT  # doesn't work
+from .oneview_module_loader import SOME_CONSTANT  # works

if the imported module is next to the current one.

pytest does some injections into PYTHONPATH which don't work with the collection loader in place so we have to do the rewrite.

test/sanity/ignore.txt in core points at moved files

Moved files out of core are still mentioned in test/sanity/ignore.txt after the migration.

Example:
ERROR: test/sanity/ignore.txt:6203:1: File 'test/units/plugins/shell/test_cmd.py' does not exist

need to add sanity ignore entries for unit dir folders

From the vmware collection created by migrate.py ...

Running sanity test 'future-import-boilerplate' with Python 3.6
ERROR: Found 47 future-import-boilerplate issue(s) which need to be resolved:
ERROR: tests/unit/mock/path.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/mock/yaml_helper.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/basic/test__symbolic_mode_to_octal.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/basic/test_deprecate_warn.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/basic/test_exit_json.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/basic/test_get_file_attributes.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/basic/test_heuristic_log_sanitize.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/basic/test_run_command.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/basic/test_safe_eval.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/basic/test_tmpdir.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/cloud/test_backoff.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/common/test_dict_transformations.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/conftest.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/ec2/test_aws.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/facts/base.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/facts/hardware/test_sunos_get_uptime_facts.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/facts/network/test_generic_bsd.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/facts/other/test_facter.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/facts/other/test_ohai.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/facts/system/test_lsb.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/facts/test_ansible_collector.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/facts/test_collector.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/facts/test_collectors.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/facts/test_facts.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/facts/test_timeout.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/facts/test_utils.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/gcp/test_auth.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/gcp/test_gcp_utils.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/gcp/test_utils.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/hwc/test_dict_comparison.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/hwc/test_hwc_utils.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/json_utils/test_filter_non_json_lines.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/net_tools/test_netbox.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/network/avi/test_avi_api_utils.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/network/ftd/test_common.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/network/ftd/test_configuration.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/network/ftd/test_device.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/network/ftd/test_fdm_swagger_parser.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/network/ftd/test_fdm_swagger_validator.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/network/ftd/test_fdm_swagger_with_real_data.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/network/ftd/test_upsert_functionality.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/parsing/test_convert_bool.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/postgresql/test_postgres.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/remote_management/dellemc/test_ome.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/xenserver/FakeAnsibleModule.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/module_utils/xenserver/FakeXenAPI.py:0:0: missing: from __future__ import (absolute_import, division, print_function)
ERROR: tests/unit/modules/utils.py:0:0: missing: from __future__ import (absolute_import, division, print_function)

Many of these would typically be ignored by the ignore file in devel https://github.com/ansible/ansible/blob/devel/test/sanity/ignore.txt#L6536-L6570

test/units/modules/utils.py future-import-boilerplate
test/units/modules/utils.py metaclass-boilerplate
test/units/modules/web_infrastructure/test_apache2_module.py future-import-boilerplate
test/units/modules/web_infrastructure/test_apache2_module.py metaclass-boilerplate
test/units/modules/web_infrastructure/test_jenkins_plugin.py future-import-boilerplate
test/units/modules/web_infrastructure/test_jenkins_plugin.py metaclass-boilerplate
test/units/parsing/utils/test_addresses.py future-import-boilerplate
test/units/parsing/utils/test_addresses.py metaclass-boilerplate
test/units/parsing/vault/test_vault.py pylint:blacklisted-name
test/units/playbook/role/test_role.py pylint:blacklisted-name
test/units/playbook/test_attribute.py future-import-boilerplate

We need to persist these ignores so that we don't have to scramble fixes or manual edits at migration time.

migrate.py does create the ignore files, but it only covers files within the plugins folder.

verify core plugins and tests after removal

verify that no 'non migrated' plugins nor tests in core depend on the collections

mostly this can work by running migration against the list of files kept in core and failing if we need to rewrite (or just log and error if log > 0)

rewrite powershell for collections

Current migration only addresses python plugins, powershell is mostly just copied but imports are not rewritten to function within the collection.
from

#Requires -Module Ansible.ModuleUtils.NameOfModuleUtil

#AnsibleRequires AnsibleCollections.namespace.collection.NameOfModuleUtil

and from

lib/ansible/module_utils/powershell/Ansible.ModuleUtils.NameOfModuleUtil.psm1

plugins/module_utils/NameOfModuleUtil.psm1