Currently, only Ubuntu Xenial is supported. The sources the apt installer uses points to Xenial endpoints. We will have to introspect on which distribution of Ubuntu we are running and then use the correct sources list.

It looks like we should pull in https://github.com/nir0s/distro since the standard library https://docs.python.org/3.5/library/platform.html#platform.linux_distribution is deprecated.

If a workspace contains a setup.py that installs pip3 dependencies, and rospkg is included in those dependencies then rospkg (rosversion etc) commands are broken in the bundle.

colcon bundle currently produces a file with the '.tar.gz' (GZip Tape ARchive) suffix. The bundle file format is becoming more complex and has more features and requirements than just a 'tar'.

In order to differentiate our bundles we should define our own file suffix. This is a common convention to reduce confusion when generating special file formats. The Debain linux distribution uses the .deb suffix for their packages, the actual file format is just 'tar'. Some other common file suffix examples are: tar '.tar', gzip '.gz' , text '.txt', Office Document '.doc'

Pros:

• Differentiates the file so it is less confusing for users

Cons:

• The underlying format (tar) is hidden, so users would have to read documentation to know they can open their bundle using tar. This is generally useful for debugging.

Initial thoughts:

• .cbf - Colcon Bundle Format - https://fileinfo.com/extension/cbf
• .cba - Colcon Bundle Archive - https://fileinfo.com/extension/cba

Colcon is a unique name and these file suffixes do not have any ubiquitous uses.

Time after time I see people with installation errors because they haven't upgraded setuptools. The general colcon ecosystem gets around this by having the colcon-common-extensions meta package. Since we aren't using a meta package we should make sure colcon-bundle and colcon-ros-bundle will install with older versions of setuptools. They should still depend on the later version of setuptools so that it gets installed for colcon-core

When there is a local repo apt doesn't copy the package into the cache directory when we call fetch_archives(). We should probably alter the way we find all the debs to install. Maybe we can get the local file path to a deb from the package in the apt cache object.

Currently, we place a lot of intermediate artifacts in the bundle folder in a user's workspace. It is confusing to find the actual bundle output. We should put all the intermediate files into bundle/cache and only place dotfiles and output.tar in the root of the bundle folder. This should reduce confusion.

Pending information from colcon/colcon-core#204 we might want to switch to sourcing local_setup.sh.

https://www.python.org/dev/peps/pep-0484/

https://travis-ci.org/colcon/colcon-bundle/jobs/622679493#L385

test/test_flake8.py F

=================================== FAILURES ===================================

_________________________________ test_flake8 __________________________________

    def test_flake8():

        style_guide = get_style_guide(extend_ignore=['D100', 'D104'],

                                      show_source=True)

        style_guide_tests = get_style_guide(

            extend_ignore=[

                'D100', 'D101', 'D102', 'D103', 'D104', 'D105', 'D107'],

            show_source=True, )

    

        stdout = sys.stdout

        sys.stdout = sys.stderr

        # implicitly calls report_errors()

        report = style_guide.check_files(

            [str(Path(__file__).parents[1] / 'colcon_bundle')])

        report_tests = style_guide_tests.check_files(

            [str(Path(__file__).parents[1] / 'tests')])

        sys.stdout = stdout

    

        total_errors = report.total_errors + report_tests.total_errors

        if total_errors:  # pragma: no cover

            # output summary with per-category counts

            print()

            report._application.formatter.show_statistics(report._stats)

            print('flake8 reported {total_errors} errors'.format_map(locals()),

                  file=sys.stderr)

    

>       assert not report.total_errors, 'flake8 reported {total_errors} ' \

                                        'errors'.format_map(locals())

E       AssertionError: flake8 reported 2 errors

E       assert not 2

E        +  where 2 = <flake8.api.legacy.Report object at 0x7ff5b99c0b00>.total_errors

test/test_flake8.py:41: AssertionError

----------------------------- Captured stdout call -----------------------------

2     D401 First line should be in imperative mood; try rephrasing

----------------------------- Captured stderr call -----------------------------

/home/travis/build/colcon/colcon-bundle/colcon_bundle/installer/__init__.py:17:1: D401 First line should be in imperative mood; try rephrasing

        """

^

/home/travis/build/colcon/colcon-bundle/colcon_bundle/verb/bundle.py:40:1: D401 First line should be in imperative mood; try rephrasing

        """

^

flake8 reported 2 errors

----------- coverage: platform linux, python 3.5.6-final-0 -----------

Name    Stmts   Miss Branch BrPart  Cover

-----------------------------------------

If the --bundle-base points to an existing directory that was created for a v1 bundle build, building with --bundle-version 2 will result in odd errors and a broken bundle.

We should add a version marker to the bundle-base directory with a 1 or 2 and check at the start of the bundle invocation to make sure that version matches up with the requested version otherwise throw an error.

Example Error: [25.298s] ERROR:colcon:colcon bundle: [Errno 2] No such file or directory: '/home/ubuntu/environment/HelloWorld/simulation_ws/bundle/dependencies.tar.gz'

We might be able to compress with a same or better compression ratio and parallelize the compression to make it much faster.

https://github.com/facebook/zstd
https://pypi.org/project/zstandard/

We require that users have keys added to their apt keys cache. When the keys aren't there an exception is thrown when we attempt to update our copy of the apt cache.

Currently when we bundle, we create cache directories on the host. Accidental mutation of these directories can cause problem in future bundling operation. Having a functionality to clean up would help customers to clean up the caches incase they face trouble in bundling due to accidental mutation of the local cache directories.

Example cmd:

colcon bundle --clean

Using apt_pkg cache doesn't find a version when searching just by name. If you split the package name into <package_name>=<version> and search <package_name>'s version_list for the version we might be able to install specific versions or verify the version against the desired version in package.xml.

https://stackoverflow.com/questions/34162051/python-apt-install-package-with-specific-version

Update:
With the work done in #117 some version support was added, but it requires workarounds in rosdep to work. The following are changes that would need to be made in colcon-bundle, colcon-ros-bundle, and colcon-ros to add full apt package version specifier support.

In colcon-ros the code located here would need to be updated to parse the version attributes supported in the package.xml format: https://www.ros.org/reps/rep-0140.html#id14. Here's an example of how we set the version metadata from Python setup.cfg packages: https://github.com/colcon/colcon-core/blob/master/colcon_core/package_identification/python.py#L142.

In colcon-ros-bundle this metadata would need to be read and then converted to the format supported in

This should be changed to the sh tuple:

We also need to make the sh regex specifically for sh..The current regex will catch bash also.

We built a bundle and had some package keys that were unable to be resolved by rosdep.

Could not find key for <dependency>

After executing rosdep update, so that the keys were there we attempted to re-bundle, but no changes were found. This is because we cache what's in the package.xml list and not what we could actually resolve. We need to make sure we are not caching packages which we weren't able to resolve.

In relation to colcon/colcon-ros-bundle#13 we should emit a message when no packages are found.

Right now, we don't explicitly create the bundle_staging directory, when we extract all the debfiles dpkg creates the directory automatically. If there are no packages then we don't run the apt installer and therefore no bundle_staging directory is created, which is the root cause of the exception in the linked issue.

ROS DISTRO: Kinetic on NVidia TX2 (arm64 architecture)
OS: Ubuntu 16.04

When we create bundle for x86 architecture we have observed that roslaunch package also get downloaded and available in bundle folder but in case of arm64 its seems that roslaunch is not available in bundle.

I have attached below colcon bundle logs for sample hello-world application on both arm64 and X86 platform for reference. Here we can see "ros-kinetic-roslaunch" on X86 colcon bundle logs but not in arm64 colcon bundle logs.

Could you please assist on this above issue.

colcon_bundle_logs_for_hello-word_application_on_arm64_platform.log

colcon_bundle_logs_for_hello-word_application_on_x86_platform.log

We are running into an issue with some packages installed via pip3 where the shebang is broken. It appears that setuptools puts hardcoded paths into their shebangs: pypa/setuptools#494.

Our shebang rewriting should support this format of shebang.

If possible it would be useful to be able to specify a rosdep formatted yaml file in the arguments to colcon bundle that would either be loaded into rosdep or used alongside rosdep to translate package dependency names into apt/pip package names.

Had an issue where I had to bundle a wheel file for a python package that was closed sources and not available on pypi.

Had to work around it by installing a local pypi server (https://pypi.org/project/pypiserver), and hosting the package with the local server.
Then added the package_name can be resolved by rosdep by adding it to a local source list, and doing a rosdep update

Then added the following command after running the local pypi server on 8080
colcon bundle --pip-args --extra-index-url http://localhost:8080/simple

That fixed the issue. Making a note here to see if can be simplified or to have record for folks who have similar issue.

We should allow users bundling their application to add lines to the setup.sh. This would enable users to expose custom environment variables and modify existing ones:

export PATH=$BUNDLE_CURRENT_PREFIX/my/custom/bin

The other idea would be to add hooks that will run custom transforms on the bundle, possibly to add symlinks into PATH etc.

Hey guys,

Have you already planned / scheduled the 0.0.19 release? I really miss the possibility to quickly install it via pip without exchanging files manually...

The current error message if the OSRF Gazebo repo key is missing looks like this:

[15.834s] ERROR:colcon.colcon_bundle.verb:Could not fetch from repositories: W:GPG error: http://packages.osrfoundation.org/gazebo/ubuntu-stable bionic InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 67170598AF249743, E:The repository 'http://packages.osrfoundation.org/gazebo/ubuntu-stable bionic InRelease' is not signed.
[15.836s] ERROR:colcon:colcon bundle: Failed to fetch from repositories. Did you set your keys correctly?

We should see if we can print out something more helpful.

The fix is to get the key from OSRF (this doesn't use HTTPS):

wget http://packages.osrfoundation.org/gazebo.key -O - | sudo apt-key add -

We don't have explicit tests for this feature. We should.

#39 exposed the fact that the user's local python environment is included in the pip freeze output. We should prevent this during pip install and pip freeze using command line options or otherwise. Connor attempted to do this but ran into issues. We should investigate further.

Currently, there is only a pip3 installer. Since ROS1 uses Python 2.7 we should also have a pip installer.

Currently, the bundle task for setup.py only supports version_eq. We should add support for all the other comparison operators.

They are defined here: https://github.com/colcon/colcon-core/blob/master/colcon_core/package_identification/python.py#L106

Hi
I am very new to colcon-bundle. I tried bundling my ros workspace with some packages using this article here. The bundle was created successfully. To test it, I copied it to another laptop with no ros installed . I extracted all the archive and when I executed following command:

BUNDLE_CURRENT_PREFIX=/home/samrin/test_ws/output/workspace source /home/samrin/test_ws/output/workspace/setup.sh

I got the following error:

not found: "/opt/ros/kinetic/local_setup.sh"
not found: "/home/bhoot/franks_ws/devel/local_setup.sh"

and my roslaunch didn't work
Can someone tell me what did i do wrong or how to resolve the error ?

I've noticed creating a bundle V2 is 2x slower (4 vs. 9 minutes) for a creating a bundle of the same size, ~700MB. The difference appears to be mostly during "Creating bundle archive..."

HelloWorld on Cloud9 M4 instance:

Bundle V1 (4 minutes):

...
Get:900 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 liburdfdom-dev amd64 0.4.1-1 [4252 B]                                 
Fetched 403 MB in 6s (5262 kB/s)                                                                                                        
Extracting apt packages...
Creating bundle archive...

real    3m54.475s
user    2m15.128s
sys     0m19.712s

Bundle V2 (9 minutes):

...
Get:900 http://us.archive.ubuntu.com/ubuntu xenial/universe amd64 liburdfdom-dev amd64 0.4.1-1 [4252 B]                                 
Fetched 403 MB in 6s (5999 kB/s)                                                                                                        
Extracting apt packages...
Creating bundle archive V2...
Archiving complete!

real    9m8.004s
user    7m38.936s
sys     0m18.836s

At some point apt imports were added to the top of apt.py. The imports should be inside the method they are used in. This file needs to load even if apt is not installed so that it can throw the nice error message we have in it.

There is a test case failure in Python 3.8 . This could be due to https://bugs.python.org/issue36268 where default tar format was changed to pax. The test passes before this change.

pytest test/test_bundlefile.py::TestBundlefile::test_bundlefile
/home/karthi/freezegun-venv/lib/python3.9/site-packages/_pytest/assertion/rewrite.py:8: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import imp
======================================== test session starts ========================================
platform linux -- Python 3.9.0a0, pytest-4.0.0, py-1.8.0, pluggy-0.12.0
rootdir: /home/karthi/colcon-bundle, inifile: setup.cfg
plugins: cov-2.7.1, flake8-0.9.1, rerunfailures-7.0, repeat-0.8.0
collected 1 item

test/test_bundlefile.py F                                                                     [100%]

============================================= FAILURES ==============================================
__________________________________ TestBundlefile.test_bundlefile ___________________________________

self = <test.test_bundlefile.TestBundlefile object at 0x7fb8c3cc0230>

    def test_bundlefile(self):
        bundle_path = os.path.join(self.tmpdir, 'test.mba')
        json_file = os.path.join(self.tmpdir, 'file')
        with open(json_file, 'w') as f:
            f.write(json.dumps(json_data))

        blocksize_file = os.path.join(self.tmpdir, 'blocksize_file')
        with open(blocksize_file, 'wb') as f:
            f.write(bytearray(tarfile.BLOCKSIZE))

        large_file = os.path.join(self.tmpdir, 'large_file')
        with open(large_file, 'wb') as f:
            large_file_size = 12 + tarfile.BLOCKSIZE * 1000
            f.write(bytearray(large_file_size))

        dep_archive = os.path.join(self.tmpdir, 'dependencies.tar.gz')
        with tarfile.open(dep_archive, 'w:gz') as a:
            a.add(json_file, arcname='file')

        other_archive = os.path.join(self.tmpdir, 'other.tar.gz')
        with tarfile.open(other_archive, 'w:gz') as a:
            a.add(json_file, arcname='file2')

        with Bundle(name=bundle_path, mode='w') as bundle:
            bundle.add_overlay_archive(dep_archive)
            bundle.add_overlay_archive(other_archive)
            bundle.add_overlay_archive(blocksize_file)
            bundle.add_overlay_archive(large_file)

        metadata_path = os.path.join(self.tmpdir, 'metadata.tar.gz')
        with tarfile.open(bundle_path, 'r:') as bundle:
            members = bundle.getmembers()
            assert 'version' == members[0].name
            assert 'metadata.tar.gz' == members[1].name
            assert 'pad' == members[2].name
            assert 'dependencies.tar.gz' == members[3].name
            # Validate the overlays start at 4MB
>           assert 4 * 1024 * 1024 == members[3].offset
E           AssertionError: assert ((4 * 1024) * 1024) == 4195328
E            +  where 4195328 = <TarInfo 'dependencies.tar.gz' at 0x7fb8c3eb03d0>.offset

test/test_bundlefile.py:60: AssertionError
===================================== 1 failed in 0.25 seconds ======================================

We should probably catch the permissions error and throw a warning. It's not fatal since something like this README doesn't need a shebang update anyways.

Extracting apt packages...
[33.832s] ERROR:colcon:colcon bundle: [Errno 13] Permission denied: '/home/ubuntu/environment/nav_stack/husky_ws/bundle/cache/bundle_staging/etc/sudoers.d/README'
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/colcon_core/command.py", line 442, in verb_main
    rc = context.args.main(context=context)
  File "/usr/local/lib/python3.5/dist-packages/colcon_bundle/verb/bundle.py", line 138, in main
    upgrade_deps_graph)
  File "/usr/local/lib/python3.5/dist-packages/colcon_bundle/verb/bundle.py", line 212, in _manage_dependencies
    include_sources=context.args.include_sources)
  File "/usr/local/lib/python3.5/dist-packages/colcon_bundle/verb/_installer_manager.py", line 98, in run_installers
    update_shebang(self.prefix_path)
  File "/usr/local/lib/python3.5/dist-packages/colcon_bundle/verb/utilities.py", line 65, in update_shebang
    with open(file_path, 'rb+') as file_handle:
PermissionError: [Errno 13] Permission denied: '/home/ubuntu/environment/nav_stack/husky_ws/bundle/cache/bundle_staging/etc/sudoers.d/README'

Running bundle against a CMake package resulted in this error:

INFO:colcon.colcon_core.location:Using config path '/home/ubuntu/.colcon'
INFO:colcon.colcon_core.location:Using log path 'log/bundle_2019-05-09_00-30-40'
Bundling workspace...
[0.817s] INFO:colcon.colcon_core.package_discovery:Crawling recursively for packages in '/home/ubuntu/environment/simulation_ws'
Reading package lists... Done
Building dependency tree... Done
Checking if dependency tarball exists...
[5.718s] INFO:colcon.colcon_bundle.verb:Checking if dependency tarball exists...
<colcon_core.package_decorator.PackageDecorator object at 0x7f8ec3b0a508>
{path: /home/ubuntu/environment/simulation_ws/src, type: cmake, name: src, dependencies: defaultdict(<class 'set'>, {'build': {'catkin'}, 'run': {'catkin'}}), hooks: [], metadata: {}}
[5.719s] ERROR:colcon:colcon bundle: 'str' object has no attribute 'name'
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/colcon_core/command.py", line 436, in verb_main
    rc = context.args.main(context=context)
  File "/usr/local/lib/python3.5/dist-packages/colcon_bundle/verb/bundle.py", line 123, in main
    context, path_context, upgrade_deps_graph)
  File "/usr/local/lib/python3.5/dist-packages/colcon_bundle/verb/bundle.py", line 173, in _manage_dependencies
    self._check_package_dependency_update(path_context, decorators)
  File "/usr/local/lib/python3.5/dist-packages/colcon_bundle/verb/bundle.py", line 315, in _check_package_dependency_update
    pkg.dependencies['run'])
  File "/usr/local/lib/python3.5/dist-packages/colcon_bundle/verb/bundle.py", line 314, in <genexpr>
    dependency_list = sorted(dependency.name for dependency in
AttributeError: 'str' object has no attribute 'name'

This is because colcon-cmake doesn't use DependencyDescriptor for its dependency metadata. We should not break if this is the case.

It would be good to have the list of packages that were removed from the list of packages to install. It should make it easier to debug when a dependency is missing.

I'm using the package raspicam_node which comes from the ubiquity robotics apt repository https://packages.ubiquityrobotics.com/ubuntu/ubiquity.

It built and ran fine, but when running colcon bundle it failed to find the package from apt throwing the following error:

Bundling workspace...
Reading package lists... Done
Building dependency tree... Done
Checking if dependency tarball exists...
Starting >>> object_tracker_robot
loading view [*default*] with rospkg loader
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 255, in __getitem__
    return self._weakref[key]
  File "/usr/lib/python3.5/weakref.py", line 131, in __getitem__
    o = self.data[key]()
KeyError: 'ros-kinetic-raspicam-node'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 259, in __getitem__
    rawpkg = self._cache[key]
KeyError: 'ros-kinetic-raspicam-node'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/colcon_core/executor/__init__.py", line 91, in __call__
    rc = await self.task(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/colcon_core/task/__init__.py", line 92, in __call__
    return await task_method(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/colcon_ros_bundle/task/ros_bundle.py", line 70, in bundle
    package_name)
  File "/usr/local/lib/python3.5/dist-packages/colcon_bundle/installer/apt.py", line 154, in add_to_install_list
    logger.info(self._cache[name].versions)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 261, in __getitem__
    raise KeyError('The cache has no package named %r' % key)
KeyError: "The cache has no package named 'ros-kinetic-raspicam-node'"
[110.747s] ERROR:colcon.colcon_core.executor.sequential:Exception in job execution 'object_tracker_robot': "The cache has no package named 'ros-kinetic-raspicam-node'"
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 255, in __getitem__
    return self._weakref[key]
  File "/usr/lib/python3.5/weakref.py", line 131, in __getitem__
    o = self.data[key]()
KeyError: 'ros-kinetic-raspicam-node'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 259, in __getitem__
    rawpkg = self._cache[key]
KeyError: 'ros-kinetic-raspicam-node'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/colcon_core/executor/sequential.py", line 49, in execute
    loop.run_until_complete(future)
  File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/usr/local/lib/python3.5/dist-packages/colcon_core/executor/__init__.py", line 91, in __call__
    rc = await self.task(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/colcon_core/task/__init__.py", line 92, in __call__
    return await task_method(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/colcon_ros_bundle/task/ros_bundle.py", line 70, in bundle
    package_name)
  File "/usr/local/lib/python3.5/dist-packages/colcon_bundle/installer/apt.py", line 154, in add_to_install_list
    logger.info(self._cache[name].versions)
  File "/usr/lib/python3/dist-packages/apt/cache.py", line 261, in __getitem__
    raise KeyError('The cache has no package named %r' % key)
KeyError: "The cache has no package named 'ros-kinetic-raspicam-node'"

The reason for this is colcon bundle has it's own list of apt sources in /etc/apt/sources.list.d (https://github.com/colcon/colcon-bundle/blob/8eb65de65d3cdd601119528331e84849b6ecba74/colcon_bundle/installer/assets/xenial.sources.list) and this list doesn't contain the ubiquty robotics repository.

I think colcon bundle should automatically pull in any custom defined apt repositories in /etc/apt/sources.list.d/ (or similar location on other OS's) so users aren't confused about this in the future.

We currently only support parsing setup.py for Python 3. We should use other metadata from the setup.py to determine which Python version the package is for and install into the correct Python installation.

It is possible to install Python 2 dependencies via package.xml declarations, but that doesn't support version locking.

https://answers.ros.org/question/194720/tab-completion/

I think this is because we are sourcing ${BUNDLE_CURRENT_PREFIX}/opt/ros/kinetic/setup.sh even if we are in a bash shell. Maybe we should check $SHELL and if it is bash then use setup.bash.

We should have intelligence that checks to see if sources.tar.gz was created and if it wasn't we should go through the process to pull down sources.

Hi, I am very interested in colcon bundle and I want to bundle cuda executable.
Can we use CUDA ROS node with colcon bundle??

As a result of #127 an incompatible version of setuptools will not be installed. We need a way to hide the irrelevant warning:

You are running Setuptools on Python 2, which is no longer
supported and
>>> SETUPTOOLS WILL STOP WORKING <<<
in a subsequent release (no sooner than 2020-04-20).

Not sure the solution would be in colcon-bundle but perhaps it's possible for colcon-bundle to change the warning configuration when it prepares the bundle with setuptools.
Refer to internal issue B9SIM-1008.

Currently, the installer metadata lists all the packages installed by a given installer. It would also be helpful to include what architecture version of the packages were downloaded. This could hopefully reduce confusion when attempting to run a bundle on an ARM architecture robot.

Need some documentation that is maintained over time, that people can refer about common bundling scenarios. Also tied to issue #119 but here are some common ones that i have had to work with.

I) Installing python2.7 packages available in pypi

• Create a source list in etc/ros/rodep/sources.list.d/ so that rosdep can resolve the package form the package.xml file.
• rosdep update, make sure you add the package as a dependency in package.xml
• colcon bundle should be able to pick it up from there.

Eg. Bundling AWSIoTPythonSDK

1. Setup source list in rosdep/sources.list.d folder, and add the details of the OS, package manager and package in the corresponding yaml file as below. Note that the contents of the file being per syntax at https://docs.ros.org/independent/api/rosdep/html/rosdep_yaml_format.html
   The following is the content of the yaml file

awsiotpythonsdk-pip:
  ubuntu:
    pip:
      packages: [AWSIoTPythonSDK]

3. Add the package name as a dependency in package.xml in the rospackage. In this case, it would be

  <depend>awsiotpythonsdk-pip</depend>

4. Now need to run the following commands, and we should be good to roll

rosdep update
rosdep install -y --from-paths src --ignore-src
colcon build
colcon bundle

5. For debugging, you can check if the package is being resolved by running some rodep db commands. Check rosdep --help for info

II) Installing whl files that are not in pypi and are closed sourced.

In some cases, you might have a python package in the form of a whl file, that is not present on pypi. In these cases, the current way to bundle the package is to locally run a pypi server, and ask the colcon bundle to look for packages in local servers.

1. Setup the local pypi server (https://pypi.org/project/pypiserver/ )
   For your wheel file that has the name my_wheel_file.whl what you have to do is

pip intall pypiserver  # this install the pypiserver
mkdir ~/packages
cp my_wheel_file.whl ~/packages/
pypi-server -p 8080 ~/packages

2. Once the local pypi server is setup. You can verify what packages are being hosted by the server at

wget http://localhost:8080/simple

3. Locally, you should now be able to install your package using

pip install my_package_in_wheel_file —extra-index-url http://localhost:8080/simple

4. Make sure that the dependency that you add in the package.xml is resolved in rosdep. Same as adding the dependency that we have done for above python packages

5. Now we can bundle it with colcon, you have to add arguments to pip , as follows

colcon bundle —pip-args —extra-index-url http://localhost:8080/simple

III) Install pip3 packages

This one would not work if you updated the rosdep file to pick pip3 package manager. In this case, you will have to

1. Create a python package file, and in the setup.py file, define the custom package requirements. You can see an example of it here at https://github.com/aws-robotics/aws-robomaker-sample-application-deepracer/blob/ros1/simulation_ws/src/sagemaker_rl_agent/setup.py

The file itself looks something like.

# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# SPDX-License-Identifier: Apache-2.0
from setuptools import setup, find_packages

# Package meta-data.
NAME = 'sagemaker_rl_agent'
REQUIRES_PYTHON = '>=3.5.0'

setup(
    name=NAME,
    version='0.0.1',
    zip_safe=False,
    packages=find_packages(),
    python_requires=REQUIRES_PYTHON,
    install_requires=[
        'boto3==1.9.23',
        'futures==3.1.1',
        'gym==0.10.5',
        'kubernetes==7.0.0',
        'minio==4.0.5',
        'numpy==1.13.3',
        'pandas==0.20.2',
        'Pillow==6.2.0',
        'pygame==1.9.3',
        'PyYAML==4.2b1',
        'redis==2.10.6',
        'rospkg==1.1.7',
        'scipy==0.19.0',
        'tensorflow==1.12.2',
        'rl-coach-slim==0.11.1'
    ],
    entry_points = {
        'console_scripts': [
            'run_rollout_rl_agent=markov.rollout_worker:main',
            'run_local_rl_agent=envs.local_worker:main'
        ],
    }
)

IV) Bundle apt packages from a custom apt source list

We will lead this with an example. In my case, I had to bundle some custom packages from clearpath, the instructions from which are available at https://wiki.ros.org/ClearpathRobotics/Packages

Per the instructions present in the link above, you can install the apt packages by upading the apt source list a

wget https://packages.clearpathrobotics.com/public.key -O - | sudo apt-key add -sudo sh -c 'echo "deb https://packages.clearpathrobotics.com/stable/ubuntu $(lsb_release -cs) main" > /etc/apt/sources.list.d/clearpath-latest.list'sudo apt-get update

On top of the above, you will have to make sure that the packages are resolved using rosdep. This is the same mechanism that you are seeing in the above two sections. An example of the rosdep yaml file is as follows:

boxer_control:
  ubuntu: ros-kinetic-boxer-control
boxer_description:
  ubuntu: ros-kinetic-boxer-description
boxer_firmware:
  ubuntu: ros-kinetic-boxer-firmware

Once you have it, make sure you add the packages in package.xml.

Now we need to update the colcon bundle souces list so that it can pickup the new apt source list.

You will need to either append to the existing source list , or point to a new one.
so for the example above, you will need to create a new file which has the existing contents ( https://github.com/colcon/colcon-bundle/blob/master/colcon_bundle/installer/assets/xenial.sources.list )
and add whatever is in /etc/apt/sources.list.d/clearpath-latest.list

now do **colcon bundle —apt-sources-list new_source_list.list **instead of just colcon bundle

V) update GAZEBO_MODEL_PATH ‘s path

If the models are present in a rospackage, and we to update the GAZEBO_MODEL_PATH, we have the option of doing it by adding some text to the package.xml file as follows
http://answers.gazebosim.org/question/6568/uri-paths-to-packages-in-the-sdf-model-file/

Which in our case would translate to adding the following at the end of the package.xml file

  <export>
    <gazebo_ros plugin_path="${prefix}/lib" gazebo_media_path="${prefix}" gazebo_model_path="${prefix}/models"/>
  </export>

When using --pip3-requirements you need to have a Python package (setup.py) in your workspace or have a dependency on a Python 3 apt package in order to have Python 3 installed for colcon-bundle to use when it attempts to install the pip3 requirements file. Currently, it fails with an error that it cannot find Python3.

Instead of requiring the use of a command line flag we should determine whether any packages have changed their dependencies before we go through the lengthy installer setup process.