Packaging with backend about mavsdk-python HOT 21 CLOSED

We could distribute dronecodesdk_backend through the distro-specific package managers or through a script (e.g. what docker-compose does for Linux), and the wrappers through the language-specific package managers (pip for Python, maven for Java, SwiftPM for Swift, ...).

FWIW for end users I'm a big fan of "one button solutions". So if someone gets swift wrapper from package manager this should also pull the backend. Starting the backend should be possible through the front end. Ditto for Python and other systems.

And note, we don't have to have same solution for all systems.

from mavsdk-python.

So a test version for linux (using manylinux1) can be used with:

pip3 install --user -i https://test.pypi.org/simple/ dronecode-sdk

This is relying on some build system work on the backend side, but the proof of concept is there and it is making good progress.

from mavsdk-python.

Just a random thought - don't ship via pip at all, distribute debian packages for Ubuntu/Debian, rpms for Fedora/Redhat and an AUR Package for archlinux

I'd prefer pip but I can see how that could be a possibility.

from mavsdk-python.

So I am currently working on compiling the backend with manylinux, will post news ASAP.

from mavsdk-python.

That's done! pip3 install mavsdk now works on Windows, macOS and Linux!

from mavsdk-python.

Just a random thought - don't ship via pip at all, distribute debian packages for Ubuntu/Debian, rpms for Fedora/Redhat and an AUR Package for archlinux

from mavsdk-python.

I'd prefer pip but I can see how that could be a possibility.

That would be also my preferred way but that depends on statically linking the backend

from mavsdk-python.

Another question is then for other wrappers (I'm thinking about Java, Swift...). In which case pip doesn't really apply.

We could also potentially have a script (I think @xvzf mentioned that, right?), like docker-compose is doing for Linux (something like sudo curl -L "https://github.com/docker/compose/releases/download/1.22.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose). I don't know how it works exactly, but it seems like they differentiate based on $(uname -s)-$(uname -m).

So setting up the SDK would be a matter of:

1. Install dronecodesdk_backend.
2. Install your favorite language wrappers (Python, Swift, Java).
3. Run dronecodesdk_backend.
4. Start using your wrappers.

It is a bit different on iOS and Android, as packaging is different (i.e. you cannot share one backend binary with different wrappers), so there we can actually provide the backend binary.

We could distribute dronecodesdk_backend through the distro-specific package managers or through a script (e.g. what docker-compose does for Linux), and the wrappers through the language-specific package managers (pip for Python, maven for Java, SwiftPM for Swift, ...).

from mavsdk-python.

We give the responsibility to the user (who can build or install the backend somehow), and distribute the python wrappers over pip.

Definitely need to supply simple solution of pre-built binaries "somehow".

But even so we should provide clear instructions on how to set this up manually for unsupported platforms.

from mavsdk-python.

Probably a silly idea, but could all this stuff be setup and run in some sort of docker package on whatever system?

from mavsdk-python.

The python package can include a static lib and backend daemon, the package can build all the dependencies including proto files in a single build step via setup.py, this has the benefit of dealing with incompatibilities between backend and client, this might mean we need to add the backend as a submodule for the build step?

I found an example of a python api binding similar to ours where they need to build and distribute static libraries when they are building the python package since they distribute everything together, they are also adding a bit more of complexity to the mix allowing dynamic libs.

https://github.com/unicorn-engine/unicorn/blob/master/bindings/python/setup.py#L275-L279

In other words, we need to look into distutils if we want to distribute as a python package through pip.

from mavsdk-python.

Probably a silly idea, but could all this stuff be setup and run in some sort of docker package on whatever system?

You mean that you would run the backend as a docker container, and connect the wrappers to it? That would solve the packaging issue (we would only have one container to maintain), but it would add its lot of networking complexity (e.g. the backend running in your docker container would likely not receive the mavlink broadcasts from the drone), and it doesn't solve the problem for people who don't want to run docker.

FWIW for end users I'm a big fan of "one button solutions". So if someone gets swift wrapper from package manager this should also pull the backend. Starting the backend should be possible through the front end. Ditto for Python and other systems.

I generally agree. My fear is just that we may end up having some binaries pulled by the language package managers (pip, maven, swiftpm, ...), but still having manual build/install instructions for unsupported platforms.

Don't get me wrong, that's almost exactly what we do on iOS and Android. But we have a limited number of platforms to support there, and their packages are made for that (e.g. making a fat binary to support all iOS devices is actually not difficult). I wouldn't know how to do that for maven or swiftpm (is it even possible?).

And finally, considering that one might want to run the backend onboard the drone, we will need to provide a way (and corresponding instructions) to install the backend standalone anyway.

It kind of sounds to me like we anyway need to find a way to distribute the backend standalone (for unsupported platforms and for running the backend onboard the drone), and having it more integrated into some languages (e.g. @mrpollo's solution above) would be some kind of convenience layer (that could therefore be solved after the backend distribution, in a per-language basis). Is that correct, or am I missing something here?

NOTE: I have no clue how a "standalone backend" would be run on Windows. On Linux/macOS, it is a matter of running $ ./backend_bin. Do you guys have insights regarding Windows?

from mavsdk-python.

Probably a silly idea, but could all this stuff be setup and run in some sort of docker package on whatever system?

That might actually not be the worst idea! We could do this and provide a build script for the backend? What do you think @JonasVautherin ?

...but it would add its lot of networking complexity (e.g. the backend running in your docker container would likely not receive the mavlink broadcasts from the drone),...

I actually did resolve that, you can forward the broadcast to the container network

from mavsdk-python.

I actually did resolve that, you can forward the broadcast to the container network

Yeah but it means that you have to actively do something regarding the network. And that we then depend on docker for running the SDK, right? And running docker on macOS and Windows is not native, is it? And what about other architectures? Not sure how it would run on ARM, for instance...

from mavsdk-python.

Yeah but it means that you have to actively do something regarding the network. And that we then depend on docker for running the SDK, right? And running docker on macOS and Windows is not native, is it? And what about other architectures? Not sure how it would run on ARM, for instance...

Okay, yes. Mac and Windows has no native support, that's right. Docker on ARM (at least ARM64) is working just fine but that is not a viable solution since we can't support all platforms with this approach.

Did you dig into statically linking the backend so it does not depend on dynamic libraries? If we can get that working we should be fine and do the same thing as @mrpollo dronekit-sitl does.

from mavsdk-python.

When I was trying the pybind solution via pip I found that it needs the whole toolchain installed, so it's far from optimal, you can see it here:
https://github.com/julianoes/DronecodeSDK-Python#installing-the-python-wrappers

from mavsdk-python.

Note to self, I should try to compile it in manylinux2010.

from mavsdk-python.

Hey guys did we decide what to do?

from mavsdk-python.

We know what to try and how to proceed but it's tricky because manylinux1 is too old to compile what we need and manylinux2010 is not released yet but it looks like it's close, so we should try to get ready for it.

And I should add manylinux are containers to build binaries that then can be distributed via pip and should run on most linux platforms.

from mavsdk-python.

Are we sure that manylinux1 is too old?

from mavsdk-python.

Hm, it's gcc 4.8, it might work.

from mavsdk-python.