Do we really need a virtualenv per component? about st2-packages HOT 9 CLOSED

We already have a common base component that is required for any service...st2common. I would like to see st2common create the virtualenv, and the other components leverage that.

from st2-packages.

In this case I vote for the least path of resistance. If using a shared virtual-env based of st2common works for all components lets use that one.

Isolation is good but in this case I really do not see the point since they are pieces of the same software.

The only piece that must work standalone is st2client.

from st2-packages.

I am a bit confused about this whole thing.

Single virtualenv is only really applicable when all the services are ran on a single server / in a single container (well potentially if you run all the containers on a single container you could do some share hacks, but that's missing the point anyway)? Or am I missing something?

If user is going to run service (package) per container / service, single virtualenv doesn't really work.

So unless I'm missing something, single virtualenv is not feasible in a standard, distributed deployment anyway.

from st2-packages.

@Kami See Patrick's point. Every service needs st2common. So we could in theory can have that host the virtualenv for the actual services.

from st2-packages.

@dennybaa (FYI available since Monday, 11th Jan) should have insight in this topic and I know his decisions are based on long research, experiments and thinking.

Instead of fixing st2 code which is indeed an option, can we switch to a single virtualenv for all components? Is this possible with dh-virtualenv?

One of @dennybaa workaround was to use st2bundle package, which has everything shared and contains all st2 components in 1 single package.
No problems with virtualenv in this approach, but it has all downsides of 1 component != 1 package, discussed before.

FYI st2bundle is already used for building Docker images.

Related discussions might be useful here:
#50
StackStorm/st2#2322
StackStorm/st2#2339

from st2-packages.

New questions:

1. Can we have a package per component but a single virtualenv from st2common for everything?
2. If it turns out that we have to have a single virtualenv per component, how do we fix packs.install?

from st2-packages.

@lakshmi-kannan Doing the bad job here with broken phone.
But to help, I found this @dennybaa vs @Kami log. Please read this discussion carefully:
https://stackstorm.slack.com/archives/stackstorm/p1450949049018860

It will bring better understanding. From that log I can assume:

1. Can we have a package per component but a single virtualenv from st2common for everything?

No.
Or hack dh-virtualenv (could be an option). See example
Every package is isolated. I think, that's advantage and one of the reasons why it works so good to have predictable, fully self-sufficient and bundled package installation.

1. If it turns out that we have to have a single virtualenv per component, how do we fix packs.install?

By bringing changes in st2, one of the ideas: StackStorm/st2#2339 (thanks @Kami for efforts)

So the best is to plan Monday meeting with @dennybaa who knows the packging at low-level and team who knows the st2 at low-level and find the consensus.

From 24 Dec:

from st2-packages.

Hello, guys @lakshmi-kannan , @DoriftoShoes, @manasdk, @armab , @Kami !

I agree with all points that @lakshmi-kannan and @manasdk says

Isolation is good but in this case I really do not see the point since they are pieces of the same software.

Yeah it's that what I also prefer to have single python virtual environment. But what it actually means:
we MUST have single package (preferably named st2), containing all components and services! Because otherwise when we use multiple packages we get multiple virtualenvs automatically, having only one venev and multiple packages IS NOT POSSIBLE.

Also I can't clearly get what @Kami says

If user is going to run service (package) per container / service, single virtualenv doesn't really work.

So unless I'm missing something, single virtualenv is not feasible in a standard, distributed deployment anyway.

It's better we give up idea of refactoring st2 code StackStorm/st2#2339. Since we don't really need different virtualenvs, since our code base uses the same versions of pip modules for all of our components (@manasdk).

Also I want to say that package != component != service, just having multiple packages with its own virtual env just for sake of "isolation" doesn't really make sense.
When isolation is required we should make ability to enable/disable components upon installation or after! There are different ways to achieve this at least what comes straight into my mind:

• answer file (so during manual even dialog is available), don't know how about rpms.
• add some functionality to st2ctl to enable/disable component...

from st2-packages.

Closing this discussion as we have made lot of progress on the bundled virtualenv.

from st2-packages.