At the moment, we're manually deploying images to Docker Hub.

CircleCI should automatically deploy images to DockerHub on any push to master.

Currently, StackStorm Docker images rely on Ubuntu 18.04 LTS with py3.6.

Because Python 3.6 is EOL in 2021, we need to switch to Ubuntu 20.04 LTS that's shipped with py3.8 for running StackStorm environment.
https://thenewstack.io/time-to-say-goodbye-python-3-6-is-end-of-life/

I noticed and issue with the nginx proxy configuration while running some more tests related to #28 (see #28 (comment))

The issue is caused by the fact that nginx resolves the upstream hosts during the startup and caches the response until the next restart. This leads to issues if any of the upstream hosts gets a new IP address due to a restart for example.

I will create a PR to address this issue.

We noticed openssh keys on few st2 docker containers which is causing security vulnerabilities when scanned.

# docker run -it stackstorm/st2stream:3.3dev bash
--
Welcome to StackStorm HA v3.3dev (Ubuntu 18.04.4 LTS GNU/Linux x86_64)
* Documentation: https://docs.stackstorm.com/
* Community: https://stackstorm.com/community-signup
* Forum: https://forum.stackstorm.com/
* Enterprise: https://stackstorm.com/#product Notice! It's recommended to use st2client container to work with StackStorm cluster.st2@938d4068b3ec:/opt/stackstorm$ cd /etc/ssh
st2@938d4068b3ec:/etc/ssh$ ls
moduli      sshd_config         ssh_host_ecdsa_key.pub  ssh_host_ed25519_key.pub  ssh_host_rsa_key.pub
ssh_config  ssh_host_ecdsa_key  ssh_host_ed25519_key    ssh_host_rsa_key          ssh_import_id

These are generated by the openssh-server package installation as a dependency to st2.deb.
They are not used as you can't run SSH server in StackStorm Docker images and don't want these keys in there, pre-packaged and distributed Docker image is not a VM.

Moved from github.com/stackstorm/st2enterprise-dockerfiles#8

We'll need to investigate the ways to cleanup the resulting Docker images.

Not sure about the docker layer squashing, but for example, after working closer, it's not the StackStorm takes a lot of space, but its package dependencies are huge.

Here is the full list of dependencies if we try to install st2enterprise under clean ubuntu:xenial (31MB):

root@0a1cd8b86d1f:/# apt-get install bwc-enterprise
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  bwc-ui dh-python file git git-man ifupdown iproute2 isc-dhcp-client isc-dhcp-common less libatm1 libbsd0 libc-dev-bin libc6-dev libdns-export162 libedit2 liberror-perl libexpat1
  libexpat1-dev libffi-dev libgdbm3 libisc-export160 libmagic1 libmnl0 libmpdec2 libperl5.22 libpopt0 libpython-dev libpython2.7 libpython2.7-dev libpython2.7-minimal
  libpython2.7-stdlib libpython3-stdlib libpython3.5-minimal libpython3.5-stdlib libssl-dev libssl-doc libwrap0 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxext6 libxmuu1
  libxtables11 libyaml-0-2 linux-libc-dev manpages manpages-dev mime-support ncurses-term netbase openssh-client openssh-server openssh-sftp-server patch perl perl-modules-5.22
  python2.7 python2.7-minimal python3 python3-chardet python3-minimal python3-pkg-resources python3-requests python3-six python3-urllib3 python3.5 python3.5-minimal rename rsync
  ssh-import-id st2 st2-auth-ldap st2flow sudo tcpd wget xauth zlib1g-dev
Suggested packages:
  libdpkg-perl gettext-base git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk gitweb git-arch git-cvs git-mediawiki git-svn ppp rdnssd iproute2-doc resolvconf
  avahi-autoipd isc-dhcp-client-ddns apparmor glibc-doc man-browser ssh-askpass libpam-ssh keychain monkeysphere rssh molly-guard ufw ed diffutils-doc perl-doc libterm-readline-gnu-perl
  | libterm-readline-perl-perl make python2.7-doc binutils binfmt-support python3-doc python3-tk python3-venv python3-setuptools python3-ndg-httpsclient python3-openssl python3-pyasn1
  python3.5-venv python3.5-doc
The following NEW packages will be installed:
  bwc-enterprise bwc-ui dh-python file git git-man ifupdown iproute2 isc-dhcp-client isc-dhcp-common less libatm1 libbsd0 libc-dev-bin libc6-dev libdns-export162 libedit2 liberror-perl
  libexpat1 libexpat1-dev libffi-dev libgdbm3 libisc-export160 libmagic1 libmnl0 libmpdec2 libperl5.22 libpopt0 libpython-dev libpython2.7 libpython2.7-dev libpython2.7-minimal
  libpython2.7-stdlib libpython3-stdlib libpython3.5-minimal libpython3.5-stdlib libssl-dev libssl-doc libwrap0 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6 libxext6 libxmuu1
  libxtables11 libyaml-0-2 linux-libc-dev manpages manpages-dev mime-support ncurses-term netbase openssh-client openssh-server openssh-sftp-server patch perl perl-modules-5.22
  python2.7 python2.7-minimal python3 python3-chardet python3-minimal python3-pkg-resources python3-requests python3-six python3-urllib3 python3.5 python3.5-minimal rename rsync
  ssh-import-id st2 st2-auth-ldap st2flow sudo tcpd wget xauth zlib1g-dev
0 upgraded, 82 newly installed, 0 to remove and 2 not upgraded.
Need to get 90.5 MB of archives.
After this operation, 360 MB of additional disk space will be used.

First step

The first dependency candidate to remove is openssh-server:

root@0a1cd8b86d1f:/# apt-get install openssh-server
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  dh-python file libbsd0 libedit2 libexpat1 libmagic1 libmpdec2 libpython3-stdlib libpython3.5-minimal libpython3.5-stdlib libwrap0 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6
  libxext6 libxmuu1 mime-support ncurses-term openssh-client openssh-sftp-server python3 python3-chardet python3-minimal python3-pkg-resources python3-requests python3-six
  python3-urllib3 python3.5 python3.5-minimal ssh-import-id tcpd wget xauth
Suggested packages:
  libdpkg-perl ssh-askpass libpam-ssh keychain monkeysphere rssh molly-guard ufw python3-doc python3-tk python3-venv python3-setuptools python3-ndg-httpsclient python3-openssl
  python3-pyasn1 python3.5-venv python3.5-doc binutils binfmt-support
The following NEW packages will be installed:
  dh-python file libbsd0 libedit2 libexpat1 libmagic1 libmpdec2 libpython3-stdlib libpython3.5-minimal libpython3.5-stdlib libwrap0 libx11-6 libx11-data libxau6 libxcb1 libxdmcp6
  libxext6 libxmuu1 mime-support ncurses-term openssh-client openssh-server openssh-sftp-server python3 python3-chardet python3-minimal python3-pkg-resources python3-requests
  python3-six python3-urllib3 python3.5 python3.5-minimal ssh-import-id tcpd wget xauth
0 upgraded, 36 newly installed, 0 to remove and 2 not upgraded.
Need to get 7,766 kB of archives.
After this operation, 44.2 MB of additional disk space will be used.

See why we do it: StackStorm/st2-packages#379. While we definitely need ssh client, can probably get rid of ssh server requirements for a Docker-friendly install.

Additionally, ssh server adds more burden to secrets handling, see from the installation:

Creating SSH2 RSA key; this may take some time ...
2048 SHA256:qIk57cnYNyPQf6O09jRosiYTJhKAszrp28GmU9oAKKI root@00dd8fb34e12 (RSA)
Creating SSH2 DSA key; this may take some time ...
1024 SHA256:6Pfoc2w3G5yEoWtNZyU+tw7LS2M12VAf0uCcMzXDbgc root@00dd8fb34e12 (DSA)
Creating SSH2 ECDSA key; this may take some time ...
256 SHA256:2AuYwxhCeRM7sj7caaaT8EPM1TqpCZ3AjMcRpb72n00 root@00dd8fb34e12 (ECDSA)
Creating SSH2 ED25519 key; this may take some time ...
256 SHA256:pcCj21tGyRQ+nqJS1KCVKvk+rSoIooLNTCXBvd6lcy4 root@00dd8fb34e12 (ED25519)

Releasing st2 v3.2.1 should lead to Docker Hub deployment for the following docker tags: 3, 3.2, 3.2.1 which is a common practice.

This way someone could rely on a wider version interval or pin to specific st2 version depending on their needs.

More info:

• https://github.com/StackStorm/st2-docker/blob/2c2032595a03394b56fbf9cc16114d823a540471/VERSIONING.md
• StackStorm/st2-docker#78
• https://github.com/StackStorm/st2-docker/blob/2c2032595a03394b56fbf9cc16114d823a540471/bin/common.sh
• https://github.com/StackStorm/st2-docker/blob/2c2032595a03394b56fbf9cc16114d823a540471/bin/deploy.sh
• https://github.com/StackStorm/st2-docker/blob/2c2032595a03394b56fbf9cc16114d823a540471/bin/gatekeeper.sh

Sign the StackStorm Docker images during the build CI/CD process so image consumers can ensure that the StackStorm images they pull from Docker Hub repository are signed.

See https://docs.docker.com/engine/reference/commandline/trust/ and https://docs.docker.com/engine/security/trust/content_trust/ for more info.

Extreme Networks opensourced EWC with Enterprise functionality https://stackstorm.com/2020/05/27/extreme-networks-donates-ewc-to-linux-foundation/

Currently StackStorm OSS Docker images are hosted on public Docker hub, while Enterprise Docker images has their own infrastructure, Registry, auth process and specific Dockerfiles code.

We'll need to rework the OSS Dockerfiles to include the code/adjustments made for ex-Enterprise Dockerfiles.
New Dockerfiles should also bundle the previously closed-source LDAP packages into image.

Remove st2resultstracker service from the packaging & startup logic as it's not needed since st2 v3.3.0 due to mistral removal.
See StackStorm/st2#5070 for more context.

This should be easy & fun task, if anyone from @StackStorm/contributors wants to help removing that.

Tie this repo needs to be integrated with StackStorm release automation workflows https://github.com/stackstorm/st2cd

Ideally is to follow existing established practice how st2 does a release with automated version pinning/upgrade and stable branch creation.

This includes:

• dev build
  • relies on latest master
  • nightly/cron builds
  • pushes latest dev images like 3.0dev
• production build
  • relies on vX.Y branch
  • pushes stable images like 2.9
• version update
• branch creation

So StackStorm release automation should update version in Makefile

Note, for simplicity and to follow K8s best practices we won't tag images with latest tag yet, offloading st2 version pinning on user's shoulders as deliberate decision. At least for this initial stage.

Somewhat related to #31

Start pushing latest Docker tags to Docker hub so users could use that instead of relying on pinned st2 version.

Gotchas:

• Do version sort, upload latest tag based on most recent real version
• Make sure to exclude dev versions, only st2 stable versions fit to be latest.
• Some scripting examples from the st2-docker project should help: https://github.com/StackStorm/st2-docker/blob/927de5adfcd5c5e95bc811a0839ed998f541b436/bin/deploy.sh#L40-L43

I see here in core.send_mail that there is a usage of the sendmail binary.

but when I use the st2actionrunner container I see that we are not installing the sendmail package.

Eventually include st2chatops Dockerfile with nodejs and hubot.

Requires making st2chatops Docker-friendly to read ENV vars outside of st2chatops.env, see blocker StackStorm/st2chatops#50.
Second is that st2chatops package can't be installed without st2 package, needs a quick fix: StackStorm/st2chatops#31.

As it is required to enable st2chatops by setting the environment variable ST2_CHATOPS_ENABLE before using the container, this should be added to the readme as a required variable.

linux pack is part of the core st2.
Need to make sure that tools and binaries used in that pack are present in base image or st2actionrunner image.

# st2 run linux.dig hostname=localhost
.
id: 5d014bb2bc1c1500014a3036
status: failed
parameters: 
  hostname: localhost
result: 
  exit_code: 1
  result: None
  stderr: "Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/python_runner/python_action_wrapper.py", line 333, in <module>
    obj.run()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/python_runner/python_action_wrapper.py", line 192, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/linux/actions/dig.py", line 44, in run
    stdout=subprocess.PIPE)
  File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
"
  stdout: ''

Moved from https://github.com/StackStorm/st2enterprise-dockerfiles/issues/47

In a normal non-dockerized deployment, typical service logs vs AUDIT logs from the services are divided into different logfiles.

In Dockerized environment the established practice is to log to stdout/stderr.
At the moment we log to stdout only INFO loglevel events.
Consider including AUDIT logs as well.

We should publish new development version of all the Docker images to Docker hub on each merge to StackStorm/st2 master or at least on a daily / weekly basis (similar to the unstable deb and rpm packages).

Right now, if I want to use latest bits from master, I need to build all those images myself and maintain them in my own registry which is not great.

A lot of other projects publish such images with dev / master / unstable tag or similar.

Depending on "Error" message as a failure indicator adds some knowledge overhead in requirements that could be broken in the future. What if 3rd Maintainer comes, edit the determine_needed_tags.sh script and doesn't add "Error" message for the failure case.

We can at least reinforce it with determine_needed_tags.sh || echo Error to fit your solution.

The comment above was originally posted by @armab in #41 (comment) and he actually has good point there. The current solution requires some extra knowledge and might confuse maintainers.

When someone SSH-in into a running container EWC/Enterprise is mentioned.

Remove those references as EWC is opensourced now:
https://stackstorm.com/2020/05/27/extreme-networks-donates-ewc-to-linux-foundation/

Is there a way to enable Mistral on this stack ?

3 out of 4 hunks FAILED -- saving rejects to file /etc/nginx/conf.d/st2-http.template.rej
The command '/bin/sh -c if [ "${ST2_VERSION#*dev}" != "${ST2_VERSION}" ]; then     ST2_BRANCH=master;   else     ST2_BRANCH=v${ST2_VERSION%.*};   fi   && echo ST2_BRANCH=${ST2_BRANCH}   && apt-get install -y patch gettext-base   && curl -sf https://raw.githubusercontent.com/StackStorm/st2/${ST2_BRANCH}/conf/nginx/st2.conf -o /etc/nginx/conf.d/st2-http.template   && cp /etc/nginx/conf.d/st2-http.template /etc/nginx/conf.d/st2-https.template   && patch /etc/nginx/conf.d/st2-http.template < /tmp/st2.conf-http.patch   && patch /etc/nginx/conf.d/st2-https.template < /tmp/st2.conf-https.patch   && rm -f /etc/nginx/conf.d/default.conf   && rm -f /tmp/st2.conf.patch   && rm -f /tmp/st2.conf-http.patch   && rm -f /tmp/st2.conf-https.patch' returned a non-zero code: 1

It looks to have started failing when the nginx conf.d was changed. Patch files in st2-dockerfiles need updating.

Since 3.0 stackstorm introduces a new service st2scheduler, - we'll need to codify it.

Related StackStorm/stackstorm-k8s#31

Problem description

I use git clone code to go locally, and I get the following error with make build
[root@node-01 st2-dockerfiles-master]# make build Failed to identify the tags to be set. No images were tagged due to an error when determining the correct tags: Error: Unexpected HTTP statuscode for https://registry.hub.docker.com exit 1 make: *** [verify_tag_update_flag] 错误 1

I got the same error when I visited https://registry.hub.docker.com/v1/repositories/stackstorm/st2/tags directly.

Moved from https://github.com/StackStorm/st2enterprise-dockerfiles/issues/57

The current/default StackStorm tries to reconnect to Mongo or RabbitMQ on failure.
This is undesired behavior in K8s environment where engine handles failover by rescheduling exited process.
So the desired configuration is to exit fast any st2 service on failure without reconnection and let K8s handle it.

Luckily, these settings are configurable in st2.conf:
https://github.com/StackStorm/st2/blob/master/conf/st2.conf.sample#L179
https://github.com/StackStorm/st2/blob/master/conf/st2.conf.sample#L109

Hardcode them in default st2.conf (during the Docker image build) to not reconnect, but exit st2 on failure.

Spot checked a few of these images on https://hub.docker.com/u/stackstorm and didn't find a release tag for v3.8.0.

Example from the st2web image tags section:

The most recent release tag was 3.7.0. Following dev images available:

• 3.8dev
• 3.9dev

Also looks like latest was published 7 mo ago (probably at v3.7.0)

I can do a PR, but wanted to ask this as a question first. I want to build a custom st2 base image and then build the remaining st2 images from that image instead of upstream stackstorm/st2:${ST2_VERSION}

For background, I went to build the docker images myself in order to get bionic/python3.6 (see #16), which required making changes to the base image. I want to give the st2 base image and all subsequent ones a tag of "3.2.0-dev-bionic", which I can do by exporting DOCKER_TAG before running "make build".

The problem is that each Dockerfile has a line of FROM stackstorm/st2:${ST2_VERSION}. Which means they won't build from my new base image, but rather upstream base image.

My workaround is to additionally pass DOCKER_TAG into each Dockerfile and update the from line.

ARG ST2_VERSION
ARG DOCKER_TAG
FROM stackstorm/st2:${DOCKER_TAG}

Then in the Makefile:

        for component in st2*; do \
                docker build \
                        --no-cache \
                        --build-arg ST2_VERSION=${ST2_VERSION} \
                        --build-arg DOCKER_TAG=${DOCKER_TAG} \
                        --tag stackstorm/$$component:${DOCKER_TAG} \

This seems reasonable to me. If someone is needing to put a specific tag on the base, they will most likely want to build their other images from that tag. If this seems reasonable or useful to others, I will go ahead and create a PR.