Basic non-ansible container preferred about docker-splunk HOT 13 CLOSED

Thanks for your feedback, @srevenant. What specific patterns do you believe are being violated? How is using the ansible binary to process YAML architecturally any different from using the bash binary to process .sh? In general, our preference was just for using a more declarative syntax and having to write less code. Keep in mind that the container is packaging an unmodified Splunk Enterprise binary, which itself isn't designed to be container-native. So certain less-than-ideal workarounds are required regardless of which command line tool you use to managing provisioning under the hood. None of this should make any difference to users, or consumers of the container itself.

As @outcoldman said, we did try creating an alpine-based container, but this didn't seem to make sense in the end because of Splunk's libc requirements. Once you build and package in a minimal version of libc into alpine, the image size isn't much smaller (a few percent). Debian is also just better tested overall, so we feel most comfortable using that as a base. Any specific reasons for wanting to use CentOS or Alpine, besides just personal preference or image size?

from docker-splunk.

Having it in the Dockerfile makes it portable and follows the community paradigm--the original idea of Dockerfile is to get away from CMS tools and just use simple self-describing steps. Taking it into another tool (any CMS) leaves the common tool and you are introducing something new. It's kind of like trying to use containers as servers, on some level. If you haven't seen 12factor.net, check it out. I've also discussed this topic in Idiomatic Container Patterns.

If you are in an environment that doesn't let you just run any vendor's tools directly from docker (any environment with regulatory compliance) then you are referencing the Dockerfiles but doing your own builds. When that service now integrates a bunch of third tools, it becomes increasingly less valuable.

As for CentOS, the same--with regulatory compliance a requirement is we have the assurance that the vendor source is as we expect. Like it or not, in the enterprise space Redhat has a bigger footprint (and a lot of downstream images use it, like AWS's Linux is just redhat/centos based as well). It's not a requirement for CentOS, just easier since we already have CentOS and Alpine in our approved vendors list.

from docker-splunk.

I just don't want ansible in a container. Period. Ever. It's nothing personal with Ansible, and has to do with the heart of what containers are. You just don't add server management tools into a container, it's just not the "idiomatic way."

It makes them a non-starter, and I can't even reference things in them to build my own containers, which makes it even harder to reference. Using a bash script is simple. It worked fine before, and doing the other stuff is basically saying you expect people to learn your framework on top of anything else they are doing currently. I understand it makes your life easier, but it makes everybody else's harder.

It just means more friction and less adoption, with more people making home brewed containers.

Ideally, you just offer a Dockerfile, and we can build our own images. I don't want to use your image, I want to build one.

from docker-splunk.

@srevenant - I would like to setup up a call to discuss your feedback. Please can you send me a note at [email protected] with possible time-slots that work for you ?

@outcoldman - Will include you in this discussion as well.

from docker-splunk.

I agree with this that.

The legacy (without ansible) image is here https://github.com/splunk/docker-splunk-legacy, but it is not officially supported

Also, will you be supporting an Alpine build soon? CentOS?

I believe Splunk statement was that their base image will be Debian.
Alpine for sure cannot be used, considering that Splunk does not work with musc libc.

from docker-splunk.

For this initial release, we intentionally decided to take a "lift and shift" approach and not make any modifications to the Splunk binaries. Taking a "traditional" enterprise application like Splunk and packaging it into a container requires very different considerations from what one faces when building a new container-native app from scratch. In particular, Splunk's configuration is both extremely flexible and complex (especially once you move from single instances to clusters), and so the translation into something that supports the "configs at runtime" pattern is non-trivial. It is certainly not possible to do this using the Dockerfile alone.

I've observed that most containers implement "configs at runtime" by writing custom code in bash or python, bundling that code into the container and then using an interpreter to execute this code via the entrypoint. The new Splunk docker container also supports "configs at runtime," but we took an admittedly non-standard approach of replacing code with YAML and using ansible as a higher-level interpreter. I argue there is no difference architecturally; it's really just an implementation detail regarding the selection of language and tool. No argument that it would be better if splunkd did this work for us, and perhaps one day we will get there.

Thanks for your input on CentOS. We have the framework in place to support different base images, but given resource constraints it was necessary to start with just one. We'll keep a watch on demand for this and consider extending it out in future releases.

from docker-splunk.

@mikedickey it is not only a question of how you configure the image but also security. Fewer dependencies you have inside your container, fewer vulnerabilities you are going to have. It is silly to have now TWO python runtimes inside of the container (one is shipped with Splunk, another one you install for ansible).

Also, this is a python version that you have inside the image

/usr/bin/python --version
Python 2.7.13

It is 2 years old (see https://www.python.org/downloads/release/python-2713/).

Even Splunk has a newer version of Python (https://www.python.org/downloads/release/python-2715/)

/opt/splunk# ./bin/splunk cmd python --version
Python 2.7.15

Also, looking at https://github.com/splunk/docker-splunk/blob/master/base/debian-9/install.sh#L16-L47 - you install wget and curl and pretty much anything that you think will be required for support.

This is cve log that I got from running https://github.com/aquasecurity/microscanner

cve.log

• ansible - 1 vulnerability
• busybox - 9 vulnerabilities
• python-crypto - 1 vulnerability
• python2.7 - 4 vulnerabilities

There are some others as well, but I assume they come with the base image. So in current splunk/splunk-7.2.0 there are at least 15 vulnerabilities just to make the bootstrap process simpler.

I would propose to have:

1. Base image (just the things, that require you to run Splunk)
2. Splunk Enterprise built on top of base
3. Splunk UF built on top of base
4. Splunk Enterprise with Orchestrator that will include ansible for people who need it
5. Splunk UF with Orchestrator that will include ansible for people who need it

In that way, we will be able to use images (2) and (3) with the orchestrators that we like, or maybe don't use any at all, and just build our own UF images on top of (2) and (3) and embed our configurations in the images.

from docker-splunk.

Hey guys, this debate is taking a turn down roads that shouldn't be in a public issues thread. Happy to facilitate some internal discussions / email / debates over webex. Reality is, we can't make every single person happy, but we can try to facilitate multiple people's processes. @amitmookerjee is currently the pm and you can reach out to him directly for some of these support / discussion points.

Bringing back to the major points @outcoldman has:

In regards to the exploit lists @outcoldman, I'm happy to remove several of these tools (like busybox) entirely, but you know full well that several things in a container need to be there to support splunk. network-tools, dns-tools, visual editors etc. Splunk will make the commitment to update to newer packages as soon as we can. (even as we speak there's internal testing on addressing what we need to do for moving to ansible 2.7) Python 2.13 that's outside of splunk is required due to ansible (and is a package dependency of ansible), and has several modules that splunk's internally packaged python does not. Please don't forget, Splunk's Python IS NOT an officially supported python and has several packages removed and modified. Would you rather us move to python3 (ansible supported) or stick with 2.7, just go to a new version? Happy to make the adjustment to either and install python ourselves instead of letting apt auto install it. Busybox is pearly to add tools you'll need to manage your splunk instances in the smallest form factor possible. Do you prefer not having these tools by default and you install your own? Busybox itself provides all of the tools, at almost a fraction of just the size of VIM itself. Welcome to some feedback here.

from docker-splunk.

@amitmookerjee @lephino I will speak mainly for myself (and I assume I share the same concern with @srevenant), I would really want to have a slim Splunk image, that does not have any support tools or orchestrator tools. I would prefer to have an image in between base and image with ansible, that has only Splunk, similar to what we had in the community supported image.

• Layer 0. Base image (just the things, that require you to run Splunk)
• Layer 1. Splunk Enterprise built on top of base
• Layer 1. Splunk UF built on top of base
• Layer 2. Splunk Enterprise with Orchestrator that will include ansible for people who need it
• Layer 2. Splunk UF with Orchestrator that will include ansible for people who need it

Do you think it is doable?

from docker-splunk.

I think that's more of a question of what Splunk support can maintain and say is validated. I'll leave @amitmookerjee to deal with the funding / support side of the house, but I don't think this is the right forum to have these discussions on. Let's get some use-case driven feedback in the PRD and go down the proper channels for this discussion and remove it from a public issues thread. Obviously what you mention is possible, it really comes down to Splunk's overall direction and I leave those discussions to the PM's. @amitmookerjee can you reach out to these two specifically and gather more requirements? Since there doesn't seem to be any current actionable item out of this thread we should probably close this and track it internally on our feature tracking system.

from docker-splunk.

@lephino to be honest, not sure if closing this thread is the best way to handle it. Can we keep it open, at least for now? And close it if it will become inactive.

If you want to schedule a call to discuss it, I will be happy to join.

from docker-splunk.

@amitmookerjee thanks!

@amitmookerjee @lephino as an idea, we can also bring back the legacy images as a lightweight image, and make the configurations to be community supported, but assuming that customers still will be able to get support for the product itself. Do you think that will be visible?

from docker-splunk.

The old github repository still exists at https://github.com/splunk/docker-splunk-legacy

@srevenant anyone who would like to build their own containers is certainly welcome to continue using that, or to just create their own from scratch. The goal of this effort is to serve customers who would rather have Splunk build, test and support it for them, end-to-end. We're trying to be transparent, but I would imagine the details of what's "under the hood" is as irrelevant for most customers as (for example) which Javascript libraries we choose to embed within Splunk's UI.

@outcoldman the security scan results are concerning. We did have the published image scanned and certified by Docker, so it's curious why these vulnerabilities would not have shown up. I recommend creating a new issue for that, but we'll dig into it either way.

I'm closing this thread out and encourage anyone else with concerns to reach out to @amitmookerjee or their sales engineers for follow-up.

from docker-splunk.