Tactical doc updates for our products in Docker / Kubernetes about stack-docs HOT 35 CLOSED

@DanRoscigno yep, happy to look over any updates. Also /cc @Crazybus here for visibility re: Helm documentation.

from stack-docs.

@tylerjl given the timing of these updates (they're getting pushed before the Helm public beta goes live) we won't be adding in Helm updates/references to the docs just yet (tracking these updates separately). Regardless, thanks so much for flagging!

from stack-docs.

Nope, all the docker commands look pretty sane. I think it serves as a good example of how to run things 🚢

from stack-docs.

Note that you cannot use the conditions within the example blocks so you will need a separate block for each command as you show above.

from stack-docs.

I've checked the examples in the repo, most of them look neat. One note though, add_docker_metadata processor should not be needed when you are using autodiscover, as autodiscover automatically enriches events: https://github.com/DanRoscigno/Beats-container-configs/blob/master/Docker/match-configs/filebeat.yml#L46

from stack-docs.

@tylerjl @exekias
What do you think about using a short lived setup pod like this to setup the dashboards, ML jobs, index pattern, etc? Similar to what I did for Docker, run the setup separate from the pod that processes data.

👍 this is what I use in my demos: https://github.com/exekias/beats-kubernetes-demo/blob/master/kubernetes/metricbeat-daemonset.yaml#L288-L350

from stack-docs.

@liztauscher : @jethr0null and I met after Kate's call. We are good to go with links from the product downloads to the docs for the Docker and Kubernetes lists up top (the checkboxes). PM, Infra, Support, and Engineering need to work out the details for adding links for Helm charts and new Docker / Kubernetes / OpenShift, etc. in the future. This will happen later.

@dedemorton : I expect the PR to go in on Monday, I think I am done but need to test things one more time on Monday. Once you and engineering look at the PR (assuming it is accepted), can you let @kate-smith know so she can plan with the web team to make the links from the Download pages to the Docker and Kubernetes docs pages?

from stack-docs.

PR is merged. I will check the docs in a few hours.

from stack-docs.

@DanRoscigno - everything set here? Webteam is ready to push the Docker & Kubernetes update live with the Helm Chart push at 8AM PT.

CC @liztauscher

from stack-docs.

Good to go.

from stack-docs.

Sure @DanRoscigno, count me in!

from stack-docs.

Links:

getting started thoughts

from stack-docs.

Checking in here - is Nov 30 still the target date here? I want to make sure it is on the calendar for Web Team. Thank you!

from stack-docs.

@DanRoscigno Nov 30 is kind of aggressive for me. I'm still wrapping up some 6.5 work and the Thanksgiving holiday is looming. It kind of depends on how far you will carry the football down the field before handing it off to me. :-) :-)

from stack-docs.

@jethr0null & @DanRoscigno - thoughts on the timeline here?

from stack-docs.

It isn't obvious to me that any updates to the docs have been made looking at the history here (I could well be missing something) @DanRoscigno would you mind giving a quick status update here for all of the checklist items you've outlined (the Helm ones excluded given that we're not making any updates until the charts are actually announced)?

from stack-docs.

@DanRoscigno - any update here? Hoping to get an idea of where we stand before meeting today - thank you!

from stack-docs.

@kate-smith , as we discussed a couple weeks ago, this work starts today. I questioned having a status meeting on the first day of the project when you scheduled it.

Can we skip the meeting if I put an update here?

from stack-docs.

@DanRoscigno - an update here would be great - thank you! I will reschedule a check in for later in the week. The preliminary deadline I have from my notes includes:

• Initial updates & quick fix → this can happen quickly post-6.5 (Week of Nov 26 - 30)
• Updates done Nov 28 → Stack/docs repo GH ticket (Dan to open)
• Web to post live Nov 29 / Nov 30 → Web ticket for downloads page

Is this timeline still reasonable? The date in the main ticket (Nov 30) above is for the web work, so the update on docs is due Nov 28.

from stack-docs.

What is web posting on Nov 29/30? Are they updating elastic.co/start, or some other page? I have been concentrating on the documentation changes.

from stack-docs.

The update in the short-term are the download pages associated with the list in the main ticket. Long-term plan is to overall update all download pages.

cc @liztauscher

An updated timeline from Dan includes:

1. Technical fixes to docs: Dan, Tyler, Carlos. Finish writing the fixes this week, have them reviewed by Tyler and Carlos week of December 3rd
2. Deb publish docs changes (Exact date TBC at the moment)
3. write updates to the web pages (one example, https://www.elastic.co/downloads/kibana). Probably start on Monday Dec 3rd --> Kate & Liz will be making any content changes / edits to relevant download pages
4. Publish the new web pages

from stack-docs.

initial Docker configs in GitHub
(I could not deal with Google Docs breaking configs with silly "smart" quotes). Will work on the rest and switch to 6.5.0.

from stack-docs.

@tylerjl @exekias

Can you please have a look at the Docker commands and configs in GitHub

File structure in the README.

I have done my testing on Docker for Mac, with Elasticsearch and Kibana running in Docker also.

I will work on the docs that wrap around the configs, and then the same for Kubernetes.

from stack-docs.

Thanks @DanRoscigno - I ran through the steps on Fedora and it all seemed to work pretty well.

• I assume that there'll eventually be documentation about trying out the hinting vs. annotation setups, etc. since the filebeat/metricbeat bash scripts have commands to try both configs. At the moment running those steps fails to run the same container invocation since filebeat/metricbeat are already claimed.
• This is probably only an issue for native Linux users, so I don't expect it would be a big issue, but metricbeat complains about config permissions on metricbeat.yml and won't start without chown root and chmod go-w on the configuration file.

Aside from that all the containers came up and the dashboards had useful metrics 👍

from stack-docs.

Thanks Tyler. Yup, working on the docs next. I will fix the permissions when it gets put into wherever the docs folks keep files for download (which maybe is in github/elastic/beats, I will check).

The sh files will not be downloadable, they will be in code blocks in the docs, so the second filebeat/metricbeat won't fail like what you saw.

Did you look at the commands? Anything embarrassing in there? I never used Docker (without Kubernetes) before other than that last thing you looked at for me)

Thanks again @tylerjl

from stack-docs.

Doc pages to link to from product download pages:

Docker:
Filebeat
Metricbeat
Heartbeat
Packetbeat
Auditbeat
Elasticsearch
Kibana

Kubernetes:
Filebeat
Metricbeat

from stack-docs.

@tylerjl @exekias
What do you think about using a short lived setup pod like this to setup the dashboards, ML jobs, index pattern, etc? Similar to what I did for Docker, run the setup separate from the pod that processes data.

from stack-docs.

@dedemorton For the Beats Running in Docker documentation pages I need to be able to specify a different run command for each Beat (there are flags that differ on each one). Is there an example of how to customize the asciidoc section shown below for each Beat?

Can I do this:

ifeval::["{beatname_lc}"=="filebeat"]
["source", "sh", subs="attributes"]
--------------------------------------------
docker run \
  --mount type=bind,source="$(pwd)"/{beatname_lc}.yml,target=/usr/share/{beatname_lc}/{beatname_lc}.yml \
  {dockerimage}
--------------------------------------------
endif::[]

ifeval::["{beatname_lc}"=="metricbeat"]
["source", "sh", subs="attributes"]
--------------------------------------------
docker run \
  --mount type=bind,source="$(pwd)"/{beatname_lc}.yml,target=/usr/share/{beatname_lc}/{beatname_lc}.yml \
  {dockerimage}
--------------------------------------------
endif::[]

ifeval::["{beatname_lc}"=="packetbeat"]
["source", "sh", subs="attributes"]
--------------------------------------------
docker run \
  --mount type=bind,source="$(pwd)"/{beatname_lc}.yml,target=/usr/share/{beatname_lc}/{beatname_lc}.yml \
  {dockerimage}
--------------------------------------------
endif::[]

ifeval::["{beatname_lc}"=="auditbeat"]
["source", "sh", subs="attributes"]
--------------------------------------------
docker run \
  --mount type=bind,source="$(pwd)"/{beatname_lc}.yml,target=/usr/share/{beatname_lc}/{beatname_lc}.yml \
  {dockerimage}
--------------------------------------------
endif::[]

ifeval::["{beatname_lc}"=="heartbeat"]
["source", "sh", subs="attributes"]
--------------------------------------------
docker run \
  --mount type=bind,source="$(pwd)"/{beatname_lc}.yml,target=/usr/share/{beatname_lc}/{beatname_lc}.yml \
  {dockerimage}
--------------------------------------------
endif::[]

and put the custom stuff in this area between the ---- lines?

--------------------------------------------
docker run \
  custom args for the beat matched in the if
  {dockerimage}
--------------------------------------------

from stack-docs.

Yes you can use conditional blocks. Let me know if you need help with the coding. Looks ok from skimming.

from stack-docs.

Docs are updated in my fork, I have to go through the other pages (hopefully just the install pages) and link to the Docker / Kubernetes pages instead of telling people to pull the images.

Support question is worked out for Beats, @acchen97 is looking into the public facing support matrix, and engineering / janitors are updating the internal support wiki. We are good to go for Metricbeat and Filebeat for Kubernetes (I crossed out the rest up top)

from stack-docs.

Replaced docker pull commands on Beats install pages with links to the docker instructions.

from stack-docs.

Kubernetes docs are also done for Beats. The only change I had to make was linking from the Getting Started page in Filebeat and MEtricbeat docs (to prevent the user from doing a bunch of downloading and configuring that would not get them headed in the right direction).

from stack-docs.

I found the *beat-docker.yml files in the root dir of each Beat @exekias , I am going to use those instead of mine since they are already there. So, I am just adding the docker run command to do the setup (dashboards etc.), fixing the run commands for processing the data, and linking to the Docker / Kubernetes pages from the Getting Started/install pages (replacing the docker pull command on the install page).

from stack-docs.

To Do (Dan)

• Double check the Elasticsearch Docker docs (I believe they are good)
• Double check the Kibana Docker docs (I believe they are good)
• Submit PR to Beats repo
  - [ ] Submit PR to Elasticsearch repo (if needed, I doubt it)
  - [ ] Submit PR to Kibana repo (if needed, I doubt it)

from stack-docs.

PR is in: elastic/beats#9391

All comments are resolved, need a review for the Auditbeat example config and then it can be merged.

from stack-docs.