RFC: Test environments for 6.0 upgrade about stack-docs HOT 10 CLOSED

++

I run a 3 MDI (master, data, ingest) + 1 KC (Kibana/coordinating) cluster (4 Ubuntu nodes total) in EC2 with Metricbeat on all of them for data. It runs this way all the time and has TLS enabled with a single multi-SAN public Let's Encrypt cert. I try to keep this cluster stable to help mimic the customer experience as much as possible-- it is this cluster that often serves as the source of the issues I file. The cluster has an LDAP realm (another common source of customer issues) that points to another EC2 instance running OpenLDAP. Logstash is on the KC node.

I often launch smaller clusters for blowing things up and usually use Metricbeat or Packetbeat since they are the quickest to get going. Automation could be better, but I am afraid of losing touch with the customer experience so I really prefer to "touch" everything.

Let's use Cross Cluster Search to tie the environments together. We should probably consider setting up Tribe since it'll be around until 7.0.

I think ML can run as default (data nodes) unless someone thinks it should be dedicated for this testing.

+1 for an OSS environment as it will help expose and verify things without X-Pack adding noise.

from stack-docs.

Thanks, @inqueue, that's great feedback. The setup you've outlined seems like a really good starting point.

Automation could be better, but I am afraid of losing touch with the customer experience so I really prefer to "touch" everything.

I think we'll want to automate the provisioning of the machines themselves, and get the initial (5.x) state under configuration management. Past that, the idea is to hand over the environments to a test group, and they'll manually upgrade to 6.0 using the instructions we provide. To my knowledge, we don't have a ton of prior art for spinning up a full stack in The Cloud. We discussed recently within the infra team, and I think this project is as good a reason as any to get some automation in place for that sort of thing.

Let's use Cross Cluster Search to tie the environments together. We should probably consider setting up Tribe since it'll be around until 7.0.

Nice catch, thanks.

from stack-docs.

You can get a full stack deployment (with TLS, and Elasticsearch in production mode) of 5.6.0 on a single machine pretty quickly by using the 5.x branch of the integration-test repo; https://github.com/elastic/integration-test/tree/5.x

There's only one instance of Elasticsearch running on this environment as a service, so it's not suitable for testing a rolling upgrade case.

I'll describe 2 ways to use it;

1. install vagrant locally, clone the 5.x repo above, run ./run.sh. This will download and start up a ubuntu 14.04 VM, download all the products to it, configure them and start them.
   Optionally edit run.sh to use a centos or w2012 VM (the Windows VM requires a vagrant login step and to be set up with the Infra team).
   To pull a specific staging build of 5.6.0 edit qa/setenv.sh.
   NOTE: if you have nodejs installed the ./run.sh will also try to open a new Chrome browser and run a set of Selenium tests. You could skip those if you want.
   This method is nice because you can save a snapshot, try some upgrade scenario, revert back to your snapshot to try a different scenario.
2. you could download the 5.x branch of the integration-test repo onto your own Ubuntu, CentOS, or Windows machine (or AWS EC2 instance) and run the ./qa./integration5.0snapshot.sh script. This will download and configure the full stack.
   I have a zoom recording of going through these steps I'll add here; ______

from stack-docs.

I should note that @LeeDr said he has some prior art in this area, particularly for running the full stack locally.

from stack-docs.

As far as the scenarios we want to test & document, I think we should add pre-5.6 OSS-only to the list.

from stack-docs.

As far as the scenarios we want to test & document, I think we should add pre-5.6 OSS-only to the list.

Added, thanks. Oversight on my part.

from stack-docs.

Four categories of users:

• Kibana only
• Kibana + X-Pack
• X-Pack only
• neither -> reindex old 2.x indices manually

from stack-docs.

Given that we are seeing issues like this (elastic/kibana#13950):

I am not sure if this is already in our plans, if not, can we also test the upgrade with indices created on 2.x or even earlier (including custom indices, .kibana index, and all our dot indices), which are then reindexed on 5.x and then test the upgrade against these indices?

from stack-docs.

We've got a sample implementation of this here: https://github.com/elastic/testing_terraform that we'd love to get comments on. See the README.md in examples/estest for some information on standing up the infra and provisioning.

from stack-docs.

I can't open (get 404) https://github.com/elastic/testing_terraform
Are the permissions OK on it? Can other people get to it?

from stack-docs.