Document upgrade order for tribe and CCS about stack-docs HOT 14 OPEN

I'm happy to update the docs once we know what the behavior/supported strategy is. This can be called out in the upgrade path builder, as well.

from stack-docs.

Hey @ppf2, for next steps:

I'll complete a Google doc with the ES upgrade notes (covering all three of your points), which will hopefully have everything @debadair needs to convert into documentation. I believe Kibana should reference the ES documentation whenever there is overlap, to keep everything up-to-date/current in the future.

from stack-docs.

Hey Pius, just some quick notes on my own testing here for CCS:

• Cluster 1 on 5.4.0 (C1)
• Cluster 2 on 5.5.1 (C2)
• CCS Cluster on 5.5.1 (CCS)

1. I first upgraded the CCS cluster to 5.6.0, to attempt a rolling upgrade of the CCS cluster to avoid downtime.

2. Next I performed a rolling upgrade of the CCS cluster to 6.0:

Caused by: java.lang.IllegalStateException: Received message from unsupported version: [5.5.1] minimal compatible version is: [5.6.0]

Caused by: java.lang.IllegalStateException: Received message from unsupported version: [5.4.0] minimal compatible version is: [5.6.0]

3. That didn't go well, CCS 6.0 needs underlying clusters to be at least 5.6.0

4. Upgraded CC1 and CC2 to 5.6.0

5. Upgraded CCS to 6.0

6. Successful upgrade

I have some more research to do, e.g. can 5.6 CCS talk to 6.0 clusters? My guess is no, but I want to confirm first.

from stack-docs.

Thx for testing :)

Caused by: java.lang.IllegalStateException: Received message from unsupported version: [5.5.1] minimal compatible version is: [5.6.0]

Hmm, we will have to fix our doc here (https://www.elastic.co/guide/en/elasticsearch/reference/6.0/modules-cross-cluster-search.html) if that is intentional and not a bug, it says CCS can work with gateway eligible nodes on 5.5 and above.

from stack-docs.

@ppf2 good catch, I'll follow up on that.

More testing notes, the following configuration didn't have any issues:

Cluster 1 on 6.0.0 (C1)
Cluster 2 on 5.5.1 (C2)
CCS Cluster on 5.6.0 (CCS)

For the next steps, I can draft an outline for this, then follow up with the ES team for additional help.

from stack-docs.

With 6.0 GA targeted for tomorrow, it is unlikely that we will have a recommendation and have this documented by then. Can we set a target to have this done before 6.0.1? :)

from stack-docs.

Hey @ppf2, yea, this will be ready post-GA, but hopefully we can get it done shortly after (although I don't know which version yet).

I'm tracking my notes here:
https://docs.google.com/document/d/1U33G4oqCp6gin0CJc3jlG10C-DH8Sfe7yXPpC1FZDpQ/edit

from stack-docs.

Even though tribe is deprecated, it is still in the product on 6.x. There will be questions around upgrading tribe node implementations for those who are not ready to switch to CCS yet. Afaik, they need to upgrade the tribe to 6.0 first before the downstream clusters because if the tribe remains on 5.x, it will not be able to join any downstream clusters that have indices created on 6.0. For example, can they upgrade tribe to 6.0 first and then do rolling restarts of downstream 5.6 clusters to 6.0? This is something we will have to sync up with dev on our recommendations. This will probably depend on the outcome of #17, but I do want to make sure that the results are documented :)

I would say that you have to do the following: (eg with clusters of 5.1 and 5.4)

• Upgrade tribe to 5.6
• Rolling upgrade of clusters to 5.6
• Upgrade indices
• Upgrade tribe to 6.0
• Rolling upgrade of clusters to 6.0

For CCS, cross major version search should work with a CCS gateway of 5.6, so:

• If you upgrade CCS to 5.6 - can talk to 5.4/5.5/5.6 and 6.x
• If you upgrade CCS to 6.0 - can talk to 5.6 and 6.x

With the security changes in x-pack for CCS, I'm not sure about the compatibility with < 5.6. @tvernum ?

@s1monw could you confirm?

from stack-docs.

Quick note, Five9 asked for a status update on this:

@s1monw can you confirm the ordering above (in @clintongormley 's comment)?

from stack-docs.

from stack-docs.

Thanks all!

@zuketo @alexfrancoeur @tbragin @debadair

What are the next steps here in terms of getting this officially documented? I see 3 main areas to cover here:

1. Users currently on tribe and for some reason they are not ready to migrate to CCS yet and want to continue to use tribe on 6.x (even though it is deprecated) for some time before they switch to CCS.

2. (Likely not too many) Users currently on CCS (5.x) and upgrading to 6.x.

3. Users currently on tribe and migrating to CCS as part of their upgrade to 6.x. I am assuming we will be recommending the same separate-CCS-cluster-in-parallel approach from the KB article Alex wrote for CCS beta here. Is this still the recommendation? Will we be updating the KB article to advertise the export/import API for Kibana? (or not ready to recommend it yet)? Seems like something we will want to start documenting officially instead of having the information in a KB article.

from stack-docs.

@debadair I have a first draft of the Google doc here: https://docs.google.com/document/d/1U33G4oqCp6gin0CJc3jlG10C-DH8Sfe7yXPpC1FZDpQ/edit#

Please let me know if there is anything that needs additional work before we move this to a PR?

from stack-docs.

Added various comments to the draft to address before the PR :)

from stack-docs.

Hi @debadair, do you have time to help with getting this added to the docs?

from stack-docs.