Feedback: Physical Cluster Replication - wrong cluster specified for pausing or canceling schedule on changefeeds after cutover about docs HOT 7 OPEN

FYI I added a comment with my perspective on the larger context to #123776 "Update scheduled changefeeds to pause if resumed on a different cluster"

from docs.

@hand-crdb the docs are actually correct. we recommend pausing or cancelling the the newly promoted cluster scheduled changefeed (backup schedules pause automatically on the newly promoted cluster). PCR was designed under the assumption that the user does not need to do anything on the original source cluster.

After pausing the schedule, to avoid the conflict problem, the user should then decide if they want to create a new schedule on the newly promoted cluster or pause the original source schedule and unpause the schedule they just paused.

We want users to assume all replicated schedules will (or should) be paused. In 24.2, the changefeed schedule on the promoted standby cluster will autometically pause, just like backup schedules.

from docs.

@msbutler In a split brain situation where the primary is still up (maybe graceful failover scenario), technically changefeeds and backups would still be running as background jobs on the primary right?

from docs.

In a split brain situation where the primary is still up (maybe graceful failover scenario), technically changefeeds and backups would still be running as background jobs on the primary right?

yes, correct.

from docs.

@kathancox as we had discussed in our sync let's document the two cases: when primary is down (dr), and if the primary is still up (graceful failover).

@hand-crdb The docs were written from the perspective of the primary cluster being offline, but we realize there may be scenarios where the primary is still running and therefore would still have background jobs. Thanks for calling this out, we will document both scenarios :)

from docs.

@alicia-l2 Just wanted to double-check: aren't we trying to avoid two sets of schedules running? Even in the graceful failover scenario you mention, if you leave the schedules running on both will there not be conflicts?

from docs.

@kathancox yup, especially when the changefeeds are going to the same place for example. Oh I see what you mean, ok maybe we don't need to separate the scenario, we could just blanket statement say you don't want two sets of schedules running

from docs.