Cards not displayed on wuffle pod restart until next background sync about wuffle HOT 7 CLOSED

I can confirm that I was properly logged into the board after restore. I see session requests in the log. I also did a hard refresh on my browser. I also agree that the restore happened correctly.

I'll try and reproduce this bug at a time it will be less disruptive for users and try and debug it further. Since there isn't an obvious answer right now, please feel free to close the issue. I'll report back if I find something concrete.

from wuffle.

Thanks for opening this issue @piyushk.

What would be interesting is to get some more details on your utilization:

• How many issues do you have on the board?
• When did background sync start and when did it complete?
• When did store dump / restore start and when did it complete?
• Which environment do you run your board in (dev/prod)?

Note that the background sync happened ~15 minutes after startup. After the background sync, cards appeared on the frontend as normal.

The only explaination I can think of at the moment is that store restore did not kick in or did not restore anything. But looking at the logs restore completed in five seconds.

from wuffle.

Do you by chance have a backup of your store dump? If so, it could be interesting to see what was inside / supposed to be restored.

from wuffle.

I'll try my best to provide details. We're running on this private repository, so I'll have to look through stuff before sharing it:

• The board had ~6000 issues before the restart. About ~5000 issues are in an archived column that most users keep collapsed -- we don't want to really show them on the board.
• Background sync started at 21:20:59 and finished at 21:21:10, so 11 seconds.

{"name":"wuffle:background-sync","hostname":"wuffle-68d756bd5b-2mprw","pid":1,"level":30,"msg":"start","time":"2021-03-08T21:20:59.728Z","v":0}
...
{"name":"wuffle:background-sync","hostname":"wuffle-68d756bd5b-2mprw","pid":1,"level":20,"issue":"REDACTED","msg":"skipping, as up-to-date","time":"2021-03-08T21:21:01.963Z","v":0}
{"name":"wuffle:background-sync","hostname":"wuffle-68d756bd5b-2mprw","pid":1,"level":20,"issue":"REDACTED","msg":"skipping, as up-to-date","time":"2021-03-08T21:21:01.967Z","v":0}
...
{"name":"wuffle:background-sync","hostname":"wuffle-68d756bd5b-2mprw","pid":1,"level":20,"msg":"cleanup start","time":"2021-03-08T21:21:10.932Z","v":0}
1901 {"name":"wuffle:background-sync","hostname":"wuffle-68d756bd5b-2mprw","pid":1,"level":20,"t":4,"msg":"cleanup done","time":"2021-03-08T21:21:10.936Z","v":0}
1902 {"name":"wuffle:background-sync","hostname":"wuffle-68d756bd5b-2mprw","pid":1,"level":30,"t":11125,"msg":"updated 1, removed 0 issues","time":"2021-03-08T21:21:10.936Z","v":0}
1903 {"name":"wuffle:background-sync","hostname":"wuffle-68d756bd5b-2mprw","pid":1,"level":30,"msg":"done","time":"2021-03-08T21:21:10.936Z","v":0}

• After the background sync, I saw all ~6000 issues back on the board. I noted that the background sync generated about ~1500 lines in the log. Interestingly, the board only shows about ~1500 issues right now. Glancing at them, they seem to all be more recently touched issues. Since I assume that missing issues are archived issues, it's not a problem.
• I do not know when the store was dumped prior to pod restart. Unfortunately, we were not shipping logs from that pod. As the logs show, the store was restored within ~5 seconds. That's about as much as I know.
• We run in a prod environment.
• There were a whole bunch of skipping, as up-to-date messages in the log during the background sync, which seems to indicate that the store was restored at startup correctly.

We did save a copy of the storedump around restart. I'll see if I can share the store dump. I may not be able to, but I'll update within the week once I look through it.

from wuffle.

Regarding background synchronization of issues: Old issues ("archived") are automatically cleaned up after some days (cf. CONFIG). So what you're seeing over there is expected behavior.

Looking at the logs I can verify what you pointed out regarding start, restore and background-sync:

• App started at 21:06:29

{"name":"wuffle:run","hostname":"wuffle-68d756bd5b-2mprw","pid":1,"level":30,"msg":"Starting Wuffle","time":"2021-03-08T21:06:29.282Z","v":0}

• As intended, restore happened as app starts 21:06:34

{"name":"wuffle:dump-store-s3","hostname":"wuffle-68d756bd5b-2mprw","pid":1,"level":30,"Bucket":"REDACTED","Key":"storedump.json","t":5244,"msg":"restored","time":"2021-03-08T21:06:34.644Z","v":0}

• Background sync starts 15 minutes later, on 21:20:59

{"name":"wuffle:background-sync","hostname":"wuffle-68d756bd5b-2mprw","pid":1,"level":30,"msg":"start","time":"2021-03-08T21:20:59.728Z","v":0}

Background synchronization is done every hour and the timer is persisted across restarts.

➡️ As we see the sync happen 15 minutes after start already, I conclude that restore worked and restored the last sync time.

I see no circumstance in which one bit of information is restored successfully and the other one (issues) is not. Are you sure you were properly logged-in into the board after restore? Sessions as well as a number of internal user-access caches do not persist.

from wuffle.

Closing this issue as you suggested. Happy to re-open if you can reproduce what you're seeing.

from wuffle.

We recently bumped our pod again, and cards disappeared for 10 minutes. This time they appeared before the next background sync happened, so my original issue report was not correct.

from wuffle.