Some analytics tables have some duplicate rows about killbill-analytics-plugin HOT 6 OPEN

We thought we didn't need to take the lock anymore (see 40662f3), since we introduced the implementation using a notification queue.

But it looks like we can still have concurrent jobs racing against each other: if two different entries (e.g. ACCOUNT_CREATION and PAYMENT_FAILED events) are picked up by two different threads on the same node for example.

To fix this, we can either:

• re-introduce the lock
• leverage the new getInProcessingNotifications queue API to delay the processing of an event for an account being refreshed already

from killbill-analytics-plugin.

If we go with 'lock' option, what we probably need is a new API tryLock. The first thread does tryLock, succeeds and proceeds. The second does tryLock, fails, and abort (since other thread is already refreshing data.

If we go with getInProcessingNotifications, i suppose we getInProcessingNotifications returns results for a given account_record_id we can also abort that refresh

from killbill-analytics-plugin.

we can also abort that refresh

We actually cannot abort, because we don't always do full refreshes (we do smart™ refreshes now!). But we could simply delay or postpone the job for that account.

from killbill-analytics-plugin.

Ah.. so refresh do not recreate the whole state for the account (just the pieces needed? )

from killbill-analytics-plugin.

As far as we can, yes: https://github.com/killbill/killbill-analytics-plugin/blob/master/src/main/java/org/killbill/billing/plugin/analytics/AnalyticsListener.java#L203

from killbill-analytics-plugin.

@kevinpostlewaite Have you tried the latest version? Did it help?

from killbill-analytics-plugin.