Set.fetch breaks if set was too large about forgetsy HOT 3 OPEN

Thanks for this report and sorry for the delay in getting back to you. I cannot delegate the limit parameter :n to the secondary set fetch as it needs to be applied after normalizing the primary set by the secondary.

One potential fix is to declare a more aggressive hi_pass_filter constant - this is used when trimming the set at query time. I could turn that into an ENV var if that helps. You can then tune that to ensure your sets never hit a size over the limit.

Another approach I use is to trim sets back to around 50k periodically (after a descending score sort).

from forgetsy.

Another option is slowing down the Hash assignment by using an iterator or other method... I can try to benchmark that to see if it will be feasible performance wise as this all runs at query time. That should allow scaling to very large number of items.

from forgetsy.

@cavvia Thanks for this great gem! I just started playing around with it and unfortunately came across the same problem as @atitan.

Did slowing down the hash assignment prove to be successful? I am trying to determine the best way to move forward with forgetsy.

from forgetsy.