Duplicates and low number of posts for past years about fbcrawl HOT 2 CLOSED

Hello @ebergam,
For the sake of debugging, I copied my test on Repubblica here https://pastebin.com/M3fKq8Ee
Have a look at line 686, here's where problems begin. The page that was retrieved clicking on "more" doesn't have the "more" link, so fbcrawl clicks on "2019" (not actually 2018) once - then it goes back to looking for the "more" link to follow - causing some duplicates that should be filtered by dupefilter.
Again, at line 796 fbcrawl clicks on "2018" and so on, causing some months of gap. Assuming that Repubblica has a consistence publishing frequency it also looks like only a handful of posts are retrieved each day (maybe sampled?), compared to the number of posts of the last days.
All this behavior depends entirely on fb's mbasic buggy interface, I don't think there is much I can do about that.

from fbcrawl.

@rugantio thanks a lot, this replies perfectly to my doubts.
Indeed, it might be a sample of the posts. I guess on the effectiveness of scraping past posts there is not much doable, whereas on a forward-collection approach it still looks the technique can yield good results. Thanks again for your very prompt reply and your work!

from fbcrawl.