Some sites stalling when using check-html-links about rocket HOT 9 CLOSED

Yeah, seems to be running fine AFAIK, it's just getting slower as time passes it seems. Let's see

Can confirm that it is going slower as times passes. Over 9000, I process roughly one file per second now.

from rocket.

Interestingly, bottleneck is not CPU nor memory apparently

from rocket.

Past the improvements that can be made, it makes me think we probably should allow for storing maps of links to avoid starting from scratch every time. You don't want your CI to take 3x more time because of dead links.

from rocket.

35.000 files is certainly a different scale 😅

why do they have sooo much? 🤔

anyways there are still a few performance improvements we can do... (some which only make sense at this scale so I didn't bother yet... also ~2s for the other use cases I tested seemed fast enough 😬)

adding some sort of permanent cache is imho tricky as then we would need to look at cache invalidation - also you would need to commit this or the ci needs to permanently cache it
potentially this would require a full link tree... yeah don't think it's worth the trouble

from rocket.

Good point.

I was thinking along those lines as well. I'll try some more sites and see where we're going with it :).

Oh, and now that you mention it the number of links could be because they're bundling autogenerated javadoc. That'd explain. I'll look at possibly adding ignore folders then.

Thanks for thinking along

from rocket.

It worked BTW, just took a while. Congrats

from rocket.

I did a quick hack with running 2 parsers in parallel... and on the 11ty-website (with around 500 documents) it improved the performance by about 30%... once I added a 3rd parser it was slower than 1 parser alone.

Seems a rule of thumb could be to have 1 parser for around 300 - 400 documents... which would mean ~100 parsers for 35.000 files 😅

that would still take a ton of time 😅

but yeah I guess checking 35.000 documents is a little out of scope

from rocket.

Nice test though! I mean the complexity of searching for links through pages and finding reference is growing exponentially with the number of pages so yeah, we should probably avoid auto generated doc, which is by design correct anyways

from rocket.

I'm going to close this as it works... it's just not fast enough if applied to 30.000+ pages 😅

working with max ~5.000 documents sounds like a valid limitation for now.
There are some more tricks that could be applied but I don't see working on them any time soon (especially as it would introduce way more complexity)

Feel free to reopen or create another issue if this needs to be tackled.
It should probably have a good use case or a nice project that needs it or it should be sponsored.

from rocket.