Importing a large linux perf file will crash the application about speedscope HOT 3 CLOSED

Hey @Goose97!

Thanks for the very clear description of the problem you're trying to solve, and the links to the specific parts of the code!

If I'm understanding the problem correctly, I think there's a simpler solution that's probably more performant.

The splitBlocks function doesn't really need to exist at all. It was just a simple way to reason about the problem.

Instead, I think we can change the importer to just operate on the output of splitLines() directly without using \n\n delimited blocks as an intermediate. This also has the benefit of not doubling the memory requirements.

If we do your proposed change, I think we end up with the 1.3GB in memory three times: once as a buffer, once as the return value of .splitLines() and once as the TextFileContent[]. If we skip the splitBlocks, then we just have the first two.

Really, ideally, we'd have only the one in the buffer and splitLines() should return an iterator instead, then we'd only have the 1.3GB in memory once. That's a separate issue than the one you're trying to solve though.

from speedscope.

Hi @jlfwong,

That makes a lot of sense. I'll try an PR, going the way you suggested.

I guess I'll go ahead and convert the splitLines() to return an iterator first (neat idea btw), then come back to this issue. Is that cool?

from speedscope.

I guess I'll go ahead and convert the splitLines() to return an iterator first (neat idea btw), then come back to this issue. Is that cool?

Yeah, if you'd like to tackle that, then please go ahead!

Please make it a separate PR from the addressing the issue described here (so one of the iterator, one for the perf file fix)

If, in the process of doing that, you realize it's kind of a pain to do, it's also fine to address the linux perf-file specific issue without solving the iterator.

from speedscope.