token too long about aptutil HOT 2 CLOSED

Currently looking into this. The root issue is that we are using bufio.Scanner with the default MaxScanTokenSize which is 64kiB. I can reproduce this in our test suite by adding a package with an unrealistically large extended description, which is technically not illegal.

This can be addressed several ways:

1. Introduce our own hard-coded MaxScanTokenSize and call Scanner.Buffer with it upon the creation of Parser in NewParser
2. Expose a configuration knob for user-defined maximum scan token size and call Scanner.Buffer to set it with the user-provided value
3. Catch bufio.ErrTooLong and dynamically increase buffer size

Option 3 is not ideal as we run into the risk of the buffer growing infinitely large.

Option 2 gives the most control, but requires a lot of elbow grease since we'll need to change a bunch of function signatures to expose MaxScanTokenSize all the way to Cacher (and we might also need to touch Mirror since it also calls apt.ExtractFileInfo. Plus, it introduces a configuration knob which might seem obscure to the user since it exposes an implementation detail which may not necessarily be relevant to them.

Option 1 doesn't give any control, but has the advantage that it does not break any of the current function signatures. It is also the most lightweight to implement.

For the time being, I'm leaning towards option 1 with an internal MaxScanTokenSize of 1 MiB, which seems to be a reasonable enough maximum size. If the buffer size issue is found to be prevalent enough, we can always consider building upon option 1 to implement the other options.

from aptutil.

The fix works great, thanks!

from aptutil.