csv-parse stream will process the whole buffer even with back pressure. about node-csv HOT 6 OPEN

I don't think it's unreasonable to expect that even if you pass the entire large dataset as a string or Buffer that it would still process it in chunks to reduce memory usage. A 300MB Buffer of CSV data could easily use several GB of RAM if you parse all the rows in advance. Putting 300MB in RAM might not be a big deal, but expecting the rows to parse in small batches so the additional memory usage is minimal.

At least, this issue pretty much took me 6 hours to figure out and find a workaround yesterday, so If filed the issue here as I could imagine others being unpleasantly surprised by it.

The workaround isn't super hard to implement once you know it's needed, but it's not very obvious that it should be.

from node-csv.

Ok, here it is:

https://www.npmjs.com/package/@sciactive/back-pressure-transform

Try it out and see if it works for you.

from node-csv.

I messed around with this for a while trying to figure out a solution, but this seems to be a case that just isn't handled by the Transform API. After the last buffer is provided to the transform, it will call _flush and close the stream and you don't get more chances to try to push the remaining data even if there was back pressure. Kind of an odd gap in the API.

Anyway, my workaround for now is to split up the incoming buffers so that they are always small, that seems to work OK. But with this issue lingering I suppose there will be others in the future who are bitten by the same issue.

from node-csv.

Is it not the responsibility of the Stream Reader (input in your case) to provide smaller chunks ? If someone doesn't control the chunk size of its input, maybe he could insert a custom transformer between the input and the parser. From the parser's standpoint, processing all the data it receives seems fair.

from node-csv.

With a bit more research I think maybe to resolve this would require implementing Duplex directly instead of using Transform. Kind of a pain, but the nodejs Transform API doesn't have any built-in concept of one-sided backpressure, it basically assumes the input and output are of a similar size and passes through all backpressure upstream.

from node-csv.

In case you're still looking for a solution here, I wrote my own back pressure aware transform stream. You can find it here:

https://github.com/sciactive/nephele/blob/37bac308b75fd660ebbbe0d93ed86b504237d20c/packages/plugin-encryption/src/BackPressureTransform.ts

And here's an example of it being used:

https://github.com/sciactive/nephele/blob/37bac308b75fd660ebbbe0d93ed86b504237d20c/packages/plugin-encryption/src/EncryptionProxyResource.ts#L115

I'm thinking about making this a separate NPM package.

from node-csv.