Very different times on table with keys vs table without keys about node-pg-copy-streams HOT 5 CLOSED

can I ask what you mean by "table with no keys" ? do you mean "empty table" vs "non empty table" ?

from node-pg-copy-streams.

You know what. It was obviously problem with my implementation. I was opening multiple streams but only from one client and I suspect one had to wait for another to finish and this is why data couldn't "flow" through the system

from node-pg-copy-streams.

ok thanks for closing this. you may still want to dig into why so much data gets buffered. I suspect you have a problem (or maybe this is a choice) with your sources not respecting the backpressure protocol (a source should stop writing into a sink when sink.write returns false)

from node-pg-copy-streams.

I know what was causing this. I only had one stream (green) opened at the same time and it was causing yellow and red to be stuck in between:

Other streams had to wait until the green stream can end (and green had to wait for readStream to end) and only then other streams opened - that is why there was such a big delay between readStream.on('end') and writeStream.on('finish').

My previous implementation used same PoolClient to instantiate all 3 streams and this is why this happened. New implementation uses 3 different instances of PoolClient to instantiate 3 streams, so now data can flow freely without waiting for other streams to finish:

And the time difference between tables was caused by the order of instantiating streams. I was getting really low times when writing only to green and really high times when writing to others

from node-pg-copy-streams.

ok. good job figuring that out !

from node-pg-copy-streams.