"Read ETIMEDOUT" on long copies. about node-pg-copy-streams HOT 4 CLOSED

I haven't tested it much at all using binary copies, but from a protocol level using binary in the copy is a lot easier than testing all the edge cases of using binary for all your queries to postgres, so I think it should be good.

As far as ETIMEOUT do you know which stream is returning you that? It might have to do with node or the PostgreSQL server timing out idle sockets. Any way to reproduce that? I understand if there isn't - it's a pain to test things that only happen once an hour 😑

from node-pg-copy-streams.

I'm not positive which one it's coming from, and I haven't found a consistent repro. Multi-hour queries through node are such a smell anyway that we're trying to work around this issue by chunking the copies instead.

The real answer for us is probably binary copies. Working on some more aggressive scale tests to get confident in them and see how much they help the perf.

from node-pg-copy-streams.

@drob can you share an update on this 2014 issue ? This kind of bug is interesting to drill on but hard to reproduce ;-)

regarding binary, you could be interested by the https://github.com/jeromew/node-pg-copy-streams-binary module I published recently if you need to transform the data you are moving around.

from node-pg-copy-streams.

I close this for now (2014). you can also test 2.0.0 which has an improvement on copy-to that make it behave better wrt the node loop.

from node-pg-copy-streams.