Getting java.lang.OutOfMemoryError: Java heap space exception about dsbulk HOT 5 CLOSED

Hi @r3econ, thanks for reaching out.

How big are the rows you are trying to load? Given that you use --connector.csv.maxCharsPerColumn -1 I suppose they are quite large.

When dealing with large rows: here are some things that work:

• Throttle DSBulk; this is best done by setting --engine.maxConcurrentQueries X where X is a small number: start with 1, if it works, that's great but the throughput will be poor; try increasing the number little by little to improve DSBulk's throughput without breaking the operation.
• Increase the heap size. This is done by setting the environment variable DSBULK_JAVA_OPTS. E.g. export DSBULK_JAVA_OPTS="-Xms1g -Xmx1g"

Hope that helps!

from dsbulk.

You can also add --dsbulk.log.sources false to lower the heap pressure.

from dsbulk.

Any updates on this @r3econ ?

from dsbulk.

Yes, thanks for the info. I managed to get it working by tweaking the params

from dsbulk.

Glad to hear! Let's close this issue then.

from dsbulk.