Disable log directory? about dsbulk HOT 4 OPEN

Hi, that could be a nice enhancement indeed. But just curious: if there is an error, how would you investigate the cause if you can't see the log files?

from dsbulk.

Ideally the error information from mapping-errors.log would print to stderr as logged ERROR messages so that the consuming program can redirect it to it's own logging as needed. The content of operations.log should go to stdout for the same reason (maybe it already does).

I'm also seeing empty files left behind in the "home" directory with the names of the tables that are being loaded. It's not a huge problem since they are empty, but it seems like the program should clean them up?

Thanks

from dsbulk.

Ideally the error information from mapping-errors.log would print to stderr as logged ERROR messages so that the consuming program can redirect it to it's own logging as needed.

That would be an option, but DSBulk creates many similar files for different kinds of errors. It would be a bit challenging to redirect everything to stderr (garbled contents).

I'm also seeing empty files left behind in the "home" directory with the names of the tables that are being loaded.

Now that's a first. Could you please give me a simple reproduction case? This is definitely not normal, DSBulk should not write to the home directory at all.

from dsbulk.

I haven't been able to reproduce the empty files issue so it may have been related to some intermittent problems I was having with the process getting killed by out of memory errors. Now that I have that resolved I'm not seeing any files left behind.

One unrelated question - the project description mentions "2-4x faster" than other bulk tools, is there any way to know what that should translate into in real numbers? I'm seeing between 1000-2000 rows/sec and I don't know if that's slow or fast? I imagine it's related to my Cassandra cluster performance.

Thanks

from dsbulk.