Processing .gz files about cobrix HOT 3 OPEN

Hi @srinubabuin ,

No, compression is not supported, and neither are inputStreams (although I'm not 100% sure what do you mean there).

The best option is to unpack the file first.

from cobrix.

Hi Yruslan,
https://github.com/AbsaOSS/cobrix/blob/master/spark-cobol/src/main/scala/za/co/absa/cobrix/spark/cobol/source/streaming/FileStreamer.scala
In this code finally we are finally reading BufferedFSDataInputStream with filePath, so here can i pass directly BufferedFSDataInputStream instead of filePath?

private var bufferedStream = new BufferedFSDataInputStream(getHadoopPath(filePath), fileSystem, startOffset, Constants.defaultStreamBufferInMB, maximumBytes)

from cobrix.

Sorry I'm not sure I understand. Keep in mind that the file will be ready in Executors, not on the driver node, and you cannot pass the stream from the driver to an executor. You need to create the stream on the executor. But you can create this stream from the file path.

Alternatively, you can use RDDs to read and uncompress input files, and then apply the record extractor to it. the example is called "Working example 3 - Using RDDs and record parsers directly" from https://github.com/AbsaOSS/cobrix

from cobrix.