Not showing deeper levels as columns. Also .count() problem. about cobrix HOT 1 CLOSED

Cobrix tries to retain the original COBOL schema when reading copybook+data file. That's why all the root level fields are struct types. You can use .option("schema_retention_policy", "collapse_root") to expand the root level of the schema. If you need a completely flat schema you can use SparkUtils.flattenSchema() function.

If you want to retain the structured nature of the data the better way to look at it is by exporting to json. Something like df.toJSON.take(10).foreach(println)

The error message is likely related to the fact the the record length does not divide the size of the binary file. Internally Cobrix uses Spark's binaryRecords() method. It requires that the total size of the file(s) be evenly divided by record size. The record size is available from the logs (the layout positions part).

Usually this error means that the copybook doesn't match the data. For instance, the copybook has missing fields. You can verify this by carefully comparing first several records reported by Spark against values shown in the mainframe.

from cobrix.