Is it possible to flatten a nested schema so all values are the root? about cobrix HOT 5 OPEN

Yes, you can use

SparkUtils.flattenSchema(df)

from cobrix.

Can we use this if our code base is in python? I believe we can use something like py4j, but is there another easier way?

from cobrix.

We haven't tried this, but you might be able to use the method from the JVM to Python gateway (sc._gateway.jvm...) the same way PySpark interacts with Scala codebase.

Alternatively, the source code for the flattening is not too big, you can covert it in Python for your use. If you want, you can also contribute the Python version of the flattening code to Cobrix, we can include as one of the examples.

from cobrix.

Sorry for the late reply. Can you show a snippet of how the call from the JVM to Python gateway would work?
Regarding contribution, I do not have the bandwidth at the moment.

from cobrix.

I actually haven't tried this specifically with the flattening code. I might check it out and let you know if it worked

I expect something like:

        sc = self.spark.sparkContext
        flat_java_df = sc._gateway.jvm.za.co.absa.cobrix.spark.cobol.utils.SparkUtils.flattenSchema(df._jdf)
        flat_df = spark.createDataFrame(flat_java_df)

ensuring Cobrix libraries are in the class path.

from cobrix.