Any plans for support spark 3.0 ? about data-flare HOT 5 CLOSED

I'd love to add support for Spark 3.0 but struggle to find the time to add that currently. I'd be very happy to help someone else add a PR for this though. Would you be interested in helping?

from data-flare.

I'd also like that. @timgent what's the biggest thing that needs to get changed?

from data-flare.

To migrate this library to scala 2.12 can be quite a project, doesn't it?

from data-flare.

Scala 2.12 is already supported and published - https://search.maven.org/artifact/com.github.timgent/data-flare_2.12/2.4.5_0.1.14/jar

To migrate to Spark 3.0 I'd probably take this kind of approach:

• First just bump the relevant versions of dependencies
• See what breaks (compilation errors and tests should catch most things) and get an idea of how much work it is to fix
• Ultimately it'd be nice to keep the library backwards compatible with Spark 2, so would then have to look at how to support both. I think SBT has support for having multiple versions of the same files/classes and it chooses between them depending on the version it is building for. If you look in build.sbt and the [build script]https://github.com/timgent/data-flare/blob/master/mini-cross-build.py you can see some of the logic there for how the different versions of the library are built already

Happy to support if you want to give it a try @kingbrown9

from data-flare.

Resolved in #147

Data-flare version will remain the same, but existing artifacts will be published that are compatible with Spark 3. They should appear in Maven central in the next day or so

from data-flare.