Changing Scala version about sahale HOT 6 CLOSED

Yeah typically we submit the slim jar to our cluster when its used as a dependency by another project. I've only used the fat jar for running example jobs so it can act as a standalone. I haven't had any trouble changing versions with the fat jar myself yet, but yes you do have to update the relevant libraries if a mismatch of major scala versions occurs.

We've discussed adding some Maven build profiles to make this a bit easier, but there are many potential combos of Scala, Scalding, Cascading, and Hadoop and I've worked on other open project where maintaining a long list of build profiles ended up making it more brittle.

All the same, I'm not against the idea if you want to submit a pull request my team and I will take a look and would give the idea another look.

Thanks for your time and interest!

from sahale.

Hey update: we are upgrading to Scala 2.10.x and Scalding 0.12 soon and hit a few minor lib and code changes needed for it. I've tagged the current release as v0.5 and will tag the next as v0.6, as well as leave comments to help users navigate the incompatible changes Scala and Scalding make across versions.

Regardless, if you rebuild flow tracker and restart the NodeJS app, then depend on the new Flowtracker for your runs, everything should work as long as you match up your internal Hadoop/Cascading/Scala/Scalding versions with Sahale/Flowtracker accordingly. Fun times.

The release will be posted Wednesday or sooner. Thanks, sorry about the headache.

from sahale.

This change is pushed out, which makes a few POM.xml changes and some (sadly) code changes to stay compatible with newer Scala/Scalding versions. this commit can be reverted if you're still on older Scalding (or Scala 2.9.x) and the other new stuff should work fine.

I've opened another Issue to perhaps set up Maven build profiles to manage the Scala 2.9/2.10 divide better. I'm closing this one for now.

from sahale.

Hey, thanks again. We have an issue open to decide what to do about this.
One issue is there are actual code changes required as the Scalding side of
things changes slightly across the Scala 2.9 => 2.10 barrier, but we could
probably shim that in the build too. Sorry for the frustration, hope you
got it worked out in the meantime? Feel free to drop me a gist of any
relevant stack traces, POM changes you made etc. and I'll take a look.

On Thu, Feb 26, 2015 at 4:07 AM, m.orazow [email protected] wrote:

After changing scala version and submitting fat jar to cluster throws
error,
NoClassDefFoundError: scala/reflect/ClassManifest

I think this happens because there is some classes still compiled with
lower scala versions.

I think, ideal would be to have cross compiled jar in maven (or other)
repositories.

What do you think?

I would be glad to help if there is anything I can do.
Best

—
Reply to this email directly or view it on GitHub
#1.

from sahale.

Hello @initialcontext, thanks for informing.
It is so great to see improvements. Since I am busy with other projects I could not look at it yet. I will deploy it with new versions and let you know if I see any issues.
All the Best!

from sahale.

You too, thanks for the feedback! Let us know if you hit more trouble.

We moved to Scala 2.10 and upgraded Scalding internally, and the current FlowTracker lib is working for us so I believe at this point the dependencies and FlowTracker itself should be working properly in a Scala 2.10 and compatible Scalding env, but I'd love to know if it doesn't!

from sahale.