refactoring org.apache.hadoop.mapred usage against org.apache.hadoop.mapreduce about cascading HOT 8 CLOSED

You can try and submit something to review, but historically you cannot mix and match api calls without instability/failure.

Moving to the new API (mr2) would be welcome, but that would be under a 5.0 release, and the last time I tried to do it I gave up after a week of effort.

from cascading.

Can you summarize the issue that showed up in the pr?

If it's about parquet, note that they dropped Cascading from the project, and I have an intermediate branch I'm trying to restore it in, but I can't get the test to work.

from cascading.

it's nothing unsolvable but keep having to have wide types that support for instance both JobConf and Configuration and then casting between the both of them to get different API. I understand that for this particular case you can cast directly from Configuration to JobConf so it's not a big deal.

That's what I've noticed so far I'm sure more things will come up over time and as you said in the general case it's not supported to mix new/old with any compatibility guarantees so it's more of a pre-caution.

from cascading.

Agree that this is very difficult. I've spent today poking around and it has a lot of moving parts. Instead of doing it all at once, this would need to be incrementally adopted because the changes are just too big otherwise.

from cascading.

Agree that this is very difficult. I've spent today poking around and it has a lot of moving parts. Instead of doing it all at once, this would need to be incrementally adopted because the changes are just too big otherwise.

this is probably why I gave up, there was no middle ground. but that was years ago, I only remember being really mad when I threw in the towel.

it's nothing unsolvable but keep having to have wide types that support for instance both JobConf and Configuration and then casting between the both of them to get different API.

So Cascading 3 moved to using Configuration and away from JobConf, in general. If there is a spot the API needs to accept Configuration to prevent a cast, let's call it out (we can always make a new API and let the old one wrap it with a cast).

Unfortunately this move/change is exactly whey adoption of Cascading 3 was limited, gaining Tez support wasn't sufficient.

from cascading.

@daniel-sudz so is it safe to say we should not release 4.5 until we have Scalding running on it?

I think this is reasonable and buys me time to get Parquet and a couple other features up into 4.1/4.5.

from cascading.

works for me @cwensel

from cascading.

closing this for now, using Configuration everywhere seems to be good with cascading from I've tested

from cascading.