Giter Club home page Giter Club logo

cascading's Introduction

Cascading

Thanks for using Cascading.

Cascading 4.5

Cascading 4.5 includes a few major changes and additions from prior major releases:

  • Support for Apache Hadoop 3. and Apache Tez 0.10.x
  • Removed Apache Hadoop 2.x and Apache Tez 0.8.x support

Cascading 4 includes a few major changes and additions from prior major releases:

  • Local mode support improved for production use
  • Moved the website to https://cascading.wensel.net/
  • Changed the Maven group name to net.wensel (from cascading)
  • Moved to GitHub Packages (WIPs) and Maven Central (releases)
  • Added native JSON support via the cascading-nested-json sub-project
  • Removed cascading-xml sub-project
  • Removed Apache Hadoop 1.x support
  • Imported deprecated Apache Parquet classes to retain support and provide enhancements

General Information:

For current WIP releases, go to: https://github.com/cwensel?tab=packages&repo_name=cascading

For project documentation and community support, visit: cascading.wensel.net

The project includes nine Cascading jar files:

  • cascading-core-x.y.z.jar - all Cascading Core class files
  • cascading-expression-x.y.z.jar - all Cascading Janino expression operations class files
  • cascading-nested-json-x.y.z.jar - all Cascading JSON operations
  • cascading-nested-x.y.z.jar - all Cascading base classes for nested data-type operations
  • cascading-local-x.y.z.jar - all Cascading Local in-memory mode class files
  • cascading-local-kafka-x.y.z.jar - all Cascading Local support for Apache Kafka
  • cascading-local-neo4j-x.y.z.jar - all Cascading Local support for Neo4j
  • cascading-local-s3-x.y.z.jar - all Cascading Local support for AWS S3
  • cascading-local-splunk-x.y.z.jar - all Cascading Local support for Splunk
  • cascading-local-hadoop3-io-x.y.z.jar - all Cascading Local in-memory mode class files used with Hadoop
  • cascading-hadoop3-common-x.y.z.jar - all Cascading Hadoop 3.x common class files
  • cascading-hadoop3-io-x.y.z.jar - all Cascading Hadoop 3.x HDFS and IO related class files
  • cascading-hadoop3-mr1-x.y.z.jar - all Cascading Hadoop 3.x MapReduce mode class files
  • cascading-hadoop3-tez-x.y.z.jar - all Cascading Hadoop 3.x Tez mode class files
  • cascading-hadoop3-tez-stats-x.y.z.jar - all Cascading Tez YARN timeline server class files
  • cascading-hadoop3-parquet-x.y.z.jar - all Cascading Parquet class files
  • cascading-hadoop3-parquet-thrift-x.y.z.jar - all Cascading Parquet Thrift class files

These class jars, along with, tests, source and javadoc jars, are all available via the Maven repository.

Local mode is where the Cascading application will run locally in memory without cluster distribution. This implementation has minimal to no robustness in low memory situations, by design.

Hadoop 3.x MR1 mode is for running on Hadoop 3.x releases.

Hadoop 3.x Tez mode is where the Cascading application should run on an Apache Tez DAG cluster.

As of Cascading 4.x, all above jar files are built against Java 1.8. Prior versions of Cascading are built against Java 1.7 and 1.6.

Local Mode

Local mode has been much improved for production use in applications that do not need to run distributed across a cluster. Specifically in applications that trivially parallelize and run within AWS Lambda or Batch applications.

See https://github.com/cwensel/cascading-local for a collection of local mode integrations.

Note this project will merge into Cascading in then next minor release.

Extensions, the SDK, and DSLs

There are a number of projects based on and extensions to Cascading available.

Visit https://cascading.org/ for links. As these projects are updated to depend on 4.x, we will update the main site.

Note many projects built and released against Cascading 3.x will work without modification with Cascading 4.x.

Versioning

Cascading stable releases are always of the form x.y.z, where z is the current maintenance release.

x.y.z releases are maintenance releases. No public incompatible API changes will be made, but in an effort to fix bugs, remediation may entail throwing new Exceptions.

x.y releases are minor releases. New features are added. No public incompatible API changes will be made on the core processing APIs (Pipes, Functions, etc), but in an effort to resolve inconsistencies, minor semantic changes may be necessary.

It is important to note that we do reserve to make breaking changes to the new query planner API through the 4.x releases. This allows us to respond to bugs and performance issues without issuing new major releases.

All releases will be maintained here: https://github.com/cwensel/cascading

WIP (work in progress) releases are fully tested builds of code not yet deemed fully stable. On every build by our continuous integration servers, the WIP build number is increased. Successful builds are then tagged and published.

The WIP releases are always of the form x.y.z-wip-n, where x.y.z will be the next stable release version the WIP releases are leading up to. n is the current successfully tested build.

The source, working branches, and tags for all WIP releases can be found here: https://github.com/cwensel/cascading

When a WIP is deemed stable and ready for production use, it will be published as a x.y.z release, and made available from Maven Central.

Writing and Running Tests

Comprehensive tests should be written against the cascading.PlatformTestCase.

When running tests built against the PlatformTestCase, the local cluster can be disabled (if enabled by the test) by setting:

-Dtest.cluster.enabled=false

From Gradle, to run a single test case:

> ./gradlew :cascading-hadoop3-mr1:platformTest --tests=*.FieldedPipesPlatformTest -i

or a single test method:

> ./gradlew :cascading-hadoop3-mr1:platformTest --tests=*.FieldedPipesPlatformTest.testNoGroup -i

Debugging the Planner

When running tests, set the following

-Dtest.traceplan.enabled=true

If you are on Mac OS X and have installed GraphViz, dot files can be converted to pdf on the fly. To enable, set:

-Dutil.dot.to.pdf.enabled=true

Optionally, for stand alone applications, statistics and tracing can be enabled selectively with the following properties:

  • cascading.planner.stats.path - outputs detailed statistics on time spent by the planner
  • cascading.planner.plan.path - basic planner information
  • cascading.planner.plan.transforms.path - detailed information for each rule

Contributing and Reporting Issues

See CONTRIBUTING.md at https://github.com/Cascading/cascading.

Using with Maven/Ivy

It is strongly recommended developers pull Cascading from Maven Central.

Alternatively, see GitHub Packages for latest WIP releases:

When creating tests, make sure to add any of the relevant above dependencies to your test scope or equivalent configuration along with the cascading-platform dependency.

Note the cascading-platform compile dependency has no classes, you must pull the tests dependency with the tests classifier.

Source and Javadoc artifacts (using the appropriate classifier) are also available through Maven.

Note that cascading-hadoop3-mr1, and cascading-hadoop3-tez have a provided dependency on the Hadoop jars so that it won't get sucked into any application packaging as a dependency, typically.

Building and IDE Integration

For most cases, building Cascading is unnecessary as it has been pre-built, tested, and published to our Maven repository (above).

To build Cascading, run the following in the shell:

> git clone https://github.com/cwensel/cascading.git
> cd cascading
> ./gradlew build

Using with Apache Hadoop

First confirm you are using a supported version of Apache Hadoop by checking the Compatibility page.

To use Cascading with Hadoop, we suggest stuffing cascading-core and cascading-hadoop3-mr1, jar files and all third-party libs into the lib folder of your job jar and executing your job via $HADOOP_HOME/bin/hadoop jar your.jar <your args>.

For example, your job jar would look like this (via: jar -t your.jar)

/<all your class and resource files>
/lib/cascading-core-x.y.z.jar
/lib/cascading-hadoop3-common-x.y.z.jar
/lib/cascading-hadoop3-mr1-x.y.z.jar
/lib/cascading-hadoop3-io-x.y.z.jar
/lib/cascading-expression-x.y.z.jar
/lib/cascading-nested-json-x.y.z.jar
/lib/<cascading third-party jar files>

Hadoop will unpack the jar locally and remotely (in the cluster) and add any libraries in lib to the classpath. This is a feature specific to Hadoop.

History and Status

Cascading was started in 2007 by Chris K Wensel.

After a series of acquisitions, it was left unsupported and unmaintained by the copyright, domain name, and GitHub organization owners.

Chris has since continued his noodling with Cascading and has been pushing changes to the original repo.

Cascading remains under the Apache License v2.0.

cascading's People

Contributors

ahmed--mohsen avatar avibryant avatar benpence avatar cchepelov avatar colinmarc avatar cwensel avatar daniel-sudz avatar dvryaboy avatar fhueske avatar fs111 avatar gianm avatar isnotinvain avatar jalkjaer avatar jamesiry avatar jenniferlin avatar julienledem avatar lukasnalezenec avatar mickaellcr avatar nahguam avatar nandorkollar avatar ohrite avatar patduin avatar rdblue avatar redshiftetl avatar sritchie avatar toktarev avatar tomwhite avatar tsdeng avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cascading's Issues

NestedRegexFilter is not building a nested pointer

Instead is using the #compile() method to generate a pointer, subsequently it is not capturing nested structures.

This operation may need to be updated to apply patterns to each child element in the pointer.allAt method.

Upgrade Janino to 3.x

The upgrade requires some refactoring as the script and expression evaluators no longer have a shared base class.

Move counter duration measurements to aggregated nanos

Step/SliceCounters sums durations in wall clock milliseconds, but some read or write operations may be very short.

The fear is we are introducing a lot of error by summing ms interval durations. Using a Stopwatch with nanos granularity, that then is periodically published on a shared timer may improve relative accuracy.

Support parallelization of child partition .close() operations

By default, we don't leverage threading server/processing side since it's a highly constrained resource.

But in local mode, and when partitioning sink data into multiple files, parallelizing the close operation makes sense as the purges may include more than one partition.

Providing hooks to wrap the cache of open partitions purge and close operations would benefit applications that were designed specifically for partitioning data if the container hosting the application can provide cpus for the threads.

`TapOutputCollector` committing partial output on task failure

According to the implementation of TapOutputCollector#close and Hadoop18TapUtil#needsTaskCommit, work is always committed as long as something exists in the work path. As a result, I am seeing partial output files committed in S3 when I use a PartitionTap and my task fails after it has written some output.

Is this expected behaviour? I am seeing this in Cascading 3.1, which I understand is a few years old now, but any information here would be of great help

Thanks

Possible to publish WIP to maven-central?

The Github Artifacts require authentication for read. This makes them hard to use seamlessly in other open source projects. Would it be possible to cross-publish to maven central or another maven-compatible repo with read enabled without auth?

HashJoin has problematic interaction with Merge

see this graph on 3.2.1
https://www.dropbox.com/s/iffadh9x7unrg5w/01-BalanceAssembly-init.dot.png?dl=0

You can see the full planner logs here:
https://www.dropbox.com/s/7qyc4a9pxtstwio/E552D2.tgz?dl=0\

We are merging two HashJoins after some Each operations. In this particular graph, it is possible to fix the issue by adding Checkpoints after all but one of the HashJoins it seems. This is not a great solution since even knowing what a graph will look like when you combine many pipes with functions is not very clear.

It would be great to have either a clear rule that we need to follow in generating the graphs, or to remove this restriction since we would like to using cascading 3 in scalding by default.

Thanks.

Casting issue in PropertyUtil

I am encountering an issue when running unit tests in a custom library that has Cadence as a dependency. This block of code has an issue when casting a Class to a String. The call to the PropertyUtil function is coming from AppProps.getApplicationJarClass. Here is the stack trace:

java.lang.ClassCastException: java.lang.Class cannot be cast to java.lang.String at cascading.property.AppProps.getApplicationJarClass(AppProps.java:113) at cascading.flow.hadoop.planner.HadoopPlanner.initialize(HadoopPlanner.java:166) at com.anon.cascading_ext.flow.LoggingHadoopPlanner.initialize(LoggingHadoopPlanner.java:75) at com.anon.cascading_ext.flow.LoggingFlowConnector.connect(LoggingFlowConnector.java:68) at cascading.flow.FlowConnector.connect(FlowConnector.java:421) at cascading.flow.FlowConnector.connect(FlowConnector.java:270) at cascading.flow.FlowConnector.connect(FlowConnector.java:215) at cascading.flow.FlowConnector.connect(FlowConnector.java:197) at com.anon.formats.mapred.inputformat.TestDirectoryLineInputFormat.testGetCurrentTapSourcePath(TestDirectoryLineInputFormat.java:61)

This could be solved by checking if defaultValue is an instance of String and using the Class.toString function. My org is in the process of upgrading from a very old version of Cascading to Cascading 4.5.0. In the version that we are using, AppProp.getApplicationJarClass actually passes a null Class object as the default value.

Is there a recommended method of fixing this casting issue?

Running into this on JDK 8, Cascading 4.5.0

Task cleanup should not look for _temporary dirs when talking to s3

Seeing this exception sometimes talking to S3. Could be a race condition on directory (key) creates on S3, but there should be no task cleaup when using S3.

    at cascading.tap.hadoop.io.TapOutputCollector.close(TapOutputCollector.java:184)
    at cascading.tuple.TupleEntrySchemeCollector.close(TupleEntrySchemeCollector.java:245)
    at cascading.tap.partition.BasePartitionTap$PartitionCollector.closeCollector(BasePartitionTap.java:205)
    at cascading.tap.partition.BasePartitionTap$PartitionCollector.close(BasePartitionTap.java:190)
    at cascading.flow.stream.element.SinkStage.cleanup(SinkStage.java:148)
    at cascading.flow.stream.graph.StreamGraph.cleanup(StreamGraph.java:187)
    at cascading.flow.local.planner.LocalStepRunner.call(LocalStepRunner.java:204)
    at cascading.flow.local.planner.LocalStepRunner.call(LocalStepRunner.java:53)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.io.FileNotFoundException: No such file or directory: s3a://bucket/test/_temporary/_attempt_002147483647_0000_m_000000_0/path/
    at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:1931)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:1822)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:1763)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.innerListStatus(S3AFileSystem.java:1585)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus(S3AFileSystem.java:1561)
    at cascading.tap.hadoop.util.Hadoop18TapUtil.moveTaskOutputs(Hadoop18TapUtil.java:326)
    at cascading.tap.hadoop.util.Hadoop18TapUtil.moveTaskOutputs(Hadoop18TapUtil.java:332)
    at cascading.tap.hadoop.util.Hadoop18TapUtil.moveTaskOutputs(Hadoop18TapUtil.java:332)
    at cascading.tap.hadoop.util.Hadoop18TapUtil.moveTaskOutputs(Hadoop18TapUtil.java:332)
    at cascading.tap.hadoop.util.Hadoop18TapUtil.moveTaskOutputs(Hadoop18TapUtil.java:332)
    at cascading.tap.hadoop.util.Hadoop18TapUtil.commitTask(Hadoop18TapUtil.java:174)
    at cascading.tap.hadoop.io.TapOutputCollector.close(TapOutputCollector.java:171)
    ... 11 more```

potential regression on scalding joinWithTiny on 4.5-wip

I've been testing scalding with newer cascading as a demo on a branch here: daniel-sudz/scalding#1.

I currently have the following bad output:

[info] - should merge and joinWithTiny shouldn't duplicate data *** FAILED ***
[info]   Set((1,3), (2,3), (3,3), (4,2)) was not equal to Set((1,2), (2,2), (3,2), (4,1)) (PlatformTest.scala:466)

it looks like there is some duplication going on considering 3 > 2 and 2 > 1. I saw that there was some previous discussion around this when cascading3 scalding branch was being developed before it got stalled. twitter/scalding#1592. The resolution there seemed to be a higher hadoop version so not really applicable here.

Not sure where to begin debugging this but would love some pointers.

Possible to keep java8 on 4.5 release?

hey @cwensel any chance we could keep java8 on the 4.5 branch at least? AWS EMR 6.X (latest release) still defaults on java8 for the hadoop3 flavors and it would be much appreciated to keep this support before they drop it. I would imagine that EMR is a very common use case for this type of library.

wrong about mapred/TaskCompletionEvent and mapreduce/TaskCompletionEvent

Exception in thread "main" java.lang.VerifyError: Inconsistent stackmap frames at branch target 83
Exception Details:
Location:
cascading/stats/hadoop/HadoopStepStats.captureDetail(Lcascading/stats/CascadingStats$Type;)V @83: aload_0
Reason:
Type '[Lorg/apache/hadoop/mapred/TaskCompletionEvent;' (current frame, locals[4]) is not assignable to '[Lorg/apache/hadoop/mapreduce/TaskCompletionEvent;' (stack map, locals[4])
Current Frame:
bci: @77
flags: { }
locals: { 'cascading/stats/hadoop/HadoopStepStats', 'cascading/stats/CascadingStats$Type', 'org/apache/hadoop/mapreduce/Job', integer, '[Lorg/apache/hadoop/mapred/TaskCompletionEvent;' }
stack: { integer }
Stackmap Frame:
bci: @83
flags: { }
locals: { 'cascading/stats/hadoop/HadoopStepStats', 'cascading/stats/CascadingStats$Type', 'org/apache/hadoop/mapreduce/Job', integer, '[Lorg/apache/hadoop/mapreduce/TaskCompletionEvent;' }
stack: { }

Hadoop version: 2.8.0

S3Tap parses bucket names incorrectly

Using an aws endpoint and, separately, an s3 URL of the form s3://my_tenant:my-bucket (including underscores)

In the S3Tap constructor (line 824 in my version) this line is causing problems:
this.bucketName = identifier.getHost();

The URI host is null. However the URI authority contains 'my_tenant:my-bucket', which is the full name of the bucket I need to access.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.