kakao / cuesheet Goto Github PK

Currently DependencyAnalyzer does not resolve dependency version conflicts while building assembly.

A simpler step to do is to make it include the latest version, and possibly provide an option to give a specific strategy for resolution

Currently CueSheet does not support submitting applications to YARN clusters that require Kerberos authentication.

The goal is to make CueSheet work in such YARN clusters, by adding Kerberos-related properties in application.conf as necessary.

Due to the nature of this framework, writing unit tests like conventional libraries is not easy.

However, for the stability of the codebase we should find a way to automate testing of basic features, such as YARN deployment, spark startup, etc.

Travis CI's sudo-enabled environment provides 7.5 GB of RAM, which may barely run ZooKeeper, HDFS, Hive, and YARN.

When a CueSheet application is running on YARN, it is typical that the JVM has duplicate JARs in the classpath. However, DependencyGraph prints too many warning messages that are very long, polluting the YARN application log.

It should shorten the warning messages, and possibly suppress or lower the log level of the message if the application is running on YARN.

When sc is instantiated in cluster mode, it does not pickup hive-conf.xml and other Hadoop configuration files as specified in spark.master configuration. Since other Hadoop configurations are already available from YARN, Hive configurations are usually the only thing that is left off.

CueSheet should pick up non-YARN Hadoop configurations when instantiating SparkContext here and here, possibly by setting sparkConf to contain Hadoop configurations when launched in cluster mode.

A workaround is possible, by explicitly specifying Hive configurations when extending CueSheet with its optional constructor parameter like:

object Example extends CueSheet({
  val conf = new org.apache.hadoop.conf.Configuration(false)
  conf.addResource(Thread.currentThread().getContextClassLoader.getResourceAsStream("quickstart/hive-site.xml"))
  conf.iterator().toSeq.map {
    e => ("spark.hadoop." + e.getKey, e.getValue)
  }
}: _*) {{

Even if -Dcuesheet.install is given, this version 0.10.0 submits a job to Yarn.

As I understand, if -Dcuesheet.install is defined, no job would be launched. But Cuesheet performs deployment then print a shell command.

Is it right?