To learn more about this application and the supported properties, please review the following link.
spring-attic / spark-yarn Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
To learn more about this application and the supported properties, please review the following link.
From @sabbyanandan on May 6, 2016 22:34
From @rejurs on April 29, 2016 20:9
Hi Team,
Could you please help me to pass the spark-defaults.conf values when we are running java -jar sparkapp-yarn/target/sparkapp-yarn-task-1.0.0.BUILD-SNAPSHOT-exec.jar.
Copied from original issue: spring-cloud/spring-cloud-task-modules#36
Copied from original issue: spring-cloud/spring-cloud-task-app-starters#18
Hi, I'm running spring-cloud-dataflow-server-yarn-1.2.0.RELEASE server in docker cloudera container with SPARK2-2.1.0.cloudera1-1.cdh5.7.0.p0.120904-el6.parcel and hadoop-2.6.0-cdh5.11.1.
Server starts without any issues, but when I'am submitting spark task it fails with:
Unrecognized option: --spark.app-jar=hdfs://app/spark/basic-spark-0.1-SNAPSHOT-jar-with-dependencies.jar Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit.
I'm registering app and creating task using:
app register --type task --name spark-yarn --uri hdfs:///app/spark/spring-cloud-starter-task-spark-yarn-1.2.0.RELEASE.jar
task create spark --definition "spark-yarn --spark.app-name=my-test-pi --spark.app-class=app.SparkESIngester --spark.app-jar=hdfs://app/spark/basic-spark-0.1-SNAPSHOT-jar-with-dependencies.jar --sark.assembly-jar=hdfs://app/spark/spark-assembly-1.6.1-hadoop2.6.0.jar --spark.app-args=10"
task launch spark
If anyone is able to provide some assistance would me immensely appreciated, as nothing on the web makes sense to resolve this and get the spark task going!!!
Thank you!
Fails fast on startup:
$ java -jar spark-yarn-task-1.1.0.RELEASE.jar
...
2017-02-06 15:56:02.682 ERROR 6683 --- [ main] o.s.boot.SpringApplication : Application startup failed
org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.context.ApplicationContextException: Unable to start EmbeddedWebApplicationContext due to missing EmbeddedServletContainerFactory bean.
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:137) ~[spring-boot-1.4.1.RELEASE.jar!/:1.4.1.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:535) ~[spring-context-4.3.3.RELEASE.jar!/:4.3.3.RELEASE]
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122) ~[spring-boot-1.4.1.RELEASE.jar!/:1.4.1.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:761) [spring-boot-1.4.1.RELEASE.jar!/:1.4.1.RELEASE]
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.