Comments (10)
christopherbozeman
commented on June 30, 2024
I was able to get Spark 1.3.1 HiveContext working with metastore on masternode in yarn-cluster by doing:
- changed master hive-site.xml to reference internal fqdn of master
- granted hive user permission in mysql auth to allow for db connections from other hosts besides the localhost
- added the following entries to the spark-submit:
--driver-class-path /home/hadoop/spark/lib/datanucleus-api-jdo-3.2.6.jar:/home/hadoop/spark/lib/datanucleu
s-core-3.2.10.jar:/home/hadoop/spark/lib/datanucleus-rdbms-3.2.9.jar:/home/hadoop/spark/classpath/emr/mysql-connector-java-5.1.30.jar:hive-site.xml --files /home/hadoop/spark/conf/
hive-site.xml
I have not tested this with Streaming, though.
from emr-bootstrap-actions.
christopherbozeman
commented on June 30, 2024
The first build of Spark 1.4.0 is up as version 1.4.0.a (see https://github.com/awslabs/emr-bootstrap-actions/blob/master/spark/VersionInformation.md and #125). You may request it by using the arguments -v 1.4.0.a. Currently, it is not the default for the bootstrap action.
from emr-bootstrap-actions.
gabrielcrowdtilt
commented on June 30, 2024
schweeet, demoing it this week.
from emr-bootstrap-actions.
erond
commented on June 30, 2024
Great, thanks!
I was just able to successfully complete a Spark 1.4.0 batch job lasting 1 hour, 30 minutes with no issues.
from emr-bootstrap-actions.
gauravtayal0
commented on June 30, 2024
Hi i am new to Emr cluster with custom bootstrap. I want to setup emr cluster with spark 1.4. Can you please let me know how can i achieve this.
from emr-bootstrap-actions.
jkleckner
commented on June 30, 2024
@tayal8, @christopherbozeman mentioned above already: You may request it by using the arguments -v 1.4.0.a. Currently, it is not the default for the bootstrap action.
from emr-bootstrap-actions.
gauravtayal0
commented on June 30, 2024
Yes i did that only, the command which i ran was:
aws emr create-cluster --name "Test cluster" --ami-version 3.8 --use-default-roles --ec2-attributes KeyName=emr --applications Name=Hive --instance-count 3 --instance-type m3.xlarge --bootstrap-actions Path=s3://support.elasticmapreduce/spark/install-spark,Args=["--v=1.4.0.a"]
But it terminates cluster giving me bootstrap failure. Is there anything wrong in this command.
from emr-bootstrap-actions.
christopherbozeman
commented on June 30, 2024
@tayal8 , the Args section would be Args=[-v,1.4.0.a]
from emr-bootstrap-actions.
gauravtayal0
commented on June 30, 2024
@christopherbozeman & @jkleckner : thanks it worked for me Args=[-v,1.4.0.a]. I am able to run spark 1.4.0.
from emr-bootstrap-actions.
christopherbozeman
commented on June 30, 2024
sparkR is now included in the 1.4.0 build version 1.4.0.b
from emr-bootstrap-actions.
Related Issues (20)
- bootstrapping opentsdb using emr-4.6.0, HBASE_HOME issue HOT 1
- Support Scala 2.11 HOT 1
- zookeeper version is invalid HOT 1
- Installing latest version of Impala on EMR HOT 10
- Permission denied error AMI 3.11.0 HOT 1
- Bootstrap for Apache Kylin HOT 3
- is there any plan to create one BA for JCE? HOT 1
- Error downloading file from Amazon S3 HOT 4
- Kafka support on EMR 5.x HOT 2
- Support jupyter notebook HOT 1
- Reading LZO files from Spark stand alone program HOT 1
- Persto 0.157.1 in EMR is facing issues regarding client side encryption AWS KMS Master Key HOT 1
- running an s3 jar file with dependencies HOT 1
- Installing latest version of Impala on EMR HOT 1
- Bootstrap for Sentry HOT 1
- Add bootstrap script to install netdata HOT 1
- sudo R command not found, when using the emR_bootstrap.sh
- Error while reading core-site.xml in elasticsearch bootstrap action HOT 1
- EMR cluster fails at boot strap HOT 1
- Bootstrap has execute failed to my shell script file on S3 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from emr-bootstrap-actions.