Deploy HDP 2.3 with bdutil about bdutil HOT 2 OPEN

Were you able to get 2.3 running using bdutil ?
I tried going Ambari 2.1 - HDP 2.3 route and it worked with some manual work at the end.

Changed the following in ambari.conf
AMBARI_VERSION='2.1.0-1409'
AMBARI_REPO="http://s3.amazonaws.com/dev.hortonworks.com/ambari/centos6/2.x/BUILDS/${AMBARI_VERSION}/ambaribn.repo"
AMBARI_STACK='HDP'
AMBARI_STACK_VERSION='2.3'

The install failed

Sun Jul 19 14:57:08 EDT 2015: Exited 1 : gcloud --project=iron-potion-771 --quiet --verbosity=info compute ssh hadoop-m --command=sudo su -l -c "cd ${PWD} && ./update-ambari-config.sh" 2>>update-ambari-config_deploy.stderr 1>>update-ambari-config_deploy.stdout --ssh-flag=-tt --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-central1-a
Sun Jul 19 14:57:08 EDT 2015: Fetching on-VM logs from hadoop-m
Warning: Permanently added '104.197.109.200' (RSA) to the list of known hosts.
Sun Jul 19 14:57:11 EDT 2015: Command failed: wait ${SUBPROC} on line 326.
Sun Jul 19 14:57:11 EDT 2015: Exit code of failed command: 1
Sun Jul 19 14:57:11 EDT 2015: Detailed debug info available in file: /tmp/bdutil-20150719-143917-aAi/debuginfo.txt
Sun Jul 19 14:57:11 EDT 2015: Check console output for error messages and/or retry your command.

Error file ended with this ..

hadoop-m:tambari_wait status: INPROGRESS
hadoop-m:tambari_wait status: INPROGRESS
hadoop-m:tambari_wait status: INPROGRESS
hadoop-m:tambari_wait status: INPROGRESS
hadoop-m:t-test: Fatal internal error
hadoop-m:tjava.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found
hadoop-m:t at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
hadoop-m:t at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2638)
hadoop-m:t at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
hadoop-m:t at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
hadoop-m:t at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
hadoop-m:t at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
hadoop-m:t at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
hadoop-m:t at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
hadoop-m:t at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:325)
hadoop-m:t at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:235)
hadoop-m:t at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:218)
hadoop-m:t at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:201)
hadoop-m:t at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
hadoop-m:t at org.apache.hadoop.fs.FsShell.run(FsShell.java:287)
hadoop-m:t at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
hadoop-m:t at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
hadoop-m:t at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)
hadoop-m:tCaused by: java.lang.ClassNotFoundException: Class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem not found
hadoop-m:t at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
hadoop-m:t at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
hadoop-m:t ... 16 more
hadoop-m:tError: There was a problem accessing your configuration bucket using the GCS
hadoop-m:tconnector. Check configuration files. Also make sure have the GCS JSON API
hadoop-m:tenabled as described at https://developers.google.com/storage/docs/json_api/.
hadoop-m:t.

However the JSON API was fine .. but the hadoop-env.sh did not have the class path update to add location of the gcs connector jar.

Added the blow to hadoop env template in ambari config guy

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/usr/local/lib/hadoop/lib/gcs-connector-1.4.1-hadoop2.jar

Seems like everything is working but I guess I might need some more storage soon :)

hadoop fs -df -h gs://hadoop1213
15/07/19 19:48:22 INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.4.1-hadoop2
Filesystem Size Used Available Use%
gs://hadoop1213/ 8.0 E 0 8.0 E 0%

from bdutil.

Were you guys able to run hdp 2.3 with the gdutil utility ??

can you please share the ambari configuration to do so ??

from bdutil.