googleclouddataproc / bdutil Goto Github PK

View Code? Open in Web Editor NEW

109.0 55.0 94.0 949 KB

[DEPRECATED] Script used to manage Hadoop and Spark instances on Google Compute Engine

Home Page: https://cloud.google.com/dataproc

License: Apache License 2.0

Shell 95.07% Python 3.04% PigLatin 1.02% Dockerfile 0.02% q 0.85%

bdutil's Introduction

This project has been deprecated. Please use Google Cloud Dataproc to create managed Apache Hadoop and Apache Spark instances on Google Compute Engine.

bdutil

bdutil is a command-line script used to manage Apache Hadoop and Apache Spark instances on Google Compute Engine. bdutil manages deployment, configuration, and shutdown of your Hadoop instances.

Requirements

bdutil depends on the Google Cloud SDK. bdutil is supported in any posix-compliant Bash v3 or greater shell.

Usage

See the QUICKSTART file in the docs directory to learn how to set up your Hadoop instances using bdutil.

Install and configure the Google Cloud SDK if you have already not done so
Clone this repository with git clone https://github.com/GoogleCloudPlatform/bdutil.git
Modify the following variables in the bdutil_env.sh file:
PROJECT - Set to the project ID for all bdutil commands. The project value will be overridden in the following order (where 1 overrides 2, and 2 overrides 3): * -p flag value, or if not specified then * PROJECT value in bdutil_env.sh, or if not specified then * gcloud default project value
CONFIGBUCKET - Set to a Google Compute Storage bucket that your project has read/write access to.
Run bdutil --help for a list of commands.

The script implements the following commands, which are very similar:

bdutil create creates and starts instances, but will not apply most configuration settings. You can call bdutil run_command_steps on instances afterward to apply configuration settings to them. Typically you wouldn't use this, but would use bdutil deploy instead.
bdutil deploy creates and starts instances with all the configuration options specified in the command line and any included configuration scripts.

Components installed

The latest release of bdutil is 1.3.5. This bdutil release installs the following versions of open source components:

Apache Hadoop - 1.2.1 (2.7.1 if you use the -e argument)
Apache Spark - 1.5.0
Apache Pig - 0.12
Apache Hive - 1.2.1

Documentation

The following documentation is useful for bdutil.

Quickstart - A guide on how to get started with bdutil quickly.
Jobs - How to submit jobs (work) to a bdutil cluster.
Monitoring - How to monitor bdutil cluster.
Shutdown - How shutdown a bdutil cluster.

bdutil's People

Contributors

Stargazers

Watchers

Forkers

mbrukman seanorama mxm ravwojdyla pmkc dennishuo ktzoumas sudheeshkatkam jirapong paramunna alunarbeach pbuszka saiganji colaberry csabatini waprin dansedov ajatix affini-tech upjohnc taragu paperdrip heathkit sduskis stev-0 eguldy paulnewson davidadamsphd ykko alexvanboxel levyx freemanbrandon nsabharwal saranreddy56 gruter kod3r pelim kevinsi4508 ibalashov robinus2 globalfishingwatch keyki tuliobraga deshpandetanmay pprado hfculver chiraggiri tmatsuo jcrutsinger digideskio semad luiz158 suresh88 dtsukiyama pankajworks rodriguezjf aman-ebay ralic surachart vkhokhla optionalg rakeshakula mizie betarelease rsekhon13 wko93 maartinpii bmisra josephlim75 sowmyadagala johanesalxd vectorijk maxlip davinash mkopaniov chinphuah pavlolobachov supernova106 vijayakumar75 ketan71 surajr3 rd1981 mtpradoc izu-stanley algarecu kate-avu jschwartzenberg isabella232 qpc-github prathapreddy123 cflowe amirpeles

bdutil's Issues

bdutil fails to build hdp

gsutil md gs://cloudgenius

bdutil -p beacloudgenius -b cloudgenius -P cg -z us-central1-f -e /home/user/bdutil/platforms/hdp/ambari_env.sh generate_config my.sh

cat my.sh

import_env /home/user/bdutil/platforms/hdp/ambari_env.sh
PROJECT=beacloudgenius
CONFIGBUCKET=cloudgenius
PREFIX=cg
GCE_ZONE=us-central1-f
GCE_MASTER_MACHINE_TYPE=n1-standard-4

bdutil -e my.sh deploy
Tue Sep 22 18:32:19 PDT 2015: Using local tmp dir for staging files: /tmp/bdutil-20150922-183219-iBw
Tue Sep 22 18:32:19 PDT 2015: Using custom environment-variable file(s): bdutil_env.sh my.sh
Tue Sep 22 18:32:19 PDT 2015: Reading environment-variable file: /home/user/bdutil/bdutil_env.sh
Tue Sep 22 18:32:19 PDT 2015: Reading environment-variable file: ./my.sh
Importing dependent env file: /home/user/bdutil/platforms/hdp/ambari_env.sh
Importing dependent env file: /home/user/bdutil/platforms/hdp/ambari_manual_env.sh
Importing dependent env file: /home/user/bdutil/hadoop2_env.sh
Importing dependent env file: /home/user/bdutil/platforms/hdp/ambari.conf
Importing dependent env file: /home/user/bdutil/platforms/hdp/ambari_functions.sh
Deploy cluster with following settings?
      CONFIGBUCKET='cloudgenius'
      PROJECT='beacloudgenius'
      GCE_IMAGE='centos-6'
      GCE_ZONE='us-central1-f'
      GCE_NETWORK='default'
      PREEMPTIBLE_FRACTION=0.0
      PREFIX='cg'
      NUM_WORKERS=4
      MASTER_HOSTNAME='cg-m'
      WORKERS='cg-w-0 cg-w-1 cg-w-2 cg-w-3'
      BDUTIL_GCS_STAGING_DIR='gs://cloudgenius/bdutil-staging/cg-m'
            MASTER_ATTACHED_PD='cg-m-pd'
      WORKER_ATTACHED_PDS='cg-w-0-pd cg-w-1-pd cg-w-2-pd cg-w-3-pd'
      (y/n) y
Tue Sep 22 18:32:21 PDT 2015: Checking for existence of gs://cloudgenius...
gs://cloudgenius/
Tue Sep 22 18:32:21 PDT 2015: Checking for existence of gs://hadoop-dist/hadoop-2.7.1.tar.gz...
Tue Sep 22 18:32:22 PDT 2015: Checking upload files...
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/conf/hadoop2/core-template.xml'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/conf/hadoop2/mapred-template.xml'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/conf/hadoop2/yarn-template.xml'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/conf/hadoop2/gcs-core-template.xml'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/conf/hadoop2/bigtable-hbase-site-template.xml'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/conf/hadoop2/hdfs-template.xml'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/conf/hadoop2/bq-mapred-template.xml'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/libexec/hadoop_helpers.sh'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/libexec/configure_mrv2_mem.py'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/hadoop2_env.sh'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/platforms/hdp/ambari.conf'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/platforms/hdp/ambari_functions.sh'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/libexec/hadoop_helpers.sh'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/platforms/hdp/configuration.json'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/platforms/hdp/resources/public-hostname-gcloud.sh'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/platforms/hdp/resources/thp-disable.sh'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/platforms/hdp/ambari_manual_env.sh'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/platforms/hdp/create_blueprint.py'
Tue Sep 22 18:32:22 PDT 2015: Verified '/home/user/bdutil/platforms/hdp/ambari_env.sh'
Tue Sep 22 18:32:22 PDT 2015: Generating 12 command groups...
Tue Sep 22 18:32:22 PDT 2015: Done generating remote shell scripts.
Tue Sep 22 18:32:22 PDT 2015: Creating attached worker disks: cg-w-0-pd cg-w-1-pd cg-w-2-pd cg-w-3-pd
....Tue Sep 22 18:32:23 PDT 2015: Creating attached master disk: cg-m-pd
.Tue Sep 22 18:32:23 PDT 2015: Done creating disks!
Tue Sep 22 18:32:23 PDT 2015: Waiting on async 'disks create' jobs to finish. Might take a while...
.....
Tue Sep 22 18:32:32 PDT 2015: Creating worker instances: cg-w-0 cg-w-1 cg-w-2 cg-w-3
....Tue Sep 22 18:32:32 PDT 2015: Creating master instance: cg-m
.Tue Sep 22 18:32:33 PDT 2015: Waiting on async 'instances create' jobs to finish. Might take a while...
.....
Tue Sep 22 18:33:30 PDT 2015: Instances all created. Entering polling loop to wait for ssh-ability
.....Tue Sep 22 18:33:31 PDT 2015: Waiting on async 'wait_for_ssh' jobs to finish. Might take a while...
.....
Tue Sep 22 18:33:34 PDT 2015: Instances all ssh-able
Tue Sep 22 18:33:34 PDT 2015: Staging file and script dependencies into gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/...
Copying file:///home/user/bdutil/conf/hadoop2/mapred-template.xml [Content-Type=application/xml]...
Copying file:///home/user/bdutil/conf/hadoop2/core-template.xml [Content-Type=application/xml]...
Copying file:///home/user/bdutil/conf/hadoop2/yarn-template.xml [Content-Type=application/xml]...
Copying file:///home/user/bdutil/conf/hadoop2/gcs-core-template.xml [Content-Type=application/xml]...
Copying file:///home/user/bdutil/conf/hadoop2/bigtable-hbase-site-template.xml [Content-Type=application/xml]...
Copying file:///home/user/bdutil/conf/hadoop2/hdfs-template.xml [Content-Type=application/xml]...
Copying file:///home/user/bdutil/conf/hadoop2/bq-mapred-template.xml [Content-Type=application/xml]...
Uploading   ...aging/cg-m/20150922-183219-iBw/core-template.xml: 749 B/749 B    
Copying file:///home/user/bdutil/libexec/hadoop_helpers.sh [Content-Type=text/x-sh]...
Uploading   ...ing/cg-m/20150922-183219-iBw/mapred-template.xml: 5.43 KiB/5.43 KiB    
Uploading   ...aging/cg-m/20150922-183219-iBw/yarn-template.xml: 2.81 KiB/2.81 KiB    
Copying file:///home/user/bdutil/libexec/configure_mrv2_mem.py [Content-Type=text/x-python]...
Copying file:///home/user/bdutil/hadoop2_env.sh [Content-Type=text/x-sh]...
Uploading   ...g/cg-m/20150922-183219-iBw/gcs-core-template.xml: 2.62 KiB/2.62 KiB    
Uploading   ...0922-183219-iBw/bigtable-hbase-site-template.xml: 1.35 KiB/1.35 KiB    
Uploading   ...aging/cg-m/20150922-183219-iBw/hdfs-template.xml: 2.72 KiB/2.72 KiB    
Uploading   .../cg-m/20150922-183219-iBw/bq-mapred-template.xml: 769 B/769 B     
Uploading   ...aging/cg-m/20150922-183219-iBw/hadoop_helpers.sh: 5.98 KiB/5.98 KiB    
Uploading   ...-staging/cg-m/20150922-183219-iBw/hadoop2_env.sh: 2.92 KiB/2.92 KiB    
Uploading   ...g/cg-m/20150922-183219-iBw/configure_mrv2_mem.py: 7.28 KiB/7.28 KiB    
Copying file:///home/user/bdutil/platforms/hdp/ambari.conf [Content-Type=application/octet-stream]...
Copying file:///home/user/bdutil/platforms/hdp/ambari_functions.sh [Content-Type=text/x-sh]...
Copying file:///home/user/bdutil/libexec/hadoop_helpers.sh [Content-Type=text/x-sh]...
Copying file:///home/user/bdutil/platforms/hdp/configuration.json [Content-Type=application/json]...
Copying file:///home/user/bdutil/platforms/hdp/resources/public-hostname-gcloud.sh [Content-Type=text/x-sh]...
Uploading   ...til-staging/cg-m/20150922-183219-iBw/ambari.conf: 2.56 KiB/2.56 KiB    
Copying file:///home/user/bdutil/platforms/hdp/resources/thp-disable.sh [Content-Type=text/x-sh]...
Copying file:///home/user/bdutil/platforms/hdp/ambari_manual_env.sh [Content-Type=text/x-sh]...
Copying file:///home/user/bdutil/platforms/hdp/create_blueprint.py [Content-Type=text/x-python]...
Copying file:///home/user/bdutil/platforms/hdp/ambari_env.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/hadoop-env-setup.sh [Content-Type=text/x-sh]...
Uploading   ...ing/cg-m/20150922-183219-iBw/ambari_functions.sh: 3.62 KiB/3.62 KiB    
Uploading   ...aging/cg-m/20150922-183219-iBw/hadoop_helpers.sh: 5.98 KiB/5.98 KiB    
Uploading   ...-m/20150922-183219-iBw/public-hostname-gcloud.sh: 174 B/174 B    
Uploading   ...ging/cg-m/20150922-183219-iBw/configuration.json: 863 B/863 B    
Uploading   ...ing/cg-m/20150922-183219-iBw/hadoop-env-setup.sh: 30.36 KiB/30.36 KiB    
Uploading   ...-staging/cg-m/20150922-183219-iBw/thp-disable.sh: 451 B/451 B    
Uploading   ...l-staging/cg-m/20150922-183219-iBw/ambari_env.sh: 1.82 KiB/1.82 KiB    
Uploading   ...ing/cg-m/20150922-183219-iBw/create_blueprint.py: 4.12 KiB/4.12 KiB    
Uploading   ...ng/cg-m/20150922-183219-iBw/ambari_manual_env.sh: 3.47 KiB/3.47 KiB    
Copying file:///tmp/bdutil-20150922-183219-iBw/deploy-ssh-master-setup.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/deploy-core-setup.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/deploy-master-nfs-setup.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/deploy-client-nfs-setup.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/deploy-ssh-worker-setup.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/deploy-start.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/install_connectors.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/deploy_start2.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/ambari-setup.sh [Content-Type=text/x-sh]...
Uploading   ...m/20150922-183219-iBw/deploy-ssh-master-setup.sh: 1.59 KiB/1.59 KiB    
Uploading   ...m/20150922-183219-iBw/deploy-client-nfs-setup.sh: 1.53 KiB/1.53 KiB    
Uploading   ...staging/cg-m/20150922-183219-iBw/deploy-start.sh: 1.27 KiB/1.27 KiB    
Uploading   ...m/20150922-183219-iBw/deploy-master-nfs-setup.sh: 3.84 KiB/3.84 KiB    
Uploading   ...m/20150922-183219-iBw/deploy-ssh-worker-setup.sh: 1.37 KiB/1.37 KiB    
Uploading   ...staging/cg-m/20150922-183219-iBw/ambari-setup.sh: 9.22 KiB/9.22 KiB    
Uploading   ...taging/cg-m/20150922-183219-iBw/deploy_start2.sh: 1.41 KiB/1.41 KiB    
Uploading   ...g/cg-m/20150922-183219-iBw/install_connectors.sh: 6.19 KiB/6.19 KiB    
Uploading   ...ng/cg-m/20150922-183219-iBw/deploy-core-setup.sh: 28.41 KiB/28.41 KiB    
Copying file:///tmp/bdutil-20150922-183219-iBw/install-gcs-connector-on-ambari.sh [Content-Type=text/x-sh]...
Uploading   ...22-183219-iBw/install-gcs-connector-on-ambari.sh: 1.47 KiB/1.47 KiB    
Copying file:///tmp/bdutil-20150922-183219-iBw/install-ambari-components.sh [Content-Type=text/x-sh]...
Copying file:///tmp/bdutil-20150922-183219-iBw/update-ambari-config.sh [Content-Type=text/x-sh]...
Uploading   ...20150922-183219-iBw/install-ambari-components.sh: 3.9 KiB/3.9 KiB    
Uploading   ...cg-m/20150922-183219-iBw/update-ambari-config.sh: 2.88 KiB/2.88 KiB    
Tue Sep 22 18:33:36 PDT 2015: Downloading staging files onto VMs...
Tue Sep 22 18:33:36 PDT 2015: Invoking on workers: gcloud --quiet components update gsutil; gsutil -m cp gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/core-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/mapred-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/yarn-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/gcs-core-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/bigtable-hbase-site-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hdfs-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/bq-mapred-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop_helpers.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/configure_mrv2_mem.py gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop2_env.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari.conf gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_functions.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop_helpers.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/configuration.json gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/public-hostname-gcloud.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/thp-disable.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_manual_env.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/create_blueprint.py gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_env.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop-env-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-ssh-master-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-core-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-master-nfs-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-client-nfs-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-ssh-worker-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-start.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install_connectors.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy_start2.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install-gcs-connector-on-ambari.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/update-ambari-config.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install-ambari-components.sh . && chmod 755 *
....Tue Sep 22 18:33:37 PDT 2015: Invoking on master: gcloud --quiet components update gsutil; gsutil -m cp gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/core-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/mapred-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/yarn-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/gcs-core-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/bigtable-hbase-site-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hdfs-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/bq-mapred-template.xml gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop_helpers.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/configure_mrv2_mem.py gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop2_env.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari.conf gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_functions.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop_helpers.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/configuration.json gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/public-hostname-gcloud.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/thp-disable.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_manual_env.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/create_blueprint.py gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_env.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop-env-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-ssh-master-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-core-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-master-nfs-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-client-nfs-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-ssh-worker-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-start.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install_connectors.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy_start2.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari-setup.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install-gcs-connector-on-ambari.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/update-ambari-config.sh gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install-ambari-components.sh . && chmod 755 *
.Tue Sep 22 18:33:37 PDT 2015: Waiting on async 'ssh' jobs to finish. Might take a while...
.....
Tue Sep 22 18:33:51 PDT 2015: Uploads of shell scripts finished, deleting staging files...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/core-template.xml...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/mapred-template.xml...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/yarn-template.xml...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/gcs-core-template.xml...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/bigtable-hbase-site-template.xml...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hdfs-template.xml...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/bq-mapred-template.xml...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop_helpers.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/configure_mrv2_mem.py...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop2_env.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari.conf...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_functions.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop_helpers.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/configuration.json...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/public-hostname-gcloud.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/thp-disable.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_manual_env.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/create_blueprint.py...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari_env.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/hadoop-env-setup.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-ssh-master-setup.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-core-setup.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-master-nfs-setup.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-client-nfs-setup.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-ssh-worker-setup.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy-start.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install_connectors.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/deploy_start2.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/ambari-setup.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install-gcs-connector-on-ambari.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/update-ambari-config.sh...
Removing gs://cloudgenius/bdutil-staging/cg-m/20150922-183219-iBw/install-ambari-components.sh...
CommandException: 1 files/objects could not be removed.
Tue Sep 22 18:33:55 PDT 2015: Warning. Failed to delete all staging files.
Tue Sep 22 18:33:55 PDT 2015: Invoking on workers: ./ambari-setup.sh
....Tue Sep 22 18:33:56 PDT 2015: Invoking on master: ./ambari-setup.sh
.Tue Sep 22 18:33:56 PDT 2015: Waiting on async 'ssh' jobs to finish. Might take a while...
.....
Tue Sep 22 18:35:34 PDT 2015: Step 'ambari-setup,ambari-setup' done...
Tue Sep 22 18:35:34 PDT 2015: Invoking on master: ./deploy-master-nfs-setup.sh
.Tue Sep 22 18:35:34 PDT 2015: Waiting on async 'ssh' jobs to finish. Might take a while...
.
Tue Sep 22 18:35:41 PDT 2015: Step 'deploy-master-nfs-setup,*' done...
Tue Sep 22 18:35:41 PDT 2015: Invoking on workers: ./deploy-client-nfs-setup.sh
....Tue Sep 22 18:35:42 PDT 2015: Invoking on master: ./deploy-client-nfs-setup.sh
.Tue Sep 22 18:35:42 PDT 2015: Waiting on async 'ssh' jobs to finish. Might take a while...
.....
Tue Sep 22 18:35:55 PDT 2015: Step 'deploy-client-nfs-setup,deploy-client-nfs-setup' done...
Tue Sep 22 18:35:55 PDT 2015: Invoking on master: ./install-ambari-components.sh
.Tue Sep 22 18:35:55 PDT 2015: Waiting on async 'ssh' jobs to finish. Might take a while...
Tue Sep 22 19:36:14 PDT 2015: Exited 1 : gcloud --project=beacloudgenius --quiet --verbosity=info compute ssh cg-m --command=sudo su -l -c "cd ${PWD} && ./install-ambari-components.sh" 2>>install-ambari-components_deploy.stderr 1>>install-ambari-components_deploy.stdout --ssh-flag=-tt --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-central1-f
Tue Sep 22 19:36:14 PDT 2015: Fetching on-VM logs from cg-m
Warning: Permanently added '104.197.16.133' (RSA) to the list of known hosts.
Tue Sep 22 19:36:16 PDT 2015: Command failed: wait ${SUBPROC} on line 326.
Tue Sep 22 19:36:16 PDT 2015: Exit code of failed command: 1
Tue Sep 22 19:36:16 PDT 2015: Detailed debug info available in file: /tmp/bdutil-20150922-183219-iBw/debuginfo.txt
Tue Sep 22 19:36:16 PDT 2015: Check console output for error messages and/or retry your command.
(foobar)

Unable to start Ambari NameNode Security Exception

When deploying ambari, script was stuck at install-ambari-components.sh
On logging into the UI and checking the error, the main problem seems to be that the NameNode (of the master instance) is not starting due to java security exception.
I have tried to copy the appropriate oracle policy files (for JDK 7) into the security folders of the jre on all the master and worker instances and have rebooted using 'sudo reboot' on all the instance. But starting the components for the master instance (in ambari UI), still fails with same security exception.
On further investigation, it seems that the jdk used in the instance is not oracle but openJDK. But this means that this security exception should not be an issue since openJDK seems to have unlimited policy files.

Is there some other configuration/ workarounds that would help with deploying ambari/ HDP on GCP?

Or are there better ways to deploy HDP on GCP?

(I am using free tier, with option -n 2 -m n1-standard-2 on deploying ambari at zone us-central1-a)

The log with the main exception is as below:

18/02/09 02:29:42 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=true 18/02/09 02:29:42 INFO blockmanagement.BlockManager: dfs.block.access.key.update.interval=600 min(s), dfs.block.access.token.lifetime=600 min(s), dfs.encrypt.data.transfer.algorithm=null 18/02/09 02:29:42 ERROR namenode.NameNode: Failed to start namenode. java.lang.ExceptionInInitializerError at javax.crypto.KeyGenerator.nextSpi(KeyGenerator.java:341) at javax.crypto.KeyGenerator.<init>(KeyGenerator.java:169) at javax.crypto.KeyGenerator.getInstance(KeyGenerator.java:224) at org.apache.hadoop.security.token.SecretManager.<init>(SecretManager.java:143) at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.<init>(BlockTokenSecretManager.java:120) at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.<init>(BlockTokenSecretManager.java:111) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createBlockTokenSecretManager(BlockManager.java:437) at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.<init>(BlockManager.java:324) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:744) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:704) at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1125) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1571) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1709) Caused by: java.lang.SecurityException: Can not initialize cryptographic mechanism at javax.crypto.JceSecurity.<clinit>(JceSecurity.java:94) ... 13 more Caused by: java.lang.SecurityException: Cannot locate policy or framework files! at javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:317) at javax.crypto.JceSecurity.access$000(JceSecurity.java:50) at javax.crypto.JceSecurity$1.run(JceSecurity.java:86) at java.security.AccessController.doPrivileged(Native Method) at javax.crypto.JceSecurity.<clinit>(JceSecurity.java:83) ... 13 more 18/02/09 02:29:42 INFO util.ExitUtil: Exiting with status 1 18/02/09 02:29:42 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at hadoop-m.c.hdp-989.internal/10.128.0.3 ************************************************************/ yes: standard output: Broken pipe yes: write error

more than one non-boot standard PD?

It wouldn't be too difficult to do myself, but just wanted to make sure I am correct in assessing that the tool supports provisioning one boot PD and one non-boot standard PD per machine?

I saw an option for attaching SSDs, but what I want is to attach multiple standard PDs to each worker during the deployment

gsutil failing

Hi,

Since yesterday, and haven't changed anything on our side, we are getting the error message printed bellow when running 'gsutil' on machines deployed using bdutil.
We can fix this problem by running 'gcloud' command on the newly created machines.
It seems to be a problem introduced by the latest version of gsutil and affect bdutil as it does a "gcloud --quiet components update gsutil" at some point.
We are using the latest version (1.1.0), but it affects previous versions as well.

xxxx_xxxx_xxx@hadoop-m:~$ gsutil
Traceback (most recent call last):
File "/usr/local/bin/../share/google/google-cloud-sdk/bin/bootstrapping/gsutil.py", line 59, in
bootstrapping.CommandStart('gsutil', component_id='gsutil')
File "/usr/local/share/google/google-cloud-sdk/bin/bootstrapping/bootstrapping.py", line 188, in CommandStart
metrics.Executions(command_name, version)
File "/usr/local/share/google/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/metrics.py", lin
e 230, in Executions
_CollectGAEvent('Executions', command_name, version_string)
File "/usr/local/share/google/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/metrics.py", lin
e 193, in _CollectGAEvent
collector = _MetricsCollector.GetCollector()
File "/usr/local/share/google/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/metrics.py", lin
e 51, in GetCollector
_MetricsCollector._instance = _MetricsCollector()
File "/usr/local/share/google/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/metrics.py", lin
e 87, in init
('cid', _MetricsCollector._GetCID()),
File "/usr/local/share/google/google-cloud-sdk/bin/bootstrapping/../../lib/googlecloudsdk/core/metrics.py", lin
e 111, in _GetCID
with open(uuid_path, 'w') as f:
IOError: [Errno 2] No such file or directory: '/home/xxxx_xxxx_xxx/.config/gcloud/.metricsUUID'

Error Deploy bdutil : Invalid value for field resource.networkInterfaces

Please kindly help,
I already configure bdutil_env.sh, and there's no field for resource.networkInterfaces.
when I exec: ./bdutil -e extensions/flink/flink_env.sh deploy

this error always happens:

******************* gcloud compute stdout *******************

******************* gcloud compute stderr *******************
WARNING: Image aliases are deprecated and will be removed in a future version. Please use --image-family=debian-8 and --image-project=debian-cloud instead.
WARNING: Image aliases are deprecated and will be removed in a future version. Please use --image-family=debian-8 and --image-project=debian-cloud instead.
WARNING: Image aliases are deprecated and will be removed in a future version. Please use --image-family=debian-8 and --image-project=debian-cloud instead.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Invalid value for field 'resource.networkInterfaces[0]': ''. Subnetwork should be specified for custom subnetmode network

ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Invalid value for field 'resource.networkInterfaces[0]': ''. Subnetwork should be specified for custom subnetmode network

ERROR: (gcloud.compute.instances.create) Could not fetch resource:
 - Invalid value for field 'resource.networkInterfaces[0]': ''. Subnetwork should be specified for custom subnetmode network


************ ERROR logs from gcloud compute stderr ************
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
ERROR: (gcloud.compute.instances.create) Could not fetch resource:

bdutil - on server stop/start - hdfs not able to find fs

Instead of having to delete the bdutil, we were hoping to stop and restart the server to avoid having to setup the spark applications from start. Also, on delete it deletes hdfs and we loose all the model data that we generated. However, on restart hdfs goes into safe mode and does not recognize the fs.

Following are the two messages that we see at this time:
Safe mode is ON. The reported blocks 0 needs additional 10 blocks to reach the threshold 0.9990 of total blocks 10. The number of live datanodes 0 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.

Zero blocklocations for /Cluster/WSSE. Name node is in safe mode. The reported blocks 0 needs additional 10 blocks to reach the threshold 0.9990 of total blocks 10. The number of live datanodes 0 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached.

Also, i tried to force it out of safemode but got the following message:
vibhu@anintelcluster:/home/hadoop/hadoop-install/bin$ ./hdfs dfsadmin -safemode leave
15/07/03 19:42:27 INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.4.0-hadoop2
safemode: FileSystem gs://anintelcluster/ is not an HDFS file system
Usage: hdfs dfsadmin [-safemode enter | leave | get | wait]

Please let us know if we don't have the option to stop/start - or is there something we can do on restart and get hdfs out of safe mode.

Instance Metadata

Is there a way to pass metadata to the provisioned instances?

In particular I'd like to pass a "startup-script-url" metadata parameter to the instances that bdutil creates:

--metadata startup-script-url=startup-script-url=gs://bucket/bootstrap.sh

so I can kick off a custom configuration of my nodes

This feature is documentation here:
https://cloud.google.com/sdk/gcloud/reference/compute/instances/create

Cluster creation is failing with "Failed to find image for alias [debian-7-backports]..."

When attempting to create a cluster, bdutil will fail with the following message:
ERROR: (gcloud.compute.instances.create) Failed to find image for alias [debian-7-backports] in public image project [debian-cloud].

The simplest workaround right now is to change the image in bdutil_env, or specify a -i command line flag, using 'backports-debian-7-wheezy-v20160531' as the image name.

Spark eventlog directory points to GCS even if default_fs is set to hdfs

Right now spark.eventLog.dir gets set to a GCS path regardless of what DEFAULT_FS is set for deployment; this means if a deployment intentionally disables GCS accessibility, e.g. by removing external IP addresses, then even an HDFS-only setup doesn't work for Spark.

The temporary workaround is to manually edit spark.eventLog.dir on the master's /home/hadoop/spark-install/conf/spark-defaults.conf to something like hdfs:///spark-eventlog-base and to run hadoop fs -mkdir -p hdfs:///spark-eventlog-base, or to set spark.eventLog.enabled to false.

We can fix this to automatically derive the right path based on the default filesystem. Unfortunately Spark doesn't appear to correctly pick up the fs.default.name automatically for schemeless paths, possibly because of classloading ordering issues so that the path is resolved before default core-site.xml has been loaded; schemeless settings end up with something like:

java.lang.IllegalArgumentException: Log directory file:/spark-eventlog-base/dhuo-noip-m does not exist.

bdutil ./hadoop-validate-setup.sh - MAPRED job stuck

I have been struggling with getting the bdutil based gce master only server to run mapred jobs.
./bdutil deploy -e hadoop2_env.sh --bucket anintelcluster -n 0 -P anintelcluster
Above command sets up the hadoop2 on the master server, i can see the yarn page.
However, once I run the hadoop validate command, the job gets stuck.
15/06/23 18:58:34 INFO gcs.GoogleHadoopFileSystemBase: GHFS version: 1.4.0-hadoop2
15/06/23 18:58:36 INFO client.RMProxy: Connecting to ResourceManager at anintelcluster-m/10.240.98.45:8032
15/06/23 18:58:38 INFO terasort.TeraSort: Generating 100000 using 0
15/06/23 18:58:40 INFO mapreduce.JobSubmitter: number of splits:0
15/06/23 18:58:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1435085683024_0001
15/06/23 18:58:42 INFO impl.YarnClientImpl: Submitted application application_1435085683024_0001
15/06/23 18:58:42 INFO mapreduce.Job: The url to track the job: http://anintelcluster-m:8088/proxy/application_1435085683024_0001/
15/06/23 18:58:42 INFO mapreduce.Job: Running job: job_1435085683024_0001

hortonworks on mac os 10.12.6

Failing when trying to set up hortonworks on mac has anyone see this problem

Fri 1 Sep 2017 07:59:08 BST: Command failed: wait ${SUBPROC} on line 326.
Fri 1 Sep 2017 07:59:08 BST: Exit code of failed command: 1
Fri 1 Sep 2017 07:59:08 BST: Detailed debug info available in file: /tmp/bdutil-20170901-075837-upC/debuginfo.txt
Fri 1 Sep 2017 07:59:08 BST: Check console output for error messages and/or retry your command.

******************* gcloud compute stdout *******************

******************* gcloud compute stderr *******************
ERROR: (gcloud.compute.disks.create) Could not fetch resource:

Quota 'DISKS_TOTAL_GB' exceeded. Limit: 2048.0

ERROR: (gcloud.compute.disks.create) Could not fetch resource:

Quota 'DISKS_TOTAL_GB' exceeded. Limit: 2048.0

ERROR: (gcloud.compute.disks.create) Could not fetch resource:

Quota 'DISKS_TOTAL_GB' exceeded. Limit: 2048.0

ERROR: (gcloud.compute.disks.create) Could not fetch resource:

Quota 'DISKS_TOTAL_GB' exceeded. Limit: 2048.0

ERROR: (gcloud.compute.disks.create) Could not fetch resource:

Quota 'DISKS_TOTAL_GB' exceeded. Limit: 2048.0

************ ERROR logs from gcloud compute stderr ************
ERROR: (gcloud.compute.disks.create) Could not fetch resource:
ERROR: (gcloud.compute.disks.create) Could not fetch resource:
ERROR: (gcloud.compute.disks.create) Could not fetch resource:
ERROR: (gcloud.compute.disks.create) Could not fetch resource:
ERROR: (gcloud.compute.disks.create) Could not fetch resource:

Bump to hadoop 0.10.1

Could you bump the GCloud bucket and flink_env to flink 0.10.1?

Cheers!

gcloud compute ssh tunneling refusing connection

Hello,

I'm trying to set up a secure ssh tunnel by running this command in the terminal
gcloud compute ssh --zone=
--ssh-flag="-D 1080" --ssh-flag="-N" --ssh-flag="-n"
Then chrome
--proxy-server="socks5://localhost:1080"
--host-resolver-rules="MAP * 0.0.0.0 , EXCLUDE localhost"
--user-data-dir=/tmp/
to access Ambari UI page.

The ssh debug says:
channel 7: open failed: connect failed: connection refused.

Any idea what is going wrong or where am I making mistake?

Thanks in advanced.

subnetwork error

WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.
WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.
WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.
ERROR: (gcloud.compute.instances.create) Could not fetch resource:

Invalid value for field 'resource.networkInterfaces[0]': ''. No default subnetwork was found in the region of the instance.

ERROR: (gcloud.compute.instances.create) Could not fetch resource:

Invalid value for field 'resource.networkInterfaces[0]': ''. No default subnetwork was found in the region of the instance.

ERROR: (gcloud.compute.instances.create) Could not fetch resource:

Invalid value for field 'resource.networkInterfaces[0]': ''. No default subnetwork was found in the region of the instance.

************ ERROR logs from gcloud compute stderr ************
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
ERROR: (gcloud.compute.instances.create) Could not fetch resource:
ERROR: (gcloud.compute.instances.create) Could not fetch resource:

Add a "version" command|flag

Google documentation references 'versions' of bdutil:

https://cloud.google.com/hadoop/bdutil#components-installed
https://cloud.google.com/hadoop/downloads#command-line_utilities

There is a version property albeit somewhat hidden:

bdutil --help | grep "version"
bdutil version: 1.3.5-SNAPSHOT

BUT bdutil has no explicit command|flag that surfaces the version:

Recommend: add a "version" command|flag

ambari deployment broken since mid-Nov due to upstream config change

Currently, "./bdutil -e ambari deploy" fails about 10 minutes into the Ambari installation step with something like:

Thu Dec 10 10:21:11 PST 2015: Waiting on async 'ssh' jobs to finish. Might take a while...
Thu Dec 10 10:32:05 PST 2015: Exited 1 : gcloud --project=project --quiet --verbosity=info compute ssh dhuo-bdutil-m --command=sudo su -l -c "cd ${PWD} && ./install-ambari-components.sh" 2>>install-ambari-components_deploy.stderr 1>>install-ambari-components_deploy.stdout --ssh-flag=-tt --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-east1-c

Inside the debuginfo.txt that bdutil prints out, you may find something like:

dhuo-bdutil-m:  ==> install-ambari-components_deploy.stderr <==
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: PENDING
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: PENDING
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: PENDING
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: PENDING
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: PENDING
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  ambari_wait status: INPROGRESS
dhuo-bdutil-m:  Ambari operiation failed with status: FAILED

Logging into the Ambari GUI and clicking on failed operations shows the ResourceManager failed to come up; digging up ResourceManager startup logs shows something like:

2015-12-10 18:31:59,431 INFO  service.AbstractService (AbstractService.java:noteFailure(272)) - Service org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler failed in state INITED; cause: java.lang.IllegalArgumentException: Illegal capacity of -1.0 for node-label=default in queue=root, valid capacity should in range of [0, 100].
java.lang.IllegalArgumentException: Illegal capacity of -1.0 for node-label=default in queue=root, valid capacity should in range of [0, 100].
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueueCapacity(CapacitySchedulerConfiguration.java:465)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity(CapacitySchedulerConfiguration.java:477)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUtils.java:143)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils.java:122)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCSQueue.java:99)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:242)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:109)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:100)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:589)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:465)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:297)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:326)
  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
  at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:576)
  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1016)
  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:269)
  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1237)
2015-12-10 18:31:59,433 INFO  service.AbstractService (AbstractService.java:noteFailure(272)) - Service RMActiveServices failed in state INITED; cause: java.lang.IllegalArgumentException: Illegal capacity of -1.0 for node-label=default in queue=root, valid capacity should in range of [0, 100].
java.lang.IllegalArgumentException: Illegal capacity of -1.0 for node-label=default in queue=root, valid capacity should in range of [0, 100].
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.internalGetLabeledQueueCapacity(CapacitySchedulerConfiguration.java:465)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerConfiguration.getLabeledQueueCapacity(CapacitySchedulerConfiguration.java:477)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadCapacitiesByLabelsFromConf(CSQueueUtils.java:143)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueUtils.loadUpdateAndCheckCapacities(CSQueueUtils.java:122)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupConfigurableCapacities(AbstractCSQueue.java:99)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.setupQueueConfigs(AbstractCSQueue.java:242)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setupQueueConfigs(ParentQueue.java:109)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.<init>(ParentQueue.java:100)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:589)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:465)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:297)
  at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:326)
  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
  at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:576)
  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1016)
  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:269)
  at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
  at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1237)

problem deploy hdp cluster

Hello,

When I launch the deploy command : ./bdutil -e platforms/hdp/ambari_env.sh deploy

The deployment end with errors :

..Wed Feb 11 15:59:12 CET 2015: Command failed: wait ${SUBPROC} on line 308.
Wed Feb 11 15:59:12 CET 2015: Exit code of failed command: 1
Wed Feb 11 15:59:12 CET 2015: Detailed debug info available in file: /tmp/bdutil-20150211-155631-ezD/debuginfo.txt
Wed Feb 11 15:59:12 CET 2015: Check console output for error messages and/or retry your command.
iMac-de-****:bdutil-vc a****$ cat /tmp/bdutil-20150211-154518-AeG/debuginfo.txt

When I explore the error file /tmp/bdutil-20150211-154518-AeG/debuginfo.txt

************ ERROR logs from gcloud compute stderr ************
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [1].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [1].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [1].

Problem seem come internally from bdutil generated command :

gcloud --project=xxxxxxxx --quiet --verbosity=info compute ssh hadoop-w-0 --command=exit 0 --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=europe-west1-c

Some one know how resolve this issue ?

Thanks

Cluster creation can fail if apt-get update has warnings

When running apt-get update any warnings produced, such as failed package index downloads, do not cause the exit status of apt-get to be non-zero and as a result do not cause bdutil retry logic to kick in.

works with centos-6 but not centos-7

This fails with centos-7 but not centos-6.

./bdutil -e platforms/hdp/ambari_env.sh deploy
Sat Mar 21 23:29:19 CST 2015: Using local tmp dir for staging files: /tmp/bdutil-20150321-232919-pZT
Sat Mar 21 23:29:19 CST 2015: Using custom environment-variable file(s): bdutil_env.sh platforms/hdp/ambari_env.sh
Sat Mar 21 23:29:19 CST 2015: Reading environment-variable file: ./bdutil_env.sh
Sat Mar 21 23:29:19 CST 2015: Reading environment-variable file: platforms/hdp/ambari_env.sh
Importing dependent env file: ./platforms/hdp/ambari_manual_env.sh
Importing dependent env file: ./hadoop2_env.sh
Importing dependent env file: ./platforms/hdp/ambari.conf
Importing dependent env file: ./platforms/hdp/ambari_functions.sh
Sat Mar 21 23:29:19 CST 2015: No explicit GCE_MASTER_MACHINE_TYPE provided; defaulting to value of GCE_MACHINE_TYPE: n1-standard-4
Deploy cluster with following settings?
      CONFIGBUCKET='net-xbhxxxxx'
      PROJECT='zeta-matrixxxxxxx'
      GCE_IMAGE='centos-7'
      GCE_ZONE='europe-west1-b'
      GCE_NETWORK='default'
      PREFIX='hadoop'
      NUM_WORKERS=4
      MASTER_HOSTNAME='hadoop-m'
      WORKERS='hadoop-w-0 hadoop-w-1 hadoop-w-2 hadoop-w-3'
      BDUTIL_GCS_STAGING_DIR='gs://net-bh/bdutil-staging/hadoop-m'
            MASTER_ATTACHED_PD='hadoop-m-pd'
      WORKER_ATTACHED_PDS='hadoop-w-0-pd hadoop-w-1-pd hadoop-w-2-pd hadoop-w-3-pd'
      (y/n) y

It seems to be an ssh problem. /tmp/bdutil-20150321-232919-pZT/debuginfo.txt:

ssh: connect to host 104.155.62.245 port 22: Connection refused
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

Ambari extension with stack HDP 1.3

I was able to deploy ambari_env.sh following the tutorial. However, because I need hadoop1 to run some previous job, I have tried to install the HDP 1.3 setting the variable AMBARI_STACK_VERSION='1.3' in ambari.conf.

bdutil got stuck at
Invoking on master: ./install-ambari-components.sh Waiting on async 'ssh' jobs to finish.
Might take a while....

Looking at install-ambari-components_deploy.stdout it says:
Provisioning ambari cluster.
{ "status" : 400,
"message" : "Unable to update configuration property with topology information. Component 'JOBTRACKER' is not mapped to any host group or is mapped to multiple groups."}

while install-ambari-components_deploy.stderr shows a loop printing

ambari_wait status:
curl: no URL specified! curl: try 'curl --help' or 'curl --manual' for more information

Is stack 1.3 supposed to work? I was thinking that also the variable AMBARI_SERVICES needs to be proper adjusted for 1.3, and if so, what values it takes?

Can't install with Ambari 2.5+

I tried upgrading to a newer version of Ambari (2.5 or 2.6) and every time my HDFS NameNode ends up failing.

It turns out, the problem ends up being with the default configuration recommendation provided by ambari, it sets the namenode directory to /mnt/pd1/hadoop/hdfs/namenode,/hadoop/hdfs/namenode which is two directories, but really the same directory since /hadoop is just a symlink to /mnt/pd1/hadoop. Therefore, when NameNode starts up, it tries to 'lock' the directory twice, but can't, since it sees itself as already having locked it.

I've deployed Ambari using the ambari_manual environment instead (where it just deploys ambari, and no services). This works initially, but when you go to install HDFS (unless you manually change the config) you get the same problem.

issue with ssh

I got this error when deploying bdutil with Ambari
https://gist.github.com/meodorewan/ab0d48e8e32a64732d5c

it seems that I cant ssh to my instances with

" gcloud --project=abivin-clusters --quiet --verbosity=info compute ssh hadoop-t-m --command=sudo su -l -c "cd ${PWD} && ./ambari-setup.sh" 2>>ambari-setup_deploy.stderr 1>>ambari-setup_deploy.stdout --ssh-flag=-tt --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=asia-east1-c"

command

bdutil confusingly reports "using project ${GCE_ZONE}"

loginfo "Using project '${GCE_ZONE}'"

Recommend:

loginfo "Using zone '${GCE_ZONE}'"

https://github.com/GoogleCloudPlatform/bdutil/blob/aafa2a2f6cdf56073ef6b5af1bdf00343cfcbf88/bdutil#L607

testing spark 1.5 .. issue with tar

I pointed the SHARK_HADOOP2_TARBALL_URI to spark-1.5.0-bin-hadoop2.6.tgz file I downloaded and places on gcs.
Downloading file:///home/hadoop/spark-1.5.0-bin-hadoop2.6.tgz: 302.98 MiB/302.98 MiB
hadoop-m:t
hadoop-m:tgzip: stdin: not in gzip format
hadoop-m:ttar: Child returned status 1
hadoop-m:ttar: Error is not recoverable: exiting now
hadoop-m:t.

I am able to run tar -C spark -xzvf spark-1.5.0-bin-hadoop2.6.tgz on my mac where I downloaded the file, however it does throw and error
tar: copyfile unpack (spark-1.5.0-bin-hadoop2.6/python/test_support/sql/orc_partitioned/SUCCESS.crc) failed: No such file or directory

I saw this issue mentioned here -> http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Apache-Spark-1-5-0-RC2-tp13826p13893.html

Any idea how to resolve this?

bdutil deploy fails with ssh error

Hello,

I am trying to setup 2 node HDP with n1-standard-2 instance in us-central1-a zone with 300GB PD. I am receiving error while deploying using bdutil. The issue is related to SSH.

After the installation fails, to debug, when I try to do ssh to these instances, at first they give me error - but after 1/2 retries, it does get connected.

I have tried this multiple times and it fails exactly at the same place.

Any idea what is going wrong or where am I making mistake?

Thanks in advanced.

Tue Sep 6 23:26:16 IST 2016: Exited 1 : gcloud --project=hdp-00-142609 --quiet --verbosity=info compute ssh hadoop-w-1 --command=tail -vn 30 *.stderr --ssh-flag=-n
--ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-central1-a
.Tue Sep 6 23:26:16 IST 2016: Command failed: wait ${SUBPROC} on line 326.
Tue Sep 6 23:26:16 IST 2016: Exit code of failed command: 255
Tue Sep 6 23:26:16 IST 2016: Detailed debug info available in file: /tmp/bdutil-20160906-232348-lc1/debuginfo.txt

Tue Sep 6 23:26:16 IST 2016: Check console output for error messages and/or retry your command.

debuginfo.txt

Hadoop does not seem to have been installed

I am using Cygwin and trying to create an HDP using the command ./bdutil -e platforms/hdp/ambari_env.sh deploy . All my instances and disks are built fine, and I am able to SSH into the machine, but there is absolutely no Hadoop stack there at all. I can see by running yum search that there is no haddop repository enabled, so assume that is the problem.

There is no error message indicating anything had failed. I get the following output:

  CONFIGBUCKET='xxx'
  PROJECT='xxxx'
  GCE_IMAGE='centos-6'
  GCE_ZONE='europe-west1-b'
  GCE_NETWORK='default'
  PREEMPTIBLE_FRACTION=0.0
  PREFIX='hadoop'
  NUM_WORKERS=3
  MASTER_HOSTNAME='hadoop-m'
  WORKERS='hadoop-w-0 hadoop-w-1 hadoop-w-2'
  BDUTIL_GCS_STAGING_DIR='gs://osm_hadoop/bdutil-staging/hadoop-m'
        MASTER_ATTACHED_PD='hadoop-m-pd'
  WORKER_ATTACHED_PDS='hadoop-w-0-pd hadoop-w-1-pd hadoop-w-2-pd'
  (y/n) y
Sun, Aug 09, 2015  1:52:35 PM: Checking for existence of gs://osm_hadoop...
gs://osm_hadoop/
Sun, Aug 09, 2015  1:52:45 PM: Checking for existence of gs://hadoop-dist/hadoop-2.6.0.tar.gz...
Sun, Aug 09, 2015  1:52:55 PM: Checking upload files...
Sun, Aug 09, 2015  1:52:55 PM: Verified './conf/hadoop2/bigtable-hbase-site-template.xml'
Sun, Aug 09, 2015  1:52:55 PM: Verified './conf/hadoop2/bq-mapred-template.xml'
Sun, Aug 09, 2015  1:52:55 PM: Verified './conf/hadoop2/core-template.xml'
Sun, Aug 09, 2015  1:52:55 PM: Verified './conf/hadoop2/gcs-core-template.xml'
Sun, Aug 09, 2015  1:52:55 PM: Verified './conf/hadoop2/hdfs-template.xml'
Sun, Aug 09, 2015  1:52:55 PM: Verified './conf/hadoop2/mapred-template.xml'
Sun, Aug 09, 2015  1:52:55 PM: Verified './conf/hadoop2/yarn-template.xml'
Sun, Aug 09, 2015  1:52:55 PM: Verified './libexec/hadoop_helpers.sh'
Sun, Aug 09, 2015  1:52:55 PM: Verified './libexec/configure_mrv2_mem.py'
Sun, Aug 09, 2015  1:52:55 PM: Verified './hadoop2_env.sh'
Sun, Aug 09, 2015  1:52:55 PM: Verified './platforms/hdp/ambari.conf'
Sun, Aug 09, 2015  1:52:56 PM: Verified './platforms/hdp/ambari_functions.sh'
Sun, Aug 09, 2015  1:52:56 PM: Verified './libexec/hadoop_helpers.sh'
Sun, Aug 09, 2015  1:52:56 PM: Verified './platforms/hdp/configuration.json'
Sun, Aug 09, 2015  1:52:56 PM: Verified './platforms/hdp/resources/public-hostname-gcloud.sh'
Sun, Aug 09, 2015  1:52:56 PM: Verified './platforms/hdp/resources/thp-disable.sh'
Sun, Aug 09, 2015  1:52:56 PM: Verified './platforms/hdp/ambari_manual_env.sh'
Sun, Aug 09, 2015  1:52:56 PM: Verified './platforms/hdp/create_blueprint.py'
Sun, Aug 09, 2015  1:52:56 PM: Generating 12 command groups...
Sun, Aug 09, 2015  1:52:59 PM: Done generating remote shell scripts.
Sun, Aug 09, 2015  1:52:59 PM: Creating attached worker disks: hadoop-w-0-pd hadoop-w-1-pd   hadoop-w-2-pd
...Sun, Aug 09, 2015  1:53:00 PM: Creating attached master disk: hadoop-m-pd
.Sun, Aug 09, 2015  1:53:00 PM: Done creating disks!
 Sun, Aug 09, 2015  1:53:00 PM: Waiting on async 'disks create' jobs to finish. Might take a while...
 ....
Sun, Aug 09, 2015  1:53:15 PM: Creating worker instances: hadoop-w-0 hadoop-w-1 hadoop-w-2
...Sun, Aug 09, 2015  1:53:16 PM: Creating master instance: hadoop-m
.Sun, Aug 09, 2015  1:53:16 PM: Waiting on async 'instances create' jobs to finish. Might take a while... 
....
Sun, Aug 09, 2015  1:54:13 PM: Instances all created. Entering polling loop to wait for ssh-ability

I will try on Linux and test, assume it may be something to do with Cygwin.

unable to deploy ambari using bdutil

for the the following command I am getting the following error:
'.' is not recognized as an internal or external command,
operable program or batch file.

unable to deploy ambari using bdutil

everytime i use "./bdutil -e ambari deploy" command, i get below errors

******************* gcloud compute stdout *******************
******************* gcloud compute stderr *******************
WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more infor
mation, see: https://developers.google.com/compute/docs/disks#pdperformance.
WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more infor
mation, see: https://developers.google.com/compute/docs/disks#pdperformance.
WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more infor
mation, see: https://developers.google.com/compute/docs/disks#pdperformance.
WARNING: You have selected a disk size of under [200GB]. This may result in poor I/O performance. For more infor
mation, see: https://developers.google.com/compute/docs/disks#pdperformance.
ERROR: (gcloud.compute.disks.create) Could not fetch resource:

Insufficient Permission
ERROR: (gcloud.compute.disks.create) Could not fetch resource:
Insufficient Permission
ERROR: (gcloud.compute.disks.create) Could not fetch resource:
Insufficient Permission
ERROR: (gcloud.compute.disks.create) Could not fetch resource:
Insufficient Permission
ERROR: (gcloud.compute.disks.create) Could not fetch resource:
Insufficient Permission
************ ERROR logs from gcloud compute stderr ************
ERROR: (gcloud.compute.disks.create) Could not fetch resource:
ERROR: (gcloud.compute.disks.create) Could not fetch resource:
ERROR: (gcloud.compute.disks.create) Could not fetch resource:
ERROR: (gcloud.compute.disks.create) Could not fetch resource:
ERROR: (gcloud.compute.disks.create) Could not fetch resource:
******************* Exit codes and VM logs *******************
Mon May 8 22:33:26 UTC 2017: Exited 1 : gcloud --project=vijay-167021 --quiet --verbosity=info compute disks cr
eate --size=100 --type=pd-standard hadoop-w-0-pd --zone=zone(unset)
Mon May 8 22:33:26 UTC 2017: Exited 1 : gcloud --project=vijay-167021 --quiet --verbosity=info compute disks cr
eate --size=100 --type=pd-standard hadoop-w-1-pd --zone=zone(unset)
"debuginfo.txt" 37L, 2679C

Please assist to resolve this problem.

specifying machine names run_command target

Hi,

I am experiencing some known issue as described here. When I try to deploy some 5 nodes cluster, I get repeated errors $worker/$master not yet sshable.... I had some free trial so I upgraded it so that click to deploy works fine. But this process creates machine names with common prefix but with random suffix. So that when I am not able to run an bdutil run_command script. Is there a way to do it ? On bdutil master branch script, it seems not to be possible

Nonissue..delete

Command failed: wait ${SUBPROC} on line 326

Updated the "GCE_SERVICE_ACCOUNT" with "compute-rw" as well. Still facing the issue of command failed while executing "./bdutil -e platforms/hdp/ambari_env.sh deploy".

Also not able to access the ambari console by performing the tunneling.

Support for extensions on hadoop 2

Most extensions are only supported on hadoop 1, are there plans to extend support to Hadoop 2 ?

Does not seem to working with Custom network

Using custom network requires the subnetwork to be specified as part of the create instance command. It does not look like we the command accepts a subnetwork

To reproduce

Create a Custom network and subnetwork in GCP
Use the custom network while running the bdutil.
Log messages says that it requires a subnetwork
Could not find any way to specify a subnetwork

Logs:
`

******************* gcloud compute stdout *******************

******************* gcloud compute stderr *******************
ERROR: (gcloud.compute.instances.create) Some requests did not succeed:

Invalid value for field 'resource.networkInterfaces[0]': ''. Subnetwork should be specified for custom subnetmode network

************ ERROR logs from gcloud compute stderr ************
ERROR: (gcloud.compute.instances.create) Some requests did not succeed:
`

Upgrade bdutil to Debian 8

Debian 7 is no longer receiving updates as of last week. In order to receive security fixes and general improvements, bdutil should upgrade to a debian-8 image.

bdutil doesn't work with Ubuntu images

Reported on StackOverflow, I'm able to reproduce the problem at the latest 1.2.1 release. There appear to be multiple issues leading to the incompatibility.

Support installing the GCS connector after deploying Ambari using the Web UI

Currently ambari_manual_env.sh allows the user to use Ambari to manually install HDP components, but it lacks the GCS connector. Manually installing the connector afterwards is rather difficult.

The ambari_env.sh post-install scripts need to be slightly reworked to work on a manual deployment.

Command failed line 326, and SSH Key

Hi all,
New user to bdutil, running on cygwin in win 7, followed instructions in readme, encountered this error:
Command failed: wait ${SUBPROC} on line 326.
Exit code of failed command: 1

Please reference this gist.

Any suggestions?

I am able to execute the socksproxy or ssh tunnel, but am unable to get them to work. Here is the terminal output while trying ssh.

Thanks!

platforms/hdp/create_blueprint.py - No module named argparse

@seanorama During the ./bdutil -e platforms/hdp/ambari_env.sh deploy process, when I reach install-ambari-components.sh and the python blueprint script I am getting an import error for some reason. Running on a fresh VM with ubuntu 14.04. Seems the OS includes python 2.7.6 which already has argparse, importing it from the interpreter works fine and reports the dependency is satisfied if I try to install python-argparse.

This is probably something pretty basic but I'm not having luck fixing it, I haven't made any other python installations

******************* gcloud compute stdout *******************
NAME          ZONE          SIZE_GB TYPE        STATUS
hadoop-w-0-pd us-central1-a 200     pd-standard READY
NAME          ZONE          SIZE_GB TYPE        STATUS
hadoop-w-1-pd us-central1-a 200     pd-standard READY
NAME        ZONE          SIZE_GB TYPE        STATUS
hadoop-m-pd us-central1-a 200     pd-standard READY
NAME          ZONE          SIZE_GB TYPE        STATUS
hadoop-w-2-pd us-central1-a 200     pd-standard READY
NAME       ZONE          MACHINE_TYPE  INTERNAL_IP  EXTERNAL_IP     STATUS
hadoop-w-2 us-central1-a n1-standard-2 10.240.12.69 173.255.114.219 RUNNING
NAME       ZONE          MACHINE_TYPE  INTERNAL_IP    EXTERNAL_IP     STATUS
hadoop-w-0 us-central1-a n1-standard-2 10.240.252.129 130.211.142.191 RUNNING
NAME     ZONE          MACHINE_TYPE  INTERNAL_IP   EXTERNAL_IP   STATUS
hadoop-m us-central1-a n1-standard-2 10.240.227.38 146.148.75.56 RUNNING
NAME       ZONE          MACHINE_TYPE  INTERNAL_IP    EXTERNAL_IP    STATUS
hadoop-w-1 us-central1-a n1-standard-2 10.240.215.199 146.148.71.206 RUNNING

******************* gcloud compute stderr *******************
Created [https://www.googleapis.com/compute/v1/projects/csab-hdp-00/zones/us-central1-a/disks/hadoop-w-0-pd].
Created [https://www.googleapis.com/compute/v1/projects/csab-hdp-00/zones/us-central1-a/disks/hadoop-w-1-pd].
Created [https://www.googleapis.com/compute/v1/projects/csab-hdp-00/zones/us-central1-a/disks/hadoop-m-pd].
Created [https://www.googleapis.com/compute/v1/projects/csab-hdp-00/zones/us-central1-a/disks/hadoop-w-2-pd].
Created [https://www.googleapis.com/compute/v1/projects/csab-hdp-00/zones/us-central1-a/instances/hadoop-w-2].
Created [https://www.googleapis.com/compute/v1/projects/csab-hdp-00/zones/us-central1-a/instances/hadoop-m].
Created [https://www.googleapis.com/compute/v1/projects/csab-hdp-00/zones/us-central1-a/instances/hadoop-w-0].
Created [https://www.googleapis.com/compute/v1/projects/csab-hdp-00/zones/us-central1-a/instances/hadoop-w-1].
Warning: Permanently added '146.148.71.206' (RSA) to the list of known hosts.
Warning: Permanently added '173.255.114.219' (RSA) to the list of known hosts.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
Warning: Permanently added '130.211.142.191' (RSA) to the list of known hosts.
Warning: Permanently added '146.148.75.56' (RSA) to the list of known hosts.
Warning: Permanently added '146.148.71.206' (RSA) to the list of known hosts.
Warning: Permanently added '173.255.114.219' (RSA) to the list of known hosts.
Warning: Permanently added '130.211.142.191' (RSA) to the list of known hosts.
Warning: Permanently added '146.148.71.206' (RSA) to the list of known hosts.
Warning: Permanently added '173.255.114.219' (RSA) to the list of known hosts.
Warning: Permanently added '146.148.75.56' (RSA) to the list of known hosts.
Connection to 146.148.75.56 closed.
Connection to 146.148.71.206 closed.
Connection to 173.255.114.219 closed.
Connection to 130.211.142.191 closed.
Warning: Permanently added '130.211.142.191' (RSA) to the list of known hosts.
Warning: Permanently added '146.148.71.206' (RSA) to the list of known hosts.
Warning: Permanently added '173.255.114.219' (RSA) to the list of known hosts.
Warning: Permanently added '146.148.75.56' (RSA) to the list of known hosts.
Connection to 173.255.114.219 closed.
Connection to 130.211.142.191 closed.
Connection to 146.148.71.206 closed.
Connection to 146.148.75.56 closed.
Warning: Permanently added '146.148.75.56' (RSA) to the list of known hosts.
Connection to 146.148.75.56 closed.
Warning: Permanently added '130.211.142.191' (RSA) to the list of known hosts.
Warning: Permanently added '146.148.71.206' (RSA) to the list of known hosts.
Warning: Permanently added '146.148.75.56' (RSA) to the list of known hosts.
Warning: Permanently added '173.255.114.219' (RSA) to the list of known hosts.
Connection to 146.148.75.56 closed.
Connection to 130.211.142.191 closed.
Connection to 146.148.71.206 closed.
Connection to 173.255.114.219 closed.
Warning: Permanently added '146.148.75.56' (RSA) to the list of known hosts.
Connection to 146.148.75.56 closed.
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [1].

************ ERROR logs from gcloud compute stderr ************
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [1].

******************* Exit codes and VM logs *******************
Wed May  6 10:32:28 CDT 2015: Exited 1 : gcloud --project=csab-hdp-00 --quiet --verbosity=info compute ssh hadoop-w-1 --command=exit 0 --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-central1-a
Wed May  6 10:32:29 CDT 2015: Exited 1 : gcloud --project=csab-hdp-00 --quiet --verbosity=info compute ssh hadoop-w-2 --command=exit 0 --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-central1-a
Wed May  6 10:35:08 CDT 2015: Exited 1 : gcloud --project=csab-hdp-00 --quiet --verbosity=info compute ssh hadoop-m --command=sudo su -l -c "cd ${PWD} && ./install-ambari-components.sh" 2>>install-ambari-components_deploy.stderr 1>>install-ambari-components_deploy.stdout --ssh-flag=-tt --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-central1-a
hadoop-m:   Wed May  6 10:35:08 CDT 2015: Running gcloud --project=csab-hdp-00 --quiet --verbosity=info compute ssh hadoop-m --command=tail -vn 30 *.stderr --ssh-flag=-n --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-central1-a
hadoop-m:   ==> ambari-setup_deploy.stderr <==
hadoop-m:   safe_format_and_mount: � 624/1600��������� 625/1600��������� 626/1600��������� 627/1600��������� 628/1600��������� 629/1600��������� 630/1600��������� 631/1600��������� 632/1600��������� 633/1600��������� 634/1600��������� 635/1600��������� 636/1600��������� 637/1600��������� 638/1600��������� 639/1600��������� 640/1600��������� 641/1600��������� 642/1600��������� 643/1600��������� 644/1600��������� 645/1600��������� 646/1600��������� 647/1600��������� 648/1600��������� 649/1600��������� 650/1600��������� 651/1600��������� 652/1600��������� 653/1600��������� 654/1600��������� 655/1600��������� 656/1600��������� 657/1600��������� 658/1600��������� 659/1600��������� 660/1600��������� 661/1600��������� 662/1600��������� 663/1600��������� 664/1600��������� 665/1600��������� 666/1600��������� 667/1600��������� 668/1600��������� 669/1600��������� 670/1600��������� 671/1600��������� 672/1600��������� 673/1600��������� 674/1600��������� 675/1600��������� 676/1600��������� 677/1600��������� 678/1600��������� 679/1600��������� 680/1600�����
hadoop-m:   safe_format_and_mount: ���� 681/1600��������� 682/1600��������� 683/1600��������� 684/1600��������� 685/1600��������� 686/1600��������� 687/1600��������� 688/1600��������� 689/1600��������� 690/1600��������� 691/1600��������� 692/1600��������� 693/1600��������� 694/1600��������� 695/1600��������� 696/1600��������� 697/1600��������� 698/1600��������� 699/1600��������� 700/1600��������� 701/1600��������� 702/1600��������� 703/1600��������� 704/1600��������� 705/1600��������� 706/1600��������� 707/1600��������� 708/1600��������� 709/1600��������� 710/1600��������� 711/1600��������� 712/1600��������� 713/1600��������� 714/1600��������� 715/1600��������� 716/1600��������� 717/1600��������� 718/1600��������� 719/1600��������� 720/1600��������� 721/1600��������� 722/1600��������� 723/1600��������� 724/1600��������� 725/1600��������� 726/1600��������� 727/1600��������� 728/1600��������� 729/1600��������� 730/1600��������� 731/1600��������� 732/1600��������� 733/1600��������� 734/1600��������� 735/1600��������� 736/1600��������� 737/1600��
hadoop-m:   safe_format_and_mount: ������� 738/1600��������� 739/1600��������� 740/1600��������� 741/1600��������� 742/1600��������� 743/1600��������� 744/1600��������� 745/1600��������� 746/1600��������� 747/1600��������� 748/1600��������� 749/1600��������� 750/1600��������� 751/1600��������� 752/1600��������� 753/1600��������� 754/1600��������� 755/1600��������� 756/1600��������� 757/1600��������� 758/1600��������� 759/1600��������� 760/1600��������� 761/1600��������� 762/1600��������� 763/1600��������� 764/1600��������� 765/1600��������� 766/1600��������� 767/1600��������� 768/1600��������� 769/1600��������� 770/1600��������� 771/1600��������� 772/1600��������� 773/1600��������� 774/1600��������� 775/1600��������� 776/1600��������� 777/1600��������� 778/1600��������� 779/1600��������� 780/1600��������� 781/1600��������� 782/1600��������� 783/1600��������� 784/1600��������� 785/1600��������� 786/1600��������� 787/1600��������� 788/1600��������� 789/1600��������� 790/1600��������� 791/1600��������� 792/1600��������� 793/1600��������� 794/160
hadoop-m:   safe_format_and_mount: 0��������� 795/1600��������� 796/1600��������� 797/1600��������� 798/1600��������� 799/1600��������� 800/1600��������� 801/1600��������� 802/1600��������� 803/1600��������� 804/1600��������� 805/1600��������� 806/1600��������� 807/1600��������� 808/1600��������� 809/1600��������� 810/1600��������� 811/1600��������� 812/1600��������� 813/1600��������� 814/1600��������� 815/1600��������� 816/1600��������� 817/1600��������� 818/1600��������� 819/1600��������� 820/1600��������� 821/1600��������� 822/1600��������� 823/1600��������� 824/1600��������� 825/1600��������� 826/1600��������� 827/1600��������� 828/1600��������� 829/1600��������� 830/1600��������� 831/1600��������� 832/1600��������� 833/1600��������� 834/1600��������� 835/1600��������� 836/1600��������� 837/1600��������� 838/1600��������� 839/1600��������� 840/1600��������� 841/1600��������� 842/1600��������� 843/1600��������� 844/1600��������� 845/1600��������� 846/1600��������� 847/1600��������� 848/1600��������� 849/1600��������� 850/1600��������� 851/
hadoop-m:   safe_format_and_mount: 1600��������� 852/1600��������� 853/1600��������� 854/1600��������� 855/1600��������� 856/1600��������� 857/1600��������� 858/1600��������� 859/1600��������� 860/1600��������� 861/1600��������� 862/1600��������� 863/1600��������� 864/1600��������� 865/1600��������� 866/1600��������� 867/1600��������� 868/1600��������� 869/1600��������� 870/1600��������� 871/1600��������� 872/1600��������� 873/1600��������� 874/1600��������� 875/1600��������� 876/1600��������� 877/1600��������� 878/1600��������� 879/1600��������� 880/1600��������� 881/1600��������� 882/1600��������� 883/1600��������� 884/1600��������� 885/1600��������� 886/1600��������� 887/1600��������� 888/1600��������� 889/1600��������� 890/1600��������� 891/1600��������� 892/1600��������� 893/1600��������� 894/1600��������� 895/1600��������� 896/1600��������� 897/1600��������� 898/1600��������� 899/1600��������� 900/1600��������� 901/1600��������� 902/1600��������� 903/1600��������� 904/1600��������� 905/1600��������� 906/1600��������� 907/1600��������� 9
hadoop-m:   safe_format_and_mount: 08/1600��������� 909/1600��������� 910/1600��������� 911/1600��������� 912/1600��������� 913/1600��������� 914/1600��������� 915/1600��������� 916/1600��������� 917/1600��������� 918/1600��������� 919/1600��������� 920/1600��������� 921/1600��������� 922/1600��������� 923/1600��������� 924/1600��������� 925/1600��������� 926/1600��������� 927/1600��������� 928/1600��������� 929/1600��������� 930/1600��������� 931/1600��������� 932/1600��������� 933/1600��������� 934/1600��������� 935/1600��������� 936/1600��������� 937/1600��������� 938/1600��������� 939/1600��������� 940/1600��������� 941/1600��������� 942/1600��������� 943/1600��������� 944/1600��������� 945/1600��������� 946/1600��������� 947/1600��������� 948/1600��������� 949/1600��������� 950/1600��������� 951/1600��������� 952/1600��������� 953/1600��������� 954/1600��������� 955/1600��������� 956/1600��������� 957/1600��������� 958/1600��������� 959/1600��������� 960/1600��������� 961/1600��������� 962/1600��������� 963/1600��������� 964/1600��������
hadoop-m:   safe_format_and_mount: � 965/1600��������� 966/1600��������� 967/1600��������� 968/1600��������� 969/1600��������� 970/1600��������� 971/1600��������� 972/1600��������� 973/1600��������� 974/1600��������� 975/1600��������� 976/1600��������� 977/1600��������� 978/1600��������� 979/1600��������� 980/1600��������� 981/1600��������� 982/1600��������� 983/1600��������� 984/1600��������� 985/1600��������� 986/1600��������� 987/1600��������� 988/1600��������� 989/1600��������� 990/1600��������� 991/1600��������� 992/1600��������� 993/1600��������� 994/1600��������� 995/1600��������� 996/1600��������� 997/1600��������� 998/1600��������� 999/1600���������1000/1600���������1001/1600���������1002/1600���������1003/1600���������1004/1600���������1005/1600���������1006/1600���������1007/1600���������1008/1600���������1009/1600���������1010/1600���������1011/1600���������1012/1600���������1013/1600���������1014/1600���������1015/1600���������1016/1600���������1017/1600���������1018/1600���������1019/1600���������1020/1600���������1021/1600�����
hadoop-m:   safe_format_and_mount: ����1022/1600���������1023/1600���������1024/1600���������1025/1600���������1026/1600���������1027/1600���������1028/1600���������1029/1600���������1030/1600���������1031/1600���������1032/1600���������1033/1600���������1034/1600���������1035/1600���������1036/1600���������1037/1600���������1038/1600���������1039/1600���������1040/1600���������1041/1600���������1042/1600���������1043/1600���������1044/1600���������1045/1600���������1046/1600���������1047/1600���������1048/1600���������1049/1600���������1050/1600���������1051/1600���������1052/1600���������1053/1600���������1054/1600���������1055/1600���������1056/1600���������1057/1600���������1058/1600���������1059/1600���������1060/1600���������1061/1600���������1062/1600���������1063/1600���������1064/1600���������1065/1600���������1066/1600���������1067/1600���������1068/1600���������1069/1600���������1070/1600���������1071/1600���������1072/1600���������1073/1600���������1074/1600���������1075/1600���������1076/1600���������1077/1600���������1078/1600��
hadoop-m:   safe_format_and_mount: �������1079/1600���������1080/1600���������1081/1600���������1082/1600���������1083/1600���������1084/1600���������1085/1600���������1086/1600���������1087/1600���������1088/1600���������1089/1600���������1090/1600���������1091/1600���������1092/1600���������1093/1600���������1094/1600���������1095/1600���������1096/1600���������1097/1600���������1098/1600���������1099/1600���������1100/1600���������1101/1600���������1102/1600���������1103/1600���������1104/1600���������1105/1600���������1106/1600���������1107/1600���������1108/1600���������1109/1600���������1110/1600���������1111/1600���������1112/1600���������1113/1600���������1114/1600���������1115/1600���������1116/1600���������1117/1600���������1118/1600���������1119/1600���������1120/1600���������1121/1600���������1122/1600���������1123/1600���������1124/1600���������1125/1600���������1126/1600���������1127/1600���������1128/1600���������1129/1600���������1130/1600���������1131/1600���������1132/1600���������1133/1600���������1134/1600���������1135/160
hadoop-m:   safe_format_and_mount: 0���������1136/1600���������1137/1600���������1138/1600���������1139/1600���������1140/1600���������1141/1600���������1142/1600���������1143/1600���������1144/1600���������1145/1600���������1146/1600���������1147/1600���������1148/1600���������1149/1600���������1150/1600���������1151/1600���������1152/1600���������1153/1600���������1154/1600���������1155/1600���������1156/1600���������1157/1600���������1158/1600���������1159/1600���������1160/1600���������1161/1600���������1162/1600���������1163/1600���������1164/1600���������1165/1600���������1166/1600���������1167/1600���������1168/1600���������1169/1600���������1170/1600���������1171/1600���������1172/1600���������1173/1600���������1174/1600���������1175/1600���������1176/1600���������1177/1600���������1178/1600���������1179/1600���������1180/1600���������1181/1600���������1182/1600���������1183/1600���������1184/1600���������1185/1600���������1186/1600���������1187/1600���������1188/1600���������1189/1600���������1190/1600���������1191/1600���������1192/
hadoop-m:   safe_format_and_mount: 1600���������1193/1600���������1194/1600���������1195/1600���������1196/1600���������1197/1600���������1198/1600���������1199/1600���������1200/1600���������1201/1600���������1202/1600���������1203/1600���������1204/1600���������1205/1600���������1206/1600���������1207/1600���������1208/1600���������1209/1600���������1210/1600���������1211/1600���������1212/1600���������1213/1600���������1214/1600���������1215/1600���������1216/1600���������1217/1600���������1218/1600���������1219/1600���������1220/1600���������1221/1600���������1222/1600���������1223/1600���������1224/1600���������1225/1600���������1226/1600���������1227/1600���������1228/1600���������1229/1600���������1230/1600���������1231/1600���������1232/1600���������1233/1600���������1234/1600���������1235/1600���������1236/1600���������1237/1600���������1238/1600���������1239/1600���������1240/1600���������1241/1600���������1242/1600���������1243/1600���������1244/1600���������1245/1600���������1246/1600���������1247/1600���������1248/1600���������12
hadoop-m:   safe_format_and_mount: 49/1600���������1250/1600���������1251/1600���������1252/1600���������1253/1600���������1254/1600���������1255/1600���������1256/1600���������1257/1600���������1258/1600���������1259/1600���������1260/1600���������1261/1600���������1262/1600���������1263/1600���������1264/1600���������1265/1600���������1266/1600���������1267/1600���������1268/1600���������1269/1600���������1270/1600���������1271/1600���������1272/1600���������1273/1600���������1274/1600���������1275/1600���������1276/1600���������1277/1600���������1278/1600���������1279/1600���������1280/1600���������1281/1600���������1282/1600���������1283/1600���������1284/1600���������1285/1600���������1286/1600���������1287/1600���������1288/1600���������1289/1600���������1290/1600���������1291/1600���������1292/1600���������1293/1600���������1294/1600���������1295/1600���������1296/1600���������1297/1600���������1298/1600���������1299/1600���������1300/1600���������1301/1600���������1302/1600���������1303/1600���������1304/1600���������1305/1600��������
hadoop-m:   safe_format_and_mount: �1306/1600���������1307/1600���������1308/1600���������1309/1600���������1310/1600���������1311/1600���������1312/1600���������1313/1600���������1314/1600���������1315/1600���������1316/1600���������1317/1600���������1318/1600���������1319/1600���������1320/1600���������1321/1600���������1322/1600���������1323/1600���������1324/1600���������1325/1600���������1326/1600���������1327/1600���������1328/1600���������1329/1600���������1330/1600���������1331/1600���������1332/1600���������1333/1600���������1334/1600���������1335/1600���������1336/1600���������1337/1600���������1338/1600���������1339/1600���������1340/1600���������1341/1600���������1342/1600���������1343/1600���������1344/1600���������1345/1600���������1346/1600���������1347/1600���������1348/1600���������1349/1600���������1350/1600���������1351/1600���������1352/1600���������1353/1600���������1354/1600���������1355/1600���������1356/1600���������1357/1600���������1358/1600���������1359/1600���������1360/1600���������1361/1600���������1362/1600�����
hadoop-m:   safe_format_and_mount: ����1363/1600���������1364/1600���������1365/1600���������1366/1600���������1367/1600���������1368/1600���������1369/1600���������1370/1600���������1371/1600���������1372/1600���������1373/1600���������1374/1600���������1375/1600���������1376/1600���������1377/1600���������1378/1600���������1379/1600���������1380/1600���������1381/1600���������1382/1600���������1383/1600���������1384/1600���������1385/1600���������1386/1600���������1387/1600���������1388/1600���������1389/1600���������1390/1600���������1391/1600���������1392/1600���������1393/1600���������1394/1600���������1395/1600���������1396/1600���������1397/1600���������1398/1600���������1399/1600���������1400/1600���������1401/1600���������1402/1600���������1403/1600���������1404/1600���������1405/1600���������1406/1600���������1407/1600���������1408/1600���������1409/1600���������1410/1600���������1411/1600���������1412/1600���������1413/1600���������1414/1600���������1415/1600���������1416/1600���������1417/1600���������1418/1600���������1419/1600��
hadoop-m:   safe_format_and_mount: �������1420/1600���������1421/1600���������1422/1600���������1423/1600���������1424/1600���������1425/1600���������1426/1600���������1427/1600���������1428/1600���������1429/1600���������1430/1600���������1431/1600���������1432/1600���������1433/1600���������1434/1600���������1435/1600���������1436/1600���������1437/1600���������1438/1600���������1439/1600���������1440/1600���������1441/1600���������1442/1600���������1443/1600���������1444/1600���������1445/1600���������1446/1600���������1447/1600���������1448/1600���������1449/1600���������1450/1600���������1451/1600���������1452/1600���������1453/1600���������1454/1600���������1455/1600���������1456/1600���������1457/1600���������1458/1600���������1459/1600���������1460/1600���������1461/1600���������1462/1600���������1463/1600���������1464/1600���������1465/1600���������1466/1600���������1467/1600���������1468/1600���������1469/1600���������1470/1600���������1471/1600���������1472/1600���������1473/1600���������1474/1600���������1475/1600���������1476/160
hadoop-m:   safe_format_and_mount: 0���������1477/1600���������1478/1600���������1479/1600���������1480/1600���������1481/1600���������1482/1600���������1483/1600���������1484/1600���������1485/1600���������1486/1600���������1487/1600���������1488/1600���������1489/1600���������1490/1600���������1491/1600���������1492/1600���������1493/1600���������1494/1600���������1495/1600���������1496/1600���������1497/1600���������1498/1600���������1499/1600���������1500/1600���������1501/1600���������1502/1600���������1503/1600���������1504/1600���������1505/1600���������1506/1600���������1507/1600���������1508/1600���������1509/1600���������1510/1600���������1511/1600���������1512/1600���������1513/1600���������1514/1600���������1515/1600���������1516/1600���������1517/1600���������1518/1600���������1519/1600���������1520/1600���������1521/1600���������1522/1600���������1523/1600���������1524/1600���������1525/1600���������1526/1600���������1527/1600���������1528/1600���������1529/1600���������1530/1600���������1531/1600���������1532/1600���������1533/
hadoop-m:   safe_format_and_mount: 1600���������1534/1600���������1535/1600���������1536/1600���������1537/1600���������1538/1600���������1539/1600���������1540/1600���������1541/1600���������1542/1600���������1543/1600���������1544/1600���������1545/1600���������1546/1600���������1547/1600���������1548/1600���������1549/1600���������1550/1600���������1551/1600���������1552/1600���������1553/1600���������1554/1600���������1555/1600���������1556/1600���������1557/1600���������1558/1600���������1559/1600���������1560/1600���������1561/1600���������1562/1600���������1563/1600���������1564/1600���������1565/1600���������1566/1600���������1567/1600���������1568/1600���������1569/1600���������1570/1600���������1571/1600���������1572/1600���������1573/1600���������1574/1600���������1575/1600���������1576/1600���������1577/1600���������1578/1600���������1579/1600���������1580/1600���������1581/1600���������1582/1600���������1583/1600���������1584/1600���������1585/1600���������1586/1600���������1587/1600���������1588/1600���������1589/1600���������15
hadoop-m:   safe_format_and_mount: 90/1600���������1591/1600���������1592/1600���������1593/1600���������1594/1600���������1595/1600���������1596/1600���������1597/1600���������1598/1600���������1599/1600���������done                            
hadoop-m:   safe_format_and_mount: Creating journal (32768 blocks): done
hadoop-m:   safe_format_and_mount: Writing superblocks and filesystem accounting information: done
hadoop-m:   safe_format_and_mount: 
hadoop-m:   safe_format_and_mount: This filesystem will be automatically checked every 34 mounts or
hadoop-m:   safe_format_and_mount: 180 days, whichever comes first.  Use tune2fs -c or -i to override.
hadoop-m:   safe_format_and_mount: Running: mount -o discard,defaults /dev/disk/by-id/google-persistent-disk-1 /mnt/pd1
hadoop-m:   which: no apt-get in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)
hadoop-m:   warning: rpmts_HdrFromFdno: Header V4 RSA/SHA1 Signature, key ID 07513cad: NOKEY
hadoop-m:   Importing GPG key 0x07513CAD:
hadoop-m:    Userid: "Jenkins (HDP Builds) <[email protected]>"
hadoop-m:    From  : http://public-repo-1.hortonworks.com/ambari/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
hadoop-m:   nohup: ignoring input
hadoop-m:   
hadoop-m:   ==> bootstrap.stderr <==
hadoop-m:   Downloading file://./thp-disable.sh:                             0 B/451 B    
Downloading file://./ambari_manual_env.sh:                       3.47 KiB/3.47 KiB    
hadoop-m:   Copying gs://csab-hdp-00/bdutil-staging/hadoop-m/20150506-103131-78z/create_blueprint.py...
hadoop-m:   Downloading file://./create_blueprint.py:                        0 B/4.1 KiB    
Downloading file://./thp-disable.sh:                             451 B/451 B    
hadoop-m:   Copying gs://csab-hdp-00/bdutil-staging/hadoop-m/20150506-103131-78z/hadoop-env-setup.sh...
hadoop-m:   Downloading file://./hadoop-env-setup.sh:                        0 B/31.25 KiB    
Downloading file://./create_blueprint.py:                        4.1 KiB/4.1 KiB    
hadoop-m:   Downloading file://./hadoop-env-setup.sh:                        31.25 KiB/31.25 KiB    
hadoop-m:   Copying gs://csab-hdp-00/bdutil-staging/hadoop-m/20150506-103131-78z/deploy-ssh-master-setup.sh...
hadoop-m:   Downloading file://./deploy-ssh-master-setup.sh:                 0 B/1.59 KiB    
Downloading file://./deploy-ssh-master-setup.sh:                 1.59 KiB/1.59 KiB    
hadoop-m:   Copying gs://csab-hdp-00/bdutil-staging/hadoop-m/20150506-103131-78z/deploy-client-nfs-setup.sh...
hadoop-m:   Downloading file://./deploy-client-nfs-setup.sh:                 0 B/1.52 KiB    
Copying gs://csab-hdp-00/bdutil-staging/hadoop-m/20150506-103131-78z/deploy-master-nfs-setup.sh...
hadoop-m:   Downloading file://./deploy-master-nfs-setup.sh:                 0 B/3.59 KiB    
Copying gs://csab-hdp-00/bdutil-staging/hadoop-m/20150506-103131-78z/deploy-ssh-worker-setup.sh...
hadoop-m:   Downloading file://./deploy-ssh-worker-setup.sh:                 0 B/1.37 KiB    
Downloading file://./deploy-master-nfs-setup.sh:                 3.59 KiB/3.59 KiB    
hadoop-m:   Downloading file://./deploy-client-nfs-setup.sh:                 1.52 KiB/1.52 KiB    
hadoop-m:   Copying gs://csab-hdp-00/bdutil-staging/hadoop-m/20150506-103131-78z/deploy-start.sh...
hadoop-m:   Downloading file://./deploy-start.sh:                            0 B/1.27 KiB    
Downloading file://./deploy-ssh-worker-setup.sh:                 1.37 KiB/1.37 KiB    
hadoop-m:   Copying gs://csab-hdp-00/bdutil-staging/hadoop-m/20150506-103131-78z/install_connectors.sh...
hadoop-m:   Downloading file://./install_connectors.sh:                      0 B/7.71 KiB    
Downloading file://./deploy-start.sh:                            1.27 KiB/1.27 KiB    
hadoop-m:   Copying gs://csab-hdp-00/bdutil-staging/hadoop-m/20150506-103131-78z/deploy_start2.sh...
hadoop-m:   Downloading file://./deploy_start2.sh:                           0 B/1.41 KiB    
Downloading file://./install_connectors.sh:                      7.71 KiB/7.71 KiB    
hadoop-m:   Copying gs://csab-hdp-00/bdutil-staging/hadoop-m/20150506-103131-78z/update-ambari-config.sh...
hadoop-m:   Downloading file://./update-ambari-config.sh:                    0 B/2.87 KiB    
Copying gs://csab-hdp-00/bdutil-staging/hadoop-m/20150506-103131-78z/install-gcs-connector-on-ambari.sh...
hadoop-m:   Downloading file://./install-gcs-connector-on-ambari.sh:         0 B/1.47 KiB    
Copying gs://csab-hdp-00/bdutil-staging/hadoop-m/20150506-103131-78z/ambari-setup.sh...
hadoop-m:   Downloading file://./ambari-setup.sh:                            0 B/8.3 KiB    
Downloading file://./deploy_start2.sh:                           1.41 KiB/1.41 KiB    
hadoop-m:   Copying gs://csab-hdp-00/bdutil-staging/hadoop-m/20150506-103131-78z/install-ambari-components.sh...
hadoop-m:   Downloading file://./install-ambari-components.sh:               0 B/3.78 KiB    
Downloading file://./update-ambari-config.sh:                    2.87 KiB/2.87 KiB    
hadoop-m:   Downloading file://./ambari-setup.sh:                            8.3 KiB/8.3 KiB    
hadoop-m:   Downloading file://./install-ambari-components.sh:               3.78 KiB/3.78 KiB    
hadoop-m:   Downloading file://./install-gcs-connector-on-ambari.sh:         1.47 KiB/1.47 KiB    
hadoop-m:   Copying gs://csab-hdp-00/bdutil-staging/hadoop-m/20150506-103131-78z/deploy-core-setup.sh...
hadoop-m:   Downloading file://./deploy-core-setup.sh:                       0 B/29.73 KiB    
Downloading file://./deploy-core-setup.sh:                       29.73 KiB/29.73 KiB    
hadoop-m:   
hadoop-m:   ==> deploy-client-nfs-setup_deploy.stderr <==
hadoop-m:   which: no apt-get in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)
hadoop-m:   which: no apt-get in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)
hadoop-m:   
hadoop-m:   ==> deploy-master-nfs-setup_deploy.stderr <==
hadoop-m:   which: no systemctl in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)
hadoop-m:   which: no apt-get in (/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)
hadoop-m:   Existing lock /var/run/yum.pid: another copy is running as pid 2830.
hadoop-m:   Another app is currently holding the yum lock; waiting for it to exit...
hadoop-m:     The other application is: yum
hadoop-m:       Memory :  28 M RSS (220 MB VSZ)
hadoop-m:       Started: Wed May  6 15:34:44 2015 - 00:01 ago
hadoop-m:       State  : Uninterruptible, pid: 2830
hadoop-m:   
hadoop-m:   ==> install-ambari-components_deploy.stderr <==
hadoop-m:   Traceback (most recent call last):
hadoop-m:     File "create_blueprint.py", line 6, in <module>
hadoop-m:       import argparse
hadoop-m:   ImportError: No module named argparse
hadoop-m:   .

HowTo configure defaultFS for hadoop on singlenode/yarn setup

Hi - I have built a gce structure using ./bdutil deploy --bucket anintelclustergen1-m-disk -n 2 -P anintelcluster -e extensions/spark/spark_on_yarn_env.sh.

In the bucket paraments, both in command and bdutil_evn.sh, I have specified a non-boot bucket.
In the core-site.xml (under hadoop/etc) on the master, it show the xml with the correct bucket value under defaultFS.
However, the hadoop console (50070) does not show the nonboot bucket attached, but shows the boot disk attached on the name node.

Node Last contact Admin State Capacity Used Non DFS Used Remaining Blocks Block pool used Failed Volumes Version
anintelcluster.c.anintelcluster.internal:50010 (10.240.0.2:50010) 0 In Service 98.4 GB 28 KB 6.49 GB 91.91 GB 0 28 KB (0%) 0 2.7.1

Is it possible to specify a non-boot bucket with the singlenode setup?
If not, what needs to be done to be able to specify the non-boot disk, which will both get attached to instance as read/write and also be used by hadoop for storage etc?

Deploy HDP 2.3 with bdutil

Hi,
I'm trying to deploy HDP 2.3 with bdutil.

I've set these configuration values

AMBARI_REPO="http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.0.1/ambari.repo"
AMBARI_STACK_VERSION='2.3'

in ambari.conf.

The deployment fails with

Mon Jun 29 10:26:52 CEST 2015: Invoking on master: ./install-ambari-components.sh
.Mon Jun 29 10:26:53 CEST 2015: Waiting on async 'ssh' jobs to finish. Might take a while...
Mon Jun 29 10:26:56 CEST 2015: Exited 1 : gcloud --project=XXXXXXX --quiet --verbosity=info compute ssh hdp22-m --command=sudo su -l -c "cd ${PWD} && ./install-ambari-components.sh" 2>>install-ambari-components_deploy.stderr 1>>install-ambari-components_deploy.stdout --ssh-flag=-tt --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=europe-west1-d
Mon Jun 29 10:26:56 CEST 2015: Fetching on-VM logs from hdp22-m
INFO: Refreshing access_token
Warning: Permanently added '130.211.X.X' (RSA) to the list of known hosts.
Mon Jun 29 10:26:57 CEST 2015: Command failed: wait ${SUBPROC} on line 311.
Mon Jun 29 10:26:57 CEST 2015: Exit code of failed command: 1
Mon Jun 29 10:26:57 CEST 2015: Detailed debug info available in file: /tmp/bdutil-20150629-102300-GFV/debuginfo.txt
Mon Jun 29 10:26:57 CEST 2015: Check console output for error messages and/or retry your command.

/tmp/bdutil-20150629-102300-GFV/debuginfo.txt contains:

hdp22-m:        ==> install-ambari-components_deploy.stderr <==
hdp22-m:        Traceback (most recent call last):
hdp22-m:          File "<string>", line 1, in <module>
hdp22-m:        ImportError: No module named argparse
hdp22-m:        Traceback (most recent call last):
hdp22-m:          File "./create_blueprint.py", line 122, in <module>
hdp22-m:            main()
hdp22-m:          File "./create_blueprint.py", line 118, in main
hdp22-m:            args.custom_configuraton)
hdp22-m:          File "./create_blueprint.py", line 76, in create_blueprints
hdp22-m:            configuration_recommendation = json.load(conf_recommendation_file)
hdp22-m:          File "/usr/lib64/python2.6/json/__init__.py", line 267, in load
hdp22-m:            parse_constant=parse_constant, **kw)
hdp22-m:          File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
hdp22-m:            return _default_decoder.decode(s)
hdp22-m:          File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
hdp22-m:            obj, end = self.raw_decode(s, idx=_w(s, 0).end())
hdp22-m:          File "/usr/lib64/python2.6/json/decoder.py", line 338, in raw_decode
hdp22-m:            raise ValueError("No JSON object could be decoded")
hdp22-m:        ValueError: No JSON object could be decoded
hdp22-m:        .

Does the hdp module in bdutil support HDP 2.3 ?

bdutil's "-f" flag doesn't work for some confirmation prompts

The -f/--force flag is supposed to skip all confirmation prompts, but it doesn't correctly skip the prompts for running as root or for overwriting an existing config in generate_config.

Stability of script for HDP platform

This isn't meant as a criticism, as I realise there are 1,000 possible things that could be going wrong, but this script seems to only successfully deploy a cluster in around 1 in 5 attempts.

The exception seems to be different each time, but the common ones are:
at upload of config scripts:

Uploading   ...20150811-000113-Hq6/install-ambari-components.sh: 3.9 KiB/3.9 KiB
CommandException: 1 files/objects could not be transferred.

when running deploy scripts on master / workers:

Mon, Aug 10, 2015 11:55:27 PM: Exited 1 : gcloud --project=yyyy --quiet --verbosity=info compute   ssh hadoop-w-1 --command=sudo su -l -c "cd ${PWD} && ./ambari-setup.sh" 2>>ambari-setup_deploy.stderr 1>>ambari-setup_deploy.stdout --ssh-flag=-tt --ssh-flag=-oServerAliveInterval=60 --  ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=europe-west1-b
 Mon, Aug 10, 2015 11:55:28 PM: Fetching on-VM logs from hadoop-w-1
 Warning: Permanently added 'x.y.z.m' (RSA) to the list of known hosts.
...Mon, Aug 10, 2015 11:57:43 PM: Command failed: wait ${SUBPROC} on line 326.

during the ambari-components install

 Mon, Aug 10, 2015 11:43:54 PM: Step 'deploy-client-nfs-setup,deploy-client-nfs-setup' done...

Mon, Aug 10, 2015 11:43:54 PM: Invoking on master: ./install-ambari-components.sh
../bdutil: line 318: 10548 Segmentation fault sleep '0.5'

By their nature they are hard to reproduce, as I am running the same script each time.

bdutil extenstion for Apache Tajo

Hi,
It seems that yshan's contribution is waiting for review. His contribution is a bdutil extention for Apache Tajo, an open source SQL-on-Hadoop system (http://tajo.apache.org). The extension enables Google cloud users to deploy and setup Tajo cluster and start their big data analysis on GCP.

Can you please review the request?
#67

Youngkyong

ERROR: (gcloud.compute.ssh) could not fetch resource: The resource 'projects/recommendationshashiproject/zones/us-central1-a/instances/hadoop-m' was not found

Hi All,
I am creating project recommendation on google cloud, in that I am setting up
Download the bdutil repository through below link
git clone https://github.com/GoogleCloudPlatform/bdutil.git
Set the variable in bdutil_env.sh file
Project ID, configbucket and ZONE.
CONFIGBUCKET='unilogrootbucket'(I have created a empty bucket ihttps://console.cloud.google.com/storage)
PROJECT='recommendationshashiproject'(https://console.cloud.google.com/home/)
GCE_ZONE='us-central1-a'

Below is the command used to run bdutil
. /bdutil deploy -e extensions/spark/spark_env.sh

. /bdutil shell

But I got error like
ERROR: (gcloud.compute.ssh) could not fetch resource:
The resource 'projects/recommendationshashiproject/zones/us-central1-a/instances/hadoop-m' was not found
complete error log
shashi_kumarmirle@recommendationshashiproject:~/bdutil-master$ ./bdutil deploy -e extensions/spark/spark_env.sh

Thu Oct 26 17:41:33 IST 2017: Using local tmp dir for staging files: /tmp/bdutil-20171026-174133-7kt
Thu Oct 26 17:41:33 IST 2017: Using custom environment-variable file(s): bdutil_env.sh extensions/spark/spark_env.sh
Thu Oct 26 17:41:33 IST 2017: Reading environment-variable file: ./bdutil_env.sh
Thu Oct 26 17:41:33 IST 2017: Reading environment-variable file: extensions/spark/spark_env.sh
Thu Oct 26 17:41:33 IST 2017: No explicit GCE_MASTER_MACHINE_TYPE provided; defaulting to value of GCE_MACHINE_TYPE: n1-standard-4
Deploy cluster with following settings?
CONFIGBUCKET='unilogrootbucket'
PROJECT='recommendationshashiproject'
GCE_IMAGE=''
GCE_IMAGE_PROJECT='debian-cloud'
GCE_IMAGE_FAMILY='debian-8'
GCE_ZONE='us-central1-a'
GCE_NETWORK='default'
GCE_TAGS='bdutil'
PREEMPTIBLE_FRACTION=0.0
PREFIX='hadoop'
NUM_WORKERS=2
MASTER_HOSTNAME='hadoop-m'
WORKERS='hadoop-w-0 hadoop-w-1'
BDUTIL_GCS_STAGING_DIR='gs://unilogrootbucket/bdutil-staging/hadoop-m'
(y/n) y
Thu Oct 26 17:41:36 IST 2017: Checking for existence of gs://unilogrootbucket...
gs://unilogrootbucket/
Thu Oct 26 17:41:38 IST 2017: Checking for existence of gs://hadoop-dist/hadoop-1.2.1-bin.tar.gz...
Thu Oct 26 17:41:41 IST 2017: Checking upload files...
Thu Oct 26 17:41:41 IST 2017: Verified './conf/hadoop1/bq-mapred-template.xml'
Thu Oct 26 17:41:41 IST 2017: Verified './conf/hadoop1/core-template.xml'
Thu Oct 26 17:41:41 IST 2017: Verified './conf/hadoop1/gcs-core-template.xml'
Thu Oct 26 17:41:41 IST 2017: Verified './conf/hadoop1/hdfs-template.xml'
Thu Oct 26 17:41:41 IST 2017: Verified './conf/hadoop1/mapred-health-check.sh'
Thu Oct 26 17:41:41 IST 2017: Verified './conf/hadoop1/mapred-template.xml'
Thu Oct 26 17:41:41 IST 2017: Verified './libexec/hadoop_helpers.sh'
Thu Oct 26 17:41:41 IST 2017: Generating 10 command groups...
Thu Oct 26 17:41:41 IST 2017: Done generating remote shell scripts.
Thu Oct 26 17:41:41 IST 2017: Creating worker instances: hadoop-w-0 hadoop-w-1
..Thu Oct 26 17:41:41 IST 2017: Creating master instance: hadoop-m
.Thu Oct 26 17:41:42 IST 2017: Waiting on async 'instances create' jobs to finish. Might take a while...
Thu Oct 26 17:42:06 IST 2017: Exited 1 : gcloud --project=recommendationshashiproject --quiet --verbosity=info compute instances create hadoop-w-1 --machine-type=n1-standard-4 --image-family=debian-8 --image-project=debian-cloud --network=default --tags=bdutil --scopes storage-full --boot-disk-type=pd-standard --zone=us-central1-a
Thu Oct 26 17:42:06 IST 2017: Exited 1 : gcloud --project=recommendationshashiproject --quiet --verbosity=info compute instances create hadoop-w-0 --machine-type=n1-standard-4 --image-family=debian-8 --image-project=debian-cloud --network=default --tags=bdutil --scopes storage-full --boot-disk-type=pd-standard --zone=us-central1-a
Thu Oct 26 17:42:07 IST 2017: Exited 1 : gcloud --project=recommendationshashiproject --quiet --verbosity=info compute instances create hadoop-m --machine-type=n1-standard-4 --image-family=debian-8 --image-project=debian-cloud --network=default --tags=bdutil --scopes storage-full --boot-disk-type=pd-standard --zone=us-central1-a
Thu Oct 26 17:42:07 IST 2017: Command failed: wait ${SUBPROC} on line 326.
Thu Oct 26 17:42:07 IST 2017: Exit code of failed command: 1
Thu Oct 26 17:42:07 IST 2017: Detailed debug info available in file: /tmp/bdutil-20171026-174133-7kt/debuginfo.txt
Thu Oct 26 17:42:07 IST 2017: Check console output for error messages and/or retry your command.
shashi_kumarmirle@recommendationshashiproject:~/bdutil-master$ ./bdutil shell
Thu Oct 26 17:42:20 IST 2017: Using local tmp dir for staging files: /tmp/bdutil-20171026-174220-tyx
Thu Oct 26 17:42:20 IST 2017: Using custom environment-variable file(s): bdutil_env.sh
Thu Oct 26 17:42:20 IST 2017: Reading environment-variable file: ./bdutil_env.sh
Thu Oct 26 17:42:20 IST 2017: No explicit GCE_MASTER_MACHINE_TYPE provided; defaulting to value of GCE_MACHINE_TYPE: n1-standard-4
Thu Oct 26 17:42:20 IST 2017: Running gcloud --project=recommendationshashiproject --quiet --verbosity=info compute ssh hadoop-m --command= --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-central1-a
ERROR: (gcloud.compute.ssh) Could not fetch resource:

The resource 'projects/recommendationshashiproject/zones/us-central1-a/instances/hadoop-m' was not found

Thu Oct 26 17:42:22 IST 2017: Exited 1 : gcloud --project=recommendationshashiproject --quiet --verbosity=info compute ssh hadoop-m --command= --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-central1-a
Thu Oct 26 17:42:22 IST 2017: Command failed: return ${exitcode} on line 453.
Thu Oct 26 17:42:22 IST 2017: Exit code of failed command: 1
Thu Oct 26 17:42:22 IST 2017: Detailed debug info available in file: /tmp/bduti
Thu Oct 26 17:42:22 IST 2017: Check console output for error messages and/or retry your command.
shashi_kumarmirle@recommendationshashiproject:~/bdutil-master$ ./bdutil shell
Thu Oct 26 18:05:09 IST 2017: Using local tmp dir for staging files: /tmp/bdutil-20171026-180509-mSH
Thu Oct 26 18:05:09 IST 2017: Using custom environment-variable file(s): bdutil_env.sh
Thu Oct 26 18:05:09 IST 2017: Reading environment-variable file: ./bdutil_env.sh
Thu Oct 26 18:05:09 IST 2017: No explicit GCE_MASTER_MACHINE_TYPE provided; defaulting to value of GCE_MACHINE_TYPE: n1-standard-4
Thu Oct 26 18:05:09 IST 2017: Running gcloud --project=recommendationshashiproject --quiet --verbosity=info compute ssh hadoop-m --command= --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-central1-a
ERROR: (gcloud.compute.ssh) Could not fetch resource:

The resource 'projects/recommendationshashiproject/zones/us-central1-a/instances/hadoop-m' was not found

Thu Oct 26 18:05:11 IST 2017: Exited 1 : gcloud --project=recommendationshashiproject --quiet --verbosity=info compute ssh hadoop-m --command= --ssh-flag=-oServerAliveInterval=60 --ssh-flag=-oServerAliveCountMax=3 --ssh-flag=-oConnectTimeout=30 --zone=us-central1-a
Thu Oct 26 18:05:11 IST 2017: Command failed: return ${exitcode} on line 453.
Thu Oct 26 18:05:11 IST 2017: Exit code of failed command: 1
Thu Oct 26 18:05:11 IST 2017: Detailed debug info available in file: /tmp/bdutil-20171026-180509-mSH/debuginfo.txt
Thu Oct 26 18:05:11 IST 2017: Check console output for error messages and/or retry your command.

add boot-disk-size option on instance creation

The script creates instances with default boot disk size(10GB), but there is no env variable to modify this value.
It would be handy to add the option
--boot-disk-size=${DISK_BOOT_SIZE_GB}
in the function create_cluster

How to create HDP cluster with High Availability?

Is there any way we can use bdutil to create HA Hadoop cluster ?

'hadoop-m' not yet sshable (1); sleeping 10

I'm using google trail account. when i execute the command ./bdutil -e platforms/hdp/ambari_env.sh deploy it throwed the following error on master and worker nodes.
'hadoop-m' not yet sshable (1); sleeping 10
The complete output of the above command is
[root@sandbox bdutil]# ./bdutil -e platforms/hdp/ambari_env.sh deploy Thu Sep 17 09:20:40 UTC 2015: Using local tmp dir for staging files: /tmp/bdutil-20150917-092040-XpJ Thu Sep 17 09:20:40 UTC 2015: Using custom environment-variable file(s): bdutil_env.sh platforms/hdp/ambari_env.sh Thu Sep 17 09:20:40 UTC 2015: Reading environment-variable file: ./bdutil_env.sh Thu Sep 17 09:20:40 UTC 2015: Reading environment-variable file: platforms/hdp/ambari_env.sh Importing dependent env file: ./platforms/hdp/ambari_manual_env.sh Importing dependent env file: ./hadoop2_env.sh Importing dependent env file: ./platforms/hdp/ambari.conf ./platforms/hdp/ambari.conf: line 1: s: command not found Importing dependent env file: ./platforms/hdp/ambari_functions.sh Thu Sep 17 09:20:40 UTC 2015: No explicit GCE_MASTER_MACHINE_TYPE provided; defaulting to value of GCE_MACHINE_TYPE: n1-standard-2 Deploy cluster with following settings? CONFIGBUCKET='hadoopbucket001' PROJECT='hadoop-001-1071' GCE_IMAGE='centos-6' GCE_ZONE='us-central1-a' GCE_NETWORK='default' PREEMPTIBLE_FRACTION=0.0 PREFIX='hadoop' NUM_WORKERS=2 MASTER_HOSTNAME='hadoop-m' WORKERS='hadoop-w-0 hadoop-w-1' BDUTIL_GCS_STAGING_DIR='gs://hadoopbucket001/bdutil-staging/hadoop-m' MASTER_ATTACHED_PD='hadoop-m-pd' WORKER_ATTACHED_PDS='hadoop-w-0-pd hadoop-w-1-pd' (y/n) y Are you sure you want to run the command as root? (y/n)y Thu Sep 17 09:20:49 UTC 2015: Checking for existence of gs://hadoopbucket001... gs://hadoopbucket001/ Thu Sep 17 09:20:56 UTC 2015: Checking for existence of gs://hadoop-dist/hadoop-2.7.1.tar.gz... Thu Sep 17 09:20:59 UTC 2015: Checking upload files... Thu Sep 17 09:20:59 UTC 2015: Verified './conf/hadoop2/bigtable-hbase-site-template.xml' Thu Sep 17 09:20:59 UTC 2015: Verified './conf/hadoop2/gcs-core-template.xml' Thu Sep 17 09:20:59 UTC 2015: Verified './conf/hadoop2/core-template.xml' Thu Sep 17 09:20:59 UTC 2015: Verified './conf/hadoop2/yarn-template.xml' Thu Sep 17 09:20:59 UTC 2015: Verified './conf/hadoop2/hdfs-template.xml' Thu Sep 17 09:20:59 UTC 2015: Verified './conf/hadoop2/bq-mapred-template.xml' Thu Sep 17 09:20:59 UTC 2015: Verified './conf/hadoop2/mapred-template.xml' Thu Sep 17 09:20:59 UTC 2015: Verified './libexec/hadoop_helpers.sh' Thu Sep 17 09:20:59 UTC 2015: Verified './libexec/configure_mrv2_mem.py' Thu Sep 17 09:20:59 UTC 2015: Verified './hadoop2_env.sh' Thu Sep 17 09:20:59 UTC 2015: Verified './platforms/hdp/ambari.conf' Thu Sep 17 09:20:59 UTC 2015: Verified './platforms/hdp/ambari_functions.sh' Thu Sep 17 09:20:59 UTC 2015: Verified './libexec/hadoop_helpers.sh' Thu Sep 17 09:20:59 UTC 2015: Verified './platforms/hdp/configuration.json' Thu Sep 17 09:20:59 UTC 2015: Verified './platforms/hdp/resources/public-hostname-gcloud.sh' Thu Sep 17 09:20:59 UTC 2015: Verified './platforms/hdp/resources/thp-disable.sh' Thu Sep 17 09:20:59 UTC 2015: Verified './platforms/hdp/ambari_manual_env.sh' Thu Sep 17 09:20:59 UTC 2015: Verified './platforms/hdp/create_blueprint.py' Thu Sep 17 09:20:59 UTC 2015: Generating 12 command groups... Thu Sep 17 09:21:00 UTC 2015: Done generating remote shell scripts. Thu Sep 17 09:21:00 UTC 2015: Creating attached worker disks: hadoop-w-0-pd hadoop-w-1-pd ..Thu Sep 17 09:21:00 UTC 2015: Creating attached master disk: hadoop-m-pd .Thu Sep 17 09:21:00 UTC 2015: Done creating disks! Thu Sep 17 09:21:01 UTC 2015: Waiting on async 'disks create' jobs to finish. Might take a while... ... Thu Sep 17 09:21:11 UTC 2015: Creating worker instances: hadoop-w-0 hadoop-w-1 ..Thu Sep 17 09:21:11 UTC 2015: Creating master instance: hadoop-m .Thu Sep 17 09:21:11 UTC 2015: Waiting on async 'instances create' jobs to finish. Might take a while... ... Thu Sep 17 09:22:02 UTC 2015: Instances all created. Entering polling loop to wait for ssh-ability ...Thu Sep 17 09:22:03 UTC 2015: Waiting on async 'wait_for_ssh' jobs to finish. Might take a while... Thu Sep 17 09:22:09 UTC 2015: 'hadoop-w-0' not yet sshable (1); sleeping 10. Thu Sep 17 09:22:09 UTC 2015: 'hadoop-m' not yet sshable (1); sleeping 10. Thu Sep 17 09:22:09 UTC 2015: 'hadoop-w-1' not yet sshable (1); sleeping 10. ...Thu Sep 17 09:22:26 UTC 2015: 'hadoop-w-0' not yet sshable (1); sleeping 10. Thu Sep 17 09:22:26 UTC 2015: 'hadoop-m' not yet sshable (1); sleeping 10. Thu Sep 17 09:22:26 UTC 2015: 'hadoop-w-1' not yet sshable (1); sleeping 10. ...Thu Sep 17 09:22:43 UTC 2015: 'hadoop-m' not yet sshable (1); sleeping 10. Thu Sep 17 09:22:43 UTC 2015: 'hadoop-w-0' not yet sshable (1); sleeping 10. Thu Sep 17 09:22:43 UTC 2015: 'hadoop-w-1' not yet sshable (1); sleeping 10. ...Thu Sep 17 09:22:59 UTC 2015: 'hadoop-w-0' not yet sshable (1); sleeping 10. Thu Sep 17 09:23:00 UTC 2015: 'hadoop-w-1' not yet sshable (1); sleeping 10. Thu Sep 17 09:23:01 UTC 2015: 'hadoop-m' not yet sshable (1); sleeping 10. ...Thu Sep 17 09:23:15 UTC 2015: 'hadoop-w-0' not yet sshable (1); sleeping 10. Thu Sep 17 09:23:18 UTC 2015: 'hadoop-w-1' not yet sshable (1); sleeping 10. Thu Sep 17 09:23:18 UTC 2015: 'hadoop-m' not yet sshable (1); sleeping 10. ...Thu Sep 17 09:23:31 UTC 2015: 'hadoop-w-0' not yet sshable (1); sleeping 10. Thu Sep 17 09:23:36 UTC 2015: 'hadoop-m' not yet sshable (1); sleeping 10. Thu Sep 17 09:23:38 UTC 2015: 'hadoop-w-1' not yet sshable (1); sleeping 10. ...Thu Sep 17 09:23:49 UTC 2015: 'hadoop-w-0' not yet sshable (1); sleeping 10. Thu Sep 17 09:23:51 UTC 2015: 'hadoop-m' not yet sshable (1); sleeping 10. Thu Sep 17 09:23:56 UTC 2015: 'hadoop-w-1' not yet sshable (1); sleeping 10. ...Thu Sep 17 09:24:07 UTC 2015: 'hadoop-w-0' not yet sshable (1); sleeping 10. Thu Sep 17 09:24:08 UTC 2015: 'hadoop-m' not yet sshable (1); sleeping 10. Thu Sep 17 09:24:11 UTC 2015: 'hadoop-w-1' not yet sshable (1); sleeping 10. ...Thu Sep 17 09:24:24 UTC 2015: 'hadoop-w-0' not yet sshable (1); sleeping 10. Thu Sep 17 09:24:24 UTC 2015: 'hadoop-m' not yet sshable (1); sleeping 10. Thu Sep 17 09:24:28 UTC 2015: 'hadoop-w-1' not yet sshable (1); sleeping 10. ...Thu Sep 17 09:24:42 UTC 2015: 'hadoop-w-0' not yet sshable (1); sleeping 10. Thu Sep 17 09:24:42 UTC 2015: 'hadoop-m' not yet sshable (1); sleeping 10. Thu Sep 17 09:24:46 UTC 2015: 'hadoop-w-1' not yet sshable (1); sleeping 10. Thu Sep 17 09:24:52 UTC 2015: Node 'hadoop-w-0' did not become ssh-able after 10 attempts Thu Sep 17 09:24:52 UTC 2015: Node 'hadoop-m' did not become ssh-able after 10 attempts Thu Sep 17 09:24:56 UTC 2015: Node 'hadoop-w-1' did not become ssh-able after 10 attempts Thu Sep 17 09:24:56 UTC 2015: Command failed: wait ${SUBPROC} on line 326. Thu Sep 17 09:24:56 UTC 2015: Exit code of failed command: 1 Thu Sep 17 09:24:56 UTC 2015: Detailed debug info available in file: /tmp/bdutil-20150917-092040-XpJ/debuginfo.txt Thu Sep 17 09:24:56 UTC 2015: Check console output for error messages and/or retry your command.

And the error log file /tmp/bdutil-20150917-092040-XpJ/debuginfo.txt output is

******************* gcloud compute stdout *******************
NAME        ZONE          SIZE_GB TYPE        STATUS
hadoop-m-pd us-central1-a 1500    pd-standard READY
NAME          ZONE          SIZE_GB TYPE        STATUS
hadoop-w-1-pd us-central1-a 1500    pd-standard READY
NAME          ZONE          SIZE_GB TYPE        STATUS
hadoop-w-0-pd us-central1-a 1500    pd-standard READY
NAME       ZONE          MACHINE_TYPE  PREEMPTIBLE INTERNAL_IP   EXTERNAL_IP    STATUS
hadoop-w-0 us-central1-a n1-standard-2             10.240.90.191 173.255.112.33 RUNNING
NAME     ZONE          MACHINE_TYPE  PREEMPTIBLE INTERNAL_IP   EXTERNAL_IP     STATUS
hadoop-m us-central1-a n1-standard-2             10.240.152.66 130.211.160.182 RUNNING
NAME       ZONE          MACHINE_TYPE  PREEMPTIBLE INTERNAL_IP    EXTERNAL_IP    STATUS
hadoop-w-1 us-central1-a n1-standard-2             10.240.237.109 104.197.79.138 RUNNING

******************* gcloud compute stderr *******************
/root/y/google-cloud-sdk/./lib/googlecloudsdk/compute/lib/base_classes.py:9: DeprecationWarning: the sets module is deprecated
  import sets
/root/y/google-cloud-sdk/./lib/googlecloudsdk/compute/lib/base_classes.py:9: DeprecationWarning: the sets module is deprecated
  import sets
/root/y/google-cloud-sdk/./lib/googlecloudsdk/compute/lib/base_classes.py:9: DeprecationWarning: the sets module is deprecated
  import sets
Created [https://www.googleapis.com/compute/v1/projects/hadoop-001-1071/zones/us-central1-a/disks/hadoop-m-pd].
Created [https://www.googleapis.com/compute/v1/projects/hadoop-001-1071/zones/us-central1-a/disks/hadoop-w-1-pd].
Created [https://www.googleapis.com/compute/v1/projects/hadoop-001-1071/zones/us-central1-a/disks/hadoop-w-0-pd].
/root/y/google-cloud-sdk/./lib/googlecloudsdk/compute/lib/base_classes.py:9: DeprecationWarning: the sets module is deprecated
  import sets
WARNING: We noticed that you are using space-separated lists, which are deprecated. Please transition to using comma-separated lists instead (try "--disk name=hadoop-w-0-pd,mode=rw"). If you intend to use [mode=rw] as positional arguments, put the flags at the end.
/root/y/google-cloud-sdk/./lib/googlecloudsdk/compute/lib/base_classes.py:9: DeprecationWarning: the sets module is deprecated
  import sets
WARNING: We noticed that you are using space-separated lists, which are deprecated. Please transition to using comma-separated lists instead (try "--disk name=hadoop-m-pd,mode=rw"). If you intend to use [mode=rw] as positional arguments, put the flags at the end.
/root/y/google-cloud-sdk/./lib/googlecloudsdk/compute/lib/base_classes.py:9: DeprecationWarning: the sets module is deprecated
  import sets
WARNING: We noticed that you are using space-separated lists, which are deprecated. Please transition to using comma-separated lists instead (try "--disk name=hadoop-w-1-pd,mode=rw"). If you intend to use [mode=rw] as positional arguments, put the flags at the end.
Created [https://www.googleapis.com/compute/v1/projects/hadoop-001-1071/zones/us-central1-a/instances/hadoop-w-0].
Created [https://www.googleapis.com/compute/v1/projects/hadoop-001-1071/zones/us-central1-a/instances/hadoop-m].
Created [https://www.googleapis.com/compute/v1/projects/hadoop-001-1071/zones/us-central1-a/instances/hadoop-w-1].
/root/y/google-cloud-sdk/./lib/googlecloudsdk/compute/lib/base_classes.py:9: DeprecationWarning: the sets module is deprecated
  import sets
/root/y/google-cloud-sdk/./lib/googlecloudsdk/compute/lib/base_classes.py:9: DeprecationWarning: the sets module is deprecated
  import sets
Warning: Permanently added '104.197.79.138' (RSA) to the list of known hosts.^M
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).^M
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]. See https://cloud.google.com/compute/docs/troubleshooting#ssherrors for troubleshooting hints.
/root/y/google-cloud-sdk/./lib/googlecloudsdk/compute/lib/base_classes.py:9: DeprecationWarning: the sets module is deprecated
  import sets
/root/y/google-cloud-sdk/./lib/googlecloudsdk/compute/lib/base_classes.py:9: DeprecationWarning: the sets module is deprecated
  import sets
Warning: Permanently added '173.255.112.33' (RSA) to the list of known hosts.^M
Warning: Permanently added '130.211.160.182' (RSA) to the list of known hosts.^M
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).^M
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]. See https://cloud.google.com/compute/docs/troubleshooting#ssherrors for troubleshooting hints.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).^M
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]. See https://cloud.google.com/compute/docs/troubleshooting#ssherrors for troubleshooting hints.
/root/y/google-cloud-sdk/./lib/googlecloudsdk/compute/lib/base_classes.py:9: DeprecationWarning: the sets module is deprecated
  import sets
Warning: Permanently added '104.197.79.138' (RSA) to the list of known hosts.^M
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).^M
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]. See https://cloud.google.com/compute/docs/troubleshooting#ssherrors for troubleshooting hints.

************ ERROR logs from gcloud compute stderr ************
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]. See https://cloud.google.com/compute/docs/troubleshooting#ssherrors for troubleshooting hints.
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]. See https://cloud.google.com/compute/docs/troubleshooting#ssherrors for troubleshooting hints.
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]. See https://cloud.google.com/compute/docs/troubleshooting#ssherrors for troubleshooting hints.
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]. See https://cloud.google.com/compute/docs/troubleshooting#ssherrors for troubleshooting hints.
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]. See https://cloud.google.com/compute/docs/troubleshooting#ssherrors for troubleshooting hints.
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]. See https://cloud.google.com/compute/docs/troubleshooting#ssherrors for troubleshooting hints.
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]. See https://cloud.google.com/compute/docs/troubleshooting#ssherrors for troubleshooting hints.
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]. See https://cloud.google.com/compute/docs/troubleshooting#ssherrors for troubleshooting hints.
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255]. See https://cloud.google.com/compute/docs/troubleshooting#ssherrors for troubleshooting hints.

Can anyone help me to sort out the issue ??
Where it is going wrong ??
Please help me..

googleclouddataproc / bdutil Goto Github PK

bdutil's Introduction

This project has been deprecated. Please use Google Cloud Dataproc to create managed Apache Hadoop and Apache Spark instances on Google Compute Engine.

bdutil

Requirements

Usage

Components installed

Documentation

bdutil's People

Contributors

Stargazers

Watchers

Forkers

bdutil's Issues

Tue Sep 6 23:26:16 IST 2016: Check console output for error messages and/or retry your command.

Recommend Projects

Recommend Topics

Recommend Org