Comments (6)
I can try to repro this. While bdutil doesn't have built-in commands for "stopping" and "starting" a cluster, there are various flavors of stopping/starting which are indeed supported.
How did you "stop" and "restart" your cluster? Did you only stop/restart your master, or did you do it on every node? Are you using Local SSDs, and are you using USE_ATTACHED_PDS?
from bdutil.
This example was based on single node using the following command:
./bdutil deploy -e hadoop2_env.sh --bucket anintelcluster -e single_node_env.sh -e spark_env.sh -P anintelcluster
All I did was top the single node. However, when I bring it up, even though the hadoop and spark services are running, my application is not able to access them via configured URL e.g. hadoop URL is hdfs://anintelcluster:8020.
The way I stopped was using the developer console - Compute>VM>select VM>Stop/Start.
Re: HDD - I use whatever is default for the bdutil script - no changes to that.
from bdutil.
Alright, I did manage to repro. There might be some race-condition that's more specific to a single-node environment during startup. I was able to repair HDFS simply by running:
/home/hadoop/hadoop-install/sbin/stop-all.sh
/home/hadoop/hadoop-install/sbin/start-all.sh
When I did that I was able to access my old data inside HDFS again that I added prior to the stop/start. I'll check on why the automatic restart didn't work.
EDIT: Forgot to mention, you have to run those scripts as user hadoop
, so first run sudo su hadoop
.
from bdutil.
Alright, tracked down the bug to configure_startup_processes.sh and spark_configure_startup_processes.sh not correctly adding worker daemons in the case of single-node clusters. This bug only affects clusters which use single_node_env.sh. I've successfully tested a fix and sent a pull request which we'll merge shortly: #42
You can optionally clone from my fork in the meantime to check it out: https://github.com/dennishuo/bdutil
Or it should it make it into the master within a couple days.
from bdutil.
#42 has been merged, issue should be fixed now if you use bdutil at head revision.
from bdutil.
Confirm bug fixed ..
from bdutil.
Related Issues (20)
- gcloud compute ssh tunneling refusing connection HOT 1
- Can't install with Ambari 2.5+ HOT 2
- hortonworks on mac os 10.12.6
- ERROR: (gcloud.compute.ssh) could not fetch resource: The resource 'projects/recommendationshashiproject/zones/us-central1-a/instances/hadoop-m' was not found HOT 5
- subnetwork error HOT 1
- Error Deploy bdutil : Invalid value for field resource.networkInterfaces HOT 1
- Unable to start Ambari NameNode Security Exception
- Command failed: wait ${SUBPROC} on line 326 HOT 2
- unable to deploy ambari using bdutil
- Nonissue..delete
- Command failed line 326, and SSH Key HOT 2
- Cluster creation can fail if apt-get update has warnings
- How to create HDP cluster with High Availability? HOT 1
- Cluster creation is failing with "Failed to find image for alias [debian-7-backports]..."
- Upgrade bdutil to Debian 8
- Does not seem to working with Custom network HOT 2
- bdutil deploy fails with ssh error
- Add a "version" command|flag
- bdutil confusingly reports "using project ${GCE_ZONE}"
- unable to deploy ambari using bdutil HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bdutil.