randerzander / docker-hdp Goto Github PK
View Code? Open in Web Editor NEWDockerized HDP Cluster
Dockerized HDP Cluster
When try to start my sandbox 2.6.4, the process go Ok until the last step: start the cleanuo of temporary directories. Fo some reason, start the process but it doesn't end. Could you help me?
Thanks a bunch for working on this project! I've been trying to get Ambari working on Docker and this seems to be right what I need. I do have a couple of concerns though.
As you can see in the screenshot here, everything works fine up to this point, after which it remains at "Preparing to Deploy: 3 of 82 tasks completed." indefinitely.
I've grabbed the logs (/var/log/ambari-server/ambari-server.log) from the ambari-server
container here: stacktrace.txt
It seems like it's encountering this:
ERROR [ambari-client-thread-77] ContainerResponse:423 - The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
org.eclipse.jetty.io.EofException: early EOF
Which looks similar to a bug that was fixed in 2013 (https://issues.apache.org/jira/browse/AMBARI-2024).
For reference, below are details on the host environment:
[root@centos7]# cat /etc/*release
CentOS Linux release 7.3.1611 (Core)
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
CentOS Linux release 7.3.1611 (Core)
CentOS Linux release 7.3.1611 (Core)
[root@centos7]# docker --version
Docker version 1.12.4, build 1564f02
[root@centos7]# docker-compose --version
docker-compose version 1.9.0, build 2585387
[root@centos7]#docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hdp/worker latest 1fbfcfe795b3 About an hour ago 7.545 GB
hdp/master latest 1fbfcfe795b3 About an hour ago 7.545 GB
hdp/ambari-server latest decb06169b34 About an hour ago 2.001 GB
hdp/postgres latest a4c3d3f4978b About an hour ago 309.8 MB
postgres latest 0e24dd8079dc 4 weeks ago 264.8 MB
centos 6 8315978ceaaa 10 weeks ago 194.6 MB
[root@centos7]#docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
976101ed60d9 hdp/worker "/bin/sh -c /start.sh" About an hour ago Up About an hour 0.0.0.0:6667->6667/tcp, 0.0.0.0:8042->8042/tcp, 0.0.0.0:8983->8983/tcp, 0.0.0.0:16020->16020/tcp, 0.0.0.0:16030->16030/tcp, 0.0.0.0:50010->50010/tcp, 0.0.0.0:50020->50020/tcp, 0.0.0.0:50030->50030/tcp, 0.0.0.0:50070->50070/tcp, 0.0.0.0:50075->50075/tcp, 0.0.0.0:50470->50470/tcp, 0.0.0.0:50475->50475/tcp, 0.0.0.0:45455->45454/tcp compose_dn0.dev_1
f21dfa771b9a hdp/master "/bin/sh -c /start.sh" About an hour ago Up About an hour 0.0.0.0:2181->2181/tcp, 0.0.0.0:3000->3000/tcp, 0.0.0.0:6080->6080/tcp, 0.0.0.0:6083->6083/tcp, 0.0.0.0:6182-6183->6182-6183/tcp, 0.0.0.0:8000->8000/tcp, 0.0.0.0:8020->8020/tcp, 0.0.0.0:8025->8025/tcp, 0.0.0.0:8030->8030/tcp, 0.0.0.0:8050->8050/tcp, 0.0.0.0:8088->8088/tcp, 0.0.0.0:8141->8141/tcp, 0.0.0.0:8188->8188/tcp, 0.0.0.0:8190->8190/tcp, 0.0.0.0:8443->8443/tcp, 0.0.0.0:8744->8744/tcp, 0.0.0.0:9000->9000/tcp, 0.0.0.0:9933->9933/tcp, 0.0.0.0:9995->9995/tcp, 0.0.0.0:9999-10000->9999-10000/tcp, 0.0.0.0:10015->10015/tcp, 0.0.0.0:10200->10200/tcp, 0.0.0.0:11000->11000/tcp, 0.0.0.0:11443->11443/tcp, 0.0.0.0:16000->16000/tcp, 0.0.0.0:16010->16010/tcp, 0.0.0.0:19888->19888/tcp, 0.0.0.0:45454->45454/tcp, 0.0.0.0:50090->50090/tcp compose_master0.dev_1
3583312fe42d hdp/ambari-server "/bin/sh -c /start.sh" About an hour ago Up About an hour 0.0.0.0:8080->8080/tcp compose_ambari-server.dev_1
a96797ec8ae6 hdp/postgres "/docker-entrypoint.s" About an hour ago Up About an hour 5432/tcp compose_postgres.dev_1
The steps I took to deploy were as follows:
Clone your repo
https://github.com/randerzander/docker-hdp.git
Build the container images
docker-compose -f examples/compose/multi-container.yml build
Create containers from the images
docker-compose -f examples/compose/multi-container.yml up
Log into Ambari on it's UI using username/password
as admin/admin
, give it master0.dev
and dn0.dev
as target hosts and use the manual registration option. Then stay with the default configuration that Ambari suggests (the only deviation being deselecting Atlas, since I didn't need it) and getting to the deploy page only for it to get stuck on the Preparing to deploy prompt.
Here is the build log for this run: buildlog.txt
It's also worth mentioning that there are a bunch of warnings on the host registration page, some of which are related to iptables being up and ntpd not running on the nodes, I'm not sure if that's related, but I changed nothing and kept going with the prompts, assuming defaults. Secondly, when we move past the Customize Services page, Ambari does prompt about Consistency Check Failed
as shown below:
Any direction or assistance you could provide would be greatly appreciated! Thanks!
INFO 2017-03-29 19:14:00,756 ActionQueue.py:118 - Adding EXECUTION_COMMAND for role check_host for service null of cluster null to the queue.
INFO 2017-03-29 19:14:00,770 ActionQueue.py:254 - Executing command with id = 1-0, taskId = 3 for role = check_host of cluster null.
INFO 2017-03-29 19:14:00,770 ActionQueue.py:295 - Command execution metadata - taskId = 3, retry enabled = False, max retry duration (sec) = 0, log_output = True
WARNING 2017-03-29 19:14:00,772 CommandStatusDict.py:128 - [Errno 2] No such file or directory: '/var/lib/ambari-agent/data/output-3.txt'
WARNING 2017-03-29 19:14:00,784 FileCache.py:168 - Error occurred during cache update. Error tolerate setting is set to true, so ignoring this error and continuing with current cache. Error detai
ls: ('Can not invalidate cache directory {0}: {1}', '/var/lib/ambari-agent/cache/custom_actions', "[Errno 39] Directory not empty: '/var/lib/ambari-agent/cache/custom_actions/scripts'")
INFO 2017-03-29 19:14:00,813 PythonExecutor.py:130 - Command ['/usr/bin/python',
I have put a sleep command in the script so that the HDP nodes wait for 2 minutes after ambari-server is up.
Upon checking the docker exec -it bash, I am able to find the file. But upon registering the nodes in ambari server, I am getting error
While trying to install Amabari server getting below error.
Step 6 : ADD scripts/start.sh /start.sh
scripts/start.sh: no such file or directory
Not sure if I am missing something.
If you manually install your stack and create your own blueprint. Sometimes the heartbeat comes by and the ambari-server gives off an error about SSL and NetUtil.java and unable to connec to ambari-server:8440.
dn0.dev_1 | WARNING 2017-07-24 16:46:42,591 NetUtil.py:101 - Failed to connect to https://ambari-server.dev:8440/ca due to [Errno 111] Connection
refused
dn0.dev_1 | WARNING 2017-07-24 16:46:42,591 NetUtil.py:124 - Server at https://ambari-server.dev:8440 is not reachable, sleeping for 10 seconds...dn0.dev_1 | INFO 2017-07-24 16:46:52,591 NetUtil.py:70 - Connecting to https://ambari-server.dev:8440/ca
dn0.dev_1 | WARNING 2017-07-24 16:46:52,593 NetUtil.py:101 - Failed to connect to https://ambari-server.dev:8440/ca due to [Errno 111] Connection
refused
dn0.dev_1 | WARNING 2017-07-24 16:46:52,593 NetUtil.py:124 - Server at https://ambari-server.dev:8440 is not reachable, sleeping for 10 seconds...
docker exec -i -t <containerID> /bin/bash
export JAVA_HOME=/usr/jdk64/jdk_1.8_version_number
That "version_number" should be whatever you find under /usr/jdk64. So please make sure to enter it exactly like jdk_1.8_0112 or whatever.
If you don't have that in that path, Then do:
yum search jdk
find the openjdk1.8 and do yum install openjdk1.8.......
sed -i "s/${ambariSchemaVersion}/2.4.2.0/g" /Ambari-DDL-Postgres-CREATE.sql
the above command getting executed as postgres but since it doesn't have access to permission to / folder, it is permission denied error
sed: couldn't open temporary file Permission denied
seeing this:
Caused by: org.postgresql.util.PSQLException: ERROR: null value in column "cluster_host_info" violates not-null constraint
ambari-server.dev_1 | Detail: Failing row contains (5, 2, null, 1494282593153, -1, 0, null, Install Components, INTERNAL_REQUEST, null, -1, null, null).
Trying to do the basic multi node and this is what I get. This happens when you install new services and try to deploy.
Error in first step :
docker-compose -f examples/compose/single-container.yml build
==>
Total download size: 571 M
Installed size: 637 M
Downloading Packages:
http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.1.0/storm/storm_2_6_1_0_129-1.1.0.2.6.1.0-129.x86_64.rpm: [Errno 12] Timeout on http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.1.0/storm/storm_2_6_1_0_129-1.1.0.2.6.1.0-129.x86_64.rpm: (28, 'Operation too slow. Less than 1 bytes/sec transfered the last 30 seconds')
Trying other mirror.
Error Downloading Packages:
storm_2_6_1_0_129-1.1.0.2.6.1.0-129.x86_64: failure: storm/storm_2_6_1_0_129-1.1.0.2.6.1.0-129.x86_64.rpm from HDP-2.6.1.0: [Errno 256] No more mirrors to try.
ERROR: Service 'dn0.dev' failed to build: The command '/bin/sh -c yum install -y storm_*' returned a non-zero code: 1
in 100-ambari.sh
shouldnt it be
instead of
Otherwise http://127.0.0.1:8080/api/v1/clusters/dev/upgrades is running in troubles.
Can this project be used to create a real distributed hadoop cluster?
Not sure but ambari is keep complaining that disk is used 99% not sure why
We are getting below error:
postgres.dev_1 | FATAL: role "ambari" does not exist
abari-server.dev_1 | No errors were found.
ambari-server.dev_1 | ERROR: Exiting with exit code 1
ambari-server.dev_1 | REASON: Database check failed to complete. Please check /var/log/ambari-server/ambari-server.log and /var/log/ambari-server/ambari-server-check-database.log for more information.
Also not able to open GUI http://localhost:8080
Thanks
Vinod
I created a single node ambari and when I try to deploy using the ambari UI, I'm able to add the nodes but when the preparing deploy starts it stucks on task 3 forever. No errors, nothing happens. Any ideas?
Hi,
I was able to bring this up in two node cluster using Swarm 1.13. Just wanted to share how it was done for all
First create a swarm by
docker swarm init --advertise-addr 10.1.0.4
And then use the command provided to join the other hosts
git clone https://github.com/sarath-mec/smks-docker-hdp.git
We used a combination of
a) Docker attachable Overlay Network for swarm
b) docker stack deploy for psql and ambari-server
c) docker-compose for running individual nodes
Used this approach as privileged for nodes is not supported for docker swarm services
But we can create an overlay attachable network and run the containers into the swarm network
All containers are discoverable by hostname by using this approach
#both manager and worker hosts to pull images
docker pull sarath4mec/node
docker pull sarath4mec/ambari-server
docker pull sarath4mec/postgres
#create volume directories in both nodes
sudo su
cd /home/sarath_mec
rm -rd hdfs
rm -rd postgresql
mkdir hdfs
mkdir postgresql
mkdir hdfs/dn0
mkdir hdfs/dn1
mkdir postgresql/data
chmod 777 hdfs
chmod 777 hdfs/dn0
chmod 777 hdfs/dn1
chmod 777 postgresql/data
cd smks-docker-hdp
#create network
docker network create -d overlay --subnet 10.0.9.0/24 --attachable hdfs
#stack deploy postgres and ambari in both manager and worker host
docker stack deploy --compose-file examples/compose/swarm-psql-ambari-container.yml hdfs
#inspect docker network from both nodes
docker network inspect hdfs
#see the network and virtual ip assigned from manager node
docker service inspect --format='{{json .Endpoint.VirtualIPs}}' hdfs_postgres
docker service inspect --format='{{json .Endpoint.VirtualIPs}}' hdfs_ambari-server
#stack deploy individually psql and ambari
docker stack deploy --compose-file examples/compose/swarm-psql-container.yml hdfs
docker stack deploy --compose-file examples/compose/swarm-ambari-container.yml hdfs
#docker-compose and bring up dn0.dev in swarm manager host
docker-compose -f examples/compose/swarm-dn0-container.yml up -d
#docker-compose and bring up dn0.dev in swarm worker worker host
docker-compose -f examples/compose/swarm-dn1-container.yml up -d
#docker-compose and bring up dn0.dev in swarm manager host
docker-compose -f examples/compose/swarm-dn0-container-resume.yml up -d
#docker-compose and bring up dn0.dev in swarm worker worker host
docker-compose -f examples/compose/swarm-dn1-container-resume.yml up -d
#Clear everything
docker stack rm hdfs
#manager host
docker stop compose_dn0_1
docker rm compose_dn0_1
#worker host
docker stop compose_dn1_1
docker rm compose_dn1_1
docker network rm hdfs
Thanks,
Sarath
Hbase master uses container DNS to find the hostname of regionserver.
And that does not return the value dn0.dev. So the master initialization fails. Is there any solution to this?
Getting below error when starting Hive Meta Store
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
sorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
NestedThrowablesStackTrace:
org.postgresql.util.PSQLException: This ResultSet is closed.
at org.postgresql.jdbc2.AbstractJdbc2ResultSet.checkClosed(AbstractJdbc2
ResultSet.java:2656)
at org.postgresql.jdbc2.AbstractJdbc2ResultSet.setFetchSize(AbstractJdbc
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.