ansible-role-hadoop's Introduction

Hadoop Cluster Role

Installs and configure Hadoop system (version 2.X) in a cluster of nodes.

Role Variables

The variables that can be passed to this role and a brief description about them are as follows:

# The type of the node: slave or master or resourcemanager or nodemanager or datanode or namenode 
hadoop_type_of_node: slave
# Hadoop base directory to install the software
hadoop_home: /opt/hadoop-2.X
# List of servers to download the hadoop code
hadoop_mirrors: [ "http://mirror.cc.columbia.edu/pub/software/apache/hadoop/core/stable/",
		"http://ftp.osuosl.org/pub/apache/hadoop/core/stable/",
		"http://apache.rediris.es/hadoop/core/stable/",
		"http://apache.uvigo.es/hadoop/common/stable/" ]
# Hadoop version to install
hadoop_version: latest
# A dictionary with a set of properties to set in the core-site.xml
hdfs_props: {}
# A dictionary with a set of properties to set in the yarn-site.xml
yarn_props: {}
# List of slave nodes
hadoop_slave_nodes: "{{ groups['all']|map('extract', hostvars,'ansible_hostname')|list }}"

Example Playbook

This an example of how to install the Cluster:

In the "Worker Nodes"

  roles:
    - { role: 'grycap.hadoop', hadoop_master: 'MASTER_NODE_NAME_OR_IP' }

In the "Manager Node"

  roles:
    - { role: 'grycap.hadoop', hadoop_master: 'MASTER_NODE_NAME_OR_IP', hadoop_type_of_node: 'master'}

Contributing to the role

In order to keep the code clean, pushing changes to the master branch has been disabled. If you want to contribute, you have to create a branch, upload your changes and then create a pull request.
Thanks

ansible-role-hadoop's People

Contributors

Stargazers

Watchers

ansible-role-hadoop's Issues

Problem with "Set JAVA_HOME in hadoop-env.sh"

Hi,
There is a problem in the task "Set JAVA_HOME in hadoop-env.sh" on Ubuntu trusty.
The console log:

` TASK [grycap.hadoop : Set JAVA_HOME in hadoop-env.sh] **************************************************************************************************************************
fatal: [10.6.14.40]: FAILED! => {"msg": "No file was found when using with_first_found. Use the 'skip: true' option to allow this task to be skipped if no files are found"}
to retry, use: --limit @/Users/Zsomx/hadoop-installer---ansible/hadoop_namenode.retry

PLAY RECAP *********************************************************************************************************************************************************************
10.6.14.40 : ok=9 changed=7 unreachable=0 failed=1 `

My playbook is a simple one:
`- hosts: hadoop_namenode
become: yes
vars:
namenode_ip: 10.6.14.40
version: '2.9.0'
roles:

{ role: 'grycap.hadoop', hadoop_master: '{{namenode_ip}}', hadoop_type_of_node: 'master', hadoop_version: '{{version}}' } `

Any suggestions?

Thanks!

Problem in the format step

The step

command: "{{ hadoop_home }}/bin/hdfs namenode -format creates=/tmp/hadoop-root/dfs/name"
when: (hadoop_type_of_node == 'master') or (hadoop_type_of_node == 'namenode')

Uses as test the "/tmp/hadoop-root/dfs/name" directory, instead of the one provided in the hdfs_props collection (dfs.datanode.data.dir)

Example Playbook for 1 node master and 1 node slave

Hello ¿ could you write a simple example to create a playbook for 1 master and 1 node (sharing datanode, resourcenode, etc), in order to check if your ansible role works? Without ansible-galaxy?

- hosts: hadoop_namenode
   become: yes
   vars:
       namenode_ip: 192.168.10.10
       datanode_ip: 192.168.10.10
       version: '2.8.0'
   roles:
      { role: 'grycap.hadoop', hadoop_master: '{{namenode_ip}}', hadoop_type_of_node: 'master', hadoop_version: '{{version}}' }
      { role: 'grycap.hadoop', hadoop_master: '{{datanode_ip}}', hadoop_version: '{{version}}' }

Regards.

Recommend Projects

grycap / ansible-role-hadoop Goto Github PK

ansible-role-hadoop's Introduction

Hadoop Cluster Role

Role Variables

Example Playbook

Contributing to the role

ansible-role-hadoop's People

Contributors

Stargazers

Watchers

Forkers

ansible-role-hadoop's Issues

Problem with "Set JAVA_HOME in hadoop-env.sh"

Problem in the format step

Example Playbook for 1 node master and 1 node slave

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent