akuehlka / dockerized-hadoop Goto Github PK

These are docker images of Apache Hadoop HDFS service.

The base image provides a custom entrypoint gets an argument for the type of hdfs server to run:

• namenode
• datanode

The base image provides a custom entrypoint that uses environment variables to set hadoop configuration file properties.

Environment variables must be in the form <PREFIX>_<HADOOP_PROPERTY>.

With PREFIX one of the following:

- CORE_CONF: /etc/hadoop/core-site.xml
- HDFS_CONF: /etc/hadoop/hdfs-site.xml
- YARN_CONF: /etc/hadoop/yarn-site.xml
- HTTPFS_CONF: /etc/hadoop/httpfs-site.xml
- KMS_CONF: /etc/hadoop/KMS-site.xml

And the HADOOP_PROPERTY should be provided with the following replacements:

. => _
_ => __
- => ___

For example:

fs.defaultFS property of core-site.xml file should be provided as the environment variable:

CORE_CONF_fs_defaultFS

dfs.replication of hdfs-site.xml file should be provided as:

HDFS_CONF_dfs_replication

The image provides de following default configuration as environment variables:

The environment variable MULTIHOMED_NETWORK configures [multihomed networks].

Image also accepts configuration through simple environment variable that translates into specific hadoop configuration variables.

• HDFS_NAMENODE_URL in the form of 'hdfs://NAMENODE_HOST:NAMENODE_PORT'

We provide two docker-compose files to deploy a 3-datanode and a single-datanode hdfs clusters.

docker-compose up -d

Then, Hdfs UI is available at:

• http://localhost:50070 for hadoop 2.x
• http://localhost:9870 for hadoop 3.x

To undeploy:

docker-compose -f tests/docker-compose.yml  down

We have testing scripts for single and multiple datanode deployments, just provide the name of the docker image to test as environment variable IMAGE_NAME. for example:

cd scripts/tests
IMAGE_NAME=gradiant/hdfs:2.7.7 ./test-hdfs-single-datanode.sh
IMAGE_NAME=gradiant/hdfs:2.7.7 ./test-hdfs-multiple-datanodes.sh

They deploy a hdfs cluster, create a folder in hdfs, copy a file from local to the folder, and remove file and folder. Finally they stop and remove the cluster containers.