jomenxiao / kafka-ansible Goto Github PK

This repo deploy Zookeeper/Kafka with Ansible

Default Version

Above information in repo file roles/download/templates/kafka_packages.yml.j2

1. First of all
2. Modify inventory.ini file
3. Prepare
4. Deploy zookeeper/kafka
5. Start zookeeper/kafka
6. Stop kafka/zookeeper
7. Manual start/stop zookeeper/kafka
8. Test
9. Expansion zookeeper/kafka

• Install ansible
  • deb apt-get install ansible -y
  • rpm yum install ansible -y
• Clone repo
  • git clone https://github.com/jomenxiao/kafka-ansible.git
• Change directoy
  • cd kafka-ansible

• Zookeeper informations

  • myid is unique integer,range 1-255; documents:zookeeper office website
  • deploy_dir deployment directory
  • one line mean one process
  • example: zk_1 ansible_host=172.17.8.201 deploy_dir=/home/tidb/zk_deploy myid=1

• Kafka informations

  • kafka_port for client connect
  • id is the kafka's broker id. It must be set to a unique integer for each broker.
  • one line mean one process
  • example: kafka1_1 ansible_host=172.17.8.201 deploy_dir=/home/tidb/kafka_deploy1 kafka_port=9091 id=1

• Localhost create some deployment directory
• Localhost create ansible's facts
• Localhost download zookeeper/kafka file
• Remote host create deployment directory
• Remote host modify kernel params
• Remote host install necessary packages
  • setting in repo directory roles/packages/packagesfiles
  • example: java package

ansible-playbook -i inventory.ini prepare.yml

• Copy zookeeper/kafka deployment packages to remote host
• Modify zookeeper/kafka's configure file
• Generate zookeeper/kafka start/stop scripts

ansible-playbook -i inventory.ini deploy.yml

• First start zookeeper
• Start kafka

ansible-playbook -i inventory.ini start.yml

• Firest stop kafka
• Stop zookeeper

ansible-playbook -i inventory.ini stop.yml

• Zookeeper
  • cd $deploy_dir/scripts && ./run_zookeeper.sh start|status|stop"
• Kafka
  • cd $deploy_dir/scripts && ./run_kafka.sh start|stop"

• Ansible tools directory
  • Start consumer tools/kafka-console-consumer -brokers="172.17.8.201:9091,172.17.8.201:9092,172.17.8.202:9091,172.17.8.202:9092,172.17.8.203:9091,172.17.8.203:9092" -topic=test
  • Start producer tools/kafka-console-producer -brokers="172.17.8.201:9091,172.17.8.201:9092,172.17.8.202:9091,172.17.8.202:9092,172.17.8.203:9091,172.17.8.203:9092" -topic=test -value=world -key=hello

• Add host/process informations to inventory.ini file
• ansible-playbook -i inventory.ini prepare.yml --diff
• ansible-playbook -i inventory.ini deploy.yml --diff
• ansible-playbook -i inventory.ini start.yml --diff

zk_deploy/
├── backup
├── data
│   ├── myid
│   ├── version-2
│   └── zookeeper_server.pid
├── datalog
│   └── version-2
├── log
│   └── zookeeper.out
├── package
│   └── zookeeper-3.4.11
├── scripts
│   └── run_zookeeper.sh
└── zk -> /home/tidb/zk_deploy/package/zookeeper-3.4.11

kafka_deploy1/
├── backup
├── datalog
│   ├── cleaner-offset-checkpoint
│   ├── __consumer_offsets-2
│   ├── log-start-offset-checkpoint
│   ├── meta.properties
│   ├── recovery-point-offset-checkpoint
│   ├── replication-offset-checkpoint
│   └── test-1
├── kafka -> /home/tidb/kafka_deploy1/package/kafka_2.12-1.0.0
├── log
│   ├── controller.log
│   ├── kafka-authorizer.log
│   ├── kafka-request.log
│   ├── kafkaServer-gc.log.0.current
│   ├── kafkaServer.out
│   ├── log-cleaner.log
│   ├── server.log
│   └── state-change.log
├── package
│   └── kafka_2.12-1.0.0
└── scripts
    └── run_kafka.sh

• Restart kafka need 6 seconds interval