idealista / kafka_role Goto Github PK

Prerequisites

• [ X] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

Not removable topics only consider the first one in the list

Expected behavior: Consider all the filters in the list

Actual behavior: Only filter the first one

Reproduces how often: Always

Versions

1.13.1

Prerequisites

• [x ] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

There is an error while checking the topics to alter

Versions

1.8.0

Additional Information

Seems to be the flatten filter used

Prerequisites

• [x ] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

The second time the topics configuration is launched, it's trying to alter the new topics

Steps to Reproduce

1. Run role with kafka_topics tag and a new topic
2. Run again with a new topic

Expected behavior: The second topic is created without problem

Actual behavior: Error while trying to alter tjhe second topic

Reproduces how often: always

Versions

1.7.0

Change default install path to avoid conflicts

Description

Enable agent usage

Prerequisites

• [X ] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

Rename the role in order to accomplish Ansible Galaxy standards

Steps to Reproduce

N/A

Expected behavior: [What you expect to happen]
N/A
Actual behavior: [What actually happens]
N/A
Reproduces how often: [What percentage of the time does it reproduce?]
N/A

Versions

All

Additional Information

N/A

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

It's not possible to modify some configuration variables like:

kafka.offsets.topic.replication.factor
transaction.state.log.replication.factor
transaction.state.log.min.isr

Steps to Reproduce

1. Run the role trying to modify some previous variables

Expected behavior: I expect these values from variables can be modified

Current behavior: I can't modify theses values automatically

Reproduces how often: 100%

Versions

<=1.3.1

Prerequisites

• [ X] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

Now not removable topics are those who begins with __. It would be great to provide also a list or a regex.

Prerequisites

• [x ] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

Use goss instead of testinfra for molecule testing

Versions

1.5.0

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

Add the ability of providing service, server and log4j templates via playbook.

Prerequisites

• [ X] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

Remove space before extra properties

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

• Kafka instances cannot access zookeeper because of landrush. I think we should use vagrant-hostmanager instead of landrush.

Steps to Reproduce

1. Execute molecule test using Vagrant
2. Access to a kafka instance using 'molecule login'
3. Try to ping Zookeeper or open /var/log/kafka/kafka-server.log and search for Unable to connect to zookeeper server 'zookeeper.vm:2181

Expected behavior: Zookeeper should be reached

Actual behavior: Zookeeper cannot be reached

Reproduces how often: 100%

Versions

All

Description

[Description of the issue]
The task to configure topics use deprecated --zookeeper parameter.
Needs to change to --bootstrap-server parameter instead.

Link/s to the specific code that needs a refactor (if applicable)

[Links]
https://github.com/idealista/kafka_role/blob/develop/tasks/topic_config.yml
https://github.com/idealista/kafka_role/blob/develop/tasks/topics.yml

I saw this repo that maybe it's useful. https://github.com/StephenSorriaux/ansible-kafka-admin

Description

There are some version unavailable under uvigo repositories. Is not possible to install Kafka v2.1.1 because is not present in that repo.

kafka_mirror variable cannot be overwritten because is declared in vars/main.yml file and has higher precedence than grup vars.

That versions could be found on Apache Historical Archive: https://archive.apache.org/dist/kafka/

Why is this needed?

Because actually is not possible to install versions < 2.2.0 using uvigo mirror

Additional Information

http://apache.uvigo.es/kafka/

Prerequisites

• [ X ] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

It would be nice to use idealista/zookeeper in unit tests

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

The role fails using the default kafka version, that is no longer available.

Steps to Reproduce

1. molecule test
   Expected behavior: [What you expect to happen]
   molecule finished ok
   Actual behavior: [What actually happens]
   fatal: [kafka2]: FAILED! => {"changed": false, "msg": "Failure downloading http://apache.uvigo.es/kafka/2.0.0/kafka_2.12-2.0.0.tgz, HTTP Error 404: Not Found"}
   Reproduces how often: [What percentage of the time does it reproduce?]
   Always.

Versions

All.

Additional Information

N/A

Prerequisites

• [ x] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

Enable extra properties in server.properties file

Prerequisites

• [ X] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

It'ld be nice to provide the KAFKA_JVM_PERFORMANCE_OPTS to override the default values

Prerequisites

• [x ] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

Let topìcs be configured in hte role

Versions

1.5.0

Prerequisites

• [* ] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

The scala version of the kafka package is in vars folder and cannot be override. It must be moved to default folder.

Prerequisites

• [x ] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

In the topics configuration step, it is trying to remove __consumer_offsets kafka internal topic

Steps to Reproduce

1. Run the role with kafka_topics tag

Expected behavior: Not try to delete internal topic

Actual behavior: Tries to remove __consumer_offsets

Reproduces how often: Always

Versions

1.6.0

Prerequisites

• [x ] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

Remove old kafka installations to avoid library conflicts

Versions

1.5.0

delete.topic.enable property can't be setted

Prerequisites

• [x ] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

Enable topic configuration parameters.
https://docs.confluent.io/current/installation/configuration/topic-configs.html

The role cannot be deployed to Ansible Galaxy because of missing username in role dependendies

Prerequisites

• [x ] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

Unexpected result while removing topics in python 3

Versions

1.8.0

Prerequisites

• [x ] Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

Enable/disable GC logging using configuration parameters

Versions

1.5.0

auto.create.topics.enable property can't be setted :)

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

In the topics configuration step, it is trying to remove __consumer_offsets kafka internal topic

Steps to Reproduce

1. Run the role with kafka_topics tag

Expected behavior: Not try to delete internal topic

Actual behavior: Tries to remove __transaction_state

**Reproduces how often:**Always

Versions

1.7.0

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:
  • Checked that your issue isn't already filled: https://github.com/issues?utf8=✓&q=is%3Aissue+user%3Aidealista
  • Checked that there is not already provided the described functionality

Description

Testinfra tests don't pass under Vagrant but in Docker:

___________________ test_kafka_service[ansible://kafka3.vm] ____________________

File = <class 'testinfra.modules.base.GNUFile'>
Service = <class 'testinfra.modules.base.SystemdService'>
Socket = <class 'testinfra.modules.base.LinuxSocketSS'>
AnsibleVars = {u'java_implementation': u'openjdk', u'kafka_host_name': u'127.0.0.1', u'kafka_hosts': [{u'host': u'kafka1.vm', u'id': 1}, {u'host': u'kafka2.vm', u'id': 2}, {u'host': u'kafka3.vm', u'id': 3}], u'kafka_port': 9999, ...}
Hostname = 'kafka3.vm'

    def test_kafka_service(File, Service, Socket, AnsibleVars, Hostname):
        if "kafka" in Hostname:
            host = AnsibleVars["kafka_host_name"]
            port = AnsibleVars["kafka_port"]
            assert File("/lib/systemd/system/kafka.service").exists
            assert Service("kafka").is_enabled
            assert Service("kafka").is_running
>           assert Socket("tcp://" + host + ":" + str(port)).is_listening
E           AssertionError: assert False
E            +  where False = <socket tcp://127.0.0.1:9999>.is_listening
E            +    where <socket tcp://127.0.0.1:9999> = <class 'testinfra.modules.base.LinuxSocketSS'>(((('tcp://' + u'127.0.0.1') + ':') + '9999'))
E            +      where '9999' = str(9999)

tests/test_ansible.py:56: AssertionError 

Seems to be a problem with the network interfaces because Kafka instances cannot connect to Zookeeper:

[2018-02-27 14:35:07,551] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to zookeeper server 'zookeeper.vm:2181' with timeout of 6000 ms
	at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1233)
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:157)
	at org.I0Itec.zkclient.ZkClient.<init>(ZkClient.java:131)
	at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:106)
	at kafka.utils.ZkUtils$.apply(ZkUtils.scala:88)
	at kafka.server.KafkaServer.initZk(KafkaServer.scala:326)
	at kafka.server.KafkaServer.startup(KafkaServer.scala:187)
	at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:39)
	at kafka.Kafka$.main(Kafka.scala:67)
	at kafka.Kafka.main(Kafka.scala)

But the zookeeper service is up & running (listening in port 2181):

vagrant@zookeeper:~$ netstat -tulpan
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp        0    176 10.0.2.15:22            10.0.2.2:49603          ESTABLISHED -                   
tcp6       0      0 :::22                   :::*                    LISTEN      -                   
tcp6       0      0 :::39223                :::*                    LISTEN      -                   
tcp6       0      0 :::2181                 :::*                    LISTEN      -                   
udp        0      0 0.0.0.0:68              0.0.0.0:*                           -                   
udp        0      0 0.0.0.0:68              0.0.0.0:*                           - 

Steps to Reproduce

Could be reproduced executing tests using molecule test

Expected behavior: Tests passing

Actual behavior: Tests don't pass

Reproduces how often: 100%

Versions

1.3.x

Additional Information

Ansible version: 2.4.2.0
Vagrant version: 1.8.6
Molecule version: 1.25.0
Testinfra version: 1.11.0

Kafka went 1.0 (https://www.confluent.io/blog/apache-kafka-goes-1-0/). We should set it as the role default version.

Description

Default mirror http://apache.uvigo.es/kafka only has 2.6.1 and 2.7.0 versions (default is 2.6.0):

Tests (and actual installation) are failing to download kafka.

Steps to Reproduce

1. Create pipenv environment ( pipenv sync molecule test )
2. Log into environment (pipenv shell)
3. Run the test showing log (MOLECULE_NO_LOG="false" molecule converge)
4. Fail with error:
   Failure downloading http://apache.uvigo.es/kafka/2.6.0/kafka_2.12-2.6.0.tgz, HTTP Error 404: Not Found

Expected behavior:
Install kafka

Actual behavior:
Failure to download the installation file

Reproduces how often:
Everytime

Environment

• The release version/s you are using:
• OS: Ubuntu
• Others:
  Default version of kafka (2.6.0) or previous (e.g. 2.1.1)

Additional Information

You could find previous versions to 2.6.1 in https://archive.apache.org/dist/kafka

The broker id is setted by default to 0.
We must provide an easy way to set the id for multiple brokers.