jcustenborder / puppet-confluent Goto Github PK
View Code? Open in Web Editor NEWPuppet Module for installing and configuring the Confluent Platform
Home Page: http://www.confluent.io
License: Apache License 2.0
Puppet Module for installing and configuring the Confluent Platform
Home Page: http://www.confluent.io
License: Apache License 2.0
Because of automatic tagging, https://puppet.com/docs/puppet/5.3/lang_tags.html#automatic-tagging the tag "confluent" gets applied to all modules included in a profile and applies the order specified in init.pp to all resources. This breaks runs when using this module and for e.g forge sssd module in a role/profile setup. I've tested the changes in #67 in our environment and it works for us.
Example log output:
Failed to apply catalog: Found 2 dependency cycles: (Exec[authconfig-mkhomedir] => File[sssd.conf] => Service[oddjobd] => Class[Sssd::Dependencies] => Class[Sssd::Config] => File[sssd.conf])
Our role:
class role::confluent::kafka::standalone_control_center {
include profile::base
include profile::java
include profile::confluent::standalone::control_center
include profile::yum::confluent
}
Profile:
class profile::confluent::control::center {
include ::confluent::control::center
include ::profile::nginx
nginx::resource::server { 'confluent-control-center':
server_name => [ $facts['networking']['fqdn'] ],
listen_port => 80,
proxy => 'http://localhost:9021/',
}
}
If you're not able to reproduce and need/want a debug output I can provide you an anonymized one.
This is the sssd module we're using https://forge.puppet.com/sgnl05/sssd
I was helping someone new to puppet setup this module, and we found that several of the documentation examples have issues. I might be able to convince him to submit a PR, or if not, I will try to fit it in. Just in case someone else is trying to figure out why things are not working when they paste in the example code.
I will try to itemize the ones I see for now:
'1'
, which is a "String" ... This results in the following error:Error: Evaluation Error: Error while evaluating a Function Call, Class[Confluent::Kafka::Broker]: parameter 'broker_id' expects an Integer value, got String (file: /etc/puppetlabs/code/environments/vagrant/site/profile/manifests/confluent_kafka/broker.pp, line: 3, column: 3) on node kafka01.vagrant.local`)
It can be solved by removing the quotes around the 1, like:
confluent::zookeeper::zookeeper_id: 1
or
class{'confluent::zookeeper':
zookeeper_id => 1,
}
NOTE: I feel like there should be some "automated" way to determine this ID by default, but that is another story.
confluent::kafka::broker::zookeeper_connect:
- zookeeper01:2181
- zookeeper02:2181
- zookeeper03:2181
key: value
. Like:confluent::zookeeper::config:
server.1: zookeeper01:2888:3888
server.2: zookeeper02:2888:3888
server.3: zookeeper03:2888:3888
confluent::systemd::unit is hardcoding the path for systemd unit files to /usr/lib/systemd/system/
. This most likely is not correct on Debian and Ubuntu.
Hi,
first of all thanks for this module. I've been playing with it for some days now and it works great! Saved me a lot of work!
I have the Control Center up and running and I wanted to see the broker health metrics. I see, your default config contains
'confluent.support.metrics.enable' => true,
However, in order to actually see metrics, I needed to do two more things:
Firstly, extend the broker config by this:
"metric.reporters" => "io.confluent.metrics.reporter.ConfluentMetricsReporter",
and furthermore, add a
package { 'confluent-rebalancer':
ensure => latest,
tag => 'confluent',
}
Am I missing something, or is this the right way to do it? Would you want a patch to extend that in your module?
Cheers,
Constantin
Confluent Kafka 5.0 was released at the end of July, it would be great for this module to support that version.
https://www.confluent.io/blog/introducing-confluent-platform-5-0
If the server.1
, server.2
and server.3
Hiera properties are not provided (or mapped incorrectly) Zookeeper will start in standalone mode. The way this Puppet module is structured, three standalone Zookeepers is an erroneous state.
This is due to the fact that the ZK manifest will proceed with or without the server list when creating the conf file for a ZK instance. According to the way ZK interpret its config, the absence of a server list will implicitly assume standalone mode. This will result in a triple split-brain that leads to data corruption among the Brokers.
Zookeeper v3.5.0 adds a config standaloneEnabled
that will stop ZK from assuming standalone mode when no server list is specified. As of currently, this is not yet available in the Confluent Platform and this Puppet module. Until then, I suggest that the zookeeper.pp profile explicitly checks for the presence of the server list configuration and fail hard without it.
It would be helpful if you would use git tags to "tag" each release. That way, when we are looking at the code in GitHub, we can ensure we are seeing the same code that is deployed to the puppet server by R10k/code-manager.
Thanks in advance,
Tommy
In
puppet-confluent/manifests/params.pp
Line 174 in 9bc2da5
~tommy
This will ensure that the network is up and online rather than just up.
Last master branch broke puppet 3.x compatibility
Error 400 on SERVER: Syntax error at 'Integer'; expected ')'
Var type declaration only works with puppet 4.x
Windows isn't supported.
The config is currently configuring offset.storage.file.filename=/tmp/connect.offsets
. A directory needs to be created and the offsets placed there. This will allow it survive a reboot.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.