jrh3k5 / flume-agent-cookbook Goto Github PK

If the directory (e.g., /opt/flume/my_agent) does not exist but /opt/flume does, the mv command on Centos apparently does not create the missing directory.

To give the consumer more flexibility over how a Flume agent can be installed, it would be desirable to support an option to allow the user to override the default flume archive URL attribute in the resource such that if not specified, it would fallback to the default attribute.

Chef's service resource, at least on Chef 10.28.0, seems to ignore the return codes of the service status. Since we can't force Chef to ignore what it thinks is the operational status of the service, we should merely shell out the command to restart the agent.

Hailing from its days as a Maven project, the provider for the Flume agent references a non-existent ${cookbookName} variable. This should be removed.

To allow for the incorporation of things like health checks into the startup of a Flume agent, a consumer should be allowed to specify a script that gets executed following the startup of a Flume agent.

When unpacking Flume plugin tarballs, the cookbook assumes that the resulting directory will follow the convention of either:

• my-plugin-1.2.3
• my-plugin-1.2.3-SNAPSHOT

However, this does not account for when the directory is simply "my-plugin", and the cookbook will fail when trying to mv "my-plugin" to "my-plugin" because it expects to find "my-plugin-1.2.3".

As such, if the directory name matches the desired endstate, then don't error out (obviously).

Presently, the resource defined in the {{flume_agent}} cookbook does not tell Chef that it supports the "status" command:

# Configure the service
service attributes["serviceName"] do
  supports :start => true, :stop => true, :restart => true
  action [ :enable ]
end

Chef can only try to make guesses about the status of the service without this.

Presently, this is what the flume_agent cookbook does to install Flume:

     tar zxf #{Chef::Config[:file_cache_path]}/#{tarFilename}
     mv #{tarBasename}/* #{attributes["installDir"]}
     rm -rf #{tarBasename}

This is all well-and-good for a fresh install, but the mv command will not overwrite an existing directory with the contents of the tarball (say, for example, when upgrading from Flume 1.4.0 to 1.5.0).

To rectify this, we should make sure that directory is empty before attempting to mv the contents of the TAR to the Flume installation:

     tar zxf #{Chef::Config[:file_cache_path]}/#{tarFilename}
     rm #{attributes["installDir]}/*
     mv #{tarBasename}/* #{attributes["installDir"]}
     rm -rf #{tarBasename}

Flume agent properties files can contain sensitive information (e.g., database passwords) which we would not want to show up in the chef-client logs. These files should be marked as "sensitive".

The shell script used to check the status of the Flume agent builds an array of running processes matching the service and then performs this check:

if [ ${#pids[@]} -ne 1 ]; then
    echo "An instance of the Flume agent (see: $PID_FILE) that is not being tracked by this script was found."
    return 4
else

In the event that the PID file contained only IDs for processes that are not running, this array will be empty and the script will falsely report that there is a running instance of the agent that is not being managed by the script.

Right now, the cookbook uses Vagrant (and not kitchen) to run its tests. Lives would generally be made easier if the cookbook used kitchen.

We should add Vagrant tests

The service script for the installed Flume agent should return more meaningful status codes, such as those defined here.

We should only delete the Flume installation directory if the Flume installation version is actually changing.

Currently, the cookbook for the Flume agent is nested beneath a cookbooks/ folder. This is mostly to allow for the provisioning of a test cookbook, but there are ways that can be accomplished without complicated nesting like that.

As the Flume agent does not provide the ability to isolate plugins from the classpath of the Flume agent itself, there can arise dependency conflicts (such as Flume's usage of Guava 10.0.1). It would be helpful to have a hook into the agent provider itself to remove these conflicting libraries.

Occasionally, the Flume agent will, to Chef's eye, appear to be running, even if it is not.

We occasionally see "blips" of process IDs where the dump of the pgrep command to the PID file results in multiple PIDs in the file, though only one is still alive. The service script should tolerate these cases and only return an "unknown" state if no PIDs are found at all.

There have been some issues that I've seen of cookbooks (especially ones that declare resource providers) working on Chef 12.5.1. This cookbook should be validated to work on Chef 12.5.1.