Hi there,

Having a dig around the source code, it looks like it is only possible to set a universal forwarder as a deployment client. Is it possible to have the same deploy-poll command available for a heavy forwarder build as well? Is there any reason why you wouldn't do this?

Cheers

Per the code

# 'systemd-unit-file-name' option because cli's versions older than 7.2.2 do
# not implement systemd in boot-start command.
- name: "Copy splunk unit file - Linux (systemd)"
  template:
    src: Splunkd.service.j2
    dest: /etc/systemd/system/Splunkd.service
    owner: "{{ privileged_user }}"
    group: "{{ privileged_user }}"
    mode: 0644
  become: yes
  become_user: "{{ privileged_user }}"
  when: ansible_system is match("Linux") and pid1.stdout.find('systemd') != -1

For cli versions after 7.2.2 we should use the systemd-unit-file-name method.

#51 #52 #50 #22

Setting the Splunk secret is a key functionality we use.

While yes, it can break your Splunk when you change your splunk secret after shutting down and starting up your container or splunk instance. This is normally not something you would change on a regular basis.

The way it was implemented was, if it was empty or undefined, it should leave it to the default auto generation behaviour of Splunk. If it is defined, it should be set before splunk launch.

In order to restore the task, which change would you like to see?

hi ,
Need to disable disable forwarder agent from splunk web console

pleae suggest

On splunk_common role, it would be helpful to decide whether to enable Splunk as a Linux service for those non-Dockerized installations. A feature toggle flag would be required to control whether to enable this installation. Implement Linux platform for now.

Setting http_port in default.yml does nothing. The port is always 8000.

There is a critical issue with the last task in the splunk-ansible/roles/splunk_search_head/tasks/search_head_clustering.yml file. This performs the action to configure all search head cluster nodes as deployment client towards the deployer. The deployment server and deployer functionality get mixed, so you get into a situation where potentially 2 processes are managing the same target client. I have seen entire search head clusters going down due to similar situations, so this something to get rid of.

Seeing the following error output in the logs.

cat: /opt/ansible/inventory/messages.txt: No such file or directory

Similar use case to #206 where a splunk image is spun up with a standalone role that connects to a license master. The ansible task calls another task that performs a http request to the license master URL using the splunk install's own admin password. This fails in our environment because our license master container has a different admin password set.

Like the title says. I have a need to be able to enable splunktcp-ssl instead of regular splunktcp on indexer discovery inputs. Trying to override this after the fact breaks idempotency.

Look at the following...

I'd assume this would prevent upgrades from being applied. E.g. deploying splunk/splunk:7.2.1 and then splunk/splunk:7.2.3 wouldn't actually install the new version of splunk. Is that correct? Is that expected?

Upgrading our kubernetes setup from Splunk 7.2.4 -> 7.3.2 and I'm seeing issues with the SHC cluster. search_head_clustering.yml

For some reason the captain-0 thinks it part of the cluster and will not let other members join. I did try and do what it suggested by removing it as a member, but it would remove.

We have a captain-0 and four search heads search-0,search-1,search-2,search-3. For some reason the captain joins as a member well before the others. You see the output from ansible below.

TASK [splunk_search_head : Boostrap SHC captain] ***********************************************************************
task path: /opt/ansible/roles/splunk_search_head/tasks/search_head_clustering.yml:64
ok: [localhost] => {
    "attempts": 1,
    "changed": false,
    "cmd": [
        "/opt/splunk/bin/splunk",
        "bootstrap",
        "shcluster-captain",
        "-servers_list",
        "https://search-0:8089,https://search-1:8089,https://search-2:8089,https://search-3:8089,https://search-4:8089",
        "-auth",
        "admin:xxxxxxxxxxxxxxxx"
    ],
    "delta": "0:00:00.476436",
    "end": "2019-10-09 19:18:58.602434",
    "failed_when_result": false,
    "invocation": {
        "module_args": {
            "_raw_params": "/opt/splunk/bin/splunk bootstrap shcluster-captain -servers_list 'https://search-0:8089,https://search-1:8089,https://search-2:8089,https://search-3:8089,https://search-4:8089' -auth 'admin:xxxxxxxxxxxxxxxx'",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": true
        }
    },
    "rc": 22,
    "start": "2019-10-09 19:18:58.125998"
}

STDERR:

server=https://captain-0.captain-0.slo.svc.cluster.local:8089, error=This node seems to have already joined another cluster with below members: 'https://captain-0.captain-0.slo.svc.cluster.local:8089'. First remove the member from the old cluster. Then run 'splunk clean raft' on the member to reuse it in a new cluster;

MSG:

non-zero return code```

The indexer discovery pass4symmkey is not properly synchronized between the master instance and the SHC. I've had to force a shared splunk.secret to get forwarding to work from the SHC.

Hey, super small, boarder line pedantic, but... when I set HIDE_PASSWORD=false, I'm still getting censored, e.g.

TASK [Download post-setup playbooks] *******************************************
fatal: [localhost]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
...ignoring

Looking at the following, I'd assume I'd see the output...

I've spent the last couple of days hunting down docs on how to use the roles. While putting in this issue, I found another ticket which directed users to the docker-splunk project. For pete's sake, add that link to this project's docs so that other poor bastards like myself aren't wasting hours.

It would be really helpful if this used the retry count from default.yml. In a standard kubernetes deployed environment (and others I'm sure) there's no guarantee as to the order in which containers will come up. While typically, the license should be downloaded from the same container as default.yml (e.g. https://github.com/heroku/docker-splunk-nginx-defaults/) and with #19 it should therefore always be available at this time. Some might choose to have a separate container (or other http location) serve the license. Allowing for more reties to avoid error on a race condition would be helpful.

• We need a SAML configuration/ ACLs ansible play feature
• We need an index creation ansible play feature
• We need alert configuration management for pagerduty for indexers and search heads

This is critical to our enterprise deployment of Splunk, because we need these features to be deterministic through the use of docker-compose 2.0 environment variables.

Thank you

The sanity checks of ansible 2.7.0 require variables be assigned before referencing:

TASK [splunk_common : include_tasks] ********************************************************************************************************
fatal: [localhost]: FAILED! => {"reason": "Invalid options for include_tasks: license_master_host\n\nThe error appears to have been in '/opt/ansible/roles/splunk_common/tasks/add_splunk_license.yml': line 41, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Set as license slave\n ^ here\n"}
to retry, use: --limit @/opt/ansible/ansible-retry/site.retry

Is an example where '/opt/ansible/roles/splunk_common/tasks/add_splunk_license.yml': line 41 is referencing a group that has not been created when no environment variable has been set. Splunk-Ansible needs to be updated to create empty lists, and null vars for all variables it uses.

Until this issue is resolved, please use ansible 2.6.4.

Hey team,

When setting the SPLUNK_DEPLOYMENT_SERVER variable, the Ansible play uses the s2s_port instead of the svc_port

command: "{{splunk.exec}} set deploy-poll {{ splunk.deployment_server }}:{{ splunk.s2s_port }} -auth admin:{{ splunk.password }}"

should be:

command: "{{splunk.exec}} set deploy-poll {{ splunk.deployment_server }}:{{ splunk.svc_port }} -auth admin:{{ splunk.password }}"

https://docs.splunk.com/Documentation/Splunk/7.2.3/Updating/Configuredeploymentclients#Use_the_CLI

Also not sure if it is expected that the "add deployment server" task is duplicated here??

https://github.com/splunk/splunk-ansible/blob/develop/roles/splunk_universal_forwarder/tasks/main.yml#L7

There seems to be a race condition with the Apply the cluster bundle to the Splunk cluster master task. If the indexers haven't completed booting, this will fail. I think this need a retry.

The command fails with "unauthorized."

I'm working a distributed deployment, and in my case I was starting with the Splunk AMI for AWS. This AMI has splunk-ansible already installed which is great and saves some steps, but the behavior of using that AMI is to create the admin password for splunk based on the created instance id.

I modified the playbook to copy instance-specific default.yml files. Within each default.yml, I can define the unique parts such as password. I noticed when I did my deploy including an indexer cluster that it was failing during peering the search head to the cluster master. Specifically, in peer_cluster_master.yml, there is an include to wait for the cluster master instance. That task is using the in-scope splunk user and password, which in this case is the admin user and password for the search head. This obviously creates a 401: Unauthorized when the admin password on the CM is different than SH.

There are ways to solve this, but I was curious if this was an assumption about using this for a distributed setup.

It would be really awesome if https://github.com/splunk/splunk-ansible/blob/develop/roles/splunk_common/tasks/set_as_license_slave.yml#L7 has ignore_errors: yes. In local setups and lower environments we might not have a real license configured. Currently I'm using the post install hook added on #31 to bypass the standard slave set like so

    ---
    - name: CUSTOM Set node as license slave
      command: "{{ splunk.exec }} edit licenser-localslave -master_uri https://license-master:{{ splunk.svc_port }} -auth 'admin:{{ splunk.password }}'"
      register: lic_slave
      until: lic_slave.rc == 0
      retries: "{{ retry_num }}"
      delay: 3
      ignore_errors: yes
      notify:
        - Restart the splunkd service

Note that I'm hardcoding the license-master hostname instead of applying though the environment variable. By not setting SPLUNK_LICENSE_MASTER_URL, the task included in this repo isn't being run.

This allows for data loss where the input is enabled on a new indexer before indexes.conf has been deployed. When in indexer mode HEC tokens must be applied via master-apps

In order to allow more advanced inventories to be used by plays we would need to change the merging logic.
Such as 10 hosts and 3 groups. With variables defined on the groups which are the default for the hosts in those groups, with overrides by the hostvars themselves.

I see two different approaches to this.

1. In the Ansible default config set the hash_behaviour to merge instead of replace, however as per Ansible documentation this is not advised. https://docs.ansible.com/ansible/latest/reference_appendices/config.html?highlight=hash_behaviour#default-hash-behaviour
2. We could flatten out the Splunk variable. So, instead of using a dictionary full of items, this would all become independent variables. Example splunk.exec would change to splunk_exec.

Using rest causes splunk to index locally until forwarding is enabled this is an incorrect behavior

From what I see, the only option for installation on linux is the tgz version using these scripts. Was that a design decision, or just no one saw the need for .rpm or .deb? Would there be any interest in a pull request to allow the other formats?

Oy! When building from the develop branch (for both github.com/splunk/docker-splunk and github.com/splunk/splunk-ansible), I'm getting the following error on deployment.

TASK [splunk_common : Install Splunk] ******************************************
fatal: [localhost]: FAILED! => {"msg": "The conditional check 'result.rc == 0' failed. The error was: error while evaluating conditional (result.rc == 0): 'dict object' has no attribute 'rc'"}
        to retry, use: --limit @/tmp/ansible-retry/site.retry

I verified that I'm seeing this error across multiple roles.

The reason of disable https on splunkd managment port - SSL Termination is responsiility of Nginx Reverse Proxy, and I use http locally. Just I disable HTTPS splunk can't start (

1. Disable https on the splunkd management port (8089)
   in etc/system/local/server.conf

[sslConfig]
enableSplunkdSSL = false

2. build docker image

FROM splunk/splunk:latest
COPY ./splunk-config/etc/system/local/server.conf /opt/splunk/etc/system/local/server.conf

3. run the new docker image

it doesnt start

TASK [splunk_standalone : Check for required restarts] *************************
fatal: [localhost]: FAILED! => {"changed": false, "content": "", "msg": "Status code was -1 and not [200, 404]: Request failed: <urlopen error [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:661)>", "redirected": false, "status": -1, "url": "https://127.0.0.1:8089/services/messages/restart_required?output_mode=json"}
  to retry, use: --limit @/opt/container_artifact/ansible-retry/site.retry
medfitbi-staging_medfit-splunk.1.doba9vsufhyr@strikerops    |
PLAY RECAP *********************************************************************
localhost                  : ok=34   changed=12   unreachable=0    failed=1

looks like issue in hardcoded https schema in ansible task
roles/splunk_common/tasks/check_for_required_restarts.yml

For those systemv based Linux, start/stop tasks fail, since it currently only supports systemd based ones. We also need to support sytemv, for the AMI builds that make use of Amazon Linux (version 1).

ansible should become the splunk user when running any splunk command any files used by splunk should use the configured splunk user and group

Running any standalone provisioning breaks when splunk.secret is supplied.

#47

Knowing when it is in either of those situations would allows us to avoid unnecessary steps.
It considerably improves the re-startup speed of containers and machines.

Initially I thought to:
check if exec exists && build exists --> upgrade
check if exec doesn't exists && build exists --> install

However, I did not see the different between the upgrade and install path, so I had changed it to 'first run'. The latest change in #47 makes it impossible to follow the upgrade path on non-docker machines.

I traced the code and tried to figure out how to best use this repository for non-docker based environment, but I'm still confused.

The splunk-docker project seems to be using this project in the following way:

1. bake the ansible scripts (as well as ansible and python etc) all into docker image.
2. pass default.yml into the container
3. pass environment variables into container
4. container entrypoint call ansible with dynamic inventory
5. the dynamic inventory script merges baked defaults with default.xml and with environment variables
6. ansible runs local-connection on site.yml.
7. each docker container has the full information as how to provision itself along with information to talk to splunk on other docker containers.

Our environment is not docker-based. I would appreciate if splunk provides a wrapper or at least pointers how to use this repository for non-docker based environment.

Will the following idea work?

1. a wrapper ansible playbook.
   a. install prerequisite such as ansible python .. on all hosts.
   b. copy or download entire repository onto each host.
   c run ansible with site.yml and dynamic inventory script for each host (maybe using delegate_to? ) as a task:
   c1. pass ansible's env as env to each host in that task.
   d. remove remote ansible and python.

each task in each step c basically simulates a service in docker-compose.yml.

Are there any better ways to do this? The above seems to be very ugly.
Especially step c--- I don't know whether that's even possible or not.

Incorrect permissions on "Create defaults dir" site play from recent changes. Output below:

jackm-MBP-6D9FA:docker-splunk jackm$ docker run -e SPLUNK_START_ARGS=--accept-license -e SPLUNK_PASSWORD=Chang3d! splunk-debian-9:latest

PLAY [Run default Splunk provisioning] *****************************************
Friday 08 March 2019  00:04:20 +0000 (0:00:00.078)       0:00:00.078 ********** 

TASK [Gathering Facts] *********************************************************
ok: [localhost]
Friday 08 March 2019  00:04:21 +0000 (0:00:01.300)       0:00:01.378 ********** 
Friday 08 March 2019  00:04:21 +0000 (0:00:00.038)       0:00:01.416 ********** 

TASK [Create defaults dir] *****************************************************
fatal: [localhost]: FAILED! => {"changed": false, "gid": 999, "group": "splunk", "mode": "0755", "msg": "chown failed: [Errno 1] Operation not permitted: '/tmp/defaults'", "owner": "splunk", "path": "/tmp/defaults", "size": 4096, "state": "directory", "uid": 999}
	to retry, use: --limit @/opt/container_artifact/ansible-retry/site.retry

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1   

Friday 08 March 2019  00:04:22 +0000 (0:00:00.280)       0:00:01.696 ********** 
=============================================================================== 
Gathering Facts --------------------------------------------------------- 1.30s
Create defaults dir ----------------------------------------------------- 0.28s
Determine captaincy ----------------------------------------------------- 0.04s

Hi, Unable to find complete documentation for this code ? or i am missing something ?

I would like to request the ability to specify the splunk.secret as an environment variable so it does not need to be stored in defaults.yml in cases where instances need to share it.

Although the documentation is ambiguous and leaves this option open, the default port used for SHC + IDXC replication seems to be 9887. We currently have it set to 4001. Per feedback from field and PS, we should change this to 9887 so we're consistent with existing deployments.

https://docs.splunk.com/Documentation/Splunk/7.2.3/Indexer/UsetheCLI

Inputs on a clustered indexer should only be enabled via master-apps

I'm trying to setup a distributed environment. Currently we're running splunk 7.2.3 using the old style docker containers, but I'm trying to move to the the configuration.

I've started with our deployment server (since it's the least impacting server). Below is what I execute to create the deployment_server

docker run -it \
  -e SPLUNK_START_ARGS=--accept-license \
  -e SPLUNK_PASSWORD=helloworld \
  -e SPLUNK_ROLE=splunk_deployment_server \
  -e SPLUNK_APPS_URL=https://url/splunk-apps/app1.tgz,https://url/splunk-apps/app2.tgz..... \
  -p 8000:8000 \
  -p 8089:8089 \
  -v default.yml:/tmp/defaults/default.yml  splunk/splunk:7.2.5 start

The issue I come across with this (we actually haver about 50 apps we install, all custom stuff we wrote) is that it installs EVERY app on the deployment server, which TBH really confuses the server since they are intended to be installed on specific hosts defined by the server classes.

Looking at the ansible, it runs

- name: Copy installed apps to deployment-apps
  command: "cp -R {{ splunk.app_paths.default }}/{{ item }} {{ splunk.app_paths.deployment }}"
  register: copy_apps
  changed_when: copy_apps.rc == 0
  failed_when: copy_apps.rc != 0
  become: yes
  become_user: "{{ splunk.user }}"
  with_items: "{{ installed_apps }}"

Which makes sense, since that's what I'm seeing.. but I think in a deployment server role, it would make more sense to either apply the SPLUNK_APPS_URL directly to the deployment dir, OR move them to the deployment dir. That way you don't end up installing all your apps on the deployment server, and thus turning it into 100's of different roles.

default.yml missing option for root_endpoint, that could be needed if you deploying splunk in kubernetes behind ingress in subpath URL. For example it is not working with this ingress:

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: splunk
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - http:
      paths:
      - path: /splunk/
        backend:
          serviceName: master
          servicePort: splunkweb

Continuously redirecting from http://server/splunk/ to http://server/en-US/splunk/

It appears that when you have both a pre install and post install hook defined, the files defined are both written to the same location. See:
https://github.com/splunk/splunk-ansible/blob/master/execute_adhoc_plays.yml#L10

The exact issue I'm seeing is thus

1. Set pre hook to a file containing a set of plays.
2. Set post hook to a file containing a different set of plays.
3. Note that upon execution, the pre hook set of plays are executed both before and after and the post hook plays are never executed.

Note: This is a repost of splunk/docker-splunk#116, but the issue is the same for this repo.

On hub.docker.com, there's a 7.2.3 release that was cut about a month ago. However, there's no git tag to match. It would be really helpful to be able to compare what's on hub.docker.com, with the current repo state. E.g.

https://github.com/splunk/splunk-ansible/compare/<version_tag>..develop

Ability to define custom configurations via the default.yml is one of my favourite features of the install script, however, its use for defining indexes.conf is limited somewhat due to the fact that stanza order isn't preserved when yaml is imported.

Specifically, default.yml snippet written like so:

indexes:
      directory: /opt/splunk/etc/system/local/
      content:
        default:
          remotePath: "volume:remote_store/$_index_name"
          repFactor: auto
        "volume:remote_store":
          storageType: remote
          path: "s3://bucket-name"
          remote.s3.encryption: sse-s3
        alerts:
          homePath: $SPLUNK_DB/alerts/db
          thawedPath: $SPLUNK_DB/alerts/thaweddb
          coldPath: $SPLUNK_DB/alerts/colddb
        cm_events:
          homePath: $SPLUNK_DB/cm_events/db
          thawedPath: $SPLUNK_DB/cm_events/thaweddb
          coldPath: $SPLUNK_DB/cm_events/colddb
        cm_metrics:
          homePath: $SPLUNK_DB/cm_metrics/db
          thawedPath: $SPLUNK_DB/cm_metrics/thaweddb
          coldPath: $SPLUNK_DB/cm_metrics/colddb
          datatype: metric
        cm_meta:
          homePath: $SPLUNK_DB/cm_meta/db
          thawedPath: $SPLUNK_DB/cm_meta/thaweddb
          coldPath: $SPLUNK_DB/cm_meta/colddb

is rendered like so:

indexes:
      content:
        alerts:
          coldPath: $SPLUNK_DB/alerts/colddb
          homePath: $SPLUNK_DB/alerts/db
          thawedPath: $SPLUNK_DB/alerts/thaweddb
        cm_events:
          coldPath: $SPLUNK_DB/cm_events/colddb
          homePath: $SPLUNK_DB/cm_events/db
          thawedPath: $SPLUNK_DB/cm_events/thaweddb
        cm_meta:
          coldPath: $SPLUNK_DB/cm_meta/colddb
          homePath: $SPLUNK_DB/cm_meta/db
          thawedPath: $SPLUNK_DB/cm_meta/thaweddb
        cm_metrics:
          coldPath: $SPLUNK_DB/cm_metrics/colddb
          datatype: metric
          homePath: $SPLUNK_DB/cm_metrics/db
          thawedPath: $SPLUNK_DB/cm_metrics/thaweddb
        default:
          remotePath: volume:remote_store/$_index_name
          repFactor: auto
        volume:remote_store:
          path: s3://bucket_name
          remote.s3.encryption: sse-s3
          storageType: remote
      directory: /opt/splunk/etc/system/local/

I understand that yaml loader makes no guarantee around order, and you take it as you get it. Ordered dicts are going to be a feature of the language in 3.7 but that's a way away.

Nevertheless, this would be a really nice thing to be able to do. The documentation is pretty clear not all configuration can be done via default.yml so I know it's not a bug, but it would be a very helpful nice-to-have.

I've found what seems like a fairly straight-forward way to overcome this that doesn't appear to have side effects but this seemed like a rather rude PR to just request without even asking if it's ok to proceed.

Sorry guys, you did some work for me on the Deployment server, but I haven't really had any time tot test it. I had moved on from that and thought I'd try setting up a CM/Indexer cluster. We're deploying this all to Kubernetes, we usually have 2-3 indexers in a region controlled by a single CM.

The problem is simple: Currently when you define a CM, it REQUIRES the list of Indexers that will connect to it (SPLUNK_INDEXER_URL). In Kubernetes, that's actually impossible as you can't assign static IP's, plus kinda defeats the purpose of using CM Discovery (which allows you to spin up additional indexers as you wish).

Code like:

- name: Get indexer count
  set_fact:
    num_indexer_hosts: "{{ groups['splunk_indexer'] | length }}"

(And there's a test somewhere, can't find it atm, that does a quick test on each indexer to test that they are up, I think it's part of the cluster_bundle code)

This causes a massive race condition as you can't define the SPLUNK_INDEXER_URL when you don't know the count, ip's or hostnames of the indexers before they come online (And the names can change at any time).

Hey,

when I was trying to setup Splunk Heavyweight Forwarder by using the latest Docker image splunk/splunk:7.3.1. I found the ansible playbook failed to finish every time I set splunkbase_username and splunkbase_password.

Here are the errors:

ansible-playbook 2.8.5
  config file = /opt/ansible/ansible.cfg
  configured module search path = [u'/opt/ansible/library', u'/opt/ansible/apps/library', u'/opt/ansible/ansible_commands']
  ansible python module location = /usr/local/lib/python2.7/dist-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 2.7.16 (default, Apr  6 2019, 01:42:57) [GCC 8.3.0]
Traceback (most recent call last):
  File "inventory/environ.py", line 445, in <module>
    main()
  File "inventory/environ.py", line 431, in main
    getSplunkInventory(inventory)
  File "inventory/environ.py", line 85, in getSplunkInventory
    inventory["all"]["vars"] = getDefaultVars()
  File "inventory/environ.py", line 151, in getDefaultVars
    getSplunkApps(defaultVars)
  File "inventory/environ.py", line 168, in getSplunkApps
    response = urllib2_parse.urlopen("https://splunkbase.splunk.com/api/account:login/", "username={}&password={}".format(splunkbase_username, splunkbase_password))
AttributeError: 'Module_six_moves_urllib_parse' object has no attribute 'urlopen'
ERROR: Couldn't read "/opt/splunk/etc/splunk-launch.conf" -- maybe $SPLUNK_HOME or $SPLUNK_ETC is set wrong?                                                                                                                                                                    

It looks like the lib used for this urllib2_parse has been changed.

It would appear that the ansible_pre_tasks and ansible_post_tasks definitions are missing from the final version of #31 , the following lines should be included:

    vars_scope["ansible_pre_tasks"] = os.environ.get("SPLUNK_ANSIBLE_PRE_TASKS") or getValueFromDEFAULTS("ansible_pre_tasks")
    vars_scope["ansible_post_tasks"] = os.environ.get("SPLUNK_ANSIBLE_POST_TASKS") or getValueFromDEFAULTS("ansible_post_tasks")

Unless I'm missing something.

Feature Enhancement Request / Opportunity for improvement

The generate_server_classes.yml play uses a wildcard app selector. When you edit the "all" class via the UI and remove an app, it will appear to have succeeded in the UI. When splunk restarts, it picks the apps back up again and distributes them to all clients.

Example:

[serverClass:all]
whitelist.0 = *

[serverClass:all:app:*]
restartSplunkd = true

If we iterate over the apps in in /etc/deployment apps and add an explicit stanza for each app, we would maintain compatibility with the UI editor.

[serverClass:alll:app:my_foo_app]
[serverClass:alll:app:my_bar_app]
[serverClass:alll:app:my_baz_app]

I have my replication factor set to 2 and search factor to 2 however the fact is being set to 1 for both at runtime. I'm trying this on a 2 node indexer cluster and I have a branch (derived from the 7.3.1 tag) which totally removes the play which sets this to 1 if there are less than 3 indexers. The output from $ python /opt/ansible/inventory/environ.py shows the correct values of 2.

ansible verbose output

2019-08-28 19:48:30,919 p=1481 u=root | Wednesday 28 August 2019 19:48:30 -0500 (0:00:00.074) 0:00:12.404 ******
2019-08-28 19:48:30,953 p=1481 u=root | TASK [splunk_cluster_master : Get default replication factor] ******************
2019-08-28 19:48:30,953 p=1481 u=root | task path: /opt/ansible/roles/splunk_cluster_master/tasks/initialize_cluster_master.yml:6
2019-08-28 19:48:30,954 p=1481 u=root | ok: [localhost] => {
"ansible_facts": {
"idxc_replication_factor": "1",
"idxc_search_factor": "1"
},
"changed": false
}
.....
2019-08-28 19:48:32,340 p=1481 u=root | TASK [splunk_cluster_master : Set the current node as a Splunk indexer cluster master] ***
2019-08-28 19:48:32,340 p=1481 u=root | task path: /opt/ansible/roles/splunk_cluster_master/tasks/initialize_cluster_master.yml:31
2019-08-28 19:48:32,347 p=1481 u=root | changed: [localhost] => {
"attempts": 1,
"changed": true,
"cmd": [
"/opt/splunk/bin/splunk",
"edit",
"cluster-config",
"-mode",
"master",
"-replication_factor",
"1",
"-search_factor",
"1",
"-secret",
"",
"-cluster_label",
"",
"-auth",
"admin:"
],
"delta": "0:00:00.642018",
"end": "2019-08-28 19:48:32.287718",
"invocation": {
"module_args": {
"_raw_params": "/opt/splunk/bin/splunk edit cluster-config -mode master -replication_factor 1 -search_factor 1 -secret '' -cluster_label '' -auth 'admin:'",
"_uses_shell": false,
"argv": null,
"chdir": null,
"creates": null,
"executable": null,
"removes": null,
"stdin": null,
"stdin_add_newline": true,
"strip_empty_ends": true,
"warn": true
}
},
"rc": 0,
"start": "2019-08-28 19:48:31.645700"
}

idxc section in /tmp/defaults/default.yml

  idxc:
    enable: true
     label: REDACTED
     replication_factor: 2
     replication_port: 9887
     search_factor: 2
     secret: REDACTED

Resulting clustering stanza in server.conf

[clustering]
cluster_label = REDACTED
mode = master
pass4SymmKey = REDACTED
replication_factor = 1
search_factor = 1

Any ideas to enforce the values I've set in my default.yml are greatly appreciated!

9ab70c9#diff-6b53eddb7a76d7bf044b32a0ee84fc58R110
the try block is incorrectly indented resulting in failure to run inventory/environ.py within the develop branch.