stackstorm / chef-stackstorm Goto Github PK

View Code? Open in Web Editor NEW

16.0 24.0 10.0 235 KB

DEPRECATED! Community-maintained Chef Cookbook to deploy StackStorm, IFTTT for Ops

Home Page: https://supermarket.chef.io/cookbooks/stackstorm

License: Apache License 2.0

Ruby 86.38% HTML 13.62%

chef-cookbook stackstorm chef cookbooks configuration-management devops-tools automation event-driven-automation

chef-stackstorm's People

Contributors

Stargazers

Watchers

Forkers

swettk ldesiqueira vanwho cibingeorge sysbot winmillwill jvrplmlmn sanderv32 isabella232 arm4b

chef-stackstorm's Issues

Add CentOS 6 support

StackStorm works on CentOS 6, there are instructions: https://docs.stackstorm.com/install/rhel6.html

It's probably low priority to verify if cookbook requires any changes to work with CentOS 6 + kitchen.yml.

Add support for full st2.conf

Current st2.conf settings, codified in this recipe is just minimal set of recommendations.

Switch to https://github.com/StackStorm/st2/blob/master/conf/st2.conf.sample which includes more advanced configuration changes, that might be needed in a real prod StackStorm use, allowing users to adjust more settings.

Multiplatform chefspec testing

Partially split from #44

Since the cookbook behaves slightly differently on RHEL vs Debian, both sides should be tested in chefspec

Example found in mysql chef cookbook:

Use own StackStorm "init" Docker images in Test-Kitchen

Those images already have everything pre-installed to run properly default init system for specific OS.

See:
https://github.com/StackStorm/ansible-st2/blob/master/.kitchen.yml#L25-L39

Code Coverage

Code Coverage is essential quality metric. Research what's possible to do with ChefSpec to show the coverage % badge.

https://sethvargo.com/chef-recipe-code-coverage/
chef-stackstorm/spec/spec_helper.rb

Line 5 in 41270c7

at_exit { ChefSpec::Coverage.report! }
https://github.com/sethvargo/chefspec#reporting
https://coveralls.zendesk.com/hc/en-us/articles/201769485-Ruby-Rails
and
https://github.com/poise/poise-spec/blob/master/spec/spec_helper.rb
https://github.com/mburns/paramount/blob/master/test/unit/spec/spec_helper.rb

Include st2chatops install

Related to: Include st2web

Install st2chatops package, configure settings (Slack, IRC, Hipchat etc).
Allow multi-node setup (external st2chatops talking to remote st2api, st2auth, st2stream).

st2.conf is out of sync

Current st2.conf template, shipped with this cookbook is outdated.

Synchronize https://github.com/StackStorm/chef-stackstorm/blob/master/templates/default/st2.conf.erb and https://github.com/StackStorm/chef-stackstorm/blob/master/spec/recipes/config_spec.rb#L12-L79
with the original st2.conf.

CI is broken on CentOS7

Restarted the Travis build for master and CentOS7 looks like started to fail: https://travis-ci.org/StackStorm/chef-stackstorm/jobs/182408367

Probably kind of upstream changes in Mongo cookbook dependency or in Docker image.
Something to research.

Generate ~/.st2/config for CLI auth

Low priority "nice to have" feature

Generate CLI Configuration File ~/.st2/config which is used as credentials source to authenticate current OS user, allowing to run st2 commands without explicitly specifying username/password/token/key.

Bash example: https://github.com/StackStorm/st2-packages/blob/0e599a61f113bd9b77ad525738fcc33fd0748fcb/scripts/st2bootstrap-deb.sh#L230-L233

Optional setting, since not everyone needs to keep username/password pair unencrypted.

Lint Checks vs Integration Tests in 2 CIs

As the codebase and number of different Checks & Tests grows, add another CI to unload Travis from doing Lint Checks & Unit tests (build time in Travis is already 10mins+).

Some different CI (CircleCI, etc)

Cookstyle lint
Foodcritic lint
ChefSpec Unit Tests
Sequenced execution

Travis:

Test-Kitchen
Serverspec, see #29
Parallel tests across different platforms (Dockerized as is)

Not a priority though

Fix compatibility with Chef 13

CI started failing 2 days ago because Chef 12 was replaced by Chef 13 in latest ChefDK.

https://travis-ci.org/StackStorm/chef-stackstorm

Cleanup TravisCI output

Travis chefspec tests output extra lines of python and platform
This is caused by the htpasswd cookbook using the deprecated python cookbook
I will take a look at switching it

Excerpt of travis output

    should include recipe apt
python
platform
python
    should create packagecloud_repo "StackStorm/stable"
python
platform
python

Full travis output with debug rspec logging turned on: https://api.travis-ci.org/jobs/182493586/log.txt?deansi=true

Switch to MongoDB 3.2

Since StackStorm v1.6 MongoDB 3.2 is the default version,
see: https://docs.stackstorm.com/install/deb.html#install-dependencies for installation instructions.

Reflect the change in st2 Configuration Management repos (same is for StackStorm/ansible-st2#49).

LWRP for st2 pack install

Provide LWRP for managing StackStorm packs.

Example:

st2_pack 'github' do
  action :install
  ...
end

Related examples:

Hints:

poise-python branch no longer exists!

This cookbook's Berksfile references a branch of chef-openstack-mistral that doesn't exist.

cookbook 'openstack-mistral', git: 'git://github.com/sysbot/chef-openstack-mistral.git', branch: 'poise-python'

What gives?

CentOS 8 / RHEL 8 Support

Support for EL8. Needs someone to tackle.

Start here: StackStorm/ansible-st2#248

Integration Tests (InSpec)

Partially adresses #26

Serverspec/Inspec Integration tests would be nice.

There are already some examples to re-use from the:
https://github.com/StackStorm/st2-packages/blob/master/rake/spec/default/60-st2_all-services-ok_spec.rb

Release 0.5.0

Changelog changes
Obtain Chef Supermarket credentials
Get familiar with the process, release 0.5.0 [ 1 ]

https://github.com/StackStorm/chef-stackstorm/milestone/1

EL SELinux check

Try to adjust SELinux policies automatically (?) OR warn user & stop the execution, according to doc instructions:
https://docs.stackstorm.com/install/rhel7.html#adjust-selinux-policies

Remove `--always-copy` >=EL7

The below two files needs the
virtualenv_opts = --always-copy line either commented out or the value removed.

This needs validation. If StackStorm/st2#4891 is merged, we can assume it's a go.

chef-stackstorm/spec/recipes/config_spec.rb

Lines 30 to 32 in 2aa543b

 [actionrunner] 

 logging = /etc/st2/logging.actionrunner.conf 

 virtualenv_opts = --always-copy

https://github.com/StackStorm/chef-stackstorm/blob/2aa543bbb19df1c176299aea22828d7a6e6e41a6/spec/recipes/config_spec.rb#L30:L32

Drop Ubuntu 14 support since st2 v3.1

Per https://docs.stackstorm.com/roadmap.html#roadmap, StackStorm v3.1 deprecates Ubuntu14 support as it reached end of life.

Remove Ubuntu14 support from the cookbook.

Add Ubuntu 16.04 support

Ubuntu 16.04 packages are currently in unstable quality and available under the staging repos.

TODO: Include & Verify Ubuntu Xenial support within cookbook when StackStorm ships the corresponding packages. The change is probably just about 1 line in kitchen.yml

P.S. It's just a reminder, not to forget about it when we release U16 packages

Release 0.3.0

@dennybaa @lakshmi-kannan can one of you guys release what is currently in master? and/or add @sysbot and i to supermarket?

Missing st2 auth credentials generation in /etc/st2/htpasswd

Current Cookbook is missing st2 username:password configuration in /etc/st2/htpasswd, eg. there is no logic to generate contents.
It means you can't run st2 with auth enabled (default) with this Cookbook.

According to Installation Docs this could be done via htpasswd utility, but I think there must be something in Chef/Ruby libs toolset to generate contents for /etc/st2/htpasswd, to avoid installing additional apache2-utils OS package.

Example in puppet-st2
https://github.com/StackStorm/puppet-st2/blob/master/manifests/auth_user.pp

This might work: http://stackoverflow.com/questions/29837267/write-users-to-htpasswd-in-chef-recipe

Cookbook Cleanup

There are small pieces to cleanup from the old cookbook story,
like: https://github.com/StackStorm/chef-stackstorm/blob/master/recipes/_initial.rb#L3-L22 and more
(user creation, directories creation, etc).

The idea is to remove all unnecessary logic, which is already included in .deb/.rpm packaging.

Cleanup cookbook
Cleanup README

st2 services won't start due to wrong configuration path

Using the latest version of this cookbook, on CentOS 7.2, with StackStorm 2.0.0, results in most st2* services not starting up. The /etc/st2/st2.conf file is missing several options.

Most references to the logging configuration files are wrong, since they point to /etc/st2api/... and at least with 2.0.0 the configuration files are in /etc/st2/...:

[st2@localhost ~]$ DAEMON_ARGS="-k eventlet -b 127.0.0.1:9101 --workers 1 --threads 1 --graceful-timeout 10 --timeout 30"
[st2@localhost ~]$ /opt/stackstorm/st2/bin/gunicorn_pecan /opt/stackstorm/st2/lib/python2.7/site-packages/st2api/gunicorn_config.py $DAEMON_ARGS
[2016-09-06 08:31:58 +0000] [24124] [INFO] Starting gunicorn 19.6.0
[2016-09-06 08:31:58 +0000] [24124] [INFO] Listening at: http://127.0.0.1:9101 (24124)
[2016-09-06 08:31:58 +0000] [24124] [INFO] Using worker: eventlet
[2016-09-06 08:31:58 +0000] [24172] [INFO] Booting worker with pid: 24172
2016-09-06 08:31:59,122 DEBUG [-] Using config files: /etc/st2/st2.conf
2016-09-06 08:31:59,122 DEBUG [-] Using logging config: /etc/st2api/console.conf
ERROR: Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/log.py", line 185, in setup
    disable_existing_loggers=disable_existing_loggers)
  File "/usr/lib64/python2.7/logging/config.py", line 70, in fileConfig
    formatters = _create_formatters(cp)
  File "/usr/lib64/python2.7/logging/config.py", line 106, in _create_formatters
    flist = cp.get("formatters", "keys")
  File "/usr/lib64/python2.7/ConfigParser.py", line 607, in get
    raise NoSectionError(section)
NoSectionError: No section: 'formatters'
[2016-09-06 08:31:59 +0000] [24172] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/gunicorn/arbiter.py", line 557, in spawn_worker
    worker.init_process()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/gunicorn/workers/geventlet.py", line 96, in init_process
    super(EventletWorker, self).init_process()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/gunicorn/workers/base.py", line 126, in init_process
    self.load_wsgi()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/gunicorn/workers/base.py", line 136, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/pecan/commands/serve.py", line 190, in load
    return deploy(self.cfgfname)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/pecan/deploy.py", line 9, in deploy
    return load_app(config)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/pecan/core.py", line 197, in load_app
    app = module.app.setup_app(_runtime_conf, **kwargs)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2api/app.py", line 70, in setup_app
    config_args=config.config_args)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/service_setup.py", line 88, in setup
    excludes=cfg.CONF.log.excludes)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/st2common/log.py", line 195, in setup
    raise Exception(six.text_type(exc))
Exception: No section: 'formatters'
[2016-09-06 08:31:59 +0000] [24172] [INFO] Worker exiting (pid: 24172)
[2016-09-06 08:31:59 +0000] [24124] [INFO] Shutting down: Master
[2016-09-06 08:31:59 +0000] [24124] [INFO] Reason: Worker failed to boot.
[st2@localhost ~]$

Probably the packaging has changed where the configuration files are created.

Switch to MongoDB '3.4' since st2 '2.4.0'

We're switching to MongoDB 3.4 since upcoming major StackStorm release v2.4.0.
Don't forget to pin that version in Chef cookbooks as well.

See: StackStorm/st2#3592

Use encrypted Data Bags

SSH keys, auth-related username:password, DB credentials should be stored in encrypted data bags.

Refactor it to use data bags where possible to make the Cookbook more secure and production-ready.

Good request by @shortdudey123 👍

platform_version in Travis

In Travis we see confusing platform_version when running ChefSpec unit tests. It's not clear what that really means.

After some research, here is what is platform_version and how it should be:

Example found in mysql chef cookbook:

So it might be good to follow that example by running unit tests with different platform settings and remove platform hardcoding from spec_helper.rb

KitchenCI not working for Fedora 20

Same thing as #10 but for Fedora, it should be updated to Fedora 24: https://atlas.hashicorp.com/bento/boxes/fedora-24

➜  chef-stackstorm git:(master) ✗ kitchen create default-fedora-20
-----> Starting Kitchen (v1.11.1)
-----> Creating <default-fedora-20>...
       Bringing machine 'default' up with 'virtualbox' provider...
       ==> default: Box 'bento/fedora-20' could not be found. Attempting to find and install...
           default: Box Provider: virtualbox
           default: Box Version: >= 0
       The box 'bento/fedora-20' could not be found or
       could not be accessed in the remote catalog. If this is a private
       box on HashiCorp's Atlas, please verify you're logged in via
       `vagrant login`. Also, please double-check the name. The expanded
       URL and error message are shown below:

       URL: ["https://atlas.hashicorp.com/bento/fedora-20"]
       Error: The requested URL returned error: 404 Not Found
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Failed to complete #create action: [Expected process to exit with [0], but received '1'
---- Begin output of vagrant up --no-provision --provider virtualbox ----
STDOUT: Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'bento/fedora-20' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
STDERR: The box 'bento/fedora-20' could not be found or
could not be accessed in the remote catalog. If this is a private
box on HashiCorp's Atlas, please verify you're logged in via
`vagrant login`. Also, please double-check the name. The expanded
URL and error message are shown below:

URL: ["https://atlas.hashicorp.com/bento/fedora-20"]
Error: The requested URL returned error: 404 Not Found
---- End output of vagrant up --no-provision --provider virtualbox ----
Ran vagrant up --no-provision --provider virtualbox returned 1] on default-fedora-20
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

➜  chef-stackstorm git:(master) ✗

Integration Tests in Travis

Running Test-Kitchen with Docker driver in Travis would be nice.

So with that for every PR we can be sure that cookbook actually works OK on different platforms (ubuntu-14.04, centos-7.2).

KitchenCI not working for CentOS 7.0

The version of CentOS defined in Kitchen is not available in the Atlas repository:

➜  chef-stackstorm git:(master) kitchen test default-centos-70
-----> Starting Kitchen (v1.11.1)
-----> Cleaning up any prior instances of <default-centos-70>
-----> Destroying <default-centos-70>...
       Finished destroying <default-centos-70> (0m0.00s).
-----> Testing <default-centos-70>
-----> Creating <default-centos-70>...
       Bringing machine 'default' up with 'virtualbox' provider...
       ==> default: Box 'bento/centos-7.0' could not be found. Attempting to find and install...
           default: Box Provider: virtualbox
           default: Box Version: >= 0
       The box 'bento/centos-7.0' could not be found or
       could not be accessed in the remote catalog. If this is a private
       box on HashiCorp's Atlas, please verify you're logged in via
       `vagrant login`. Also, please double-check the name. The expanded
       URL and error message are shown below:

       URL: ["https://atlas.hashicorp.com/bento/centos-7.0"]
       Error: The requested URL returned error: 404 Not Found
>>>>>> ------Excpetion-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Failed to complete #create action: [Expected process to exit with [0], but received '1'
---- Begin output of vagrant up --no-provision --provider virtualbox ----
STDOUT: Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'bento/centos-7.0' could not be found. Attempting to find and install...
    default: Box Provider: virtualbox
    default: Box Version: >= 0
STDERR: The box 'bento/centos-7.0' could not be found or
could not be accessed in the remote catalog. If this is a private
box on HashiCorp's Atlas, please verify you're logged in via
`vagrant login`. Also, please double-check the name. The expanded
URL and error message are shown below:

URL: ["https://atlas.hashicorp.com/bento/centos-7.0"]
Error: The requested URL returned error: 404 Not Found
---- End output of vagrant up --no-provision --provider virtualbox ----
Ran vagrant up --no-provision --provider virtualbox returned 1] on default-centos-70
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

➜  chef-stackstorm git:(master)

It should be updated to CentOS 7.2.

'bundle' should include st2web

Enhancement request: bundle recipe should include st2web & nginx

Supermarket Quality Metrics

https://github.com/chef-cookbooks/cookbook-quality-metrics/tree/master/quality-metrics
Things to not forget about:

	[actionrunner]
	logging = /etc/st2/logging.actionrunner.conf
	virtualenv_opts = --always-copy