WSGI for st2api and st2auth about st2-packages HOT 11 CLOSED

@armab Thank you for opening tickets for @enykeev . It's makes life a lot clearer 👍

from st2-packages.

updated version at d8c29d8

from st2-packages.

As noted in https://stackstorm.slack.com/archives/stackstorm/p1453440097029836:

There are several approaches to handle these sockets & permissions:

• 1.) Run st2api and st2auth startup scripts with default setuid/setgid (from root) to do the thing like ensuring directory exists/create /var/run/st2 with proper permissions in pre-start hook. And launching the main process like setuidgid st2 command from st2 user.
• 2.) Running st2api service to ensure /var/run/st2 (from root) with child service start on starting st2api (from st2 user).
• 3.) Use in app-like directory for example: /opt/stackstorm or /var/lib/st2. Looks easier, since we don't need to ensure/create every time directory like /var/run/st2 which is not persisted between reboots in /run & /var/run.

Solution for 3.) is in WIP here (debian worked/tested for now):
https://github.com/StackStorm/st2-packages/compare/master...armab:feature/st2api-st2auth-gunicorn?expand=1

Will try to run gunicorn / uwsgi in HTTP mode later, so st2bundle could remain self-sufficient.

from st2-packages.

Since we rely on http sockets rather then unix sockets, everything ^^ is much simpler now.
Glad the discussion in Slack happened and packages are still self-sufficient without nginx as strict requirement for minimal install 👍

Discussion: https://stackstorm.slack.com/archives/stackstorm/p1453468313030218

from st2-packages.

https://stackstorm.slack.com/archives/stackstorm/p1453497004030576

from st2-packages.

@enykeev that bug when st2api is not running - is still there, opened: #106

Spent some cycles on CircleCI SSH to catch what's going on, - couldn't identify what's wrong. I'll test tomorrow on AWS some very small instance, maybe could catch if issue is related to resources.

Played a bit with those nice Ruby Rspec tests here: #107 That will help us a bit.
Also updated TODO in the first message, did tests on Wheezy (AWS).

As you know, just to repeat, since that PR #100 is merged, we can test that feature via apt-get/yum install from staging unstable repos.

from st2-packages.

Now good manual testing is only needed here.

from st2-packages.

Did some dumb performance tests with Apache Benchmark for both API + Auth.
Overall it was very slow for me: 20req/sec on default non-tuned config/machine, especially considering that only 1 gunicorn worker is used, eg 1 CPU core from 4 was in use (3.5Ghz laptop CPU core in my case).

But so far it never died (all processes were alive) when I requested API /v1/actions endpoint with Auth enabled 10 000 times with 100 parallelism.
I did several runs with different configs for an hour maybe. For example started first with optimistic 1M requests)

# ab -H "X-Auth-Token: 250e2a32ef63410d99b2c03c62cdba8c" -n 10000 -c 100 http://localhost:9101/v1/actions

This is ApacheBench, Version 2.3 <$Revision: 1528965 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        gunicorn/19.4.5
Server Hostname:        localhost
Server Port:            9101

Document Path:          /v1/actions
Document Length:        46130 bytes

Concurrency Level:      100
Time taken for tests:   511.676 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      466580000 bytes
HTML transferred:       461300000 bytes
Requests per second:    19.54 [#/sec] (mean)
Time per request:       5116.757 [ms] (mean)
Time per request:       51.168 [ms] (mean, across all concurrent requests)
Transfer rate:          890.49 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       2
Processing:   175 5092 4379.3   3307   34933
Waiting:      175 5092 4379.3   3306   34933
Total:        177 5093 4379.3   3307   34933

Percentage of the requests served within a certain time (ms)
  50%   3307
  66%   3721
  75%   5497
  80%   7499
  90%  11827
  95%  14704
  98%  18304
  99%  20553
 100%  34933 (longest request)

cc @lakshmi-kannan

Closing, as gunicorn + st2api/st2auth looks at least stable with your fix.
Tested on Ubuntu14

from st2-packages.

https://stackstorm.slack.com/archives/stackstorm/p1454608632005060

from st2-packages.

@armab they will never fail. You are right about test lol :)
Actions are queued and responses are got at the same time. With the same success you could invoke background actions from CLI.

from st2-packages.

I'd argue about theory and how it happens in practice, especially "they will never fail" :)
There are always 100500 reasons to get fail from API/AUTH, sitting behind another layer of complexity like gunicorn.

from st2-packages.