Very high CPU usage about starman HOT 12 CLOSED

You should provide more information such as Perl version and the operating system. You should also dig what it is doing with ptrace or equivalent.

Makefile.PL failure is normal - you should install Module::Install::ReadMeFromPod and such to install from git.

from starman.

ok, I've make it.

My runtime environment:
Perl 5.10.1
CentOS 5.4 x86_64
MongoDB driver
Starman(git master)
Cache::Memcached::Fast (0.19)
Dancer(git master)

Last night, I'd benchmark it agin,

As result, starman performance it less bit than Plack::FCGI.
It can handel ~39-40 qps, FCGI is 41-44 qps.

I move the application to another dedicated server.
Currently 20 starman workers, cpu usage is 97-100%.

Because memcache cache hit ratio is very high, I think it''s not
MongoDB problem.

Now, I'd make a choice between starman and fcgi.

Do you have any case of production?
How about the performance in real-world?

Thanks.

from starman.

Starman does work really good in the production. It sounds like your application has an issue in a preforked environment and it's you who should diagnose the problem to figure out the high CPU usage.

from starman.

I benchmark again, use this suite:

1. install Dancer (git master)

2. Generate a dancer hello_world app
   dancer -a demo
   now, test:
   ./demo.pl

   ab -c 100 -n 1000 http://127.0.0.1:3000/

result:

Server Software:
Server Hostname: 127.0.0.1
Server Port: 3000

Document Path: /
Document Length: 620 bytes

Concurrency Level: 100
Time taken for tests: 1.466 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 698000 bytes
HTML transferred: 620000 bytes
Requests per second: 682.34 #/sec
Time per request: 146.555 ms
Time per request: 1.466 [ms](mean, across all concurrent requests)
Transfer rate: 465.11 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 1.4 0 6
Processing: 43 139 17.3 139 174
Waiting: 43 139 17.2 139 174
Total: 49 140 16.4 139 178

Next, run dancer psgi use starman:

starman --port 3000 --workers 1

result:

Server Software:
Server Hostname: 127.0.0.1
Server Port: 3000

Document Path: /
Document Length: 620 bytes

Concurrency Level: 100
Time taken for tests: 4.738 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 754000 bytes
HTML transferred: 620000 bytes
Requests per second: 211.07 #/sec
Time per request: 473.780 ms
Time per request: 4.738 [ms](mean, across all concurrent requests)
Transfer rate: 155.42 [Kbytes/sec] received

Last, I increment workers:
starman --port 3000 --workers 6

Server Software:
Server Hostname: 127.0.0.1
Server Port: 3000

Document Path: /
Document Length: 620 bytes

Concurrency Level: 100
Time taken for tests: 3.522 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 754000 bytes
HTML transferred: 620000 bytes
Requests per second: 283.97 #/sec
Time per request: 352.153 ms
Time per request: 3.522 [ms](mean, across all concurrent requests)
Transfer rate: 209.09 [Kbytes/sec] received

Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 1 1.9 0 14
Processing: 98 336 44.8 333 447
Waiting: 97 335 44.6 333 446
Total: 103 337 44.2 335 448

ok, a bit better, but not enough.

And, compare to dancer standalone server, starman cpu usage is more higher.

My test machine:
Snow Leopard 10.6.3,
Macbook Pro, 2.33 Core2Duo, 4GB memory.

from starman.

You're testing on a localhost and compare a single worker (dancer) with multiple workers, and the number of workers is low as compared to -cｃ. This is not a realistic benchmark.

Again, you keep saying the cpu is high, and you NEVER have supplied the ptrace/strace output to show what is causing the CPU spin.

from starman.

I known this is NOT a realistic benchmark. BUT, as your "Performance" section of Starman introduction, your benchmark is also invoke as '"ab -c 10 -t 1 -k", so, I just
want to replay the case. And if you could replay my bench case, and give me some
different results. That's will be good.

Any 'hello' test case is useless, I known that ;-)

I never use dtrace, and just deal with dtruss.

from starman.

I get almost the same result and I'm sure this is Dancer's problem.

Take a look at Dancer's code base - Dancer::Handler::Standalone and Dancer::Handler::PSGI. In ::PSGI it calls unnecessary Plack::Request->new to get the headers which gives a reasonable overhead.

So you're comparing two different applications in two different servers.

If you want to compare Starman with a different server, use the app.psgi with other servers such as HTTP::Server::PSGI, instead of demo.pl.

I never use dtrace, and just deal with dtruss.

So what? Until you provide the data to show me what is causing the "very high CPU usage" I can't help.

from starman.

Any 'hello' test case is useless, I known that ;-)

No, Hello benchmark is useful, to get the potential maximum performance of the servers.

However, if your application's maximum performance is 40-50 qps, that's much slower than the server's potential, so deciding the server which does 7000qps or 5000qps does not make much difference anyway. In that case, the decision of servers should be made in terms of memory usage or features rather than the performance.

from starman.

Thanks a lot.
So, this not Starman problem, I'll dive into Dancer.
My another psgi based Plack, it run fine on Starman, great job!

from starman.

Good. Well you brought up a few different problems in one case.

1. You compared the raw performance of demo Dancer app between demo.pl and Starman. As I said, that's not a good comparison.
2. You reported that your mongo/memcached application gives high CPU in Starman. Since we don't have any system trace for that, we can't dig deeper here.

Also, a couple of months ago i supplied a patch for Dancer to eliminate the Standalone vs PSGI branch on my github fork but it never took off because of the compatibility issue. They said they will incorporate the changes in the next major upgrade. The performance issue about the headers could be addressed when you report that.

Good luck.

from starman.

Cooool!
I merged your patch, and now most of my problem was gone!
Live qps goes up to 1000+, cpu usage also down to 50%.
Thanks!

from starman.

Hah, that's nice! You should definitely report that back to the Dancer community to push the code changes back.

from starman.