performance problems... about cm_redissession HOT 26 CLOSED

I think it'd be hard to isolate a case using New Relic and know for sure if there was session locking going on or not, but that is definitely the most likely scenario. The other possibility would be an actual read issue with Redis, but the read timeout should limit the delays caused by timeouts so for 80s this doesn't seem to be the case. The redis_session.log file should report any broken locks.

So I am not convinced this is a Redis/Cm_RedisSession issue as I have myself logged occasional long page loads for example when adding a product to a category during peak traffic. I think the Mage_Index indexers lock some tables that the category pages apparently must wait on in this case. We ended up working around by finding an alternative to reduce usage of categories and problem went away.

Redis garbage collects as records expire so you can simply reduce the session expire time in Magento config. You can also reduce bot_lifetime even more if you aren't worried about possible side effects (can't think of any myself).

from cm_redissession.

Hi Colin,

As I said, I was very sceptical of the results I was seeing, as tracing php processes running the new relic plugin shows a very similar format to that of php with xdebug running, and that can really screw stuff up!

And the customer makes extreme use of categories as well...

I will refer back to the customer on this one - good to know others have seen the same result.

from cm_redissession.

Hey Collin,
I am going to have 3 lac visitors on my website and my redis session server is almost full, Out of 17GB RAM redis uses 16gb, can you suggest what needs to be done here as I don't want to loose sessions.

from cm_redissession.

• Use (better) compression
• Fix Magento session models to not store so much redundant data
• Audit your code to make sure you aren't wasting space with session storage
• Reduce session lifetime (e.g. is 30 days really necessary?)
• Reduce session lifetime for bots

Lastly, Redis may not be right for you. MongoDb would probably be where I'd start looking if Redis can't meet your needs.

from cm_redissession.

An interesting aside to this, an something I'd never measured before. It turns out that the frontend cache being served by redis on this platform was running at an average 1.5MB/s, which did surprise me being that high. I'm analysing further, but it could be the number of requests, rather than the volume that's causing the problems.

from cm_redissession.

Hi Colin, we are experiencing the same escenario as @sholdowa with the waits after Cm_RedisSession_Model_Session::read. Also Im able to identify the issue thru New Relic.

In our case, the overall transaction takes more than 30secs in some cases, and I see that theres a loop in the transactions trying to run Cm_RedisSession_Model_Session::read. Each try, waits 2.5 secs, after that, it retries again (in this example up to 12 retries).

Our config:

        <redis_session>             
            <host>127.0.0.1</host>  
            <port>6379</port>
            <timeout>2.5</timeout>  
            <persistent></persistent>
            <db>12</db>
            <compression_threshold>2048</compression_threshold> 
            <compression_lib>gzip</compression_lib>           
        <force_standalone>1</force_standalone>
            <log_broken_locks>0</log_broken_locks>            
            <max_concurrency>10</max_concurrency>
            <break_after_frontend>5</break_after_frontend>
            <break_after_adminhtml>30</break_after_adminhtml>            
            <bot_lifetime>7200</bot_lifetime>                 
        </redis_session>

This store also has some crazy add to cart ajax stuff, and an ajax infinite scroll, so many concurrent requests trying to access the same session I guess. My suspicious is that while theres a process adding a product to the cart (in example) theres another process trying to get the same session (an infinite scroll request), and the config comes in play to stop the wait at 2.5secs.

Would this make sense?

from cm_redissession.

Hello everyone,

Thank you @colinmollenhour for this great contribution ! Please pardon my late reply, but I think my input is relevant to this matter as it clears out things.
I am using the following stack: Varnish -> Nginx -> 2x PHP-FPM nodes + 1 Memcached + 2x Redis.
My sessions are stored in one Memcached node and I'm using Redis only for Magento's backend cache.
My point is that I am also experiencing the same problem with the sessions locking, therefore that demonstrates that this is not a Redis problem. It's a Magento problem which gets out of proportions in combination with the PHP Memcached module.
I am also using NewRelic and seeing exactly what you guys mentioned. Hoping to get more insight out of this, I even wrote two Munin plugins that graph the slow requests times and their corresponding CPU usage and tried correlating that to other events.
The problem is that with every opening page in Magento you get two ajax requests: one for the cart and one for the logged in / logged out check. So, particularly in an e-commerce website, you have people opening more than one tab... I for example, I just hold my CTRL down and go crazy on eBay (smile).
I would be curious if someone managed to find a solution to this Magento problem.
Thank you.

from cm_redissession.

Starting with the latest commit (2 months ago: 0a03d8c) it is possible to disable locking for a specific request. Not sure offhand what the proper event would be, but for example you could add a preDispatch event that disables locking for the read-only ajax requests. Or in index.php you could add a hack based on the request url. This should not be necessary but if you need a quick solution it may be worth a try.

The way I've tested session locking is to hold ctrl and click on a single link 20 times as fast as I possibly can, or open the media browser in the adminhtml (it generates thumbnails real-time). I am not able to reproduce such long delays using this method and can observe that the locking is working as each tab/image loads one by one. So I definitely won't deny that the locking algorithm could be improved, but I have not been able to create a reproducible scenario to observe a bug in the locking yet (with the current version).

from cm_redissession.

We find out an (maybe) interesting aspect:

Long running PHP processes caused a problem in Redis session handlung in connection with old redis server version.

First we checked that a Magento API request took 1 second on our stage system and 30 seconds on live. We traced it via New Relic. The entry looked like

Cm_RedisSession_Model_Session::read 1 30,100 ms 98%

Same with some not cached search requests:

Cm_RedisSession_Model_Session::read 1 30,100 ms 98%
Mage_Catalog_Model_Layer::getFilterableAttributes 3 11,500 ms 32%
Bubble_Elasticsearch_Model_Resource_Fulltext_Collection::addSearchFilter 1 10,700 ms 30%

Problems accoured with redis 2.6.17 and not under 2.8.9.

from cm_redissession.

@wsteinbach that 30-second timing reads to me as a session lock. Please review the configuration options and consider disabling locking or decreasing timeouts.

from cm_redissession.

Thank you very much. It works with

<disable_locking>1</disable_locking>

BTW:
The Docu https://github.com/colinmollenhour/Cm_RedisSession is a little bit confusing to me:

<disable_locking>0</disable_locking> <!-- Disable session locking entirely. -->

suggests to me '0' disables locking but I think '1' = true does it?!

from cm_redissession.

Locking is considered a core feature so it was safer to add "disable_locking" with 0 being the implicit value which retains backwards compatibility than use "enable_locking" in which case backwards compatibility would be broken. "Disable: yes" means disabled is pretty clear IMO.

I don't know what your original issue was, perhaps a fatal error was occurring and causing sessions to be left locked?

from cm_redissession.

Would be great to get some further information on this. We get these occurring quite regularly with no identifiable pattern.

What are your thoughts around disabling locking @colinmollenhour? There aren't any fatal errors causing sessions to remain locked from what we've investigated so far.

from cm_redissession.

With locking disabled multiple requests could overwrite each other's session data. If that is fine with you then disabling will certainly resolve your issue.

What is "quite regularly"? If it is 1:100,000 hits then I wouldn't say that is quite regular.. If it is quite regular then you should be able to collect more data to go by for tracking down the issue. E.g. if it happens while persisting database to disk then you could setup replication and disable persistence on the master.

from cm_redissession.

We are also experiencing this with the Session::read.

Sometimes in mundane calls;

ALWAYS on return from Sagepay using ebizmarts sagepay suite pro.

I'm tempted to give the disable_locking fix a go, but concerned we could cause more problems than we have currently?

from cm_redissession.

Interesting. Nothing comes to mind regarding why a particular extension would be able to trigger a session read issue, but it sounds like you've at least identified a starting point to dig further. If you don't want to dig further then either disable locking just for those requests somehow (e.g. define the constant in index.php when the request path is matched) or set a much lower break_after_frontend to just reduce how long it waits for the lock before breaking it.

from cm_redissession.

we're going to give the upgraded module a go and deploy that tomorrow. It seems to suggest it will work on our staging site, but this is not under the same user load as production (obviously!)

Will know more tomorrow!

from cm_redissession.

👍 for a fairly low break_after_frontend value. I’ve had better results with that config as low as 4.

from cm_redissession.

updating to the latest version of the code has stopped our return time from being 30s + ~4s down to ~4s in total.

Much better and as anticipated.

from cm_redissession.

Hi @Bobspadger,
we've been following this thead for a little while now, and can you clarify what code you've updated to the latest versions.

Many Thanks

from cm_redissession.

we went from Magento 1.9.1.0 codebase to the latest on github :)

from cm_redissession.

0 app/code/community/Cm/RedisSession/Model/Session.php
+47 -15 app/code/community/Cm/RedisSession/etc/config.xml
+32 -0 app/etc/modules/Cm_RedisSession.xml
+2 -0 lib/Credis/.gitignore
+375 -55 lib/Credis/Client.php
+317 -0 lib/Credis/Cluster.php
+23 -0 lib/Credis/LICENSE
+171 -0 lib/Credis/README.markdown
+239 -0 lib/Credis/Sentinel.php
+23 -0 lib/Credis/composer.json
+21 -0 lib/Credis/phpunit.xml
+3 -0 lib/Credis/tests/.gitignore
+224 -0 lib/Credis/tests/CredisClusterTest.php
+173 -0 lib/Credis/tests/CredisSentinelTest.php
+29 -0 lib/Credis/tests/CredisStandaloneClusterTest.php
+8 -0 lib/Credis/tests/CredisStandaloneSentinelTest.php
+37 -0 lib/Credis/tests/CredisStandaloneTest.php
+351 -0 lib/Credis/tests/CredisTest.php
+35 -0 lib/Credis/tests/SetUpBefore.php
+35 -0 lib/Credis/tests/TearDownAfter.php
+100 -0 lib/Credis/tests/keys.test
+5 -0 lib/Credis/tests/redis-2.conf
+5 -0 lib/Credis/tests/redis-3.conf
+5 -0 lib/Credis/tests/redis-4.conf
+6 -0 lib/Credis/tests/redis-auth.conf
+5 -0 lib/Credis/tests/redis-master.conf
+6 -0 lib/Credis/tests/redis-sentinel.conf
+6 -0 lib/Credis/tests/redis-slave.conf
+6 -0 lib/Credis/tests/redis-socket.conf
+9 -0 lib/Credis/tests/redis_config.json
+1 -0 lib/Credis/tests/sentinel_config.json

from cm_redissession.

@Bobspadger Many thanks... that appears to have solve the issue for us as well...

from cm_redissession.

For anyone looking on here to know WHY @Bobspadger saw times being decreased from 30s -> ~4s, I believe it is because the default of 30sec was being used due to this issue:

#61

from cm_redissession.

With locking disabled multiple requests could overwrite each other's session data. If that is fine with you then disabling will certainly resolve your issue.

Can you please elaborate on what this means? Is it that one user can overwrite another user's or that one user's session could be overwritten by concurrent requests from their browser (loading page + AJAX). Either way, how would you test for this in production and how risky is it to enable it?

from cm_redissession.

Necro... :)

The session id is the key that is locked so it is the latter. Likelihood depends on your application but in Magento there is not all that much important data in the sessions, mostly the session messages and the quote id.

Closing conversation.

from cm_redissession.