Opcache errors after deployment with symlink switch about cachetool HOT 15 CLOSED

Are you resetting the cache after symlinking? Can you post the deploy script (a slimmed down version).

from cachetool.

Yes, the reset happens after the symlink switch. But we also tried before and even after the switch at the same time to no avail.

We are using Rocketeer for deployment so there is not really a "script", only a configuration and a set of tasks which are run at specific stages during the deployment.

We've hooked the Opcache reset directly after the symlink switch so it is really the last thing that is done during deployment. (Except of cleanup of old releases.)

Anyways here's a vastly slimmed down excerpt of the last deployment log:

...
[user@host] (connection) No migrations performed.
connection/0 |=> Sharing file /var/www/project/releases/20160216181832/foo
$ ln -s /var/www/project/shared/foo /var/www/project/releases/20160216181832/foo-temp
$ mv -Tf /var/www/project/releases/20160216181832/foo-temp /var/www/project/releases/20160216181832/foo
...
connection/0 |=> Successfully deployed release 20160216181832
connection/0 |-- Closure fired by deploy.after [~1.45s]
$ cd /var/www/project/releases/20160216181832
$ /usr/bin/php /var/www/project/releases/20160216181832/typo3cms cache:flush --force
$ /usr/bin/php /var/www/project/releases/20160216181832/typo3cms cache:warmup
[user@host] (connection) Flushed all caches.
[user@host] (connection) Warmed up the following caches: classes, package manager, tca, ext_tables, ext_localconf
connection/0 |-- Closure fired by deploy.after [~1.79s]
$ cd /var/www/project/releases/20160216181832
$ /usr/bin/php /var/www/project/releases/20160216181832/bin/cachetool opcache:status
[user@host] (connection) ...
connection/0 |-- ClearOpcodeCacheTask (Performs tasks on opcode caches) fired by deploy.after
$ cd /var/www/project/releases/20160216181832
$ /usr/bin/php /var/www/project/releases/20160216181832/bin/cachetool opcache:reset
connection/0 |-- Closure fired by deploy.after [~1.79s]
$ cd /var/www/project/releases/20160216181832
$ /usr/bin/php /var/www/project/releases/20160216181832/bin/cachetool opcache:status
[user@host] (connection) ...
connection/0 | Cleanup (Clean up old releases from the server)
$ rm -rf /var/www/project/releases/20160212170134

from cachetool.

Try clearing the realpath cache before or after the opcache reset:
cachetool stat:clear

from cachetool.

@gordalina Didn't think of that, thanks I'll give it a try.

from cachetool.

No luck, still getting errors after deployment. :-X I just saw this in the PHP-FPM logs though:

[18-Feb-2016 10:35:06] WARNING: [pool www] child 16153 said into stderr: "zend_mm_heap corrupted"
[18-Feb-2016 10:35:06] WARNING: [pool www] child 16153 exited with code 1 after 61793.224488 seconds from start
[18-Feb-2016 10:35:06] NOTICE: [pool www] child 25421 started

from cachetool.

That looks like a very specific error, did you try these solutions?

from cachetool.

@gordalina I've also searched for this and saw the same suggestions but I fear none of them will help.

As for USE_ZEND_ALLOC=0 the manual says this:

Note: USE_ZEND_ALLOC=0 in the environment will stop the memory manager from functioning, all allocations fall back on the default system allocators which can be useful for debugging leaks.

This sounds more like a debugging measure and I'm not sure if this is something one wants in a production environment...

from cachetool.

Did you try reloading php-fpm after cachetool? It restarts the workers a gracefully.

from cachetool.

This would be one of the remaining options which I tried to avoid due to requiring sudo. So far mod_realdoc also doesn't see to yield the expected results.

Do you mean that there will be no downtime when restarting PHP-FPM?

from cachetool.

Have a look here

from cachetool.

@gordalina Thanks, so this is about reload vs. restart. Seems like it isn't that graceful either and the SO comment only mentions APC so I'm not sure about the Opcache behavior here.

from cachetool.

Not sure if I can be of further assistance. I'm closing this issue.
Feel free to reopen any time, or post a comment if you've found a solution.

from cachetool.

@gordalina Thanks already for taking time to give me pointers here. I now went with PHP-FPM restart and so far it looks OK-ish in 90% of the deployments. I still saw an error once but I'll keep watching.

from cachetool.

Heh after few days of using cachetool I also started seeing this kind of behavior in my environment. Once cache is cleared – all require_* calls fails.

from cachetool.

Hopefully this would solve the issue: zendtech/ZendOptimizerPlus#126 (comment)

opcache.use_cwd = 1
opcache.revalidate_path = 1

from cachetool.