humanmade / cavalcade Goto Github PK

Cavalcade prevents the rescheduling of an existing job to a new frequency as the change in scheduling is not recognised.

Including the interval in the connector's real key will resolve the problem.

To reproduce the error, run the following:

$stamp = time() + HOUR_IN_SECONDS;
wp_schedule_event( $stamp, 'hourly', 'cavalcade_repeat' );
wp_schedule_event( $stamp, 'twicedaily', 'cavalcade_repeat' );

var_dump( _get_cron_array()[ wp_next_scheduled( 'cavalcade_repeat' ) ] );

The final line will show the job is scheduled to run hourly.

I'm trying to switch to Cavalcade for our multisite, but every time I start runner, it crashes after running the first set of jobs with the following error:

Sep 15 14:28:18 server php: PHP Notice:  Undefined offset: 0 in /opt/Cavalcade-Runner/lib/Runner.php on line 211
Sep 15 14:28:18 server php: PHP Fatal error:  Call to a member function is_done() on a non-object in /opt/Cavalcade-Runner/lib/Runner.php on line 212

I've been running it in the foreground, but it doesn't really output anything more useful:

 cavalcade
[27] Running wp cavalcade run 27 --url='f.q.d.n/site1/' (check_plugin_updates-user-role-editor-pro a:0:{})
[27] Started worker
[27] out:
[27] err:
[27] ret: 0
[28] Running wp cavalcade run 28 --url='f.q.d.n/site2/' (check_plugin_updates-user-role-editor-pro a:0:{})
[28] Started worker
[29] Running wp cavalcade run 29 --url='f.q.d.n/site2/' (wp_scheduled_delete a:0:{})
[29] Started worker

Actual hostnames and URLs replaced to protect the innocent, etc. Meanwhile, I'm also looking into why the stupid User Role Editor Pro checks for plugin updates on every single site.

Since out of the box monitoring is missing it would be nice to have the possibility of automatically retrying failed jobs. Along with a timeout this would have prevented almost all issues we've had.

Last evening, at exactly 9 PM EDT (2 AM GMT), hourly jobs started running continuously--instead of being scheduled for the next hour. This caused a major load event and we had to stop Cavalcade everywhere. Sample log showing the issue:

20413   wp_privacy_delete_old_export_files      2018-11-03 19:14:32     completed
20413   wp_privacy_delete_old_export_files      2018-11-03 20:14:32     completed
20413   wp_privacy_delete_old_export_files      2018-11-03 21:14:34     completed
20413   wp_privacy_delete_old_export_files      2018-11-03 21:14:36     completed
20413   wp_privacy_delete_old_export_files      2018-11-03 21:14:37     completed
20413   wp_privacy_delete_old_export_files      2018-11-03 21:14:39     completed
20413   wp_privacy_delete_old_export_files      2018-11-03 21:14:40     completed
20413   wp_privacy_delete_old_export_files      2018-11-03 21:14:41     completed
20413   wp_privacy_delete_old_export_files      2018-11-03 21:14:42     completed

This morning, it seems like we can get cavalcade happy by running wp site list --field=url | xargs -n 1 -I % wp cron event run --due-now --url=% to bring all jobs into the present. The WordPress site in question uses UTC internally; the host servers (both application and database) are on EDT/EST.

Hi,

We are using zero downtime deployments for our project (Deployer). When deploying some new code the worker seems to run into problems. All jobs continuously fail with the message "Failed to shutdown worker.". I figure this has something to do with with paths/symlink changing when a new deployment is executed.

The only fix I found for now is to stop the supervisor job completely en restart it again, I was wondering if this is a known problem and if there would be a more low level solution. As restarting supervisor requires root and is therefor a pain to integrate in our deployment logic.

Thanks in advance.

Parse error: syntax error, unexpected '[', expecting ')' in /opt/wordpress/wp-content/plugins/Cavalcade-master/inc/namespace.php on line 22

@rmccue Cavalcade looks great and attacks a real problem with wp-cron and larger multisites. Is this ready to give a try with the expectations of a few minor issues but nothing crazy?

I'd be happy to give our feedback on trying to use this if you need some beta testers to help the project.

It seems that perhaps the Cavalcade WP plugin doesn't support adding a job using a class with an invokable as the callback?

http://php.net/manual/en/language.oop5.magic.php#object.invoke

I have the following code to add my invokable class:

if ( ! wp_next_scheduled( Model\MyExampleInvokableClass::class ) ) {
    wp_schedule_event( time(), 'daily', Model\MyExampleInvokableClass::class );
}

This seems to work fine using the standard WP Cron system and I'm able to verify that the job is queued using the 'Advanced Cron Manager' plugin (https://wordpress.org/plugins/advanced-cron-manager/).

However, when Cavalcade is enabled, these jobs disappear from the jobs list in WP Admin and also aren't visible when I do wp cavalcade jobs.

I don't see anything in PHP error logs or cavalcade log either...

Thanks! 😃

WordPress 4.9 + Redis Object Cache

Events multiply every two seconds

The WP-CLI commands have sparse user-facing documentation, which does not give the user everything they need to know.

For example, the @synopsis tag tells me that the jobs command has a status filter, but not what the valid values for it are. I have to dig through the code to figure out that they're -- as far as I can tell -- waiting, running, failed, and completed.

Adding option descriptions and examples would result in a much better UX.

Ran into an interesting scenario today where a daily recurring job failed last week and it wasn't queued again / re-run.

In the cavalcade_jobs table, the status was 'failed'

• wp_next_scheduled( $job ) returned the date for the next time after that failed run when it should have run (The plugin queries for jobs which are status != 'complete' to return in the cron array)
• Cavalcade failed to run the job the next day as the job failed, and it was only looking for status = 'waiting'.

This ended up in a situation where WordPress thought the job was queued and Cavalcade didn't attempt to run it.

In this case, I manually reset the status field for the jobs in question to waiting and it continued on it's way.

I just noticed that DOING_CRON isn't defined when running jobs, this could cause unexpected behaviour for some jobs.

While running through WP-CLI, it won't be possible to define it before bootstrap occurs (AFAIK), but defining it late might be better than not at all.

Jobs containing args that contain multibyte unicode characters such as emoji can't be run as after they're serialised they do not unserialise correctly.

I am using one long process which takes about an hour to get completed. but the process get in failed status before it get completed.

Specifically this: https://core.trac.wordpress.org/ticket/32656.
https://core.trac.wordpress.org/ticket/15148#comment:22

Can you provide step to install you plugin ?
I get error when install on mu plugin
PHP Fatal error: Class 'WP_CLI_Command' not found in /var/www/wp-content/mu-plugins/class-command.php on line 8

I can run wp --info
PHP binary: /usr/bin/php5
PHP version: 5.5.9-1ubuntu4.7
php.ini used: /etc/php5/cli/php.ini
WP-CLI root dir: phar://wp-cli.phar
WP-CLI global config:
WP-CLI project config:
WP-CLI version: 0.20.1

what next should I do?

thank you

Currently get_next_job() does two things wrong IMHO:

• Doesn't specify a LIMIT 1 - which while implied by the fetchObject() call, still causes all waiting jobs to be returned to the client buffer
• Doesn't specify a ORDER BY - What this causes is that when jobs get backlogged, jobs run in an order as determined by their id rather than by their order of nextrun, which can result in some unexpected behaviours in cron jobs (if a specific time, say an hour after another job was specified, and can cause some unexpected behaviours in cron jobs)

The ORDER BY shouldn't be an issue, except when backlogged by something such as #37

In WordPress/wordpress-develop@e73af26 meaningful return values were introduced for cron functions, often these return the result of update_option() for the cron array.

As the Cavalcade connector short circuits the cron array from updating this always returns a false. Until the connector is updated to use the hooks added in WordPress/wordpress-develop@a32ea2d testing the return values will fail.

Blame is an ugly word but it should be noted both of the commits linked above were done by the same person. Me.

I'm setting up Cavalcade in a hosting environment where MySQL is running in local time - UTC-5, in my case. This is causing a mismatch: WP and mu-plugins/cavalcade are recording the job (properly) in UTC, while the runner's nextrun < NOW() query is offset by 5 hours, causing jobs to be run 5 hours late.

I realize that the ideal situation is to use UTC everywhere, but I don't control the environment, and certain parts of the infrastructure are shared with a couple other resources.

I've fixed this in my local codebase by putting some offset logic in the /connector/namespace.php methods. I'm wondering whether you'd be interested in an enhancement that makes it possible to set a flag (a constant or something) so that Cavalcade will perform this offset automatically if the flag is set.

Feel free to close if you feel like this would be reinforcing a bad practice! And thanks for the cool tool 😎

Hi Ryan/Humanmade

First of all thanks for creating this, have been looking for something like this for a while to have better job/queue management.

Coming from Laravel I have the feature request to run tasks sync on local/staging environments for easier development/debugging.

An other workaround suggestion would also be appreciated!

Thanks!

Seeing a potential issue where (in this example) scheduled post cavalcade jobs are duplicated causing a race condition and posts don't publish.

Will update upon further investigation

The Cavalcade tables can be created prior to WP Core tables.

A couple of times I have seen the Cavalcade tables created in the database before the options table exists. Since Cavalcade started storing it's own DB version in the options table, this can now cause problems.

Prior to bootstrapping, a check to ensure the options table is created and populated will prevent this. For example:

if ( ! get_option( 'db_version' ) ) {
  return;
}

Running wp cavalcade jobs --status=completed will result in the command fetching potentially millions of rows, which takes forever (if it even finishes) and doesn't provide meaningful results to the user.

A default limit of 10 or 20 and a --limit parameter seems like a good solution.

Currently cavalcade doesn't support wp_get_schedule() properly - it always returns __fake_schedule. This isnt' a problem for WordPress normally, however it has caused issues with the latest Jetpack release.

Consider the following code, boiled down from https://plugins.trac.wordpress.org/browser/jetpack/tags/4.4/sync/class.jetpack-sync-actions.php?marks=295-302#L289

if ( ! wp_next_scheduled( $hook ) ) {
    wp_schedule_event( time(), 'twicedaily', $hook );
} else if ( 'twicedaily' != wp_get_schedule( $hook ) ) { // It was queued as 'daily' previously
    wp_clear_scheduled_hook( $hook );
    wp_schedule_event( time(), 'twicedaily', $hook );
}

Cavalcade will return __fake_schedule for wp_get_schedule() and cause it to endlessly clear the hook and requeue it again.

As we've run into this on WordPress.org, I've temporarily put the following in place: dd32@fbd23d2
Unfortunately as pointed out in the commit, this is only a temporary solution, in the event that two schedules with identical timings, but different names, there's a good chance that it'll return the incorrect schedule.

I don't really see a way around this other than storing the schedule name in the database which a job is attached to, that probably means an extra field in the jobs table.

The cron array may be accessed prior to the plugins_loaded action firing, the most likely candidates are:

• muplugins_loaded
• plugin_loaded added in WP 5.1
• muplugin_loaded added in WP 5.1

Running Connector\bootstrap() as the plugin is required will filter the cron array as soon as the code is available.

On the downside, the is_installed() check would probably need to be included in the filters but as it's cached, this shouldn't cause too much of a problem.

Hello,

My question is about current big installions that will migrate to WP native cron to Calvalcade cron system.
Is it possible ? I think we can

1. remove filters of Calvacade
2. get the events schedules
3. re-add the filters
4. reschedule everything through the WordPress API

but is it safe ?

Regards,

Follow up to discussion in PR #91

Cavalcade uses a non-persistent cache for the cavalcade-jobs group. As the runner triggers WP CLI tasks within the plugin, invalidating on UPDATE, INSERT and DELETE should be a relatively simple task.

Proposal:

• Remove cavalcade-jobs from the non-persistent groups
• Consider whether cavalcade-jobs should be a global or per site

Hey, this is Luca from FFX. Our paths cross again =)

I can see both CAVALCADE_JOB_ID and DOING_CRON constants are defined in Command::run() just before the actual execution of the scheduled hook.

That means I cannot access them during, say, any init or admin_init hook.

Use case is an event which must come after the completion of another event.
example:
Event_A calls wp_schedule_single_event( time(), 'event_b' );
Event_A shows up as completed in the log but event_ b never shows up in the log (neither the DB log or a piped to log from the cavalcade runner).

There's a chance that you might end up with a stack of jobs running at once which can throttle a server. We should consider adding a command to stop jobs from running.

@rmccue does that sound like a good idea? Can you forsee any issues or reasons not to add a command for that?

Can your docs please mention that pcntl must be installed?
http://php.net/manual/en/book.pcntl.php

Task
When setting and getting Cron jobs, make use of the new WP filters: pre_schedule_event, pre_reschedule_event, pre_unschedule_event, pre_clear_scheduled_hook, pre_unschedule_hook, pre_get_scheduled_event and pre_get_ready_cron_jobs.

Background
In WordPress 5.1, new filters will be added to preflight event scheduling to allow for custom Cron storage systems.

The filters were added in WordPress#32656 (see WordPress/wordpress-develop@a32ea2d) and WordPress#45797 (see WordPress/wordpress-develop@0bc2059).

Hey,

Not quite sure where to start debugging this tbh, but we've been having issues of tasks not getting cleared properly. We've also had a few times where we had to reenable plugins to make scheduled tasks run.

Feel free to push me in the right direction for getting some useful information that can be used for debugging this.

Some info from the developer of Product Bundles:

I can see that tasks are completing, which rules out a problem with

WC_PB_DB_Sync_Task_Runner::time_exceeded or
WC_PB_DB_Sync_Task_Runner::memory_exceeded.

So WC_PB_DB_Sync_Task_Runner::task is returning false, which is correct.

However, there's likely a problem here: https://cl.ly/3q293d3y1s1Z

Tasks look like they are not being cleared out of the options table correctly

That's something you need to look into because the WC updater relies on the same library -- if it doesn't work correctly, you'll likely have issues with updating WooCommerce down the road.

Can you perhaps debug this on a standard / local server environment without any complex caching or task scheduling software running?

You should see this instead of the same task being processed over and over:

2017-10-31T07:48:52+00:00 INFO Sync complete.

If this is a problem that affects your live site only, it might be hard to diagnose it -- and something my support team won't be able to help with.

I am not very familiar with Cavalcade either -- if it's a caching or Cavalcade related issue, I'd have trouble figuring out the exact cause without looking at their code. Something like that could take days to debug :)

Can you share your findings after looking into it on the staging site?

Keep an eye on that log -- it should not keep running over and over with every request.

Tasks are stored in your options table -- they should be called:

wp_wc_pb_db_sync_task_runner_batch_xxx

If you have a lot of these queued, it might take some time until they get processed, but eventually you should see a

I am running a version of Altis+Chassis+Wordpress. There is an issue with the Cavalcade module: it throws errors when I have the dubugger open. See screenshot.

I am assuming Cavalcade is a dependency of one of the Altis modules. How do I fix this and confirm that Cavalcade is doing whatever it is supposed to do?
This is the Altis part of my composer.json file:

"require-dev": {
        "altis/core": "dev-master",
        "altis/local-chassis": "dev-master",
        "altis/dev-tools": "dev-master",
        "humanmade/coding-standards": "^0.7.0"
    },
    "autoload-dev": {
        "files": []
    },
    "extra": {
        "installer-paths": {
            "content/plugins/{$name}/": [
                "type:wordpress-plugin"
            ],
            "content/mu-plugins/{$name}": [
                "type:wordpress-muplugin"
            ],
            "content/themes/{$name}/": [
                "type:wordpress-theme"
            ]
        },
        "altis": {
            "modules": {
                "local-chassis": {
                    "hosts": [
                        "fresenius.local"
                    ],
                    "multisite": "Yes",
                    "extensions": [
                        "sequelpro",
                        "xdebug",
                        "memcache",
                        "phpini"
                    ]
                }
            }
        }
    }

On our install, sometimes we see the same job for the same hook being duplicated multiple times with the same nextrun timestamp.

Regardless of my example, instead of using a SQL query to delete a job or multiple jobs, it would be handy if there was a wp-cli command that allowed us to remove said jobs.

If this sounds like a good idea, I recommend wp cavalcade job delete for this new subcommand, but that means a wp cavalcade job command should be created as this currently doesn't exist.

What do you think?

Running composer install will also check for php extensions and versions, adding these to composer.json would help with checking dependencies

Hi!

We are using Cavalcade for a crucial part of our system. Can we be 100% certain that failed jobs will remain in the cavalcade_jobs table so we can re-run them at a later point in time, or is there any chance that they might disappear?

Thanks!

Similar to #18, we ran into an issue on WordPress.org where one of cavalcade daemons was killed unexpectantly which left a bunch of jobs in an unknown state.

The jobs were marked as running, but there were no workers to manage those jobs anymore. This resulted in jobs not running for a few hours/days until they were detected and restarted.

There should be some way for a job to be detected as no-longer-running or that it's daemon is MIA.

In this case, I simply restarted the jobs: UPDATE wp_cavalcade_jobs SET status = 'waiting' WHERE status = 'running' AND nextrun <= '2016-12-06'

Can I clear the table or delete records with a status of completed?

After what I can only describe as an intense session of debugging with @rmccue we discovered that some jobs were hanging when the pipe's buffer filled up from stderr / stdout of a running job. This could be something like a cron job doing an excessive amount of error_logging.

The buffer size is relatively small by default, and I believe the runner only reads from the pipe once the child process has completed. This means the sys call to write to the pipe will hang when full.

As a "workaround" we removed the error_log calls from the job, and if need be it's possible to cat /proc/{process id of wp-cli}/fd/2 to flush the pipe buffer.

Adding a cron job with the code wp_schedule_event( 0, 'daily', 'cavalcade_daily' ); will cause the job to run constantly until it's run once for each day since 1/1/1970. Once the job has caught up it will run at midnight each day, as expected.

Adding a command such as wp cavalcade job restart {id} --skip-backlog will allow users to skip the backlog and ensure the job only runs once per day.

Follow up to humanmade/Cavalcade-Runner#53

Cavalcade is adding 100-150ms to each page load, even with DISABLE_WP_CRON.

Do you guys have any insights on that? Could we an in-memory cache maybe?

Hey, so we're using a WP multisite setup, and noticed that when a site is deleted, Cavalcade doesn't clean out the jobs for these non-existent sites.

add_action('delete_blog', 'cleanup')

function cleanup($site_id) {
// remove jobs owned by $site_id
}

When attempting to install WP on my wp-local-docker environment, I get the following error:

WordPress database error Table 'wordpress.wp_options' doesn't exist for query INSERT INTO `wp_options` (`option_name`, `option_value`, `autoload`) VALUES ('cavalcade_db_version', '2', 'no') ON DUPLICATE KEY UPDATE `option_name` = VALUES(`option_name`), `option_value` = VALUES(`option_value`), `autoload` = VALUES(`autoload`) made by include('phar:///usr/local/bin/wp/php/boot-phar.php'), include('phar:///usr/local/bin/wp/php/wp-cli.php'), WP_CLI\bootstrap, WP_CLI\Bootstrap\LaunchRunner->process, WP_CLI\Runner->start, WP_CLI\Runner->load_wordpress, require('wp-settings.php'), do_action('plugins_loaded'), WP_Hook->do_action, WP_Hook->apply_filters, HM\Cavalcade\Plugin\bootstrap, HM\Cavalcade\Plugin\create_tables, update_site_option, update_network_option, add_network_option, add_option

The problem appears to be line 98 in inc/bootstrap.php, within the create_tables function:
update_site_option( 'cavalcade_db_version', DATABASE_VERSION );

The error is being caused by exactly what it says on the tin -- the wp_options table is not in existence at the time that this function call is made. It's possible that this is something wrong with my docker environment, but I note that no other plugins cause issues.

System details:

• Docker (Debian/Ubuntu)
• PHP 7.1
• WP 4.9.7, multisite

When I try to run wp cavalcade log on with wp-cli 1.5.1 and wp-cli 1.5.0, I'm getting this error:

https://5e8b25de-f7a8-4b18-934e-9f3b0d54042e.htmlpasta.com/

It works fine with wp cli 1.4.1 without any errors.

Any ideas ?

A ready filter would be handy that runs the pseudo SQL query SELECT jobs WHERE status='waiting' AND 'nextrun' < time().

Occasionally, jobs may get stuck. Having a high limit of something like 30 mins (configurable) would help catch these.

• Add timeout setting via wp-config constant
• Check worker timing during loop
• Kill worker if over time limit and mark job as failed
• Log worker failure reason

I dropped all 4 of the files into the mu-plugins folder
class-command.php
class-job.php
connector.php
plugin.php

And on reload immediately the site died with a 500 error
PHP Fatal error: Class 'WP_CLI_Command' not found in wp-content/mu-plugins/class-command.php on line 8

Can you please update or add detail to your install instructions?

Currently the only way to get job failure data is to look at the main cavalcade output logs, this is useful when Cavalcade is only running on a single server, or a bunch of servers with a remote syslog, but when you need to find the failure data for a job that was run on one of many different servers it requires digging through multiple server logs to find the actual data.

It'd be ideal if instead of/in addition to logging Failed to shutdown worker. that the stdout and stderr streams were logged to the database.

WordPress.org currently runs Cavalcade with a modified Runner::check_workers() so that we log the server on which the job ran, we could add this functionality ourselves but Job->output and Job->error_output are both protected fields which prevents this.
Unfortunately there isn't an easy way to override the Worker class so we can't simply extend it as we've done with the Runner (And if we wanted to put stderr/stdout into their own field in the table, rather than a blob of data into content we'd also need to override Logger which again, isn't easily possible).