Giter Club home page Giter Club logo

php-daemon's Introduction

PHP Simple Daemon

Create solid, long-running PHP daemon processes by extending the Core_Daemon class. Use a built-in timer to run your application in second or sub-second frequencies, or build servers using libraries like Socket and LibEvent. Create conventional single-process applications or choose true parallel processing in PHP with persistent background workers.

Note: For many reasons PHP is not an optimal language choice for creating servers or daemons. I created this library so if you must use PHP for these things, you can do it with ease and produce great results. But if you have the choice, Java, Python, Ruby, etc, are all better suited for this.

Note: I have not found time to invest in this project for the last few years. It's not modern PHP, but may still work for you. If anybody would like to maintain a fork please let me know and I will point to it here.

Requires:

  • PHP 5.3 or Higher
  • A POSIX compatible operating system (Linux, OSX, BSD)
  • POSIX and PCNTL Extensions for PHP

Daemon Monitoring

  • Over the last few years I've built and deployed many daemons based on this library as well as cron jobs of all sorts. Recently I launched https://cronitor.io -- a dead simple cron (and daemon!) monitoring service. With one line of code you can integrate Cronitor into your daemon and get email/sms alerts if anything goes wrong.

[NEW] Version 2.1 Released!

  • After a 9 month beta period, the code that was in the branch feature_abstract_ipc has been merged into master as v2.1
  • Version 2.0 available in the v2.0 tag.
  • Exciting features in 2.1 include:
    • A major refactoring of the Worker API includes pluggable IPC classes: Channel worker communication over any popular message queue or stick with the built-in SysV channel.
    • Improved, simpler worker debug shell with new, powerful commands and easier integration of custom breakpoints into your own worker code.
    • Centralize all the process forking, reaping and management code spread around the Core_Daemon and Core_Worker_Mediator classes into a simple ProcessManager plugin.
    • Dozens of other bug fixes and improvements towards simpler, clearer code in the core Daemon and Mediator classes.

Support & Consulting

  • Commercial support & consulting is available, including on-site support in the San Francisco Bay Area.
  • Contact me through GitHub for more details, including no-cost consultation.

##Notable Features:

  • ###We provide the event loop: Build your application free of boilerplate. You start with a working application right out of the box. Once you extend Core_Daemon and implement the 3 abstract methods (they can be empty), you have a working prototype.

  • ###Block or Clock: Your choice in 1 LOC Most daemon applications will either use a blocking API library (libevent, socket_listen, etc), or run on an internal clock with code that needs to be run every 5 seconds, or every second, or 5 times a second.

    You can implement a clock with 1 line of code or leave it out for an event loop built on async or blocking IO.

  • ###True parallel processing in PHP In a few lines of code you can create asynchronous background processes that let you keep your daemon process light and responsive. After you pass an object to the Worker API, you can call its methods normally. The API silently intercepts your method call and passes it to the object running in the background process. The call returns as soon as the Worker API intercepts it, and your daemon continues normally. When the background process completes the method call any onReturn callbacks you set are fired. And if things go wrong you've got the ability to enforce a timeout and easily retry the call.

    As an alternative to the persistent, long-running background workers, the Tasks API gives you a simple way to call any method in an ad-hoc background process that will exit when your method is complete.

    PHP Simple Daemon workers and tasks are simple and powerful multi-processing tools in a language with very few of them. (But don't go trying to build a PHP version of Node.js)

    https://github.com/shaneharter/PHP-Daemon/wiki/Worker-API

    https://github.com/shaneharter/PHP-Daemon/wiki/Task-API

  • ###Integrated Debugging Tools Debugging multi-process applications is notoriously painful and several integrated debugging tools are shipped with the library.

    Since you cannot run an application like this under xdebug or zend debugger, a debug console is provided that lets you set pseudo breakpoints in your code. Your daemon turns into an interactive shell that gives you the ability to proceed or abort as well as a dozen+ commands to figure out exactly what is happening at any given time. Dump function arguments, eval() custom code, print stack traces, the list goes on.

    In addition to the integrated debug console, the /scripts directory includes a useful signal_console app: Attach to your daemon and easily send and re-send signals. Checkout the PrimeNumbers application for an example of using a signal handler to mimic occasional real-world events.

    You'll also find the shm_console app that lets you attach to a shared memory address, scan for keys, view them, and even run a watch command that prints out a transactional log of creations, updates and deletes.

    https://github.com/shaneharter/PHP-Daemon/wiki/Debugging-workers

    https://github.com/shaneharter/PHP-Daemon/wiki/Debug-Tools

  • ###Simple Callbacks: Because decoupled is better. A simple jQuery-like API lets you add callbacks to daemon lifecycle events (think: startup, teardown, fork, etc) and create your own. Attach an event listener using on(), remove it using off(), and create your own using dispatch(). Like all PHP Simple Daemon APIs it accepts a Closure or any valid PHP Callback.

    https://github.com/shaneharter/PHP-Daemon/wiki/Using-callbacks-and-custom-events

  • ###Simple Plugins: Because code reuse is better. If you care more about building a reusable component with the ability to execute code during the daemon startup process before your application code is called than you do about decoupling, you can create a Plugin simply by implementing Core_IPlugin. Plugins are the easiest way to share code between multiple daemon applications and it can literally be implemented in 3 lines of code. We've got several general-purpose plugins on the drawing board to ship with the Core_Daemon library but currently we're shipping just one. The Ini plugin gives you an easy tool to read and validate any config files you ship with your application.

    https://github.com/shaneharter/PHP-Daemon/wiki/Creating-and-Using-Plugins

  • ###Lock files (and lock keys, and lock mutexes, and...) Several plugins are shipped with the library that implement different ways to create a lock for your daemon process. Running more than once instance of a daemon is often a problem and implementing a locking mechnism is often a headache. We've been paged at 2 AM when supervisord couldn't restart a daemon because of a stale lock file. We've bundled the Lock plugins to try to save you from that same fate. In all cases locks are self-expiring and you can chose between using a Memcache key, a lockfile, or a shared memory address. A faux lock plugin is also shipped to make your life easier during application development.

  • ###Automatic Restart Applications built with PHP Simple Daemon will automatically restart themselves after catchable fatal errors, and on a user-defined interval as a precaution to combat memory leaks, reinitialize resources, and rotate event logs. Auto-restart is only available when you run your app in "daemon mode" (eg -d at the command prompt). You're on your own when you run the app within your shell.

  • ###Built-in Event Logging The library ships with an insanely basic event log. While the features of off-the-shelf logging libraries are sometimes missed, we have a serious aversion to external dependencies and memory bloat. In any case, the built-in log provider writes messages and headers to a log file you supply. You can see an example of a simple rotator in both example daemons.

    If you have an internal logging tool, or just a favorite logging library, you can replace the internal tool as simply as overloading the log() method. Just be sure the only require parameter is the message being logged and everything internally that uses the event log -- Core_Daemon, signal handlers, the Workers API, etc -- will play nice.

    https://github.com/shaneharter/PHP-Daemon/wiki/Logging

  • ###Built-in Signal Handling Out of the box, your application will respond to 4 signals. You can add-to or overload that behavior by adding an ON_SIGNAL callback. The four built-in behaviors are:

    SIGINT kill -2 (or CTRL+C): Gracefully shutdown the daemon. Finish the current iteration of the event loop and signal any workers to finish their current tasks. If you're using workers, it could take as long as the maximum worker timeout to complete the process, though the lock file (if one is being used) is released before that so you could re-start the daemon while your workers finish up.

    SIGHUP kill -1: Gracefully restart the daemon using the process described for SIGINT. Useful after changing the application's code for example.

    SIGUSR1 kill -10: Dump a block of runtime statistics to the event log or stdout (or both, depending on how you configure logging). Includes things like current uptime, memory usage, event loop busy/idle statistics, and stats for any workers or plugins you have loaded.

    SIGCONT kill -18: If your daemon is currently blocked or sleeping, wake it up and continue. (Will always wake it up from a sleep(), may not always return from a blocking API call.)

    https://github.com/shaneharter/PHP-Daemon/wiki/Creating-Custom-Signal-Handlers

  • ###Command Line Switches You can run a '-H' help command when you run the Daemon. It will dump a help menu that looks like this, but can be easily overridden for your daemon:

Examples\PrimeNumbers\Daemon
USAGE:
 # run.php -H | -i | -I TEMPLATE_NAME [--install] | [-d] [-p PID_FILE] [--recoverworkers] [--debugworkers]
 
OPTIONS:
 -H Shows this help
 -i Print any daemon install instructions to the screen
 -I Create init/config script
    You must pass in a name of a template from the /Templates directory
    OPTIONS:
     --install
       Install the script to /etc/init.d. Otherwise just output the script to stdout.

 -d Daemon, detach and run in the background
 -p PID_FILE File to write process ID out to

 --recoverworkers
   Attempt to recover pending and incomplete calls from a previous instance of the daemon. Should be run under supervision after a daemon crash. Experimental.

 --debugworkers
   Run workers under a debug console. Provides tools to debug the inter-process communication between workers.
   Console will only be displayed if Workers are used in your daemon

php-daemon's People

Contributors

alecrabbit avatar bmatschullat avatar brunnels avatar burci avatar desarrolla2 avatar ekzobrain avatar googlerobot avatar mikulas avatar nmoskovkinmp avatar shaneharter avatar taeram avatar vitch avatar vojtabiberle avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

php-daemon's Issues

Endless Restart Loop

This is possibly related to #57.

When PHP Daemon tears itself down for an automatic reboot it invokes its own destruct method explicitly. The destruct method loops through worker and plugin aliases, invoking each of their teardown methods and removing the objects

        foreach(array_merge($this->workers, $this->plugins) as $object) {
            $this->{$object}->teardown();
            unset($this->{$object});
        }

However, the aliases are maintained in the arrays. I'm noticing that occasionally the PHP interpreter is invoking the destruct method again. In the second iteration, the workers and plugins are already unset so it attempts to call a method on an undefined property. The error that is raised causes PHP Daemon to attempt to restart itself.... Which calls the destruct method again and the whole party continues.

Because the original restart was actually successful in spawning a new instance of the daemon, you end up with one correctly functioning daemon and a few endless looping walking dead daemons.

I'm using version 2.1.1. Also, thanks for the library and the great documentation! Overall, this has been working great for me.

Socket-Server Daemon

Hi,

I'm really sorry to post such a question as an 'issue' but I can't get it working…
I want to have a daemon that cares about my socket server (php) and in the docs you've written about socket server. So I think it should work but I don't know HOW.

If I don't set a call repeat time and leave out my while(true) nothing will matter. If I leave while(true) the server doesn't respond to inputs. And without the daemon it works well with the while(true) and responds to input.

Maybe someone could write the most important things I should care about to run my socket server with your daemon.

Jan

Pluggable storage classes for the Worker API

  1. Abstract all of the SHM functionality into an SHM storage class.
  2. Create additional storage options (Redis, Memcached, etc) and implement the ability for application developers to choose a specific storage class. Create hueristic that will try to select the optional storage engine based on worker throughput and mean arg/return value size.

The idea here is that we've learned from this first release of the Worker API that shared memory has faults and is, you could say, not the most rock solid PHP API available. And if Redis or Memcached are available on the server, one or both could be a better option than SHM.

Daemon not destroying msg_queue and shm when restarting

I thought I had reported this issue before but I can't find it now.

To reproduce run in daemon mode and set a low $auto_restart_interval. Every time the daemon restarts it creates a new msg_queue and new shm mallocs for each worker. This will quickly eat up all your memory and the daemon will eventually be unable to restart.

Daemon::__destruct() does not call teardown() on all workers and plugins

Bug: This foreach loop in Daemon::__destruct() does not behave as expected:

foreach($this->workers + $this->plugins as $object) {
    $this->{$object}->teardown();
    unset($this->{$object});
}

The '+' operator performs an array union using keys, not the values. In this case the array indices may conflict, causing some array values to be excluded. See the note here: http://php.net/manual/en/language.operators.array.php#86379.

Impact: This bug prevents the teardown() function from being called on the complete set of workers and plugins, for example on the Lock_File plugin object. As a result, the lock file may not be unlinked, so restart() fails after the auto_restart_interval with the message "Core_Lock_File::set Failed. Additional Lock Detected."

Solution 1:

foreach($this->workers as $object) {
    $this->{$object}->teardown();
    unset($this->{$object});
}
foreach($this->plugins as $object) {
    $this->{$object}->teardown();
    unset($this->{$object});
}

Solution 2:

foreach(array_unique(array_merge($this->workers,$this->plugins)) as $object){
    $this->{$object}->teardown();
    unset($this->{$object});
}

Please Expose Core_Worker_Mediator->running_tasks

is_idle() is nice, but it would actually be nice to know how many tasks are running (or have a "worker slots available"), in order to more efficiently pop jobs off of a work queue in each execution state of the event loop.

Run loop stops executing in PHP 5.5.23 after task completes.

Some system information: this daemon is running on Mac OS X, using PHP 5.5 installed via homebrew. The current stable in homebrew is 5.5.23.

My daemon launches several tasks to download files from a remote host. I have a condition that occurs when, after whatever the first task is that finishes, the main runloop stops executing (or, at the very least, execute() isn't being called). All of the tasks eventually finish and I can still call log() on the parent, and if I look at a process list the daemon is still "running", but not actually doing anything and no new tasks are created.

Downgrading to PHP 5.5.20 fixes it.

I know this is a frustratingly vague bug report, so if there's anything else I can help you with I would be glad to provide assistance.

IRC Application

Is this deamon suitable for irc applications, and if so, where to start would be appreciated

Using things loaded from the ini plugin in setup()

Hi,

First up thanks for this code. Just playing around with it and it seems like a nice clear bit of code which does exactly what I need!

I've run into a problem because I am storing database config in a config file loaded by Core_Plugins_Ini. I want to create the database connection in my daemon's setup() function. However, due to the order that things happen the setup() in Core_Plugins_Ini hasn't been executed by the time my daemon's setup() function is called.

How would you normally suggest avoiding this problem? One solution would be to init the plugins before the main daemon (around https://github.com/shaneharter/PHP-Daemon/blob/master/Core/Daemon.php#L269 ) but I'm not sure if this would have unwanted side effects?

The other solution (which I'm currently using as a workaround) is to override run() to look like this:

public function run()
{
    $this->init_db();
    parent::run();
}

This seems pretty hacky though. I guess the other solution is to check in execute() whether the connection has been created and create it if now.

How do you recommend dealing with this situation?

Thanks!

Blocking mode - loop_interval not excepting NULL

In row 265 in Daemon.php the following seems to be mistaken: "if (is_numeric($this->loop_interval) == false)".

if loop_interval is NULL which is what is supposed to occur when working in blocking mode then this causes an exception. Removing this row works nicely so can you please check if this is indeed a typo or if I'm missing anything?

Thanks!

Core_Plugin_Ini::offsetSet throws exception even if $offset is valid

in

public function offsetSet($offset, $value)
    {
        if (is_scalar($offset))
            $this->contents[$offset] = $value;

        throw new Exception('Could not set INI value: $offset must be a scalar');
    }

You need to make that an if else as setting an INI will always crash with the Exception.

Project dead?

Hi!

The last commit on this project was 8 months ago, there are numerous issues without responses, no apparent development. I am using this tool heavily and am waiting for the changes promised for v2.2.

@shaneharter are you still maintaining this project? We need you :)

Thanks
tobias

Dead worker causing PHP Fatal error: Call to a member function teardown() on a non-object

If a worker process dies for some reason then after the timeout interval Daemon::__destruct() calls teardown() on a null object. The result is PHP Fatal error: Call to a member function teardown() on a non-object. This could be handled gracefully by checking that the worker object is non-null before calling teardown().

Bug:

$this->{$object}->teardown();
unset($this->{$object});

Solution:

if(isset($this->{$object})){
    $this->{$object}->teardown();
    unset($this->{$object});
}

Share memory lock broken

Description

Share memory lock plugin was broken.

Reproduce

To verification, just change the Lock_File in Example/LongPoll to Lock_Shm. You will be able to run more than one instance.

Error "The PCNTL Extension is not installed" but it is.

Hello !

The extension PNCTL is installed on my server (apache2, debian) and work on my scripts but when i try to start run.php of examples (PrimeNumbers), i get this error :

Date                  PID   Label         Message
[2013-11-10 14:57:02]  4524               Checking Dependencies... Failed:
[2013-11-10 14:57:02]  4524                 The PCNTL Extension is not installed
[2013-11-10 14:57:02]  4524               Examples\PrimeNumbers\Daemon is Shutting Down...

But effectively, in php -i and phpinfo() the PCNTL extension is enabled.

A little help or indication about this?

Thanks.

Missing version tags?

README.md indicates that version 2.1 has been released, but 2.0 is the only stable version listed on Packagist. There are a couple development 2.1 releases, but it's unclear which one is the release (if any). Can you please tag the v2.1 commit?

Defunct process

Recently I need to write a program to pull message from SQS constantly. Then I found this library to do it. I extended the abstract class Core_daemon and wrote main function of my program in execute(). When I ran the daemon process, I found the daemon process produced a lots of defunct processes. Is it normal situation? or something I‘ve done wrong?

SysV::put() Failed for call_id 2: Retrying. Error Code: 11

Hi,

I am building a solution that crawls some websites, and after a while the following messages are "dumped" in the log.

[2014-04-29 13:32:11] 6013 Workers SysV::put() Failed for call_id 2: Retrying. Error Code: 11
[29-Apr-2014 13:32:11 UTC] PHP Warning: msg_send(): msgsnd failed: Resource temporarily unavailable in /var/www/worker/includes/daemon/Core/Worker/Via/SysV.php on line 208 pid 6013

I am currently backtracing -and debugging the root cause, but also have tried to find an explanation of the Error code provided, so far I have not been able to find any documentation of the error code. Any ideas?

Non time-based loop?

Hi,

congrats on php-daemon! I've just started looking into the framework and find it quite cool and complete! Especially good the cluster-based functionality, very useful. A couple of questions/comments maybe you can help steer me:

  • my first daemon needs to poll a particular webservice queue which wasnt developed by us. Nonetheless, the webserver is "blocking", and when it reaches the last even it will "wait" 10 seconds... basically, I need a non-time-looped daemon. Would this be something possible to implement? Basically, whenever it reaches the end, start again... I could just put 1 second interval, but that would flood my e-mail box, and potentially give other complications? (Is there somewhere to disable these catches? would this be the best way?)
  • on another note, do you see any implications in porting the memcache portion of your code to use Redis instead? We have changed from memcache to redis for most of our use, and were thikning of disabling the memcache cluster, but we could always keep it for this daemon control (although i'd prefer to just port a lock plugin for redis).

Anyways, great work. I think this will be very useful in several jobs, and when combined with a message queue such as php-resque or whichever one anyone prefers, gives us a complete toolbox for work...

Core_Daemon->getOpt should load console arguments from argv

Using native getopt method from php it doesn`t let you alter the arguments list in order to get the corresponding arguments correctly.

For example, you may access the daemon from a framework, so "php bootstrap.php -d" won`t daemonize the process.

Automatic restart causes file handle to standard log file to remain open.

We encountered an issue when rotating and cleaning up log files.
It turned out that when a daemon (automatically) restarts, the log file handle is not closed.
This causes the new daemon process to inherit the handle and the file to remain open.
After the first restart the log file will be reopened causing the file to be opened twice.

Luckily on our system the new log file would get a handle value of 0 and be considered to be
STDIN and closed on subsequent restarts. We could not free up disk space by removing the file since it was still open. We had to truncate the file to free up disk space.

This can be easily fixed for the daemon. I will add a pull request for this.

Command line arguments not retained after automatic restart

I am using command line arguments to control the behavior of the daemon, but they are lost when the daemon restarts. This makes the auto-restart feature unusable for me.

The problem centers around Core_Daemon::command() where the filename of the script is built. Here some default arguments (-d --recoverworkers) are appended, but the original arguments are not mentioned at all.

Trying to run 2 PHP Daemons on the same box seems to create issues in msg q

I have 2 Damons, for 2 differnet purposes.
From ObjectMediatior.php I see the ftok is always the same, this means they both seem to want to use the same msg_q

$tmp = sys_get_temp_dir();
$ftok = sprintf($tmp . '/%s_%s', str_replace('/', '_',
$this->daemon->get('filename')), $this->alias);

Am I missing something about configuring my env to allow multple seperate
Daemons run on the same box.

Make it easier for developers to add their own Help documentation

Refactor the Core_Daemon::show_help() method to work like show_install_instructions().

Create two instance vars on Core_Daemon:

protected $help_usage = array();
protected $help_options = array();

Change the Core_Daemon::getopt() method to write to these arrays. Change the show_help() method to just implode these arrays and print them with a similar format to the existing code.

This will enable application developers to parse their own command line options by overloading getopt (and calling parent::getopt() when they're done), while writing help instructions that get displayed alongside the library-specific instructions.

"Creating Custom Signal Handlers" idea/solution

Considering the last paragraph of the wiki page:

If all of that was overly complex I apologize, but you can suffice to say this: Signal handlers can be complex and nuanced. But you should know that any code you create or spawn from within a signal handler will itself be unable to handle signals.

Would a possible solution be to create a callback of some sort and then toss it into an execution queue, to be run outside of the signal handler perhaps? It would introduce a slight delay to execution, but I believe it would allow working around that pitfall.

Allow creating constructors for the worker class

Hi i was wondering if it was possible to allow creating constructors for worker classes that implements Core_IWorker. Much like creating constructors for classes that implements \Core_ITask. It would be quite beneficial to have this feature.

Btw, i love the work you've done on PHP-Daemon. I always dreamt of an efficient creation and management of processes in PHP, and PHP-Daemon made that dream come true. Its a fantastic package overall. Many thanks for the hard work that you've done. Keep it alive and rocking !!

Very poor throughput with $loop_interval = 0

I am trying to create an application which reads messages off of a distributed queue and processes each of them. As there are very many messages coming in (thousands per second), high throughput and performance are essential for the app.

I looked into using this daemon framework as basis for the app, but found out that despite setting the $loop_interval to 0, the maximum number of iterations per second is 100. While this may be more than enough for other, less intense workloads, it is far too little for my use case.

The code comments suggest that setting the variable to 0 would result in the loop being executed continually without delay (except for the occasional ON_IDLE event), but I found out that even when with the value 0 there is a call to usleep() which wastes a huge amount of time.

The call is in Core_daemon::timer() on line 883 (as of this writing). There the process is set to sleep for 10ms, effectively limiting the maximum number of iterations per second to 100. The comments justify doing this by needing to "give the CPU a break", but 10ms is a looooong time in CPU terms.

Please reduce the length of the break or entirely remove it, so that high-throughput, high-performance apps can also benefit from this otherwise well-written framework. I suggest 100µs as delay in between iterations.

Error while running LongPoll example in OSX snowleopard

Core_Worker_Mediator::shm_init Failed. Could Not Read Header. If this problem persists, try running the daemon with the --resetworkers option

tried with the --resetworkers option

php run.php --resetworkers

but the error still persists ...

Any idea how it can be resolved?
Its running PHP5.4.8

Thanks

Integrate Gearman (or similar) to provide plugin-like feature to distribute workers across multiple servers and not just multiple process on the same server.

This is a large feature that I'm currently thinking about. I'm trying to sketch-out what the feature will look like, and how best to integrate it into the existing codebase.

Goals:

  1. Integrate an off-the-shelf queuing solution (eg Gearman, Kestrel, Beanstalkd, ActiveMQ, etc) that could be used instead of the existing SysV message queue.
  2. Take advantage of the ability an OTS queue provides to distribute workers across multiple machines in a cluster.
  3. Implement this in a way that is idiomatic to PHP Simple Daemon and provides for easily moving between queueing providers.

Implementation Thoughts:

  1. Create a new directory Core/Worker/Queues
  2. Move all of the SysV-specific code out of Core_Worker_Mediator into Core_Worker_Queues_SysV
  3. Add an optional argument onto the Core_Daemon::create_worker() method to choose a specific queue.
  4. Probably will need to create a unique interface for each queue, or possibly create a queue object in your daemon that you inject into the Worker (as the optional argument mentioned previously) or something else similar to deal with the unique needs of different queues. SysV needs a malloc for example. Gearman would need to know the ip/port of Gearmand. That sorta thing.
  5. I sorta like the dependency-injection style. That would work equally as well for workers implemented as closures as it does for IWorker objects. You would instantiate a Queue object before you create the worker, configure it however you want, and then pass it to create_worker()
  6. An emphasis will be placed on making it easy to move between different Queue types, but that may not apply to code written on v2.0 workers. It may be possible, but i'm not going to worry about that. You may have a small price if you want to upgrade to 2.1-style queues. If the penalty is too high, I'd consider creating this feature in a 3.0 branch and still supporting bug-fixes on the 2.x branch.

Any comment/input is appreciated. Is this a feature that interests you? Would you be interested in helping on any part of it? In beta testing?

I'd expect this feature to be on a Jan/Feb 2013 timeline.

multiple workers cannot have names that start with the same letter

If you have multiple workers their names can't start with the same letter. It's case sensitive. Took me all morning before I tracked down what was causing the error.

This stems from using ftok which takes a single character string on line 538 of Mediator.php. Some error checking should be added to check for this scenario and throw an exception.

$this->guid = ftok($ftok, $this->alias[0]);
$this->worker('RecordWorker', new RecordWorker(), new Core_Worker_Via_SysV(128 * 1024 * 1024));
$this->worker('RecordUpdater', new RecordUpdateWorker(), new Core_Worker_Via_SysV(110 * 1024 * 1024));
[2014-04-24 11:39:48] 11498               Application Startup Complete. Starting Event Loop.
[2014-04-24 11:39:48] 11503 RecordUpdater  Worker Process Started

Warning: msg_send(): msgsnd failed: Identifier removed in /home/br64538/workspace/daemon/lib/vendor/PHP-Daemon/Core/Worker/Via/SysV.php on line 208

Call Stack:
    0.0002     233184   1. {main}() /home/br64538/workspace/daemon/lib/daemon/run.php:0
    0.9073    6718928   2. Core_Daemon->run() /home/br64538/workspace/daemon/lib/daemon/run.php:15
    0.9075    6720472   3. SNUpdateDaemon->execute() /home/br64538/workspace/daemon/lib/vendor/PHP-Daemon/Core/Daemon.php:414
    0.9075    6720704   4. Core_Worker_ObjectMediator->getStalexChanges() /home/br64538/workspace/daemon/lib/daemon/SNUpdateDaemon.class.php:105
    0.9075    6720992   5. Core_Worker_Mediator->__call() /home/br64538/workspace/daemon/lib/daemon/SNUpdateDaemon.class.php:105
    0.9077    6760544   6. Core_Worker_Mediator->call() /home/br64538/workspace/daemon/lib/vendor/PHP-Daemon/Core/Worker/Mediator.php:851
    0.9077    6761112   7. Core_Worker_Via_SysV->put() /home/br64538/workspace/daemon/lib/vendor/PHP-Daemon/Core/Worker/Mediator.php:824
    0.9078    6763192   8. msg_send() /home/br64538/workspace/daemon/lib/vendor/PHP-Daemon/Core/Worker/Via/SysV.php:208

[2014-04-24 11:39:48] 11498 RecordWorker  SysV::put() Failed for call_id 2: Retrying. Error Code: 43
[2014-04-24 11:39:48] 11503 RecordUpdater  getStalexChanges() is Not Callable.

Lock File plugin not working properly

It is possible to launch multiple instances of daemon even when using file lock plugin. For example I run one daemon instance, than after Core_Daemod::loop_interval seconds i can launch more instances of the same daemon. I don't understand the logic of TTL checking, till it equals to Core_Daemod::loop_interval, which is usually a few seconds... I commented out lines 85-87 of Core/Lock/File.php to use this plugin for now

Daemon working with rabbitMQ dropping calls

Hi
I'm having problem of getting error:
"Dropped Call. Requeuing Call # To {$call->method}"
And "Dropped Call. Requeue threshold reached.

I can't figure out what is the root cause for that behavior happening more over this is happening occasionally in non consistent way.

The function which triggers this is "public function garbage_collector()".
I tried to do restart when it is happening, but to put the problematic call back to daemon queue by the already written line code:
$this->call($call) (and)
$this->daemon->restart(); (my additional line code)

Questions is, if after daemon restart we already have the same daemon queue state as before or it is probably being initialized?

If we don't maintain daemon's queue state between daemon restart how can I return the call to rabbitMQ that will temporary store the call until daemon will be available again to handle the call.

This should happen somehow from this scope of code (Mediator.php)

Provide a way to kill / restart daemon processes

I was thinking it would be nice to have an additional commandline option which would kill (or possibly restart) any processes (started with -d) that are currently running for a given daemon.

My use-case is because I'm still developing the code for my daemons. So occasionally I deploy new code to the server and have to do a kill pid to get the new code read in and executed. It would be nice if there was a way to do this without first looking up the pid etc...

Daemon crashes after a certain amount of time

I've build a deamon with several workers with PHP-Daemon v2.0
At a certain point of time (after approximately 8 hours with 8 workers) the Daemon reaches an "Fatal Error" and tries to restart the worker processes. There's probably something wrong with the memory allocation. Because if I restart the Worker (and automaticly flush the memory) it works like before.

See in the detail here:

[2015-06-10 05:56:04] 13940 mitchell Message Queue Error 13: Permission denied]
[2015-06-10 05:56:04] 13940 mitchell Incrementing Error Count for catchall to 24
[2015-06-10 05:56:04] 13940 mitchell Message Queue Error 13: Permission denied
[2015-06-10 05:56:04] 13940 mitchell Incrementing Error Count for catchall to 25
[2015-06-10 05:56:04] 13940 mitchell Message Queue Error 13: Permission denied
[2015-06-10 05:56:04] 13940 mitchell IPC 'catchall' Error Threshold Reached
[2015-06-10 05:56:04] 13940 mitchell Fatal Error: Worker process will restart
[2015-06-10 05:56:04] 13940 WheresAppDaemon\Daemon is Shutting Down...
[2015-06-10 05:56:07] 13940 Restart Happening Now...

Date PID Label Message
[2015-06-10 05:56:32] 18615 Invalid Alias. Identifiers must be scalar.
[2015-06-10 05:56:32] 18615 WheresAppDaemon\Daemon is Shutting Down...
[2015-06-10 05:56:34] 18615 Restart Happening Now...

Date PID Label Message
[2015-06-10 05:56:36] 18620 Invalid Alias. Identifiers must be scalar.
[2015-06-10 05:56:36] 18620 WheresAppDaemon\Daemon is Shutting Down...

Also when the Daemon tries to restarts itself I got the message Identifiers must be scalar. Maybe this is related to the fact that there's no memory left for the Daemon. I run a Vagrant hashicorp/precise32 Ubuntu machine with 2048MB of memory.

PHP Unknown: Declaration of Core_Worker_ObjectMediator::check_environment()

Hi,
I get this error when I run a example or when I execute my worker.

PHP Unknown: Declaration of Core_Worker_ObjectMediator::check_environment() should be compatible with Core_Worker_Mediator::check_environment(array $errors = Array) in lib/PHP-Daemon/Core/Worker/ObjectMediator.php on line 12 pid 2483

To reapir it I'have changed the file Daemon/Core/Worker/ObjectMediator.php

-- public function check_environment() {
-- $errors = array();

with

++ public function check_environment(Array $errors = array()) {
++ //$errors = array();

ON_SHUTDOWN hook not called during automatic restart

In my application I have registered a handler for the ON_SHUTDOWN hook, which cleans up some network connections. During regular shutdown, the hook is executed fine, but when the daemon is automatically restarted, the hook is not called, resulting in lingering connections.

The reason for this is that the hook is dispatched from the destructor of the Daemon class, which in the case of an automatic restart is called when the exit call is made in the restart() method. Unfortunately, for non-intuitive reasons, right before the exit call is made, the array containing all the callbacks is cleared, so once the destructor runs, there aren't any hooks leftover to call.

tutorial in implementing shaneharter/php-daemon

Hi,I want to try this php-daemon,but I am having difficulty on how to implement this,I created as socket listener in php and I want to apply this php-daemon to my project.because my listener will exit if there is no client connection and I re-run this by using task-scheduler in every 1 min.but I don't want to exit my script I want to run forever. I hope you can help me

Thank you in advance.

Logs don't rotate for a running process

Hi,

Since you store the handle to the log file in a static variable the file is only opened once per process. In my case I have a process which has run across a date boundary and I'm using an implementation of log_file() based on date('Ymd').

Since the handle is already there the code never re-calls log_file() and doesn't find out that there should be a new filename so it carries on writing to the old log file.

Possible solutions would be:

  • Closing the handle every call to log
  • Adding another variable which stores the filename for the current handle and if this is different to that returned by log_file() then close the handle and open a new one
  • ...or something else?

I think I prefer the second option and can implement it with a pull request if you like but maybe I'm missing something and there is a better third option?

Thanks,

Kelvin :)

Unit tests for Core_IWorkerVia

I would like to start work on adding a Core_Worker_Via_Memcache class. It would make it much simpler to implement this new class if there were some unit tests to run to ensure the results from each method call are behaving as intended.

How do I use this in my case?

Ok, to start off, I know this isn't an issue or something of that kind but since this package doesn't have a huge following on any forum or whatever, I figured I may aswel try here.

I want to make a daemon script in PHP that basically runs for every user I have in my application. I also want my users to have the ability to stop such a process whenever they want. And also start it back up again.

Is this the package I've been looking for? The average run time of the script is like 2 hours. Sometimes they run for 5 minutes and the user stops it, sometimes they run for a full week.

Thanks for those that considered answering and to those who read this unstructured message. I use everything ranging from Job Queue's to Cron Jobs but I can't seem to find something that completely suits my needs.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.