bernardphp / bernard Goto Github PK

Have been thinking about Middlewares and how they work. I like them since they are easy to create and the interface is simple, i dont like them when you have functionality that are shared between consuming and producing. Because of that i propose the following.

1. Use events where the middlewares are currently used consume, produce and exception.
2. Create a Strategy implementation for executing messages. which will be a layer between the Consumer and calling the method.
3. Remove the middleware notion.

This work make the BatchMiddleware look as following (mockup).

<?php

namespace Bernard\Listener;

use Bernard\Batch\Storage;

class BatchListener implements EventSubscriber
{
    protected $storage;

    public function __construct(Storage $storage)
    {
        $this->storage;
    }

    public function onProduce(Event $event)
    {
        if (false == $batch = $event->getEnvelope()->getStamp('batch')) {
            return;
        }

        $this->storage->register($batch);
    }

    public function onConsume(Event $event)
    {
        if (false == $batch = $event->getEnvelope()->getStamp('batch')) {
            return;
        }

        $this->storage->increment($batch, 'consumed');
    }

    public function onConsumeException(Event $event)
    {
        if (false == $batch = $event->getEnvelope()->getStamp('batch')) {
            return;
        }

        $this->storage->increment($batch, 'failed');
    }

    public static function getSubscribedEvents()
    {
        // .. subscribe to the right events.
    }
}

The convenience for this is that it will only requirering creating a single class for adding stuff of both the producer and consumer side.

the Strategy is to provide a extension point for executing jobs in different ways suchs as threads or forks with Spork.

Should we use PSR-4 when the votes passes?

The last release is the 14 April 2013 so maybe it can be great to add it. What do you think ? http://kr.github.io/beanstalkd/

It should not be a DBAL like connection but instead have methods like

createQueue
insertMessage

and so on to abstract the thing behind it away.

In some cases it exposes a security issue if the messages in a queue are not encrypted, thus can be read by a human.

Is it possible to implement encryption? Not meaning having it in the core, but having the possibility to implement it. Any ideas?

http://bernardphp.com/ any link produces 404

PSR-3 should be implemented.

I am always curious if I see such an inexplicable name. :)

Create ServiceResolvers that is Aware of a Container such as the one from Symfony or Pimple.

I was thinking to make a PR but maybe you prefer to not "pollute" the composer.json file and its packagist profile. What do you think ?

http://getcomposer.org/doc/04-schema.md#suggest

Would be useful to have RabbitMQ's exchange concept for other drivers too:

The core idea in the messaging model in RabbitMQ is that the producer never sends any messages directly to a queue. Actually, quite often the producer doesn't even know if a message will be delivered to any queue at all.

Instead, the producer can only send messages to an exchange. An exchange is a very simple thing. On one side it receives messages from producers and the other side it pushes them to queues. The exchange must know exactly what to do with a message it receives. Should it be appended to a particular queue? Should it be appended to many queues? Or should it get discarded. The rules for that are defined by the exchange type.

Source: http://www.rabbitmq.com/tutorials/tutorial-three-python.html

As Symfony\Console is the de-facto console application framework. It makes sense to require Symfony\Console for consule stuff (through the suggest block in composer.json).

Some of the class names have become very verbose

• PimpleAwareServiceResolver -> PimpleServiceResolver PimpleAwareResolver
• ConstainerAwareServiceResolver -> ContainerServiceResolver ContainerAwareResolver

Or remove the ServiceResolver suffix from all the extending resolvers.

Hi,
is there a way to pass additional arguments to receiver function? I am getting message, but for example i would like to pass console output, to make processing more verbose.

Do something like Controller::action

To create a "real" extension point we should add middleware like sidekiq for the Consumer and Producer. One kind of implementation could be https://gist.github.com/henrikbjorn/995438e67ab75c8fe243

The information returned in Driver::info() must always include the drivers name. Also when reaching a more stable version it must include the version of the driver.

• PhpRedisDriver
• PredisDriver
• DoctrineDriver
• SqsDriver
• IronMQ

In one of my current projects I use redis alot. Therefor I use phpredis, a redis-driver written in C. It can be found at https://github.com/nicolasff/phpredis. Is there any reason to not support it in Bernard?

It would be great if both clients (phpredis and Predis) could be supported and the hard dependency in composer on predis could be removed.

Dependency heavy integrations should be refactored out into its own packages.

Should take a client and a list of name => urls in the constructor, it will not be possible to create queues on the fly, they have to be declared before use.

This will also be the case for failed queue.

In order to prepare for Batches feature it is needed to add a pushMessages (bulk) method to the drivers and to the Queue (bulk).

There should be dispatched event through out the creating/deletion of messages etc. To make extending easy.

When starting with Bernard the ServiceResolver is hard to understand because a Service can be a lot of things. Instead it should be named something like "Receiver" and be explained as a simple Controller that runs asynchronously.

see https://scrutinizer-ci.com/g/henrikbjorn/Raekke/inspections/64a9d915-b68a-4b6e-b8c1-6ed7b90c16b8#inspectioncomment-2177948

It should have a name that makes it clear it is for simple use cases and have limitations.

I'm trying to figure out the proper way to use Bernard for consuming messages. I have the following code(using IronMQ driver):

    $router = new ContainerAwareRouter( $container, array( 'UpdatePostSnapshots' => 'update_snapshots_service' ) );
    $consumer = new Consumer( $router, new MiddlewareBuilder() );
    $consumer->consume( $queueFactory->create( 'post-update-snapshots-msg' ), array( 'max-runtime' => 900 ) );

I run it as a custom console command in Symfony. It works well, however IronMQ dashboard shows several thousand API requests within a few minutes(just for one consumed message).

How do I optimize this, so the worker would not go crazy over IronMQ with million requests a day for not a large queue.

It would be nice to be able to requeue a message but not resetting it to the end of the queue (if the driver supports it). This would make it easier to requeue messages we are working on when doing a hard shutdown

The format a Envelope/Message is serialized into is not compatible with Resque.

What needs to be done

• message key should be renamed to args.
• class key should be normalized by replacing \\ with :

The Core of Bernard have gotten quite fat with require-dev dependencies. Theese should be kept to a minimum for testing and the logger.

As 'Raekke' is apparently hard to pronounce for other than native Danish speakers, the project should proberly be renamed.

Currently creating the whole object graph for Bernard can be quite daunting and complex especially if you just want to get started. In order to rectify this i propose to add some sorft of builder, it would work as below.

I know it have been suggested before but have recently played around with it and changed my opinion about it :)

<?php

use Bernard\Driver\FlatFileDriver;
use Bernard\Builder;

$driver = new FlatFileDriver(__DIR__ . '/queue');

$b = Builder::create($driver)
    ->configureRouter(function ($router) {
        $router->add('MessageName', function ($message) {
            echo "i got a message.";
        });
    })
    ->configureConsumerMiddleware(function ($middlewareBuilder) {
        // add middlewares
    })
    ->configureProducerMiddleware(function ($middlewareBuilder) {
        // add middlewares
    })
;

$producer = $b->buildProducer();
$consumer = $b->buildConsumer();

The example is actually the most complex one, normally i dont think people will use this and add middlewares except for the normalt ones :)

Builder is a stupid name, open for suggestions.

What do others think?

Many drivers have support for a negativeAcknowledge. This means instead of relying on a backend implementing a timeout we can explicitly send a message back and say it could not be handled. This could be implemented in userland for doctrine, redis, file etc.

As Resque registers its workers so it possible to see what workers are running.

This would proberly allow some stuff to be removed or slimmed down ([] for arrays etc)

Some of the implemented services support to push messages with a delay specified. Would be awesome to support it.

When running the examples for a minute or so the process runs out of memory.

Could it be that the process does not free the memory used ?

Added a logger to the Consumer:consume while($this->tick(...)) loop and then it's quite clear that the memory usage only increases.

Is this only an issue with the example scripts or something jobs have to take care of ?

use case:

php phpredis.php consume
php phpredis.php produce

Would be great so see a RabbitMQ driver implemented (https://www.rabbitmq.com/).

When running example/in_memory.php or example/consumer.php signals arent processed.

Replace the manual creation of Connections with a system like Celery that creates Brokers/Connections based on a dsn.

Sidekiq has a real neat implementation, it will keep retrying with a exponential backoff.

https://github.com/mperham/sidekiq/wiki/Error-Handling

Docs are not relevant for the latest version of Bernard:

• NaiveSerializer- > SimpleSerializer
• Middleware
• etc.

It seems to me, the default interval of 5 seconds is without effect for this driver.

The 10 ms wait time between calls of "getMessages()" leads to a theoretical maximum of 100 API calls per second (of course, the requests take some time by themselves, so we're somewhat lower in practice).

Since IronMQ is priced by the number of API requests, this is a real concern.
Do you agree that this needs to be adressed? I'd be happy to provide a fix.

This would make the method names better and not like events.

<?php

class Somo
{
     public function onSendNewsletter(DefaultMessage $message)
     {
     }

     public function sendNewsletter(DefaultMessage $message)
     {
     }
}

I think it will be easier to understand if we add a get method (or something like that) especially when the consumer will consume the queue.

Replace:

$consumer->consume($queues->create('echo-time'));

By:

$consumer->consume($queues->get('echo-time'));

In the example:
https://github.com/bernardphp/bernard/blob/master/example/bootstrap.php#L76

What do you think ?

Example: https://scrutinizer-ci.com/g/scrutinizer-ci/scrutinizer/