zeromq / perlzmq Goto Github PK

The tests for this module core dump against the latest development version of libzmq.
Please look into it.

http://www.cpantesters.org/cpan/report/2ddd20d2-c664-11e4-adfb-e2d3ebda7229
http://www.cpantesters.org/cpan/report/98867f82-c67d-11e4-adfb-e2d3ebda7229
http://www.cpantesters.org/cpan/report/e42745ce-c674-11e4-b221-9e126cbd7f71

We can't use it, if we can't install it.

Someone should fix that.

Warning: Prerequisite 'Alien::FFI => 0.12' for 'PLICEASE/FFI-Platypus-0.47.tar.gz' failed when processing 'PLICEASE/Alien-FFI-0.17.tar.gz' with 'make_test => NO'. Continuing, but chances to succeed are limited.
Configuring P/PL/PLICEASE/FFI-Platypus-0.47.tar.gz with Build.PL
Can't locate Alien/FFI.pm in @inc (@inc contains: inc /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at inc/My/ModuleBuild.pm line 6.
BEGIN failed--compilation aborted at inc/My/ModuleBuild.pm line 6.
Compilation failed in require at Build.PL line 22.
BEGIN failed--compilation aborted at Build.PL line 22.
Warning: No success on command[/usr/bin/perl Build.PL --installdirs site]
PLICEASE/FFI-Platypus-0.47.tar.gz
/usr/bin/perl Build.PL --installdirs site -- NOT OK
CALID/ZMQ-FFI-1.11.tar.gz
Has already been unwrapped into directory /root/.cpan/build/ZMQ-FFI-1.11-0
Warning: Prerequisite 'FFI::Platypus => 0.33' for 'CALID/ZMQ-FFI-1.11.tar.gz' failed when processing 'PLICEASE/FFI-Platypus-0.47.tar.gz' with 'writemakefile => NO '/usr/bin/perl Build.PL --installdirs site' returned status 512'. Continuing, but chances to succeed are limited.
Configuring C/CA/CALID/ZMQ-FFI-1.11.tar.gz with Makefile.PL
Can't locate FFI/Platypus.pm in @inc (@inc contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at Makefile.PL line 17.
BEGIN failed--compilation aborted at Makefile.PL line 17.
Warning: No success on command[/usr/bin/perl Makefile.PL INSTALLDIRS=site]
CALID/ZMQ-FFI-1.11.tar.gz
/usr/bin/perl Makefile.PL INSTALLDIRS=site -- NOT OK
Failed during this command:
PLICEASE/Alien-FFI-0.17.tar.gz : make_test NO
CALID/ZMQ-FFI-1.11.tar.gz : writemakefile NO '/usr/bin/perl Makefile.PL INSTALLDIRS=site' returned status 512
PLICEASE/FFI-Platypus-0.47.tar.gz : writemakefile NO '/usr/bin/perl Build.PL --installdirs site' returned status 512

in some situations during global destruction, FFI::Platypus can dlclose libzmq before Socket/Context DEMOLISH routines have run... so when they do actually run they try to make ffi calls on a library that's no longer loaded leading to... kaboom.

this can (hopefully) be worked around by doing the FFI::Platypus attach equivlant of the FFI::Raw::Ptr trick of overriding the package DESTROY with a real underlying ffi destroy function:

    use base qw(FFI::Raw::Ptr);

    *_foo_new = FFI::Raw -> new(
      $shared, 'foo_new',
      FFI::Raw::ptr
    ) -> coderef;

    sub new {
      bless shift -> SUPER::new(_foo_new());
    }

    *DESTROY = FFI::Raw -> new(
      $shared, 'foo_free',
      FFI::Raw::void,
      FFI::Raw::ptr
    ) -> coderef;

    1;

    package main;

    my $foo = Foo -> new;

package Foo;

use FFI::Platypus::Declare;

attach [ foo_new => 'new' ] => [] => 'pointer';
attach [ foo_free => 'DESTROY' ] => [ 'pointer' ] => 'void';

1;

package main;

my $foo = Foo->new;

Module cannot install because tests failed.
Tests failed because it expects an English error message ("invalid argument") and I have a french message ("Argument invalide").

Below is an extract of cpanm logs so you can point the issue.

...
Manifying blib/man3/ZMQ::FFI::Versioner.3
Manifying blib/man3/ZMQ::FFI::ZMQ3::Socket.3
Manifying blib/man3/ZMQ::FFI::Constants.3
PERL_DL_NONLAZY=1 /home/myHome/perl5/perlbrew/perls/perl-5.14.3/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t

#   Failed test 'invalid socket type dies with EINVAL'
#   at t/errors.t line 15.
# expecting: Regexp ((?^i:invalid argument))
# found: zmq_socket: Argument invalide at (eval 40) line 5.
#  at /home/myHome/.cpanm/work/1382127763.6273/ZMQ-FFI-0.06/blib/lib/ZMQ/FFI/SocketBase.pm line 36.

#   Failed test 'invalid endpoint dies with EINVAL'
#   at t/errors.t line 21.
# expecting: Regexp ((?^i:invalid argument))
# found: zmq_connect: Argument invalide at (eval 48) line 5.
# Looks like you failed 2 tests of 2.
t/errors.t ..... 
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/2 subtests 
t/fd.t ......... ok
t/gc.t ......... ok
t/multipart.t .. ok
t/options.t .... ok
t/pubsub.t ..... ok
t/send_recv.t .. ok

This for example could be needed for non-blocking $sock->send_multipart($string, ZMQ_DONTWAIT).

root@dev-ipbx-spb-3:/usr/local/share/perl/5.10.1/ZMQ/FFI# diff -u SocketBase.pm.orig SocketBase.pm
--- SocketBase.pm.orig  2014-08-14 05:52:24.000000000 -0700
+++ SocketBase.pm       2014-08-14 05:35:25.000000000 -0700
@@ -114,7 +114,7 @@
     }

     for my $i (0..$#parts-1) {
-        $self->send($parts[$i], ZMQ_SNDMORE);
+        $self->send($parts[$i], $flags | ZMQ_SNDMORE);
     }

     $self->send($parts[$#parts], $flags);

First of all, thanks for working on this library.

Second, I was trying this out on Mac, and I noticed that it wouldn't work correctly because you explicitly search for libc.so.6 in SocketBase.pm and libzmq.so in Util.pm, and dynamic libraries end with dylib instead of so on OS X.

I'm not very familiar with libffi, so I don't know if there's a simpler way to do this, but I think you can easily fix this by just adding libzmq.dylib and libzmq.3.dylib to your @sonames list in Util.pm and also checking for libc.dylib in SocketBase.pm.

In the meantime, I've just worked around this by adding so symlinks to the dylib files on my machine.

right now this is always set to 1 in DEMOLISH, but instead we should honor the user value if set.

currently for a user to have their own linger value take effect they also need to explicitly call close/destroy themselves, which probably violates the principal of least surprise

Hi

I have DEALER to REP flow. Client has throttling (no more than 4 async requests at the same time) and monitors socket for asynchronous replies. Server is synchronous.

Client script gets stuck at random point. For example client produced 27 requests, server processed 27 requests but only 23 were received by client and throttling kicks in locking script forever. I've debugged this and IO callback on socket is not called anymore after receiving 23rd message.

Environment:

Perl 5.14.4, 64bit, no threads
ZMQ::FFI - 1.11
AnyEvent - 7.08 (no EV or other backends installed)
FFI::Platypus - 0.40

Server:

#!/usr/bin/env perl

use strict;
use warnings;

use Data::Dumper;
use ZMQ::FFI;
use ZMQ::FFI::Constants qw(ZMQ_REP);

my $context = ZMQ::FFI->new();
my $responder = $context->socket(ZMQ_REP);
$responder->bind('tcp://*:5555');

while (1) {
    my $msg = $responder->recv();
    print "processing $msg\n";
    $responder->send($msg);
}

Client

#!/usr/bin/env perl

use strict;
use warnings;

use AnyEvent;
use ZMQ::FFI;
use ZMQ::FFI::Constants qw(ZMQ_DEALER);

my $context = ZMQ::FFI->new();
my $requestor = $context->socket(ZMQ_DEALER);
$requestor->connect('tcp://localhost:5555');
# ...more servers can be added

my $tasks = 100_000;
my ($req, $rep) = (0, 0);
my ($in, $out);

$out = AnyEvent->idle(
    'cb' => sub {

        # throttle requests, only 4 at a time
        return if $req - $rep >= 4;

        # send request, emulate REQ frame
        $requestor->send_multipart(['', ++$req]);
        print "sent $req\n";

        # do this amount of tasks
        $out = undef if $req == $tasks;
    }
);

$in = AnyEvent->io(
      'fh' => $requestor->get_fd,
      'poll' => 'r',
      'cb'   => sub {

          # expect multiple answers
          while ($requestor->has_pollin) {

             # receive response
             my @msg = $requestor->recv_multipart();
             $rep++;
             print "received $rep\n";

             # finish if every task was processed
             exit if $req == $tasks and $rep == $tasks;
         }
      }
   );

AnyEvent::Loop->run();

I'm new to 0MQ so please forgive me if this is not a bug in ZMQ::FFI but bad logic in my code.

device.t hangs with Socket 2.016, does not hang after upgrade to 2.018.

If the perl used to run gen_zmq_constants.pl was build with clang then tcc will fail:

open("/usr/bin/../lib/clang/3.7.0/include/stdint.h", O_RDONLY) = 5
read(5, "/*===---- stdint.h - Standard he"..., 8192) = 8192
close(5)                                = 0
write(2, "In file included from <string>:2"..., 200In file included from <string>:2:
In file included from /home/green.sophos/ioanrogers/git/zeromq4-x/include/zmq.h:87:
/usr/bin/../lib/clang/3.7.0/include/stdint.h:31: error: function pointer expected
) = 200

As in #35, this is more of a TinyCC issue but it should probably at least be documented. Or perhaps have the build abort if it sees perl was built with clang.

Workaround: I'm using plenv so I build another perl with gcc and set it as the default perl for my zmq-ffi clone (plenv local 5.22.1-gcc)

I'm probably in a small niche by using a clang-built perl on Linux but, IIRC, clang is the default on MacOSX.

Taking this and #35 together, is it feasible/worthwhile to use something other than TinyCC?

My program uses send/recv (ZMQ_REQ,ZMQ_REP), it's OK.

But when the perl script finish, the process never finish.
Below is the stack trace, it's in zmq. My test is on lib zmq v2

(gdb) bt
#0  0x00007f66f5860c13 in poll () from /lib/libc.so.6
#1  0x00007f66f2ebcdf0 in zmq::signaler_t::wait (this=<value optimized out>, 
    timeout_=-1) at signaler.cpp:145
#2  0x00007f66f2eadaae in zmq::mailbox_t::recv (this=0x271da08, 
    cmd_=0x7fff343a1380, timeout_=-1) at mailbox.cpp:74
#3  0x00007f66f2ea759d in zmq::ctx_t::terminate (this=0x271d970) at ctx.cpp:146
#4  0x00007f66f2ec800c in zmq_term (ctx_=0x271d970) at zmq.cpp:292
#5  0x00007f66f494ed78 in ffi_call_unix64 () at src/x86/unix64.S:75
#6  0x00007f66f494e51c in ffi_call (cif=0x271abf0, fn=<value optimized out>, 
    rvalue=<value optimized out>, avalue=<value optimized out>)
    at src/x86/ffi64.c:492
#7  0x00007f66f4949d7c in XS_FFI__Raw_call ()
   from /home/jeanpat/perl5/perlbrew/perls/perl-5.14.3/lib/site_perl/5.14.3/x86_64-linux/auto/FFI/Raw/Raw.so
#8  0x0000000000492961 in Perl_pp_entersub ()
#9  0x0000000000490f03 in Perl_runops_standard ()
#10 0x0000000000434420 in Perl_call_sv ()
#11 0x000000000049c22b in Perl_sv_clear ()
#12 0x000000000049c906 in Perl_sv_free2 ()
#13 0x000000000049a88b in S_visit ()
#14 0x000000000049cdae in Perl_sv_clean_objs ()
#15 0x00000000004384a5 in perl_destruct ()
#16 0x000000000042035c in main ()
(gdb) 

Hey Dylan,

I put together some code to show the issue.

use ZMQ::FFI;
use ZMQ::FFI::Constants qw(ZMQ_REQ);
use Data::Dump qw(dump);

my $c = ZMQ::FFI->new();
my @zmqver = $c->version();

print "libzmq=@zmqver\n";

foreach (1..2) {
    my $s = $c->socket(ZMQ_REQ);

    $s->bind("ipc://test-" . $_);
    $s->close();
    $s = undef;
}

print dump($c);

Output:

libzmq=4 2 0
bless({
  _pid => 9598,
  _tid => 0,
  _version_parts => [4, 2, 0],
  context_ptr => 43981008,
  die_on_error => 1,
  last_errno => 0,
  sockets => [
    bless({
      _pid => 9598,
      _tid => 0,
      _zmq_msg_t => 44137312,
      die_on_error => 1,
      last_errno => 0,
      socket_ptr => -1,
      sockopt_sizes => { int => 4, sint64 => 8, uint64 => 8 },
      soname => "libzmq.so",
      type => 3,
    }, "ZMQ::FFI::ZMQ4_1::Socket"),
    bless({
      _pid => 9598,
      _tid => 0,
      _zmq_msg_t => 44145888,
      die_on_error => 1,
      last_errno => 0,
      socket_ptr => -1,
      sockopt_sizes => { int => 4, sint64 => 8, uint64 => 8 },
      soname => "libzmq.so",
      type => 3,
    }, "ZMQ::FFI::ZMQ4_1::Socket"),
  ],
  soname => "libzmq.so",
}, "ZMQ::FFI::ZMQ4_1::Context")

potential memory leak?

so:

use ZMQ::FFI qw(ZMQ_REQ ZMQ_REP);

instead of

use ZMQ::FFI;
use ZMQ::FFI::Constants qw(ZMQ_REQ ZMQ_REP);

Rationale
you always need one or more zmq constants, so having ZMQ::FFI implicitly import and expose available ZMQ::FFI::Constants makes sense as a convenience feature. Can still use ZMQ::FFI::Constants independently if desired.

all API methods should be declared in SocketRole

BSD:
http://www.cpantesters.org/cpan/report/1509eddc-c6b5-11e4-adfb-e2d3ebda7229
http://www.cpantesters.org/cpan/report/ef6c33f2-c64e-11e4-b221-9e126cbd7f71
http://www.cpantesters.org/cpan/report/1c8770e0-c659-11e4-b221-9e126cbd7f71
http://www.cpantesters.org/cpan/report/ed9fd3d8-c632-11e4-b221-9e126cbd7f71

Linux:
http://www.cpantesters.org/cpan/report/2de30988-c624-11e4-aee6-60e877f6a711
http://www.cpantesters.org/cpan/report/b0a58b3a-c623-11e4-8ad5-062978f6a711

I'm attempting to ZMQ::FFI to multicast some binary data but it appears the pgm (or epgm) transports are not supported.

Interestingly, the script doesn't die at the context bind but I do see this:

zmq_bind: Protocol not supported at /usr/local/share/perl/5.18.2/ZMQ/FFI/ErrorHelper.pm line 83.
        ZMQ::FFI::ErrorHelper::fatal(ZMQ::FFI::ZMQ3::Socket=HASH(0x27a7640), "zmq_bind") called at /usr/local/share/perl/5.18.2/ZMQ/FFI/ErrorHelper.pm line 52
        ZMQ::FFI::ErrorHelper::check_error(ZMQ::FFI::ZMQ3::Socket=HASH(0x27a7640), "zmq_bind", -1) called at /usr/local/share/perl/5.18.2/ZMQ/FFI/ZMQ3/Socket.pm line 213

My libzmq is the current 4.2 and ZMQ::FFI was installed via CPAN. For what it is worth ZMQ and ZMQ::LibZMQ3 modules were also installed via CPAN but I was getting an error I could not track down so I decided to switch to this module because it is what the examples use on the zeromq website (and it does seem to be more up to date).

Also, bind follows the pgm example URL in the example snippet at the bottom of http://api.zeromq.org/4-2:zmq-pgm except I use a the network device (e.g. eth0) instead of an IP. Either should work as far as I can tell.

Are pgm and/ epgm transport not supported for some reason?

Hi,

I use the perl binding zmq3. It seems there won't be a zmq4. It will be the 'zmq-fi' for zmq4.
I don't see in the documentation the function 'zmq_poll'. I'm using it a lot to watch for events. Do you plan to implement it in some way ?

Thanks for your work.

zmq_device in zeromq v2

http://www.cpantesters.org/cpan/report/82a6d9fa-c691-11e4-81ee-a225e0bfc7aa
http://www.cpantesters.org/cpan/report/81c2a190-c691-11e4-81ee-a225e0bfc7aa
http://www.cpantesters.org/cpan/report/dc0274fa-c632-11e4-a2b5-449ce0bfc7aa
http://www.cpantesters.org/cpan/report/dfe98c8a-c62c-11e4-8bfc-ee8be0bfc7aa
http://www.cpantesters.org/cpan/report/6bda7bfc-c699-11e4-adfb-e2d3ebda7229
http://www.cpantesters.org/cpan/report/6bda7bfc-c699-11e4-adfb-e2d3ebda7229

Seems to only occur on BSD. One of the CPAN testers provided this info:

I reproduced the failure on the same box using cpanminus.

Wstat: 134 means ( after 134 & 127 ) signal 6 was generated which is abort.

The box does have both libffi and libzmq installed.

libffi-3.0.13 Foreign function interface
zeromq-4.0.4 The ZeroMQ messaging library

I have a very simple proxy script, like this:

use ZMQ::FFI qw(ZMQ_PULL ZMQ_XPUB);

my $context = ZMQ::FFI->new();

my $frontend = $context->socket(ZMQ_PULL);
$frontend->bind('ipc:///tmp/frontend.ipc');

my $backend = $context->socket(ZMQ_XPUB);
$backend->bind('ipc:///tmp/backend.ipc');

$context->proxy( $frontend, $backend );

This seems to work just fine, but after proxying lots of messages the process has CPU usage of 100% and seems stuck there, even when no messages are coming through.

This is perhaps a problem with the underlying zmq_proxy function, and nothing at all to do with perl bindings, but I thought I'd trying mentioning it as an issue here nonetheless. If there is anything I am doing wrong or a potential way to resolve this please advise. Thanks!

Note: of course one way to resolve it is to use pure perl, not the proxy function, like so:

while (1) {
    $backend->send( $frontend->recv );
}

That works and doesn't cause the CPU spike.

When a context goes out of scope it will be destroyed even if there are still sockets around that use it. Segmentation faults await.

sub f {
    my $zmq = ZMQ::FFI->new();
    my $socket = $zmq->socket(ZMQ_PUB);
    $socket->connect('ipc:///tmp/x');
    $socket;
}

my $socket = f();
$socket->send('foo'); # segmentation fault

Reproduce:
See https://gist.github.com/calid/18791dd558947d04a1e5 for output showing the segfault and code to reproduce it.

Workaround:
Use EV version 4

Cause:
???

We have the following bug reported to the Debian package of ZMQ-FFI:
https://bugs.debian.org/835412

As you can see from the attached and linked to logs, t/proxy.t tends to hang
without any output.

Thanks for considering,
gregor herrmann,
Debian Perl Group

See http://www.cpantesters.org/cpan/report/500ee0e0-a20c-11e4-b8cb-1593e0bfc7aa for a sample fail report. It seems that the stringified errno message is unexpected in the test case ("Das Argument ist ungültig" vs. "Invalid argument").

Automatic build fails on some of my smokers:

Output from '/opt/perl-5.28.1/bin/perl Makefile.PL':

Can't locate FFI/Platypus.pm in @INC (you may need to install the FFI::Platypus module) (@INC contains: /var/tmp/cpansmoker-1023/2019030311/CPAN-Reporter-lib-7x_i /home/cpansand/.cpan/build/2019030311/FFI-CheckLib-0.23-1/blib/arch /home/cpansand/.cpan/build/2019030311/FFI-CheckLib-0.23-1/blib/lib /opt/perl-5.28.1/lib/site_perl/5.28.1/x86_64-linux /opt/perl-5.28.1/lib/site_perl/5.28.1 /opt/perl-5.28.1/lib/5.28.1/x86_64-linux /opt/perl-5.28.1/lib/5.28.1 .) at Makefile.PL line 16.
BEGIN failed--compilation aborted at Makefile.PL line 16.

It looks like FFI::Platypus should be specified as a configure_requires dependency in the META.* files.

gen_zmq_constants.pl ends with a segfault on SELinux enabled systems because tcc violates the execheap policy.

strace snippet:

open("/home/green.sophos/ioanrogers/.plenv/versions/5.22.1/lib/perl5/site_perl/5.22.1/auto/share/dist/Alien-TinyCC/lib/tcc/include/stddef.h", O_RDONLY) = 5
read(5, "#ifndef _STDDEF_H\n#define _STDDE"..., 8192) = 684
read(5, "", 8192)                       = 0
close(5)                                = 0
read(4, "           */\n#define ZMQ_HWM 1\n"..., 8192) = 2199
read(4, "", 8192)                       = 0
close(4)                                = 0
open("/home/green.sophos/ioanrogers/.plenv/versions/5.22.1/lib/perl5/site_perl/5.22.1/auto/share/dist/Alien-TinyCC/lib/tcc/libtcc1.a", O_RDONLY) = 4
read(4, "!<arch>\n/               0       "..., 64) = 64
lseek(4, 0, SEEK_SET)                   = 0
read(4, "!<arch>\n", 8)                 = 8
read(4, "/               0           0   "..., 60) = 60
lseek(4, 0, SEEK_CUR)                   = 68
read(4, "\0\0\0\f\0\0\1\6\0\0\1\6\0\0\1\6\0\0\1\6\0\0\1\6\0\0\1\6\0\0\1\6"..., 194) = 194
close(4)                                = 0
mprotect(0x14b1000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC) = -1 EACCES (Permission denied)
mprotect(0x14b1000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC) = -1 EACCES (Permission denied)
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_ACCERR, si_addr=0x14b1d40} ---
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault (core dumped)

SELinux violation details:
tcc_setroubleshoot.txt

This is really a TinyCC issue, but as it prevents building ZMQ::FFI there should at least be a record here for anyone else having this issue.

Workaround:

sudo setenforce 0
dzil build
sudo setenforce 1

I found that get/set on sockets are using regexps to match literal strings:

    for ($opt_type) {
        when (/^int$/) { ... }
        ...
    }

This can be converted to

    for ($opt_type) {
        when ( 'int' ) { ... }
        ...
    }

Which is 2x faster (12M/s for regexp vs 25M/s for literal on my machine).

Note that this do not apply to when (/^(binary|string)$/), in this case regexp is 10x faster than when(['binary','string']).