Giter Club home page Giter Club logo

pyczmq's Introduction

Pyczmq

Pyczmq is a Python wrapper around the CZMQ zeromq bindings.

Build Status

Overview

Pyczmq exposes CZMQ functions via the cffi library's ABI access mode. No compiler is required to use it. The czmq library is accessed with dlopen and a set of parsed function declarations.

This is still a work in progress, not all of the API is fully wrapped yet. Most of the core context, socket, socket option, polling, beacon, auth, and cert functions are provided. Some functions will probably not be wrapped, notably those that provide duplicate functionality to built-in python type or libraries like zlist, zhash, zsys, zclock, zdir, etc.

Building and Installing

There are two methods for installing pyczmq.

Option 1 - Use the Cheese Shop

This is the easiest option for most users. If you have all the dependencies then it is as simple as:

[sudo] pip install pyczmq

NOTE: This version may not be as up to date as the Github master.

Option 2 - Use Github

This option would typically be used by users and contributing developers who want access to the most up to date version.

Dependencies

On Ubuntu you need the following packages (Other OSes may use a different package name) which are typicall installed using 'sudo apt-get install package':

libffi-dev
python-dev
python-virtualenv

The following Python packages are required which are typically installed using 'sudo pip install package':

cffi
nose

Libsodium provides security for ZMQ:

git clone git://github.com/jedisct1/libsodium.git
cd libsodium
./autogen.sh
./configure && make check
sudo make install
sudo ldconfig
cd ..

The ZMQ core library:

git clone git://github.com/zeromq/libzmq.git
cd libzmq
./autogen.sh
./configure && make check
sudo make install
sudo ldconfig
cd ..

The high-level CZMQ library:

git clone git://github.com/zeromq/czmq.git
cd czmq
./autogen.sh
./configure && make check
sudo make install
sudo ldconfig
cd ..

Pyczmq Installation

git clone git://github.com/zeromq/pyczmq.git
cd pyczmq
sudo python setup.py install

Testing

The pyczmq tests are run using the nose testing package. The tests are run from the top level pyczmq directory using:

nosetests [-v]

To report an issue, use the Pyczmq issue tracker at github.com.

Details

All CZMQ functions can be accessed directly via the low-level cffi binding, ie:

  • pyczmq.C.zctx_new

  • pyczmq.C.zsocket_bind

  • pyczmq.C.zstr_send

  • etc...

Functions also have aliases in a module level namespace interface, so:

  • pyczmq.C.zctx_new is pyczmq.zctx.new (with caveat, see below)

  • pyczmq.C.zsocket_bind is pyczmq.zsocket.bind

  • pyczmq.C.zstr_send is pyczmq.zstr.send

  • etc...

For example a simple PUSH/PULL socket pipeline::

from pyzmq import zctx, zsocket, zstr, zmq

ctx = zctx.new()
push = zsocket.new(ctx, zmq.PUSH)
pull = zsocket.new(ctx, zmq.PULL)
zsocket.bind(push, 'inproc://test')
zsocket.connect(pull, 'inproc://test')
zstr.send(push, 'foo')
assert zstr.recv(pull) == 'foo'
zstr.send(push, 'foo')
zsocket.poll(pull, 1)
assert zstr.recv_nowait(pull) == 'foo'

Some of the 'new' functions in the module namespaces (like pyczmq.zctx.new) are wrappers that plug into Python's garbage collector, so you typically never need to explicitly destroy objects if you use the namespace interface.

Therefore, pyczmq.zctx.new isn't really pyczmq.C.zmq_new, but the effect is exactly the same. If you use the "raw" C binding interface pyczmq.C.zctx_new, however, you must explicitly garbage collect your own resources by calling the coresponding destroy method (pyczmq.C.zctx_destroy, etc.).

Some 'new' functions do not do this wrapping behavior, because they are meant to be destroyed by czmq. zmsg objects for example are destroyed by zmsg_send, and zframe objects have their ownership taken over by various functions in zmsg and are destroyed when the msg is sent and destroyed. If you create these objects and don't in turn call the functions that destroy them, you must explicitly destroy them yourself with zmsg.destroy or zframe.destroy.

API Documentation

The documentation for this binding can be found at readthedocs:

http://pyczmq.readthedocs.org/

pyczmq's People

Contributors

bluca avatar claws avatar gward avatar hintjens avatar michelp avatar minrk avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pyczmq's Issues

pyczmq not compatible with libzmq < 4

pyczmq isn't compatible with libzmq < 4 or older versions of czmq either I suppose (at a minumum, I think libzmq would need the zocket ipv6 stuff)

This is fine, I just thought it might be worthwhile to mention somewhere the versioning requirements. It wasn't clear to me when I first tried to install it (admittedly, I didn't follow instructions)

import pyczmq

from _cffi import C, ffi

ImportError: No module named '_cffi'

zmsg import failure

I'm using czmq e831fffe5 + libzmq ecb9770, pyczmq 8ad9d50 (all latest and greatest)

$ python
Python 2.7.6 (default, Jan 11 2014, 17:06:02)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

from pyczmq import zmq, zctx, zsocket, zmsg, zframe, zbeacon, zstr
Traceback (most recent call last):
File "", line 1, in
File "build/bdist.linux-i686/egg/pyczmq/zmsg.py", line 66, in
File "build/bdist.linux-i686/egg/pyczmq/_cffi.py", line 17, in cdef
File "/usr/local/lib/python2.7/dist-packages/cffi/api.py", line 102, in cdef
self._parser.parse(csource, override=override)
File "/usr/local/lib/python2.7/dist-packages/cffi/cparser.py", line 154, in parse
self._internal_parse(csource)
File "/usr/local/lib/python2.7/dist-packages/cffi/cparser.py", line 159, in _internal_parse
ast, macros = self._parse(csource)
File "/usr/local/lib/python2.7/dist-packages/cffi/cparser.py", line 126, in _parse
self.convert_pycparser_error(e, csource)
File "/usr/local/lib/python2.7/dist-packages/cffi/cparser.py", line 148, in convert_pycparser_error
raise api.CDefError(msg)
cffi.api.CDefError: cannot parse "int zmsg_push (zmsg_t *self, zframe_t *frame);"
:8:30: before: zframe_t

if I add

cdef('typedef struct _zframe_t zframe_t;')

to zmsg.py, the error's gone; not sure this is the right fix though.

-m

Core dump in zauth_test_zauth

Reported by Michael Haberler:

btw there's another regression relative to czmq master:

tests.test_zauth.test_zauth ... python: zsockopt.c:455: zsocket_set_curve_secretkey_bin: Assertion `rc == 0 || zmq_errno () == (156384712 + 53)' failed.
Aborted (core dumped)

pyczmq still depends on zsocket_probe_router

Which i assume was removed in 303bf148e1fb19f25276f33e55ba37b522ed7230

$ python ./test_zcert.py
Traceback (most recent call last):
  File "./test_zcert.py", line 7, in <module>
    from pyczmq import zctx, zcert
  File "/usr/local/lib/python2.7/dist-packages/pyczmq/__init__.py", line 3, in <module>
    from . import (
  File "/usr/local/lib/python2.7/dist-packages/pyczmq/zsocket.py", line 215, in <module>
    probe_router = C.zsocket_probe_router
  File "/usr/local/lib/python2.7/dist-packages/cffi/api.py", line 463, in __getattr__
    make_accessor(name)
  File "/usr/local/lib/python2.7/dist-packages/cffi/api.py", line 459, in make_accessor
    make_accessor_locked(name)
  File "/usr/local/lib/python2.7/dist-packages/cffi/api.py", line 422, in make_accessor_locked
    raise AttributeError('%s: %s' % (name, e))
AttributeError: zsocket_probe_router: "function 'zsocket_probe_router' not found in library 'libczmq.so.1': /usr/lib/x86_64-linux-gnu/libczmq.so.1: undefined symbol: zsocket_probe_router"

segfault in zbeacon.new()

not sure why - happens to me on jessie as well as lucid:

mah@wheezy:~/src/czmq$ gdb python
GNU gdb (GDB) 7.6.1 (Debian 7.6.1-1)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/...
Reading symbols from /usr/bin/python2.7...(no debugging symbols found)...done.
(gdb) r
Starting program: /usr/bin/python
warning: Could not load shared library symbols for linux-gate.so.1.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/i386-linux-gnu/i686/cmov/libthread_db.so.1".
Python 2.7.5+ (default, Sep 17 2013, 17:31:54)
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import pyczmq
pyczmq.zmq.version()
(4, 1, 0)
from pyczmq import zbeacon
service_beacon = zbeacon.new(10042)

Program received signal SIGSEGV, Segmentation fault.
zctx__socket_new (self=self@entry=0x273a, type=type@entry=0) at zctx.c:326
326 if (!self->context) {
(gdb) backtrace
#0 zctx__socket_new (self=self@entry=0x273a, type=type@entry=0) at zctx.c:326
#1 0xb774ef89 in zctx__socket_pipe (self=self@entry=0x273a) at zctx.c:364
#2 0xb7762c9e in zthread_fork (ctx=ctx@entry=0x273a, thread_fn=thread_fn@entry=0xb774a390 <s_agent_task>, args=args@entry=0x0)

at zthread.c:182

#3 0xb774ac51 in zbeacon_new (ctx=0x273a, port_nbr=137740368) at zbeacon.c:108
#4 0xb79ea58a in ffi_call_SYSV () from /usr/lib/i386-linux-gnu/libffi.so.5
#5 0xb79ea20d in ffi_call () from /usr/lib/i386-linux-gnu/libffi.so.5
#6 0xb7fccead in cdata_call (cd=0x87079e0, args=0x870dbac, kwds=0x0) at c/_cffi_backend.c:2323
#7 0x08151f9d in PyEval_EvalFrameEx ()
#8 0x081a5392 in ?? ()
#9 0x08153536 in PyEval_EvalFrameEx ()
#10 0x08189972 in PyEval_EvalCodeEx ()
#11 0x08151ac4 in PyEval_EvalFrameEx ()
#12 0x08189972 in PyEval_EvalCodeEx ()
#13 0x081efa97 in PyEval_EvalCode ()
#14 0x081f419d in ?? ()
#15 0x080ac9f9 in PyRun_InteractiveOneFlags ()
#16 0x080acb2e in PyRun_InteractiveLoopFlags ()
#17 0x080ad2ca in PyRun_AnyFileExFlags ()
#18 0x080adf5e in Py_Main ()
#19 0x080ae03d in main ()

(gdb)

error on module load (no module named _cffi) Win10 + py3.7/3.6

Error:
c:\local\python36>.\python.exe -tt c:\temp\sample.py
Traceback (most recent call last):
File "c:\temp\sample.py", line 1, in
from pyczmq import zctx, zmsg, zpoller, zsocket, zframe
File "c:\local\python36\lib\site-packages\pyczmq_init_.py", line 1, in
from _cffi import C, ffi
ModuleNotFoundError: No module named '_cffi'

To reproduce, a 1 line file with statement:
from pyczmq import zctx, zmsg, zpoller, zsocket, zframe

OS: Windows 10 Pro, AMD64
Python: 3.6.8 and 3.7.3

pip report in 3.7.3 env, lists cffi module exists:
Package Version


cffi 1.12.3
h5py 2.9.0
nose 1.3.7
numpy 1.16.4
pip 19.1.1
pycparser 2.19
pyczmq 0.0.4
pyzmq 18.0.2
setuptools 41.0.1
six 1.12.0

pip verbose for CFFI shows supported envs:
Name: cffi
Version: 1.12.3
Summary: Foreign Function Interface for Python calling C code.
Home-page: http://cffi.readthedocs.org
Author: Armin Rigo, Maciej Fijalkowski
Author-email: [email protected]
License: MIT
Location: c:\local\python\lib\site-packages
Requires: pycparser
Required-by: pyczmq
Metadata-Version: 2.1
Installer: pip
Classifiers:
Programming Language :: Python
Programming Language :: Python :: 2
Programming Language :: Python :: 2.6
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.2
Programming Language :: Python :: 3.3
Programming Language :: Python :: 3.4
Programming Language :: Python :: 3.5
Programming Language :: Python :: 3.6
Programming Language :: Python :: Implementation :: CPython
Programming Language :: Python :: Implementation :: PyPy

Tried to run the same 1-line sample.py file with python 3.6.8 -- same problem.

Adding pip packages dir to path explicitly does not solve the issue.
set PATH=c:\local\python36\Lib\site-packages;%PATH%

output of pip verbose cffi under 3.6.8:
c:\local\python36>Scripts\pip.exe show --verbose -f cffi
Name: cffi
Version: 1.12.3
Summary: Foreign Function Interface for Python calling C code.
Home-page: http://cffi.readthedocs.org
Author: Armin Rigo, Maciej Fijalkowski
Author-email: [email protected]
License: MIT
Location: c:\local\python36\lib\site-packages
Requires: pycparser
Required-by: pyczmq
Metadata-Version: 2.1
Installer: pip
Classifiers:
Programming Language :: Python
Programming Language :: Python :: 2
Programming Language :: Python :: 2.6
Programming Language :: Python :: 2.7
Programming Language :: Python :: 3
Programming Language :: Python :: 3.2
Programming Language :: Python :: 3.3
Programming Language :: Python :: 3.4
Programming Language :: Python :: 3.5
Programming Language :: Python :: 3.6
Programming Language :: Python :: Implementation :: CPython
Programming Language :: Python :: Implementation :: PyPy
Entry-points:
[distutils.setup_keywords]
cffi_modules = cffi.setuptools_ext:cffi_modules
Files:
cffi_backend.cp36-win_amd64.pyd
cffi-1.12.3.dist-info\INSTALLER
cffi-1.12.3.dist-info\LICENSE
cffi-1.12.3.dist-info\METADATA
cffi-1.12.3.dist-info\RECORD
cffi-1.12.3.dist-info\WHEEL
cffi-1.12.3.dist-info\entry_points.txt
cffi-1.12.3.dist-info\top_level.txt
cffi_init
.py
cffi_pycache__init_.cpython-36.pyc
cffi_pycache_\api.cpython-36.pyc
cffi_pycache_\backend_ctypes.cpython-36.pyc
cffi_pycache_\cffi_opcode.cpython-36.pyc
cffi_pycache_\commontypes.cpython-36.pyc
cffi_pycache_\cparser.cpython-36.pyc
cffi_pycache_\error.cpython-36.pyc
cffi_pycache_\ffiplatform.cpython-36.pyc
cffi_pycache_\lock.cpython-36.pyc
cffi_pycache_\model.cpython-36.pyc
cffi_pycache_\pkgconfig.cpython-36.pyc
cffi_pycache_\recompiler.cpython-36.pyc
cffi_pycache_\setuptools_ext.cpython-36.pyc
cffi_pycache_\vengine_cpy.cpython-36.pyc
cffi_pycache_\vengine_gen.cpython-36.pyc
cffi_pycache_\verifier.cpython-36.pyc
cffi \ _cffi_errors.h
cffi \ _cffi_include.h
cffi \ _embedding.h
cffi\api.py
cffi\backend_ctypes.py
cffi\cffi_opcode.py
cffi\commontypes.py
cffi\cparser.py
cffi\error.py
cffi\ffiplatform.py
cffi\lock.py
cffi\model.py
cffi\parse_c_type.h
cffi\pkgconfig.py
cffi\recompiler.py
cffi\setuptools_ext.py
cffi\vengine_cpy.py
cffi\vengine_gen.py
cffi\verifier.py

Listing pip details for pyczmq, shows requirement for "_cffi.py".

c:\local\python36>Scripts\pip.exe show --verbose -f pyczmq
Name: pyczmq
Version: 0.0.4
Summary: pyczmq cffi wrapper
Home-page: https://github.com/michelp/pyczmq
Author: Michel Pelletier
Author-email: [email protected]
License: LGPL v3
Location: c:\local\python36\lib\site-packages
Requires: cffi
Required-by:
Metadata-Version: 1.0
Installer:
Classifiers:
Entry-points:
Files:
pyczmq-0.0.4-py3.6.egg-info\PKG-INFO
pyczmq-0.0.4-py3.6.egg-info\SOURCES.txt
pyczmq-0.0.4-py3.6.egg-info\dependency_links.txt
pyczmq-0.0.4-py3.6.egg-info\requires.txt
pyczmq-0.0.4-py3.6.egg-info\top_level.txt
pyczmq_init_.py
pyczmq_pycache__init_.cpython-36.pyc
pyczmq_pycache__cffi.cpython-36.pyc
pyczmq_pycache_\types.cpython-36.pyc
pyczmq_pycache_\zauth.cpython-36.pyc
pyczmq_pycache_\zbeacon.cpython-36.pyc
pyczmq_pycache_\zcert.cpython-36.pyc
pyczmq_pycache_\zcertstore.cpython-36.pyc
pyczmq_pycache_\zctx.cpython-36.pyc
pyczmq_pycache_\zframe.cpython-36.pyc
pyczmq_pycache_\zloop.cpython-36.pyc
pyczmq_pycache_\zmq.cpython-36.pyc
pyczmq_pycache_\zmsg.cpython-36.pyc
pyczmq_pycache_\zpoller.cpython-36.pyc
pyczmq_pycache_\zsocket.cpython-36.pyc
pyczmq_pycache_\zstr.cpython-36.pyc
pyczmq \ _cffi.py
pyczmq\types.py
pyczmq\zauth.py
pyczmq\zbeacon.py
pyczmq\zcert.py
pyczmq\zcertstore.py
pyczmq\zctx.py
pyczmq\zframe.py
pyczmq\zloop.py
pyczmq\zmq.py
pyczmq\zmsg.py
pyczmq\zpoller.py
pyczmq\zsocket.py
pyczmq\zstr.py

CFFI module does not list "_cffi.py" file.
What's missing?

sample.txt

Switch to cffi.verify()

Currently using ABI level compatibility that opens the zmq and czmq libraries with dlopen(). CFFI provides an API level mode where a small stub python extension is built at initial import time to export the library APIs to Python. It has many portability and version management advantages but requires a compiler. Precompiled platform binaries can be optionally distributed.

Support for both is also possible and fairly trivial, any thoughts @claws or @hintjens?

test failure: AttributeError: 'FFI' object has no attribute 'new_handle'

Running on Ubuntu 13.10 (saucy), pyczmq tests fail for me:

nosetests
.........E.....
======================================================================
ERROR: tests.test_zloop.test_zloop
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/data/src/pyczmq/tests/test_zloop.py", line 37, in test_zloop
    cancel_timer_id = zloop.timer(l, 1000, 1, on_timer_event, None)
  File "/data/src/pyczmq/pyczmq/_cffi.py", line 22, in inner_f
    val = f(*args)
  File "/data/src/pyczmq/pyczmq/zloop.py", line 94, in timer
    return C.zloop_timer(loop, delay, times, handler, ffi.new_handle(arg))
AttributeError: 'FFI' object has no attribute 'new_handle'

----------------------------------------------------------------------
Ran 15 tests in 2.153s

FAILED (errors=1)

I'm using libsodium, libzmq, and czmq cloned from github yesterday (2014-01-15), pyczmq from github today (2014-01-16), all installed to a dedicated prefix (/usr/local/zmq). No virtualenv involved. Everything else is from recent Ubuntu packages. Specifically:

$ dpkg-query -W | grep "^python"
python  2.7.5-5ubuntu1
[...]
python-cffi 0.6-1
[...]
$ dpkg-query -W | grep "ffi" 
libffi-dev:amd64    3.0.13-4
libffi6:amd64   3.0.13-4
[...]

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.