Commit 412d4291 authored by SVN-Git Migration's avatar SVN-Git Migration

Imported Upstream version 2.0.10

parent b9e9de6b
include README.rst
include setup.cfg.template
recursive-include docs *
recursive-exclude docs _build
# graft docs
graft examples
graft zmq
graft perf
exclude setup.cfg
# exclude docs/_static
# exclude docs/_templates
global-exclude *.so
global-exclude *.pyc
global-exclude .git*
......@@ -5,9 +5,24 @@ PyZMQ: Python bindings for 0MQ
This package contains Python bindings for `0QM <>`_.
0MQ is a lightweight and fast messaging implementation.
Unless you specifically want to develop PyZMQ, we recommend downloading the
PyZMQ source code from our github download page here:
While you can also get the latest source code by forking our github
repository, building from the repository will require that you download and
install Cython version 0.13 or later.
Building and installation
To build and install this Python package, you will first need to build and
install the latest development version of 0MQ itself. After you have done
this, follow these steps:
......@@ -17,39 +32,95 @@ and edit the `include_dirs` and `library_dirs` fields of the ``setup.cfg``
file to point to the directories that contain the library and header file for
your 0MQ installation.
Cython is not required to build pyzmq, but it is required if you want to
develop pyzmq.
Second, run this command::
python install
$ python install
If you want to develop this package, instead of this last command do::
Cython is not required to build pyzmq from a release package, but it is
required if you want to develop pyzmq, or build directly from our repository
on GitHub.
python build_ext --inplace
python develop
This will build the C extension inplace and then put this directory on your
``sys.path``. With this setup you only have to run::
python build_ext --inplace
Generally you'll need to add the location of ``libzmq.dll`` to your ``$PATH``.
Here's Microsoft's docs: on this topic.
each time you change the ``.pyx`` files.
It is best to compile both ØMQ and PyØMQ with Microsoft Visual Studio 2008 or
above. You should not need to use mingw.
On Windows, it is easiest to simply copy ``libzmq.dll`` and ``zmq.h``
into the ``zmq`` subdirectory and set ``library_dirs`` to ``.\zmq``.
On Linux, you will need to do one of the following:
* Set ``LD_LIBRARY_PATH`` to point to the :file:`lib` directory of 0MQ.
* Set ``LD_LIBRARY_PATH`` to point to the ``lib`` directory of 0MQ.
* Build the extension using the ``-rpath`` flag::
python build_ext --rpath=/opt/zeromq-dev/lib --inplace
$ python build_ext --rpath=/opt/zeromq-dev/lib --inplace
To develop PyZMQ, you will need to install Cython, version 0.13 or greater.
After installing Cython, instead of doing ``python install`` do::
$ python build_ext --inplace
$ python develop
This will build the C extension inplace and then put this directory on your
``sys.path``. With this configuration you only have to run::
$ python build_ext --inplace
each time you change the ``.pyx`` files. To clean the sources, you can do::
$ python clean
To run the test suite after installing, just do::
$ python test
How to release PyZMQ
Currently, we are using the following steps to release PyZMQ:
* Change the version number in ```` and ``version.pyx``.
* Remove old ``MANIFEST`` and ``egg-info`` files and ``dist`` and ``build``
* Check ````.
* Register the release with pypi::
python register
* Build source distributions and upload::
python sdist --formats=zip,gztar upload
* Upload the tarball and ``.zip`` file to github.
* Branch the release::
git co -b 2.0.8 master
git push origin 2.0.8
* Tag the release::
git tag -a -m “Tagging release 2.0.8” 2.0.8
git push origin —tags
* Make sure the ``README.rst`` has an updated list of contributors.
* Announce on list.
This project was started by Brian E. Granger (ellisonbg AT gmail DOT com).
This project was started by and continues to be led by Brian E. Granger
(ellisonbg AT gmail DOT com).
The following people have contributed to the project:
......@@ -57,3 +128,13 @@ The following people have contributed to the project:
* Andrew Gwozdziewycz (git AT apgwoz DOT com)
* Fernando Perez (fernando DOT perez AT berkeley DOT edu)
* Nicholas Piel (nicholas AT nichol DOT as)
* Eugene Chernyshov (chernyshov DOT eugene AT gmail DOT com)
* Justin Riley (justin DOT t DOT riley AT gmail DOT com)
* Ivo Danihelka (ivo AT denihelka DOT net)
* Thomas Supra (tomspur AT fedoraproject DOT org)
* Douglas Creager (dcreager AT dcreager DOT net)
* Erick Tryzelaar (erick DOT tryzelaar AT gmail DOT com)
* Min Ragan-Kelley (benjaminrk AT gmail DOT com)
* Scott Sadler (github AT mashi DOT org)
* spez (steve AT hipmunk DOT com)
* Thomas Kluyver (takowl AT gmail DOT com)
......@@ -11,6 +11,9 @@ Contents:
.. toctree::
:maxdepth: 2
Indices and tables
.. PyZMQ Version compatibility doc, by Min Ragan-Kelley, 2010
.. _pyversions:
PyZMQ, Python2.5, and Python3
PyZMQ is a fairly light, low-level library, so supporting as many versions
as is reasonable is our goal. Currently, we support at least Python 2.5-3.1.
Making the changes to the codebase required a few tricks, which are documented here
for future reference, either by us or by other developers looking to support several
versions of Python.
.. Note::
It is far simpler to support 2.6-3.x than to include 2.5. Many of the significant
syntax changes have been backported to 2.6, so just writing new-style code would work
in many cases. I will try to note these points as they come up.
Many functions we use, primarily involved in converting between C-buffers and Python
objects, are not available on all supported versions of Python. In order to resolve
missing symbols, we added a header :file:`utils/pyversion_compat.h` that defines missing
symbols with macros. Some of these macros alias new names to old functions (e.g.
``PyBytes_AsString``), so that we can call new-style functions on older versions, and some
simply define the function as an empty exception raiser. The important thing is that the
symbols are defined to prevent compiler warnings and linking errors. Everywhere we use
C-API functions that may not be available in a supported version, at the top of the file
is the code:
.. sourcecode:: guess
cdef extern from "pyversion_compat.h":
This ensures that the symbols are defined in the Cython generated C-code. Higher level
switching logic exists in the code itself, to prevent actually calling unavailable
functions, but the symbols must still be defined.
Bytes and Strings
.. Note::
If you are using Python >= 2.6, to prepare your PyZMQ code for Python3 you should use
the ``b'message'`` syntax to ensure all your string literal messages will still be
:class:`bytes` after you make the upgrade.
The most cumbersome part of PyZMQ compatibility from a user's perspective is the fact
that, since ØMQ uses C-strings, and would like to do so without copying, we must use the
Py3k :class:`bytes` object, which is backported to 2.6. In order to do this in a
Python-version independent way, we added a small utility that unambiguously defines the
string types: :class:`bytes`, :class:`unicode`, :obj:`basestring`. This is important,
because :class:`str` means different things on 2.x and 3.x, and :class:`bytes` is
undefined on 2.5, and both :class:`unicode` and :obj:`basestring` are undefined on 3.x.
All typechecking in PyZMQ is done against these types:
================= ================= ====================
Explicit Type 2.x 3.x
================= ================= ====================
:obj:`bytes` :obj:`str` :obj:`bytes`
:obj:`unicode` :obj:`unicode` :obj:`str`
:obj:`basestring` :obj:`basestring` :obj:`(str, bytes)`
================= ================= ====================
.. Note::
2.5 specific
Where we really noticed the issue of :class:`bytes` vs :obj:`strings` coming up for
users was in updating the tests to run on every version. Since the ``b'bytes
literal'`` syntax was not backported to 2.5, we must call ``"message".encode()`` for
*every* string in the test suite.
.. seealso:: :ref:`Unicode discussion <unicode>` for more information on strings/bytes.
The standard C-API function for turning a C-string into a Python string was a set of
functions with the prefix ``PyString_*``. However, with the Unicode changes made in
Python3, this was broken into ``PyBytes_*`` for bytes objects and ``PyUnicode_*`` for
unicode objects. We changed all our ``PyString_*`` code to ``PyBytes_*``, which was
backported to 2.6.
.. Note::
2.5 Specific:
Since Python 2.5 doesn't support the ``PyBytes_*`` functions, we had to alias them to
the ``PyString_*`` methods in utils/pyversion_compat.h.
.. sourcecode:: c++
#define PyBytes_FromStringAndSize PyString_FromStringAndSize
#define PyBytes_FromString PyString_FromString
#define PyBytes_AsString PyString_AsString
#define PyBytes_Size PyString_Size
The layer that is most complicated for developers, but shouldn't trouble users, is the
Python C-Buffer APIs. These are the methods for converting between Python objects and C
buffers. The reason it is complicated is that it keeps changing.
There are two buffer interfaces for converting an object to a C-buffer, known as new-style
and old-style. Old-style buffers were introduced long ago, but the new-style is only
backported to 2.6. The old-style buffer interface is not available in 3.x. There is also
an old- and new-style interface for creating Python objects that view C-memory. The
old-style object is called a :class:`buffer`, and the new-style object is
:class:`memoryview`. Unlike the new-style buffer interface for objects,
:class:`memoryview` has only been backported to *2.7*. This means that the available
buffer-related functions are not the same in any two versions of Python 2.5, 2.6, 2.7, or
We have a :file:`utils/buffers.pxd` file that defines our :func:`asbuffer` and
:func:`frombuffer` functions. :file:`utils/buffers.pxd` was adapted from mpi4py_'s
:file:`asbuffer.pxi`. The :func:`frombuffer` functionality was added. These functions
internally switch based on Python version to call the appropriate C-API functions.
.. seealso:: `Python Buffer API <bufferapi>`_
.. _bufferapi:
As discussed, :class:`str` is not a platform independent type. The two places where we are
required to return native str objects are :func:`error.strerror`, and
:func:`Message.__str__`. In both of these cases, the natural return is actually a
:class:`bytes` object. In the methods, the native :class:`str` type is checked, and if the
native str is actually unicode, then we decode the bytes into unicode:
.. sourcecode:: py
# ...
b = natural_result()
if str is unicode:
return b.decode()
return b
.. Note::
This section is only relevant for supporting Python 2.5 and 3.x, not for 2.6-3.x.
The syntax for handling exceptions has `changed <PEP-3110>`_ in Python 3. The old syntax:
.. sourcecode:: py
except zmq.ZMQError, e:
is no longer valid in Python 3. Instead, the new syntax for this is:
.. sourcecode:: py
except zmq.ZMQError as e:
This new syntax is backported to Python 2.6, but is invalid on 2.5. For 2.6-3.x compatible
code, we could just use the new syntax. However, the only method we found to catch an
exception for handling on both 2.5 and 3.1 is to get the exception object inside the
exception block:
.. sourcecode:: py
except zmq.ZMQError:
e = sys.exc_info()[1]
This is certainly not as elegant as either the old or new syntax, but it's the only way we
have found to work everywhere.
.. seealso:: PEP-3110_
.. _PEP-3110:
.. _mpi4py:
\ No newline at end of file
.. PyZMQ Unicode doc, by Min Ragan-Kelley, 2010
.. _unicode:
PyZMQ and Unicode
PyZMQ is built with an eye towards an easy transition to Python 3, and part of
that is dealing with unicode objects. This is an overview of some of what we
found, and what it means for PyZMQ.
First, Unicode in Python 2 and 3
In Python < 3, a ``str`` object is really a C string with some sugar - a
specific series of bytes with some fun methods like ``endswith()`` and
``split()``. In 2.0, the ``unicode`` object was added, which handles different
methods of encoding. In Python 3, however, the meaning of ``str`` changes. A
``str`` in Python 3 is a full unicode object, with encoding and everything. If
you want a C string with some sugar, there is a new object called ``bytes``,
that behaves much like the 2.x ``str``. The idea is that for a user, a string is
a series of *characters*, not a series of bytes. For simple ascii, the two are
interchangeable, but if you consider accents and non-Latin characters, then the
character meaning of byte sequences can be ambiguous, since it depends on the
encoding scheme. They decided to avoid the ambiguity by forcing users who want
the actual bytes to specify the encoding every time they want to convert a
string to bytes. That way, users are aware of the difference between a series of
bytes and a collection of characters, and don't confuse the two, as happens in
Python 2.x.
The problems (on both sides) come from the fact that regardless of the language
design, users are mostly going to use ``str`` objects to represent collections
of characters, and the behavior of that object is dramatically different in
certain aspects between the 2.x ``bytes`` approach and the 3.x ``unicode``
approach. The ``unicode`` approach has the advantage of removing byte ambiguity
- it's a list of characters, not bytes. However, if you really do want the
bytes, it's very inefficient to get them. The ``bytes`` approach has the
advantage of efficiency. A ``bytes`` object really is just a char* pointer with
some methods to be used on it, so when interacting with, so interacting with C
code, etc is highly efficient and straightforward. However, understanding a
bytes object as a string with extended characters introduces ambiguity and
possibly confusion.
To avoid ambiguity, hereafter we will refer to encoded C arrays as 'bytes' and
abstract unicode objects as 'strings'.
Unicode Buffers
Since unicode objects have a wide range of representations, they are not stored
as the bytes according to their encoding, but rather in a format called UCS (an
older fixed-width Unicode format). On some platforms (OSX,Windows), the storage
is UCS-2, which is 2 bytes per character. On most \*ix systems, it is UCS-4, or
4 bytes per character. The contents of the *buffer* of a ``unicode`` object are
not encoding dependent (always UCS-2 or UCS-4), but they are *platform*
dependent. As a result of this, and the further insistence on not interpreting
``unicode`` objects as bytes without specifying encoding, ``str`` objects in
Python 3 don't even provide the buffer interface. You simply cannot get the raw
bytes of a ``unicode`` object without specifying the encoding for the bytes. In
Python 2.x, you can get to the raw buffer, but the platform dependence and the
fact that the encoding of the buffer is not the encoding of the object makes it
very confusing, so this is probably a good move.
The efficiency problem here comes from the fact that simple ascii strings are 4x
as big in memory as they need to be (on most Linux, 2x on other platforms).
Also, to translate to/from C code that works with char*, you always have to copy
data and encode/decode the bytes. This really is horribly inefficient from a
memory standpoint. Essentially, Where memory efficiency matters to you, you
should never ever use strings; use bytes. The problem is that users will almost
always use ``str``, and in 2.x they are efficient, but in 3.x they are not. We
want to make sure that we don't help the user make this mistake, so we ensure
that zmq methods don't try to hide what strings really are.
What This Means for PyZMQ
PyZMQ is a wrapper for a C library, so it really should use bytes, since a
string is not a simple wrapper for ``char *`` like it used to be, but an
abstract sequence of characters. The representations of bytes in Python are
either the ``bytes`` object itself, or any object that provides the buffer
interface (aka memoryview). In Python 2.x, unicode objects do provide the buffer
interface, but as they do not in Python 3, where pyzmq requires bytes, we
specifically reject unicode objects.
The relevant methods here are **socket.send/recv**, **socket.get/setsockopt**,
**socket.bind/connect**. The important consideration for send/recv and
set/getsockopt is that when you put in something, you should really get the same
object back with its partner method. We can easily coerce unicode objects to
bytes with send/setsockopt, but the problem is that the pair method of
recv/getsockopt will always be bytes, and there should be symmetry. We certainly
shouldn't try to always decode on the retrieval side, because if users just want
bytes, then we are potentially using up enormous amounts of excess memory
unnecessarily, due to copying and larger memory footprint of unicode strings.
Still, we recognize the fact that users will quite frequently have unicode
strings that they want to send, so we have added ``socket.<method>_unicode()``
wrappers. These methods simply wrap their bytes counterpart by encoding
to/decoding from bytes around them, and they all take an `encoding` keyword
argument that defaults to utf-8. Since encoding and decoding are necessary to
translate between unicode and bytes, it is impossible to perform non-copying
actions with these wrappers.
``socket.bind/connect`` methods are different from these, in that they are
strictly setters and there is not corresponding getter method. As a result, we
feel that we can safely coerce unicode objects to bytes (always to utf-8) in
these methods.
The Methods
Overview of the relevant methods:
.. py:function:: socket.bind(self, addr)
`addr` is ``bytes`` or ``unicode``. If ``unicode``,
encoded to utf-8 ``bytes``
.. py:function:: socket.connect(self, addr)
`addr` is ``bytes`` or ``unicode``. If ``unicode``,
encoded to utf-8 ``bytes``
.. py:function:: socket.send(self, object obj, flags=0, copy=True)
`obj` is ``bytes`` or provides buffer interface.
if `obj` is ``unicode``, raise ``TypeError``
.. py:function:: socket.recv(self, flags=0, copy=True)
returns ``bytes`` if `copy=True`
returns ``zmq.Message`` if `copy=False`:
`message.buffer` is a buffer view of the ``bytes``
`str(message)` provides the ``bytes``
`unicode(message)` decodes `message.buffer` with utf-8
.. py:function:: socket.send_unicode(self, unicode s, flags=0,
takes a ``unicode`` string `s`, and sends the ``bytes``
after encoding without an extra copy, via:
`socket.send(s.encode(encoding), flags, copy=False)`
.. py:function:: socket.recv_unicode(self, flags=0, encoding='utf-8')
always returns ``unicode`` string
there will be a ``UnicodeError`` if it cannot decode the buffer
performs non-copying `recv`, and decodes the buffer with `encoding`
.. py:function:: socket.setsockopt(self, opt, optval)
only accepts ``bytes`` for `optval` (or ``int``, depending on `opt`)
``TypeError`` if ``unicode`` or anything else
.. py:function:: socket.getsockopt(self, opt)
returns ``bytes`` (or ``int``), never ``unicode``
.. py:function:: socket.setsockopt_unicode(self, opt, unicode optval,
accepts ``unicode`` string for `optval`
encodes `optval` with `encoding` before passing the ``bytes`` to
.. py:function:: socket.getsockopt_unicode(self, opt, encoding='utf-8')
always returns ``unicode`` string, after decoding with `encoding`
note that `zmq.IDENTITY` is the only `sockopt` with a string value
that can be queried with `getsockopt`
......@@ -20,14 +20,14 @@
import zmq
def main(addr):
def main(addrs):
print "Connecting to: ", addr
context = zmq.Context()
socket = context.socket(zmq.SUB)
socket.setsockopt(zmq.SUBSCRIBE, "")
for addr in addrs:
print "Connecting to: ", addr
while True:
msg = socket.recv_pyobj()
......@@ -35,7 +35,7 @@ def main(addr):
if __name__ == '__main__':
import sys
if len(sys.argv) != 2:
print "usage: <address>"
if len(sys.argv) < 2:
print "usage: <address> [,<address>...]"
raise SystemExit
"""A client for the device based server."""
# Copyright (c) 2010 Brian E. Granger and Eugene Chernyshov
# This file is part of pyzmq.
# pyzmq is free software; you can redistribute it and/or modify it under
# the terms of the Lesser GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.
# pyzmq is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# Lesser GNU General Public License for more details.
# You should have received a copy of the Lesser GNU General Public License
# along with this program. If not, see <>.
import zmq
import os
from time import time
print 'Client', os.getpid()
context = zmq.Context(1)
socket = context.socket(zmq.REQ)
while True:
data = zmq.Message(str(os.getpid()))
start = time()
data = socket.recv()
print time()-start, data
"""A device based server."""
# Copyright (c) 2010 Brian E. Granger and Eugene Chernyshov
# This file is part of pyzmq.
# pyzmq is free software; you can redistribute it and/or modify it under
# the terms of the Lesser GNU General Public License as published by
# the Free Software Foundation; either version 3 of the License, or
# (at your option) any later version.
# pyzmq is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# Lesser GNU General Public License for more details.
# You should have received a copy of the Lesser GNU General Public License
# along with this program. If not, see <>.
import zmq
import os
import threading
import time
print 'Server', os.getpid()
def routine(context):
socket = context.socket(zmq.REP)
while True:
message = socket.recv()
context = zmq.Context(1)
workers = context.socket(zmq.XREQ)
clients = context.socket(zmq.XREP)
for i in range(10):
thread = threading.Thread(target=routine, args=(context, ))
zmq.device(zmq.QUEUE, clients, workers)
print "Finished"
#!/usr/bin/env python
"""A trivial ZMQ echo server using the eventloop.
* MinRK
import zmq
from zmq.eventloop import ioloop
#!/usr/bin/env python
"""Adapted to put the send in the event loop using a ZMQStream.
* MinRK
import zmq
from zmq.eventloop import ioloop, zmqstream
loop = ioloop.IOLoop.instance()
ctx = zmq.Context()
s = ctx.socket(zmq.REP)
stream = zmqstream.ZMQStream(s, loop)
def echo(msg):
print " ".join(msg)
\ No newline at end of file
#!/usr/bin/env python
"""This launches an echoing rep socket device,
and runs a blocking numpy action. The rep socket should
remain responsive to pings during this time. Use to
ping this heart, and see the responsiveness.
* MinRK
import time
import numpy
import zmq
from zmq import devices
ctx = zmq.Context()
dev = devices.ThreadDevice(zmq.FORWARDER, zmq.SUB, zmq.XREQ)
dev.setsockopt_in(zmq.SUBSCRIBE, "")