Commit 486e2087 authored by Sebastian Ramacher's avatar Sebastian Ramacher

Imported Upstream version 0.9.5

parents
Copyright 2013 Ross McFarland
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
include README.rst LICENSE test_requests_futures.py requirements-python-2.7.txt requirements-python-3.2.txt
Metadata-Version: 1.1
Name: requests-futures
Version: 0.9.5
Summary: Asynchronous Python HTTP for Humans.
Home-page: https://github.com/ross/requests-futures
Author: Ross McFarland
Author-email: rwmcfa1@neces.com
License: Apache License v2
Description: Asynchronous Python HTTP Requests for Humans
============================================
.. image:: https://travis-ci.org/ross/requests-futures.png?branch=master
:target: https://travis-ci.org/ross/requests-futures
Small add-on for the python requests_ http library. Makes use of python 3.2's
`concurrent.futures`_ or the backport_ for prior versions of python.
The additional API and changes are minimal and strives to avoid surprises.
The following synchronous code:
.. code-block:: python
from requests import Session
session = Session()
# first requests starts and blocks until finished
response_one = session.get('http://httpbin.org/get')
# second request starts once first is finished
response_two = session.get('http://httpbin.org/get?foo=bar')
# both requests are complete
print('response one status: {0}'.format(response_one.status_code))
print(response_one.content)
print('response two status: {0}'.format(response_two.status_code))
print(response_two.content)
Can be translated to make use of futures, and thus be asynchronous by creating
a FuturesSession and catching the returned Future in place of Response. The
Response can be retrieved by calling the result method on the Future:
.. code-block:: python
from requests_futures.sessions import FuturesSession
session = FuturesSession()
# first request is started in background
future_one = session.get('http://httpbin.org/get')
# second requests is started immediately
future_two = session.get('http://httpbin.org/get?foo=bar')
# wait for the first request to complete, if it hasn't already
response_one = future_one.result()
print('response one status: {0}'.format(response_one.status_code))
print(response_one.content)
# wait for the second request to complete, if it hasn't already
response_two = future_two.result()
print('response two status: {0}'.format(response_two.status_code))
print(response_two.content)
By default a ThreadPoolExecutor is created with 2 workers. If you would like to
adjust that value or share a executor across multiple sessions you can provide
one to the FuturesSession constructor.
.. code-block:: python
from concurrent.futures import ThreadPoolExecutor
from requests_futures.sessions import FuturesSession
session = FuturesSession(executor=ThreadPoolExecutor(max_workers=10))
# ...
As a shortcut in case of just increasing workers number you can pass
`max_workers` straight to the `FuturesSession` constructor:
.. code-block:: python
from requests_futures.sessions import FuturesSession
session = FuturesSession(max_workers=10)
That's it. The api of requests.Session is preserved without any modifications
beyond returning a Future rather than Response. As with all futures exceptions
are shifted (thrown) to the future.result() call so try/except blocks should be
moved there.
Working in the Background
=========================
There is one additional parameter to the various request functions,
background_callback, which allows you to work with the Response objects in the
background thread. This can be useful for shifting work out of the foreground,
for a simple example take json parsing.
.. code-block:: python
from pprint import pprint
from requests_futures.sessions import FuturesSession
session = FuturesSession()
def bg_cb(sess, resp):
# parse the json storing the result on the response object
resp.data = resp.json()
future = session.get('http://httpbin.org/get', background_callback=bg_cb)
# do some other stuff, send some more requests while this one works
response = future.result()
print('response status {0}'.format(response.status_code))
# data will have been attached to the response object in the background
pprint(response.data)
Installation
============
pip install requests-futures
.. _`requests`: https://github.com/kennethreitz/requests
.. _`concurrent.futures`: http://docs.python.org/dev/library/concurrent.futures.html
.. _backport: https://pypi.python.org/pypi/futures
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2.6
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.1
Classifier: Programming Language :: Python :: 3.2
Classifier: Programming Language :: Python :: 3.3
Asynchronous Python HTTP Requests for Humans
============================================
.. image:: https://travis-ci.org/ross/requests-futures.png?branch=master
:target: https://travis-ci.org/ross/requests-futures
Small add-on for the python requests_ http library. Makes use of python 3.2's
`concurrent.futures`_ or the backport_ for prior versions of python.
The additional API and changes are minimal and strives to avoid surprises.
The following synchronous code:
.. code-block:: python
from requests import Session
session = Session()
# first requests starts and blocks until finished
response_one = session.get('http://httpbin.org/get')
# second request starts once first is finished
response_two = session.get('http://httpbin.org/get?foo=bar')
# both requests are complete
print('response one status: {0}'.format(response_one.status_code))
print(response_one.content)
print('response two status: {0}'.format(response_two.status_code))
print(response_two.content)
Can be translated to make use of futures, and thus be asynchronous by creating
a FuturesSession and catching the returned Future in place of Response. The
Response can be retrieved by calling the result method on the Future:
.. code-block:: python
from requests_futures.sessions import FuturesSession
session = FuturesSession()
# first request is started in background
future_one = session.get('http://httpbin.org/get')
# second requests is started immediately
future_two = session.get('http://httpbin.org/get?foo=bar')
# wait for the first request to complete, if it hasn't already
response_one = future_one.result()
print('response one status: {0}'.format(response_one.status_code))
print(response_one.content)
# wait for the second request to complete, if it hasn't already
response_two = future_two.result()
print('response two status: {0}'.format(response_two.status_code))
print(response_two.content)
By default a ThreadPoolExecutor is created with 2 workers. If you would like to
adjust that value or share a executor across multiple sessions you can provide
one to the FuturesSession constructor.
.. code-block:: python
from concurrent.futures import ThreadPoolExecutor
from requests_futures.sessions import FuturesSession
session = FuturesSession(executor=ThreadPoolExecutor(max_workers=10))
# ...
As a shortcut in case of just increasing workers number you can pass
`max_workers` straight to the `FuturesSession` constructor:
.. code-block:: python
from requests_futures.sessions import FuturesSession
session = FuturesSession(max_workers=10)
That's it. The api of requests.Session is preserved without any modifications
beyond returning a Future rather than Response. As with all futures exceptions
are shifted (thrown) to the future.result() call so try/except blocks should be
moved there.
Working in the Background
=========================
There is one additional parameter to the various request functions,
background_callback, which allows you to work with the Response objects in the
background thread. This can be useful for shifting work out of the foreground,
for a simple example take json parsing.
.. code-block:: python
from pprint import pprint
from requests_futures.sessions import FuturesSession
session = FuturesSession()
def bg_cb(sess, resp):
# parse the json storing the result on the response object
resp.data = resp.json()
future = session.get('http://httpbin.org/get', background_callback=bg_cb)
# do some other stuff, send some more requests while this one works
response = future.result()
print('response status {0}'.format(response.status_code))
# data will have been attached to the response object in the background
pprint(response.data)
Installation
============
pip install requests-futures
.. _`requests`: https://github.com/kennethreitz/requests
.. _`concurrent.futures`: http://docs.python.org/dev/library/concurrent.futures.html
.. _backport: https://pypi.python.org/pypi/futures
Metadata-Version: 1.1
Name: requests-futures
Version: 0.9.5
Summary: Asynchronous Python HTTP for Humans.
Home-page: https://github.com/ross/requests-futures
Author: Ross McFarland
Author-email: rwmcfa1@neces.com
License: Apache License v2
Description: Asynchronous Python HTTP Requests for Humans
============================================
.. image:: https://travis-ci.org/ross/requests-futures.png?branch=master
:target: https://travis-ci.org/ross/requests-futures
Small add-on for the python requests_ http library. Makes use of python 3.2's
`concurrent.futures`_ or the backport_ for prior versions of python.
The additional API and changes are minimal and strives to avoid surprises.
The following synchronous code:
.. code-block:: python
from requests import Session
session = Session()
# first requests starts and blocks until finished
response_one = session.get('http://httpbin.org/get')
# second request starts once first is finished
response_two = session.get('http://httpbin.org/get?foo=bar')
# both requests are complete
print('response one status: {0}'.format(response_one.status_code))
print(response_one.content)
print('response two status: {0}'.format(response_two.status_code))
print(response_two.content)
Can be translated to make use of futures, and thus be asynchronous by creating
a FuturesSession and catching the returned Future in place of Response. The
Response can be retrieved by calling the result method on the Future:
.. code-block:: python
from requests_futures.sessions import FuturesSession
session = FuturesSession()
# first request is started in background
future_one = session.get('http://httpbin.org/get')
# second requests is started immediately
future_two = session.get('http://httpbin.org/get?foo=bar')
# wait for the first request to complete, if it hasn't already
response_one = future_one.result()
print('response one status: {0}'.format(response_one.status_code))
print(response_one.content)
# wait for the second request to complete, if it hasn't already
response_two = future_two.result()
print('response two status: {0}'.format(response_two.status_code))
print(response_two.content)
By default a ThreadPoolExecutor is created with 2 workers. If you would like to
adjust that value or share a executor across multiple sessions you can provide
one to the FuturesSession constructor.
.. code-block:: python
from concurrent.futures import ThreadPoolExecutor
from requests_futures.sessions import FuturesSession
session = FuturesSession(executor=ThreadPoolExecutor(max_workers=10))
# ...
As a shortcut in case of just increasing workers number you can pass
`max_workers` straight to the `FuturesSession` constructor:
.. code-block:: python
from requests_futures.sessions import FuturesSession
session = FuturesSession(max_workers=10)
That's it. The api of requests.Session is preserved without any modifications
beyond returning a Future rather than Response. As with all futures exceptions
are shifted (thrown) to the future.result() call so try/except blocks should be
moved there.
Working in the Background
=========================
There is one additional parameter to the various request functions,
background_callback, which allows you to work with the Response objects in the
background thread. This can be useful for shifting work out of the foreground,
for a simple example take json parsing.
.. code-block:: python
from pprint import pprint
from requests_futures.sessions import FuturesSession
session = FuturesSession()
def bg_cb(sess, resp):
# parse the json storing the result on the response object
resp.data = resp.json()
future = session.get('http://httpbin.org/get', background_callback=bg_cb)
# do some other stuff, send some more requests while this one works
response = future.result()
print('response status {0}'.format(response.status_code))
# data will have been attached to the response object in the background
pprint(response.data)
Installation
============
pip install requests-futures
.. _`requests`: https://github.com/kennethreitz/requests
.. _`concurrent.futures`: http://docs.python.org/dev/library/concurrent.futures.html
.. _backport: https://pypi.python.org/pypi/futures
Platform: UNKNOWN
Classifier: Development Status :: 5 - Production/Stable
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 2.6
Classifier: Programming Language :: Python :: 2.7
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.1
Classifier: Programming Language :: Python :: 3.2
Classifier: Programming Language :: Python :: 3.3
LICENSE
MANIFEST.in
README.rst
requirements-python-2.7.txt
requirements-python-3.2.txt
setup.py
test_requests_futures.py
requests_futures/__init__.py
requests_futures/sessions.py
requests_futures.egg-info/PKG-INFO
requests_futures.egg-info/SOURCES.txt
requests_futures.egg-info/dependency_links.txt
requests_futures.egg-info/not-zip-safe
requests_futures.egg-info/requires.txt
requests_futures.egg-info/top_level.txt
\ No newline at end of file
requests>=1.2.0
futures>=2.1.3
\ No newline at end of file
# -*- coding: utf-8 -*-
# Requests Futures
"""
async requests HTTP library
~~~~~~~~~~~~~~~~~~~~~
"""
__title__ = 'requests-futures'
__version__ = '0.9.5'
__build__ = 0x000000
__author__ = 'Ross McFarland'
__license__ = 'Apache 2.0'
__copyright__ = 'Copyright 2013 Ross McFarland'
# Set default logging handler to avoid "No handler found" warnings.
import logging
try: # Python 2.7+
from logging import NullHandler
except ImportError:
class NullHandler(logging.Handler):
def emit(self, record):
pass
logging.getLogger(__name__).addHandler(NullHandler())
# -*- coding: utf-8 -*-
"""
requests_futures
~~~~~~~~~~~~~~~~
This module provides a small add-on for the requests http library. It makes use
of python 3.3's concurrent.futures or the futures backport for previous
releases of python.
from requests_futures import FuturesSession
session = FuturesSession()
# request is run in the background
future = session.get('http://httpbin.org/get')
# ... do other stuff ...
# wait for the request to complete, if it hasn't already
response = future.result()
print('response status: {0}'.format(response.status_code))
print(response.content)
"""
from concurrent.futures import ThreadPoolExecutor
from requests import Session
from requests.adapters import DEFAULT_POOLSIZE, HTTPAdapter
class FuturesSession(Session):
def __init__(self, executor=None, max_workers=2, *args, **kwargs):
"""Creates a FuturesSession
Notes
~~~~~
* ProcessPoolExecutor is not supported b/c Response objects are
not picklable.
* If you provide both `executor` and `max_workers`, the latter is
ignored and provided executor is used as is.
"""
super(FuturesSession, self).__init__(*args, **kwargs)
if executor is None:
executor = ThreadPoolExecutor(max_workers=max_workers)
# set connection pool size equal to max_workers if needed
if max_workers > DEFAULT_POOLSIZE:
adapter_kwargs = dict(pool_connections=max_workers,
pool_maxsize=max_workers)
self.mount('https://', HTTPAdapter(**adapter_kwargs))
self.mount('http://', HTTPAdapter(**adapter_kwargs))
self.executor = executor
def request(self, *args, **kwargs):
"""Maintains the existing api for Session.request.
Used by all of the higher level methods, e.g. Session.get.
The background_callback param allows you to do some processing on the
response in the background, e.g. call resp.json() so that json parsing
happens in the background thread.
"""
func = sup = super(FuturesSession, self).request
background_callback = kwargs.pop('background_callback', None)
if background_callback:
def wrap(*args_, **kwargs_):
resp = sup(*args_, **kwargs_)
background_callback(self, resp)
return resp
func = wrap
return self.executor.submit(func, *args, **kwargs)
futures
requests>=1.2.0
[egg_info]
tag_build =
tag_date = 0
tag_svn_revision = 0
#!/usr/bin/env python
import os
import sys
import requests_futures
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
if sys.argv[-1] == 'publish':
os.system('python setup.py sdist upload')
sys.exit()
packages = [
'requests_futures',
]
requires = [
'requests>=1.2.0'
]
if sys.version_info < (3, 2):
requires.append('futures>=2.1.3')
setup(
name='requests-futures',
version=requests_futures.__version__,
description='Asynchronous Python HTTP for Humans.',
long_description=open('README.rst').read(),
author='Ross McFarland',
author_email='rwmcfa1@neces.com',
packages=packages,
package_dir={'requests_futures': 'requests_futures'},
package_data={'requests_futures': ['LICENSE', 'README.rst']},
include_package_data=True,
install_requires=requires,
license='Apache License v2',
url='https://github.com/ross/requests-futures',
zip_safe=False,
classifiers=(
'Development Status :: 5 - Production/Stable',
'Intended Audience :: Developers',
'Natural Language :: English',
'License :: OSI Approved :: Apache Software License',
'Programming Language :: Python',
'Programming Language :: Python :: 2.6',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.1',
'Programming Language :: Python :: 3.2',
'Programming Language :: Python :: 3.3',
),
)
#!/usr/bin/env python
# -*- coding: utf-8 -*-
"""Tests for Requests."""
from concurrent.futures import Future
from requests import Response
from os import environ
from requests_futures.sessions import FuturesSession
from unittest import TestCase, main
HTTPBIN = environ.get('HTTPBIN_URL', 'http://httpbin.org/')
def httpbin(*suffix):
"""Returns url for HTTPBIN resource."""
return HTTPBIN + '/'.join(suffix)
class RequestsTestCase(TestCase):
def test_futures_session(self):
# basic futures get
sess = FuturesSession()
future = sess.get(httpbin('get'))
self.assertIsInstance(future, Future)
resp = future.result()
self.assertIsInstance(resp, Response)
self.assertEqual(200, resp.status_code)
# non-200, 404
future = sess.get(httpbin('status/404'))
resp = future.result()
self.assertEqual(404, resp.status_code)
def cb(s, r):
self.assertIsInstance(s, FuturesSession)
self.assertIsInstance(r, Response)
# add the parsed json data to the response
r.data = r.json()
future = sess.get(httpbin('get'), background_callback=cb)
# this should block until complete
resp = future.result()
self.assertEqual(200, resp.status_code)
# make sure the callback was invoked
self.assertTrue(hasattr(resp, 'data'))
def rasing_cb(s, r):
raise Exception('boom')
future = sess.get(httpbin('get'), background_callback=rasing_cb)
with self.assertRaises(Exception) as cm:
resp = future.result()
self.assertEqual('boom', cm.exception.args[0])
def test_max_workers(self):
""" Tests the `max_workers` shortcut. """
from concurrent.futures import ThreadPoolExecutor
session = FuturesSession()
self.assertEqual(session.executor._max_workers, 2)
session = FuturesSession(max_workers=5)
self.assertEqual(session.executor._max_workers, 5)
session = FuturesSession(executor=ThreadPoolExecutor(max_workers=10))
self.assertEqual(session.executor._max_workers, 10)
session = FuturesSession(executor=ThreadPoolExecutor(max_workers=10),
max_workers=5)
self.assertEqual(session.executor._max_workers, 10)
def test_redirect(self):
""" Tests for the ability to cleanly handle redirects. """
sess = FuturesSession()
future = sess.get(httpbin('redirect-to?url=get'))
self.assertIsInstance(future, Future)
resp = future.result()
self.assertIsInstance(resp, Response)
self.assertEqual(200, resp.status_code)
future = sess.get(httpbin('redirect-to?url=status/404'))
resp = future.result()
self.assertEqual(404, resp.status_code)
if __name__ == '__main__':
main()
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment