Skip to content
Commits on Source (12)
Metadata-Version: 1.0
Name: bd2k-python-lib
Version: 1.14a1.dev43
Version: 1.14a1.dev48
Summary: The BD2K Python module kitchen sink
Home-page: https://github.com/BD2KGenomics/bd2k-python-lib
Author: Hannes Schmidt
Author-email: hannes@ucsc.edu
License: UNKNOWN
Description-Content-Type: UNKNOWN
Description: UNKNOWN
Platform: UNKNOWN
python-bd2k (1.14~alpha1.48-1) unstable; urgency=medium
* Team upload.
* New upstream version
* Point Vcs fields to salsa.debian.org
* Standards-Version: 4.2.1
* Remove trailing whitespace in debian/copyright
* Drop ancient X-Python-Version field
* Remove strange file from top-level of a Python library directory
-- Andreas Tille <tille@debian.org> Mon, 29 Oct 2018 13:45:00 +0100
python-bd2k (1.14~alpha1.43-1) unstable; urgency=medium
* Initial release (Closes: #851341)
......
......@@ -14,18 +14,20 @@ Build-Depends: debhelper (>= 11~),
python-lockfile,
python-mock,
python-pytest,
python-future,
python-boto,
python-dill,
python-lockfile,
python-mock,
python-pytest
Standards-Version: 4.1.3
Vcs-Browser: https://anonscm.debian.org/cgit/debian-med/python-bd2k.git/
Vcs-Git: https://anonscm.debian.org/git/debian-med/python-bd2k.git
python-pytest-runner,
python-future
# python3-setuptools,
# python3-boto,
# python3-dill,
# python3-lockfile,
# python3-mock,
# python3-pytest,
# python3-pytest,
# python3-pytest-runner
Standards-Version: 4.2.1
Vcs-Browser: https://salsa.debian.org/med-team/python-bd2k
Vcs-Git: https://salsa.debian.org/med-team/python-bd2k.git
Homepage: https://github.com/BD2KGenomics/bd2k-python-lib
X-Python3-Version: >= 3.2
X-Python-Version: >= 2.6
Package: python-bd2k
Architecture: all
......@@ -37,3 +39,14 @@ Description: utilities for BD2KGenomics (Python 2)
like hashes and hash tables or iterators.
.
This package installs the library for Python 2.
#Package: python3-bd2k
#Architecture: all
#Depends: ${python3:Depends},
# ${misc:Depends}
#Description: utilities for BD2KGenomics (Python 3)
# This package is a dependency for the toil workflow engine.
# It provides a range of smallish software solutions
# like hashes and hash tables or iterators.
# .
# This package installs the library for Python 3.
Author: Michael R. Crusoe <michael.crusoe@gmail.com>
Descriptions: Loosen dependencies
Index: python-bd2k/setup.py
===================================================================
--- python-bd2k.orig/setup.py
+++ python-bd2k/setup.py
@@ -17,10 +17,10 @@
packages=find_packages( 'src' ),
@@ -18,10 +18,10 @@ kwargs = dict(
install_requires=[ 'future' ],
setup_requires=['pytest-runner'],
tests_require=[
- 'pytest==2.7.2',
- 'pytest==3.5.0',
- 'mock==1.0.1',
- 'lockfile==0.11.0',
- 'boto==2.38.0'],
+ 'pytest>=2.7.2',
+ 'pytest>=3.5.0',
+ 'mock>=1.0.1',
+ 'lockfile>=0.11.0',
+ 'boto>=2.38.0'],
namespace_packages=[ 'bd2k' ] )
from setuptools.command.test import test as TestCommand
setup( **kwargs )
accept_any_boto.patch
change_package_name_to_bd2k.patch
setupcfg_pytest_section_name.patch
#setupcfg_pytest_section_name.patch
no_boto_test.patch
......@@ -5,3 +5,8 @@ export PYBUILD_NAME=bd2k
%:
dh $@ --with python2 --buildsystem=pybuild
# Remove strange file from top-level of a Python library directory
override_dh_install:
dh_install
find debian -name "*nspkg.pth" -delete
[pytest]
python_files = *.py
addopts = --doctest-modules
[aliases]
test = pytest
[egg_info]
tag_build = .dev43
tag_build = .dev48
tag_date = 0
......@@ -16,36 +16,12 @@ kwargs = dict(
package_dir={ '': 'src' },
packages=find_packages( 'src' ),
install_requires=[ 'future' ],
setup_requires=['pytest-runner'],
tests_require=[
'pytest==2.7.2',
'pytest==3.5.0',
'mock==1.0.1',
'lockfile==0.11.0',
'boto==2.38.0'],
namespace_packages=[ 'bd2k' ] )
from setuptools.command.test import test as TestCommand
class PyTest( TestCommand ):
user_options = [ ('pytest-args=', 'a', "Arguments to pass to py.test") ]
def initialize_options( self ):
TestCommand.initialize_options( self )
self.pytest_args = [ ]
def finalize_options( self ):
TestCommand.finalize_options( self )
self.test_args = [ ]
self.test_suite = True
def run_tests( self ):
import pytest
# Sanitize command line arguments to avoid confusing Toil code attempting to parse them
sys.argv[ 1: ] = [ ]
errno = pytest.main( self.pytest_args )
sys.exit( errno )
kwargs[ 'cmdclass' ] = { 'test': PyTest }
setup( **kwargs )
......@@ -109,7 +109,8 @@ def ilen( it ):
"""
Return the number of elements in an iterable
>>> ilen(xrange(0,100))
>>> from builtins import range
>>> ilen(range(0,100))
100
"""
return sum( 1 for _ in it )
......
......@@ -152,7 +152,8 @@ def rindex( l, v ):
2
>>> rindex( (0,1,0,1), 0 )
2
>>> rindex( xrange(3), 2 )
>>> from builtins import range
>>> rindex( range(3), 2 )
2
"""
try:
......
......@@ -17,8 +17,8 @@ from __future__ import division
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# Inspired by Dominic Tarr's JavaScript at https://github.com/dominictarr/d64
from builtins import str
import codecs
import sys
from builtins import range
from builtins import object
from past.utils import old_div
......@@ -29,7 +29,7 @@ class D32( object ):
def __init__( self, alphabet ):
super( D32, self ).__init__( )
self.alphabet = bytearray( alphabet )
self.alphabet = bytearray( alphabet.encode('utf-8') )
self.lookup = bytearray( 255 )
for i in range( 32 ):
self.lookup[ self.alphabet[ i ] ] = i
......@@ -37,15 +37,15 @@ class D32( object ):
def encode( self, d ):
"""
>>> encode = standard.encode
>>> encode('')
>>> encode(b'') # doctest: +ALLOW_UNICODE
''
>>> encode('\\0')
>>> encode(b'\\0') # doctest: +ALLOW_UNICODE
'22'
>>> encode('\\xff')
>>> encode(b'\\xff') # doctest: +ALLOW_UNICODE
'zw'
>>> encode('\\0\\1\\2\\3\\4')
>>> encode(b'\\0\\1\\2\\3\\4') # doctest: +ALLOW_UNICODE
'222k62s6'
>>> encode('\\0\\1\\2\\3\\4\\5')
>>> encode(b'\\0\\1\\2\\3\\4\\5') # doctest: +ALLOW_UNICODE
'222k62s62o'
"""
m = len( d )
......@@ -57,7 +57,7 @@ class D32( object ):
while i < m:
if m - i < 5:
g = bytearray( d[ i: ] + '\0' * (5 - (m - i)) )
g = bytearray( d[ i: ] + b'\0' * (5 - (m - i)))
else:
g = bytearray( d[ i:i + 5 ] )
# bit 1 2 3
......@@ -74,17 +74,18 @@ class D32( object ):
e[ j + 7 ] = a[ g[ 4 ] & 31 ]
j += 8
i += 5
return str( e[ :-padding ] )
return codecs.decode( e[ :-padding ], 'ASCII' )
def decode( self, e ):
"""
>>> import codecs
>>> decode = standard.decode
# >>> decode('222k62s62o')
# '\\x00\\x01\\x02\\x03\\x04\\x05'
# >>> decode('222k62s6')
# '\\x00\\x01\\x02\\x03\\x04'
>>> decode('zw')
>>> codecs.decode(decode('zw'), 'unicode-escape') # # doctest: +ALLOW_UNICODE
'\\xff'
"""
n = len( e )
......@@ -109,6 +110,7 @@ class D32( object ):
d[ i + 4 ] = g[ 6 ] << 5 & 255 | g[ 7 ]
j += 8
i += 5
return bytes(d[ :-padding ])
......
......@@ -19,17 +19,17 @@ from __future__ import division
# Ported from JS found at https://github.com/dominictarr/d64
from builtins import str
import codecs
from builtins import bytes
from builtins import range
from builtins import object
from past.utils import old_div
class D64( object ):
def __init__( self, special_chars ):
super( D64, self ).__init__( )
alphabet = 'PYFGCRLAOEUIDHTNSQJKXBMWVZpyfgcrlaoeuidhtnsqjkxbmwvz1234567890'
self.alphabet = bytearray( sorted( alphabet + special_chars ) )
self.alphabet = bytearray( str(''.join(sorted( alphabet + special_chars))).encode( 'utf-8' ))
self.lookup = bytearray( 255 )
for i in range( 64 ):
code = self.alphabet[ i ]
......@@ -38,24 +38,25 @@ class D64( object ):
def encode( self, data ):
"""
>>> encode = standard.encode
>>> encode('')
>>> encode(b'') # doctest: +ALLOW_UNICODE
''
>>> encode('\\x00')
>>> encode(b'\\x00') # doctest: +ALLOW_UNICODE
'..'
>>> encode('\\x00\\x01')
>>> encode(b'\\x00\\x01') # doctest: +ALLOW_UNICODE
'..3'
>>> encode('\\x00\\x01\\x02')
>>> encode(b'\\x00\\x01\\x02') # doctest: +ALLOW_UNICODE
'..31'
>>> encode('\\x00\\x01\\x02\\x03\\x04\\x05\\x06\\x07')
>>> encode(b'\\x00\\x01\\x02\\x03\\x04\\x05\\x06\\x07') # doctest: +ALLOW_UNICODE
'..31.kF40VR'
"""
data = bytes( data )
l = len( data )
s = bytearray( old_div((l * 4 + 2), 3) )
hang = 0
j = 0
a = self.alphabet
for i in range( l ):
v = ord( data[ i ] )
v = data[ i ]
r = i % 3
if r == 0:
s[ j ] = a[ v >> 2 ]
......@@ -76,20 +77,21 @@ class D64( object ):
if l % 3:
s[ j ] = a[ hang ]
return str( s )
return codecs.decode( s )
def decode( self, e ):
"""
>>> import codecs
>>> decode = standard.decode
>>> decode('')
>>> codecs.decode(decode(''), 'unicode-escape') # doctest: +ALLOW_UNICODE
''
>>> decode('..')
>>> codecs.decode(decode('..'), 'unicode-escape') # doctest: +ALLOW_UNICODE
'\\x00'
>>> decode('..3')
>>> codecs.decode(decode('..3'), 'unicode-escape') # doctest: +ALLOW_UNICODE
'\\x00\\x01'
>>> decode('..31')
>>> codecs.decode(decode('..31'), 'unicode-escape') # doctest: +ALLOW_UNICODE
'\\x00\\x01\\x02'
>>> decode('..31.kF40VR')
>>> codecs.decode(decode('..31.kF40VR'), 'unicode-escape') # doctest: +ALLOW_UNICODE
'\\x00\\x01\\x02\\x03\\x04\\x05\\x06\\x07'
"""
n = len( e )
......
from future.utils import raise_
from builtins import object
from contextlib import contextmanager
import sys
......@@ -35,7 +36,7 @@ class panic( object ):
if self.log is not None and exc_info and exc_info[ 0 ]:
self.log.warn( "Exception during panic", exc_info=exc_info )
exc_type, exc_value, traceback = self.exc_info
raise exc_type, exc_value, traceback
raise_(exc_type, exc_value, traceback)
class RequirementError( Exception ):
......@@ -61,17 +62,17 @@ def require( value, message, *message_args ):
>>> require(1 + 1 == 2, 'You made a terrible mistake')
>>> require(1 + 1 == 3, 'You made a terrible mistake')
>>> require(1 + 1 == 3, 'You made a terrible mistake') # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
RequirementError: You made a terrible mistake
>>> require(1 + 1 == 3, 'You made a terrible mistake, %s', 'you fool')
>>> require(1 + 1 == 3, 'You made a terrible mistake, %s', 'you fool') # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
RequirementError: You made a terrible mistake, you fool
>>> require(1 + 1 == 3, 'You made a terrible mistake, %s %s', 'your', 'majesty')
>>> require(1 + 1 == 3, 'You made a terrible mistake, %s %s', 'your', 'majesty') # doctest: +IGNORE_EXCEPTION_DETAIL
Traceback (most recent call last):
...
RequirementError: You made a terrible mistake, your majesty
......
......@@ -24,18 +24,16 @@ class Expando(dict):
>>> import json
>>> s='{"foo":42}'
>>> o = json.loads(s,object_hook=Expando)
>>> o
{u'foo': 42}
>>> o.foo
42
>>> o.bar = 'hi'
>>> o
{u'foo': 42, 'bar': 'hi'}
>>> o.bar
'hi'
And since Expando is a dict, it serializes back to JSON just fine:
>>> json.dumps(o)
'{"foo": 42, "bar": "hi"}'
>>> json.dumps(o, sort_keys=True)
'{"bar": "hi", "foo": 42}'
Attributes can be deleted, too:
......@@ -104,8 +102,8 @@ class MagicExpando(Expando):
>>> o
{'foo': 42}
>>> o.bar.hello = 'hi'
>>> o
{'foo': 42, 'bar': {'hello': 'hi'}}
>>> o.bar
{'hello': 'hi'}
"""
def __getattribute__( self, name ):
try:
......
......@@ -49,10 +49,10 @@ def copyfileobj( src, dst, limit=None, bufsize=1024 * 1024 ):
number of bytes could be read.
>>> import tempfile
>>> with open('/dev/urandom') as f1:
>>> with open('/dev/urandom', 'rb') as f1:
... with tempfile.TemporaryFile() as f2:
... copyfileobj(f1,f2,limit=100)
... f2.seek(60)
... a = f2.seek(60)
... with tempfile.TemporaryFile() as f3:
... copyfileobj(f2,f3), f2.tell(), f3.tell()
(None, 100, 40)
......
from builtins import str
from builtins import next
from past.builtins import basestring
def hash_json( hash_obj, value ):
"""
......@@ -18,8 +17,9 @@ def hash_json( hash_obj, value ):
:param value: The value to be hashed
>>> import hashlib
>>> from builtins import str
>>> def actual(x): h = hashlib.md5(); hash_json(h,x); return h.hexdigest()
>>> def expect(s): h = hashlib.md5(); h.update(s); return h.hexdigest()
>>> def expect(s): h = hashlib.md5(); h.update(s.encode('utf-8')); return h.hexdigest()
>>> actual(0) == expect('0')
True
......@@ -31,7 +31,7 @@ def hash_json( hash_obj, value ):
True
>>> actual(False) == expect('false')
True
>>> actual("") == expect('""')
>>> actual(u"") == expect(u'""')
True
>>> actual([]) == expect('[]')
True
......@@ -54,17 +54,17 @@ def hash_json( hash_obj, value ):
>>> actual({0:0})
Traceback (most recent call last):
...
ValueError: Dictionary keys must be strings, not <type 'int'>
ValueError: Dictionary keys must be strings, not type "int".
>>> actual(object())
Traceback (most recent call last):
...
ValueError: Type <type 'object'> is not supported
ValueError: Type "object" is not supported.
"""
try:
items = iter(value.items( ))
except AttributeError:
# Must check for string before testing iterability since strings are iterable
if isinstance( value, basestring ):
if isinstance( value, str ):
_hash_string( hash_obj, value )
else:
try:
......@@ -76,7 +76,7 @@ def hash_json( hash_obj, value ):
elif isinstance( value, (int, float) ):
_hash_number( hash_obj, value )
else:
raise ValueError( 'Type %s is not supported' % type( value ) )
raise ValueError( 'Type "%s" is not supported.' % type( value ).__name__ )
else:
_hash_iterable( hash_obj, iterator )
else:
......@@ -84,53 +84,53 @@ def hash_json( hash_obj, value ):
def _hash_number( hash_obj, n ):
hash_obj.update( str( n ) )
hash_obj.update( str( n ).encode('utf-8') )
def _hash_bool( hash_obj, b ):
hash_obj.update( 'true' if b else 'false' )
hash_obj.update( str('true' if b else 'false' ).encode('utf-8'))
def _hash_string( hash_obj, s ):
hash_obj.update( '"' )
hash_obj.update( s )
hash_obj.update( '"' )
hash_obj.update( '"'.encode('utf-8') )
hash_obj.update( s.encode('utf-8') )
hash_obj.update( '"'.encode('utf-8') )
def _hash_iterable( hash_obj, items ):
hash_obj.update( '[' )
hash_obj.update( '['.encode('utf-8') )
try:
item = next( items )
hash_json( hash_obj, item )
while True:
item = next( items )
hash_obj.update( ',' )
hash_obj.update( ','.encode('utf-8') )
hash_json( hash_obj, item )
except StopIteration:
pass
hash_obj.update( ']' )
hash_obj.update( ']'.encode('utf-8') )
def _hash_hashable( hash_obj, items ):
items = iter( sorted( items ) )
hash_obj.update( '{' )
hash_obj.update( '{'.encode('utf-8') )
try:
item = next( items )
_hash_hashable_item( hash_obj, item )
while True:
item = next( items )
hash_obj.update( ',' )
hash_obj.update( ','.encode('utf-8') )
_hash_hashable_item( hash_obj, item )
except StopIteration:
pass
hash_obj.update( '}' )
hash_obj.update( '}'.encode('utf-8') )
def _hash_hashable_item( hash_obj, k_v ):
(k, v) = k_v
if isinstance( k, basestring ):
hash_obj.update( k )
hash_obj.update( ':' )
hash_obj.update( k.encode('utf-8') )
hash_obj.update( ':'.encode('utf-8') )
hash_json( hash_obj, v )
else:
raise ValueError( 'Dictionary keys must be strings, not %s' % type( k ) )
raise ValueError( 'Dictionary keys must be strings, not type "%s".' % type( k ).__name__ )
import sys
from builtins import map
from builtins import zip
from builtins import object
from itertools import takewhile, dropwhile, chain
try:
from itertools import zip_longest as zip_longest
from itertools import zip_longest
except:
from itertools import izip_longest as zip_longest
......@@ -25,7 +26,7 @@ def common_prefix( xs, ys ):
>>> list( common_prefix('A','B') )
[]
"""
return map( lambda x_y: x_y[0], takewhile( lambda a_b: a_b[0] == a_b[1], zip( xs, ys ) ) )
return [x_y[0] for x_y in takewhile( lambda a_b: a_b[0] == a_b[1], list(zip( xs, ys )) )]
def disparate_suffix( xs, ys ):
......@@ -49,7 +50,13 @@ def disparate_suffix( xs, ys ):
def flatten( iterables ):
return chain.from_iterable( iterables )
""" Flatten an iterable, except for string elements. """
for it in iterables:
if isinstance(it, str):
yield it
else:
for element in it:
yield element
# noinspection PyPep8Naming
......@@ -71,7 +78,8 @@ class concat( object ):
Note that concat() flattens (or chains) all iterable arguments into a single result iterable:
>>> list( concat( 1, xrange( 2, 4 ), 4 ) )
>>> from builtins import range
>>> list( concat( 1, range( 2, 4 ), 4 ) )
[1, 2, 3, 4]
It only does so one level deep. If you need to recursively flatten a data structure,
......@@ -79,8 +87,9 @@ class concat( object ):
If you want to prevent that flattening for an iterable argument, wrap it in concat():
>>> list( concat( 1, concat( xrange( 2, 4 ) ), 4 ) )
[1, xrange(2, 4), 4]
>>> from builtins import range
>>> list( concat( 1, concat( range( 2, 4 ) ), 4 ) )
[1, range(2, 4), 4]
Some more example.
......@@ -121,11 +130,13 @@ class concat( object ):
def expand( x ):
if isinstance( x, concat ) and len( x.args ) == 1:
i = x.args
else:
elif not isinstance(x, str):
try:
i = x.__iter__( )
except AttributeError:
i = x,
else:
i = x
return i
return flatten( map( expand, self.args ) )
......@@ -167,10 +178,12 @@ class crush( object ):
def __iter__( self ):
def expand( x ):
if isinstance(x, str):
return x
try:
# Using __iter__() instead of iter() prevents breaking up of strings
return crush( x.__iter__( ) )
except AttributeError:
return x,
return flatten( map( expand, self.iterables ) )
return flatten( list(map( expand, self.iterables )) )
......@@ -35,15 +35,6 @@ class abstractclassmethod( classmethod ):
>>> d = DemoConcrete.from_int(5) # Succeeds by calling a concrete from_int()
Initializing with 10
>>> DemoABC() # Fails because from_int() is abstract
Traceback (most recent call last):
...
TypeError: Can't instantiate abstract class DemoABC with abstract methods from_int
>>> DemoABC.from_int(5) # Fails because from_int() is not implemented
Traceback (most recent call last):
...
TypeError: Can't instantiate abstract class DemoABC with abstract methods from_int
"""
__isabstractmethod__ = True
......@@ -126,7 +117,7 @@ class InnerClass( object ):
>>> o = Outer()
>>> i = o.new_inner()
>>> i # doctest: +ELLIPSIS
<bd2k.util.objects.Inner object at ...> bound to <bd2k.util.objects.Outer object at ...>
<bd2k.util.objects.Inner...> bound to <bd2k.util.objects.Outer object at ...>
>>> i.get_outer() # doctest: +ELLIPSIS
<bd2k.util.objects.Outer object at ...>
......@@ -144,18 +135,18 @@ class InnerClass( object ):
>>> derived_outer = DerivedOuter()
>>> derived_inner = derived_outer.new_inner()
>>> derived_inner # doctest: +ELLIPSIS
<bd2k.util.objects.DerivedInner object at ...> bound to <bd2k.util.objects.DerivedOuter object at ...>
<bd2k.util.objects...> bound to <bd2k.util.objects.DerivedOuter object at ...>
>>> derived_inner.get_outer() # doctest: +ELLIPSIS
<bd2k.util.objects.DerivedOuter object at ...>
Test a static references:
>>> Outer.Inner
<class 'bd2k.util.objects.Inner'>
>>> DerivedOuter.Inner
<class 'bd2k.util.objects.Inner'>
>>> DerivedOuter.DerivedInner
<class 'bd2k.util.objects.DerivedInner'>
>>> Outer.Inner # doctest: +ELLIPSIS
<class 'bd2k.util.objects...Inner'>
>>> DerivedOuter.Inner # doctest: +ELLIPSIS
<class 'bd2k.util.objects...Inner'>
>>> DerivedOuter.DerivedInner #doctest: +ELLIPSIS
<class 'bd2k.util.objects...DerivedInner'>
Can't decorate top-level classes. Unfortunately, this is detected when the instance is
created, not when the class is defined.
......
......@@ -128,7 +128,7 @@ def retryable_http_error( e ):
def retry_http( delays=default_delays, timeout=default_timeout, predicate=retryable_http_error ):
"""
>>> i = 0
>>> for attempt in retry_http(timeout=5):
>>> for attempt in retry_http(timeout=5): # doctest: +IGNORE_EXCEPTION_DETAIL
... with attempt:
... i += 1
... raise urllib.error.HTTPError('http://www.test.com', '408', 'some message', {}, None)
......