Commit 5ebf124c authored by Cédric Boutillier's avatar Cédric Boutillier

Import Upstream version 1.1.0

parents
.DS_Store
*.gem
/.vscode/
/coverage/
/Gemfile.lock
-r ./spec/spec_helper.rb
--format Fuubar
--color
# ----- CONFIGURED -----
AllCops:
TargetRubyVersion: 2.3
DisplayCopNames: true
Style/TrailingCommaInArguments:
EnforcedStyleForMultiline: comma
Style/TrailingCommaInLiteral:
EnforcedStyleForMultiline: comma
Layout/IndentArray:
EnforcedStyle: consistent
# ----- DISABLED (metrics) -----
# Cops for metrics are disabled because they should not cause tests to fail.
Metrics/AbcSize:
Enabled: false
Metrics/BlockLength:
Enabled: false
Metrics/BlockNesting:
Enabled: false
Metrics/ClassLength:
Enabled: false
Metrics/CyclomaticComplexity:
Enabled: false
Metrics/LineLength:
Enabled: false
Metrics/MethodLength:
Enabled: false
Metrics/ModuleLength:
Enabled: false
Metrics/ParameterLists:
Enabled: false
Metrics/PerceivedComplexity:
Enabled: false
# ----- DISABLED (opinionated) -----
# It does not make sense to enforce everything to have documentation.
Style/Documentation:
Enabled: false
# This does not always make sense
Style/GuardClause:
Enabled: false
language: ruby
rvm:
- "2.3"
- "2.4"
branches:
only:
- "master"
env:
global:
-
LC_ALL=en_US.UTF_8
LANG=en_US.UTF_8
cache: bundler
sudo: false
git:
depth: 10
script:
- bundle exec rake
# Contributor Covenant Code of Conduct
## Our Pledge
In the interest of fostering an open and welcoming environment, we as
contributors and maintainers pledge to making participation in our project and
our community a harassment-free experience for everyone, regardless of age, body
size, disability, ethnicity, gender identity and expression, level of experience,
nationality, personal appearance, race, religion, or sexual identity and
orientation.
## Our Standards
Examples of behavior that contributes to creating a positive environment
include:
* Using welcoming and inclusive language
* Being respectful of differing viewpoints and experiences
* Gracefully accepting constructive criticism
* Focusing on what is best for the community
* Showing empathy towards other community members
Examples of unacceptable behavior by participants include:
* The use of sexualized language or imagery and unwelcome sexual attention or
advances
* Trolling, insulting/derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or electronic
address, without explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Our Responsibilities
Project maintainers are responsible for clarifying the standards of acceptable
behavior and are expected to take appropriate and fair corrective action in
response to any instances of unacceptable behavior.
Project maintainers have the right and responsibility to remove, edit, or
reject comments, commits, code, wiki edits, issues, and other contributions
that are not aligned to this Code of Conduct, or to ban temporarily or
permanently any contributor for other behaviors that they deem inappropriate,
threatening, offensive, or harmful.
## Scope
This Code of Conduct applies both within project spaces and in public spaces
when an individual is representing the project or its community. Examples of
representing a project or community include using an official project e-mail
address, posting via an official social media account, or acting as an appointed
representative at an online or offline event. Representation of a project may be
further defined and clarified by project maintainers.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported by contacting the project team at denis.defreyne@stoneship.org. All
complaints will be reviewed and investigated and will result in a response that
is deemed necessary and appropriate to the circumstances. The project team is
obligated to maintain confidentiality with regard to the reporter of an incident.
Further details of specific enforcement policies may be posted separately.
Project maintainers who do not follow or enforce the Code of Conduct in good
faith may face temporary or permanent repercussions as determined by other
members of the project's leadership.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
available at [http://contributor-covenant.org/version/1/4][version]
[homepage]: http://contributor-covenant.org
[version]: http://contributor-covenant.org/version/1/4/
# frozen_string_literal: true
source 'https://rubygems.org'
gemspec
group :devel do
gem 'codecov', require: false
gem 'fuubar'
gem 'pry'
gem 'rake'
gem 'rspec'
gem 'rubocop', '~> 0.50'
end
The MIT License (MIT)
Copyright (c) 2017 Denis Defreyne
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
# SlowEnumeratorTools news
## 1.1.0 (2017-11-08)
Fixes:
* Handled errors in `.merge` (#3)
* Handled errors in `.batch` (#2)
Features:
* Added `.buffer` (#1)
## 1.0.0 (2017-10-21)
Changes:
* Renamed `.buffer` to `.batch`
## 1.0.0a1 (2017-10-08)
Initial release.
[![Gem version](https://img.shields.io/gem/v/slow_enumerator_tools.svg)](http://rubygems.org/gems/slow_enumerator_tools)
[![Gem downloads](https://img.shields.io/gem/dt/slow_enumerator_tools.svg)](http://rubygems.org/gems/slow_enumerator_tools)
[![Build status](https://img.shields.io/travis/ddfreyne/slow_enumerator_tools.svg)](https://travis-ci.org/ddfreyne/slow_enumerator_tools)
[![Code Climate](https://img.shields.io/codeclimate/github/ddfreyne/slow_enumerator_tools.svg)](https://codeclimate.com/github/ddfreyne/slow_enumerator_tools)
[![Code Coverage](https://img.shields.io/codecov/c/github/ddfreyne/slow_enumerator_tools.svg)](https://codecov.io/gh/ddfreyne/slow_enumerator_tools)
# SlowEnumeratorTools
_SlowEnumeratorTools_ provides tools for transforming Ruby enumerators that produce data slowly and unpredictably (e.g. from a network source):
* `SlowEnumeratorTools.merge`: given a collection of enumerables, creates a new enumerator that yields elements from any of these enumerables as soon as they become available.
* `SlowEnumeratorTools.batch`: given an enumerable, creates a new enumerable that yields batches containing all elements currently available.
* `SlowEnumeratorTools.buffer`: given an enumerable and a number, will create a buffer of that number of elements and try to fill it up with as many elements from that enumerable, so that they can be yielded immediately.
## Installation
Add this line to your application's Gemfile:
```ruby
gem 'slow_enumerator_tools'
```
And then execute:
$ bundle
Or install it yourself as:
$ gem install slow_enumerator_tools
## Usage
### `SlowEnumeratorTools.merge`
Given a collection of enumerables, creates a new enumerator that yields elements from any of these enumerables as soon as they become available.
This is useful for combining multiple event streams into a single one.
```ruby
# Generate some slow enums
enums = []
enums << 5.times.lazy.map { |i| sleep(0.1 + rand * 0.2); [:a, i] }
enums << 5.times.lazy.map { |i| sleep(0.1 + rand * 0.2); [:b, i] }
enums << 5.times.lazy.map { |i| sleep(0.1 + rand * 0.2); [:c, i] }
# Merge and print
merged_enum = SlowEnumeratorTools.merge(enums)
merged_enum.each { |e| p e }
```
Example output:
```
[:b, 0]
[:a, 0]
[:b, 1]
[:c, 0]
[:a, 1]
[:b, 2]
[:c, 1]
[:c, 2]
[:a, 2]
[:b, 3]
[:c, 3]
[:b, 4]
[:a, 3]
[:c, 4]
[:a, 4]
```
### `SlowEnumeratorTools.batch`
Given an enumerable, creates a new enumerable that yields batches containing all elements currently available.
This is useful for fetching all outstanding events on an event stream, without blocking.
```ruby
# Generate a slow enum
enum = 4.times.lazy.map { |i| sleep(0.1); i }
# Batch
batch_enum = SlowEnumeratorTools.batch(enum)
# Wait until first batch is available
# … prints [0]
p batch_enum.next
# Give it enough time for the second batch to have accumulated more elements,
# … prints [1, 2]
sleep 0.25
p batch_enum.next
# Wait until final batch is available
# … prints [3]
p batch_enum.next
```
### `SlowEnumeratorTools.buffer`
Given an enumerable and a number, will create a buffer of that number of elements and try to fill it up with as many elements from that enumerable.
This is particularly useful when reading from a slow source and writing to a slow sink, because the two will be able to work concurrently.
```ruby
# Create (fake) articles enumerator
articles =
Enumerator.new do |y|
5.times do |i|
sleep 1
y << "Article #{i}"
end
end
# Buffer
articles = SlowEnumeratorTools.buffer(articles, 5)
# Print each article
# This takes 6 seconds, rather than 10!
articles.each do |a|
sleep 1
end
```
## Development
Install dependencies:
$ bundle
Run tests:
$ bundle exec rake
## Contributing
Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/slow_enumerator_tools. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [Contributor Covenant](http://contributor-covenant.org) code of conduct.
## License
The gem is available as open source under the terms of the [MIT License](http://opensource.org/licenses/MIT).
## Code of Conduct
Everyone interacting in the SlowEnumeratorTools project’s codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/slow_enumerator_tools/blob/master/CODE_OF_CONDUCT.md).
# frozen_string_literal: true
require 'rubocop/rake_task'
require 'rspec/core/rake_task'
RSpec::Core::RakeTask.new(:spec)
RuboCop::RakeTask.new(:rubocop)
task default: :test
task test: %i[spec rubocop]
# frozen_string_literal: true
module SlowEnumeratorTools
def self.merge(es)
SlowEnumeratorTools::Merger.merge(es)
end
def self.batch(es)
SlowEnumeratorTools::Batcher.batch(es)
end
def self.buffer(es, size)
SlowEnumeratorTools::Bufferer.buffer(es, size)
end
end
require_relative 'slow_enumerator_tools/version'
require_relative 'slow_enumerator_tools/util'
require_relative 'slow_enumerator_tools/batcher'
require_relative 'slow_enumerator_tools/bufferer'
require_relative 'slow_enumerator_tools/merger'
# frozen_string_literal: true
module SlowEnumeratorTools
module Batcher
def self.batch(enum)
queue = Queue.new
t = SlowEnumeratorTools::Util.gen_collector_thread(enum, queue)
Enumerator.new do |y|
loop do
res = []
ended = false
# pop first
elem = queue.pop
if SlowEnumeratorTools::Util::STOP_OK.equal?(elem)
break
elsif SlowEnumeratorTools::Util::STOP_ERR.equal?(elem)
raise queue.pop
end
res << elem
loop do
# pop remaining
begin
elem = queue.pop(true)
rescue ThreadError
break
end
if SlowEnumeratorTools::Util::STOP_OK.equal?(elem)
ended = true
break
elsif SlowEnumeratorTools::Util::STOP_ERR.equal?(elem)
raise queue.pop
end
res << elem
end
y << res
break if ended
end
t.join
end.lazy
end
end
end
# frozen_string_literal: true
module SlowEnumeratorTools
module Bufferer
def self.buffer(enum, size)
queue = SizedQueue.new(size)
thread = SlowEnumeratorTools::Util.gen_collector_thread(enum, queue)
gen_enumerator(queue, thread)
end
def self.gen_enumerator(queue, collector_thread)
Enumerator.new do |y|
loop do
elem = queue.pop
if SlowEnumeratorTools::Util::STOP_OK.equal?(elem)
break
elsif SlowEnumeratorTools::Util::STOP_ERR.equal?(elem)
raise queue.pop
end
y << elem
end
collector_thread.join
end.lazy
end
end
end
# frozen_string_literal: true
module SlowEnumeratorTools
module Merger
def self.merge(enums)
enum = Iterator.new(enums).tap(&:start)
Enumerator.new do |y|
loop { y << enum.next }
end.lazy
end
class Iterator
def initialize(enums)
@enums = enums
@q = SizedQueue.new(5)
@done = false
end
def next
raise StopIteration if @done
nxt = @q.pop
if SlowEnumeratorTools::Util::STOP_OK.equal?(nxt)
@done = true
raise StopIteration
elsif SlowEnumeratorTools::Util::STOP_ERR.equal?(nxt)
raise @q.pop
else
nxt
end
end
def start
threads = @enums.map { |enum| spawn_empty_into(enum, @q) }
Thread.new do
threads.each(&:join)
@q << SlowEnumeratorTools::Util::STOP_OK
end
end
protected
def spawn_empty_into(enum, queue)
Thread.new do
begin
enum.each { |e| queue << e }
rescue StandardError => e
queue << SlowEnumeratorTools::Util::STOP_ERR
queue << e
end
end
end
end
end
end
# frozen_string_literal: true
module SlowEnumeratorTools
module Util
STOP_OK = Object.new
STOP_ERR = Object.new
def self.gen_collector_thread(enum, queue)
Thread.new do
begin
enum.each { |e| queue << e }
queue << STOP_OK
rescue StandardError => e
queue << STOP_ERR
queue << e
end
end
end
end
end
# frozen_string_literal: true
module SlowEnumeratorTools
VERSION = '1.1.0'
end
#!/usr/bin/env ruby
# frozen_string_literal: true
require 'fileutils'
require 'octokit'
def run(*args)
puts 'I will execute the following:'
puts ' ' + args.map { |a| a =~ /\s/ ? a.inspect : a }.join(' ')
print 'Is this correct? [y/N] '
res = gets
unless res.strip.casecmp('y').zero?
warn 'Answer was not Y; release aborted.'
exit 1
end
system('echo', *args)
system(*args)
print 'Continue? [y/N] '
res = gets
unless res.strip.casecmp('y').zero?
warn 'Answer was not Y; release aborted.'
exit 1
end
end
puts '=== Logging in to GitHub’s API…'
client = Octokit::Client.new(netrc: true)
puts
puts '=== Deleting old *.gem files…'
Dir['*.gem'].each do |fn|
puts " #{fn}…"
FileUtils.rm_f(fn)
end
puts
puts '=== Verifying presence of release date…'
unless File.readlines('NEWS.md').drop(2).first =~ / \(\d{4}-\d{2}-\d{2}\)$/
warn 'No proper release date found!'
exit 1
end
puts
puts '=== Building new gem…'
run('gem', 'build', 'slow_enumerator_tools.gemspec')
puts
puts '=== Reading version…'
require './lib/slow_enumerator_tools/version'
puts "Version = #{SlowEnumeratorTools::VERSION}"
puts
puts '=== Verifying that release does not yet exist…'
releases = client.releases('ddfreyne/slow_enumerator_tools')
release = releases.find { |r| r.tag_name == SlowEnumeratorTools::VERSION }
if release
warn 'Release already exists!'
warn 'ABORTED!'
exit 1
end
puts
puts '=== Creating Git tag…'
run('git', 'tag', '--sign', '--annotate', SlowEnumeratorTools::VERSION, '--message', "Version #{SlowEnumeratorTools::VERSION}")
puts
puts '=== Pushing Git data…'
run('git', 'push', 'origin', '--tags')
puts
puts '=== Pushing gem…'
run('gem', 'push', "slow_enumerator_tools-#{SlowEnumeratorTools::VERSION}.gem")
puts
puts '=== Reading release notes…'
release_notes =
File.readlines('NEWS.md')
.drop(4)
.take_while { |l| l !~ /^## / }
.join
puts
puts '=== Creating release on GitHub…'
sleep 3 # Give GitHub some time to detect the new tag
is_prerelease = SlowEnumeratorTools::VERSION =~ /a|b|rc/ || SlowEnumeratorTools::VERSION =~ /^0/
client.create_release(
'ddfreyne/slow_enumerator_tools', SlowEnumeratorTools::VERSION,
prerelease: !is_prerelease.nil?,
body: release_notes
)
puts
puts 'DONE!'
# frozen_string_literal: true
require_relative 'lib/slow_enumerator_tools/version'
Gem::Specification.new do |spec|
spec.name = 'slow_enumerator_tools'
spec.version = SlowEnumeratorTools::VERSION
spec.authors = ['Denis Defreyne']
spec.email = ['denis.defreyne@stoneship.org']
spec.summary = 'provides tools for transforming Ruby enumerators that produce data slowly and unpredictably'
spec.homepage = 'https://github.com/ddfreyne/slow_enumerator_tools'
spec.license = 'MIT'
spec.files = `git ls-files -z`.split("\x0")
spec.require_paths = ['lib']
spec.add_development_dependency 'bundler', '~> 1.15'
end
# frozen_string_literal: true
describe SlowEnumeratorTools::Batcher do
let(:wrapped) do
Enumerator.new do |y|
5.times do |i|
y << i
sleep 0.2
end
end
end
subject do
described_class.batch(wrapped)
end
example do
expect(subject.next).to eq([0])
end
example do
subject.next
expect(subject.next).to eq([1])
end
example do
subject
sleep 0.25
expect(subject.next).to eq([0, 1])
end
example do
subject
sleep 0.45
expect(subject.next).to eq([0, 1, 2])
end
context 'empty enumerable' do
let(:wrapped) do
Enumerator.new do |y|
end
end
it 'returns nothing' do
expect { subject.next }.to raise_error(StopIteration)
end
end
context 'instant-erroring enumerable' do
let(:wrapped) do
Enumerator.new do |_y|
raise 'boom'
end
end
it 'returns nothing' do
expect { subject.next }.to raise_error(RuntimeError, 'boom')
end
end
context 'error in taken elements' do
let(:wrapped) do
Enumerator.new do |y|
y << 1
y << 2