Commit 56ade412 authored by Sebastien Badia's avatar Sebastien Badia

Imported Upstream version 1.0.2

parent 333e70f9
AllCops:
DisplayCopNames: true
Metrics/BlockNesting:
Max: 2
......@@ -24,7 +27,7 @@ Metrics/ModuleLength:
Max: 120
Metrics/ParameterLists:
Max: 3
Max: 5
CountKeywordArgs: true
Metrics/AbcSize:
......
......@@ -8,6 +8,7 @@ rvm:
- 2.0.0
- 2.1
- 2.2
- 2.3.0
- jruby
- jruby-head
- ruby-head
......
This diff is collapsed.
......@@ -20,11 +20,11 @@ end
group :test do
gem "backports"
gem "coveralls"
gem "simplecov", ">= 0.9"
gem "json", ">= 1.8.1"
gem "rspec", "~> 3.0"
gem "simplecov", ">= 0.9"
gem "json", ">= 1.8.1"
gem "rubocop", "= 0.35.1"
gem "rspec", "~> 3.0"
gem "rspec-its"
gem "rubocop"
gem "yardstick"
gem "certificate_authority"
end
......
Copyright (c) 2011-2015 Tony Arcieri, Erik Michaels-Ober, Alexey V. Zapparov, Zachary Anker
Copyright (c) 2011-2016 Tony Arcieri, Erik Michaels-Ober, Alexey V. Zapparov, Zachary Anker
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
......
# ![http.rb](https://raw.github.com/httprb/http.rb/master/logo.png)
[![Gem Version](https://badge.fury.io/rb/http.svg)](http://rubygems.org/gems/http)
[![Build Status](https://secure.travis-ci.org/httprb/http.svg?branch=master)](http://travis-ci.org/httprb/http)
[![Gem Version](https://badge.fury.io/rb/http.svg)](https://rubygems.org/gems/http)
[![Build Status](https://secure.travis-ci.org/httprb/http.svg?branch=master)](https://travis-ci.org/httprb/http)
[![Code Climate](https://codeclimate.com/github/httprb/http.svg?branch=master)](https://codeclimate.com/github/httprb/http)
[![Coverage Status](https://coveralls.io/repos/httprb/http/badge.svg?branch=master)](https://coveralls.io/r/httprb/http)
[![MIT licensed](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/httprb/http/blob/master/LICENSE.txt)
## About
http.rb is an easy-to-use client library for making requests from Ruby. It uses
a simple method chaining system for building requests, similar to Python's [Requests].
HTTP (The Gem! a.k.a. http.rb) is an easy-to-use client library for making requests
from Ruby. It uses a simple method chaining system for building requests, similar to
Python's [Requests].
Under the hood, http.rb uses [http_parser.rb], a fast HTTP parsing native
extension based on the Node.js parser and a Java port thereof. This library
......@@ -55,6 +56,10 @@ Top three reasons:
Benchmarks performed using excon's benchmarking tool
DISCLAIMER: Most benchmarks you find in READMEs are crap,
including this one. These are out-of-date. If you care about
performance, benchmark for yourself for your own use cases!
## Help and Discussion
If you need help or just want to talk about the http.rb,
......@@ -68,7 +73,7 @@ You can join by email by sending a message to:
If you believe you've found a bug, please report it at:
https://github.com/httprb/http.rb/issues
https://github.com/httprb/http/issues
## Installation
......@@ -263,42 +268,58 @@ HTTP.accept(:json).get("https://github.com/httprb/http/commit/HEAD")
This adds the appropriate Accept header for retrieving a JSON response for the
given resource.
### Reuse HTTP connection: HTTP Keep-Alive
If you have many successive requests against the same host, you better want to
reuse the same connection again and again:
If you need to make many successive requests against the same host,
you can create client with persistent connection to the host:
```ruby
contents = []
targets = %w(Hypertext_Transfer_Protocol Git GitHub Linux Hurd)
HTTP.persistent('http://en.wikipedia.org') do |http|
targets.each { |target| contents << http.get("/wiki/#{target}") }
``` ruby
begin
# create HTTP client with persistent connection to api.icndb.com:
http = HTTP.persistent "http://api.icndb.com"
# issue multiple requests using same connection:
jokes = 100.times.map { http.get("/jokes/random").to_s }
ensure
# close underlying connection when you don't need it anymore
http.close if http
end
```
If the optional code block is given, it will be passed the client with
persistent connection to the host as an argument and `client.close` will be
automatically called when the block terminates.
The value of the block will be returned:
``` ruby
jokes = HTTP.persistent "http://api.icndb.com" do |http|
100.times.map { http.get("/jokes/random").to_s }
end
```
### Celluloid::IO Support
##### NOTICE
http.rb makes it simple to make multiple concurrent HTTP requests from a
Celluloid::IO actor. Here's a parallel HTTP fetcher combining http.rb with
Celluloid::IO:
You must consume response before sending next request via persistent connection.
That means you need to call `#to_s`, `#parse` or `#flush` on response object.
In the example above we used `http.get("/jokes/random").to_s` to get response
bodies. That works perfectly fine, because `#to_s` reads off the response.
```ruby
require "celluloid/io"
require "http"
Sometimes you don't need response body, or need whole response object to
access it's status, headers etc instead. You can either call `#to_s` to
make sure response was flushed and then use response object itself, or use
`#flush` (syntax sugar for `#tap(&:to_s)` that will do that for you:
class HttpFetcher
include Celluloid::IO
def fetch(url)
HTTP.get(url, socket_class: Celluloid::IO::TCPSocket)
``` ruby
contents = HTTP.persistent "http://en.wikipedia.org" do |http|
%w(Hypertext_Transfer_Protocol Git GitHub Linux Hurd).map do
http.get("/wiki/#{target}").flush
end
end
```
There's a little more to it, but that's the core idea!
* [Full parallel HTTP fetcher example](https://github.com/httprb/http/wiki/Parallel-requests-with-Celluloid%3A%3AIO)
* See also: [Celluloid::IO](https://github.com/celluloid/celluloid-io)
### Timeouts
......@@ -339,8 +360,9 @@ versions:
* Ruby 2.0.0
* Ruby 2.1.x
* Ruby 2.2.x
* Ruby 2.3.x
* JRuby 1.7.x
* JRuby 9000
* JRuby 9000+
If something doesn't work on one of these versions, it's a bug.
......@@ -370,5 +392,5 @@ dropped.
## Copyright
Copyright (c) 2011-2015 Tony Arcieri, Erik Michaels-Ober, Alexey V. Zapparov, Zachary Anker.
Copyright (c) 2011-2016 Tony Arcieri, Erik Michaels-Ober, Alexey V. Zapparov, Zachary Anker.
See LICENSE.txt for further details.
#!/usr/bin/env ruby
#
# Example of using the HTTP Gem with Celluloid::IO
# Make sure to 'gem install celluloid-io' before running
#
# Run as: bundle exec examples/parallel_requests_with_celluloid.rb
#
require "celluloid/io"
require "http"
class HttpFetcher
include Celluloid::IO
def fetch(url)
# Note: For SSL support specify:
# ssl_socket_class: Celluloid::IO::SSLSocket
HTTP.get(url, :socket_class => Celluloid::IO::TCPSocket)
end
end
fetcher = HttpFetcher.new
urls = %w(http://ruby-lang.org/ http://rubygems.org/ http://celluloid.io/)
# Kick off a bunch of future calls to HttpFetcher to grab the URLs in parallel
futures = urls.map { |u| [u, fetcher.future.fetch(u)] }
# Consume the results as they come in
futures.each do |url, future|
# Wait for HttpFetcher#fetch to complete for this request
response = future.value
puts "Got #{url}: #{response.inspect}"
end
# Suppress Celluloid's shutdown messages
# Otherwise the example is a bit noisy :|
exit!
......@@ -23,5 +23,3 @@ module HTTP
alias_method :[], :headers
end
end
Http = HTTP unless defined?(Http)
......@@ -83,7 +83,10 @@ module HTTP
# @option options [Float] :write Write timeout
# @option options [Float] :connect Connect timeout
def timeout(klass, options = {})
klass, options = :per_operation, klass if klass.is_a? Hash
if klass.is_a? Hash
options = klass
klass = :per_operation
end
klass = case klass.to_sym
when :null then HTTP::Timeout::Null
......@@ -166,29 +169,22 @@ module HTTP
branch default_options.with_follow opts
end
# @deprecated will be removed in 1.0.0
# @see #follow
alias_method :with_follow, :follow
# Make a request with the given headers
# @param headers
def headers(headers)
branch default_options.with_headers(headers)
end
# @deprecated will be removed in 1.0.0
# @see #headers
alias_method :with, :headers
# @deprecated will be removed in 1.0.0
# @see #headers
alias_method :with_headers, :headers
# Make a request with the given cookies
def cookies(cookies)
branch default_options.with_cookies(cookies)
end
# Force a specific encoding for response body
def encoding(encoding)
branch default_options.with_encoding(encoding)
end
# Accept the given MIME type(s)
# @param type
def accept(type)
......@@ -197,10 +193,7 @@ module HTTP
# Make a request with the given Authorization header
# @param [#to_s] value Authorization header value
def auth(value, opts = nil)
# shim for deprecated auth(:basic, opts).
# will be removed in 0.8.0
return basic_auth(opts) if :basic == value
def auth(value)
headers Headers::AUTHORIZATION => value.to_s
end
......@@ -229,19 +222,9 @@ module HTTP
@default_options = HTTP::Options.new(opts)
end
# @deprecated Will be removed in 1.0.0; Use `#default_options#headers`
# Get headers of HTTP options
def default_headers
default_options.headers
end
# Set headers of HTTP options
# @deprecated Will be removed in 1.0.0; Use `#headers`
# @param headers
def default_headers=(headers)
@default_options = default_options.dup do |opts|
opts.headers = headers
end
# Set TCP_NODELAY on the socket
def nodelay
branch default_options.with_nodelay(true)
end
private
......
......@@ -32,13 +32,19 @@ module HTTP
body = make_request_body(opts, headers)
proxy = opts.proxy
req = HTTP::Request.new(verb, uri, headers, proxy, body)
res = perform req, opts
req = HTTP::Request.new(
:verb => verb,
:uri => uri,
:headers => headers,
:proxy => proxy,
:body => body
)
res = perform(req, opts)
return res unless opts.follow
Redirector.new(opts.follow).perform req, res do |request|
perform request, opts
Redirector.new(opts.follow).perform(req, res) do |request|
perform(request, opts)
end
end
......@@ -61,11 +67,12 @@ module HTTP
end
res = Response.new(
@connection.status_code,
@connection.http_version,
@connection.headers,
Response::Body.new(@connection),
req.uri
:status => @connection.status_code,
:version => @connection.http_version,
:headers => @connection.headers,
:connection => @connection,
:encoding => options.encoding,
:uri => req.uri
)
@connection.finish_response if req.verb == :head
......@@ -73,9 +80,7 @@ module HTTP
res
rescue
# On any exception we reset the conn. This is a safety measure, to ensure
# we don't have conns in a bad state resulting in mixed requests/responses
close if persistent?
close
raise
end
......
......@@ -20,21 +20,24 @@ module HTTP
# @param [HTTP::Request] req
# @param [HTTP::Options] options
# @raise [HTTP::ConnectionError] when failed to connect
def initialize(req, options)
@persistent = options.persistent?
@keep_alive_timeout = options[:keep_alive_timeout].to_f
@keep_alive_timeout = options.keep_alive_timeout.to_f
@pending_request = false
@pending_response = false
@failed_proxy_connect = false
@parser = Response::Parser.new
@socket = options[:timeout_class].new(options[:timeout_options])
@socket.connect(options[:socket_class], req.socket_host, req.socket_port)
@socket = options.timeout_class.new(options.timeout_options)
@socket.connect(options.socket_class, req.socket_host, req.socket_port, options.nodelay)
send_proxy_connect_request(req)
start_tls(req, options)
reset_timer
rescue SocketError, SystemCallError => e
raise ConnectionError, "failed to connect: #{e}"
end
# @see (HTTP::Response::Parser#status_code)
......@@ -104,7 +107,7 @@ module HTTP
set_keep_alive
rescue IOError, Errno::ECONNRESET, Errno::EPIPE => e
raise IOError, "problem making HTTP request: #{e}"
raise ConnectionError, "failed to read headers: #{e}"
end
# Callback for when we've reached the end of a response
......@@ -148,14 +151,14 @@ module HTTP
def start_tls(req, options)
return unless req.uri.https? && !failed_proxy_connect?
ssl_context = options[:ssl_context]
ssl_context = options.ssl_context
unless ssl_context
ssl_context = OpenSSL::SSL::SSLContext.new
ssl_context.set_params(options[:ssl] || {})
ssl_context.set_params(options.ssl || {})
end
@socket.start_tls(req.uri.host, options[:ssl_socket_class], ssl_context)
@socket.start_tls(req.uri.host, options.ssl_socket_class, ssl_context)
end
# Open tunnel through proxy
......
......@@ -2,6 +2,9 @@ module HTTP
# Generic error
class Error < StandardError; end
# Generic Connection error
class ConnectionError < Error; end
# Generic Request error
class RequestError < Error; end
......
......@@ -51,9 +51,6 @@ module HTTP
Array(value).each { |v| @pile << [name, v.to_s] }
end
# @deprecated Will be removed in 1.0.0
alias_method :append, :add
# Returns list of header values if any.
#
# @return [Array<String>]
......
......@@ -43,11 +43,13 @@ module HTTP
:timeout_class => self.class.default_timeout_class,
:timeout_options => {},
:socket_class => self.class.default_socket_class,
:nodelay => false,
:ssl_socket_class => self.class.default_ssl_socket_class,
:ssl => {},
:keep_alive_timeout => 5,
:headers => {},
:cookies => {}
:cookies => {},
:encoding => nil
}
opts_w_defaults = defaults.merge(options)
......@@ -66,9 +68,13 @@ module HTTP
end
end
def_option :encoding do |encoding|
self.encoding = Encoding.find(encoding)
end
%w(
proxy params form json body follow response
socket_class ssl_socket_class ssl_context ssl
socket_class nodelay ssl_socket_class ssl_context ssl
persistent keep_alive_timeout timeout_class timeout_options
).each do |method_name|
def_option method_name
......@@ -91,15 +97,6 @@ module HTTP
!persistent.nil?
end
# @deprecated
def [](option)
send(option)
rescue
warn "[DEPRECATED] `HTTP::Options#[:#{option}]` was deprecated. " \
"Use `HTTP::Options##{option}` instead."
nil
end
def merge(other)
h1 = to_hash
h2 = other.to_hash
......@@ -119,7 +116,7 @@ module HTTP
def to_hash
hash_pairs = self.class.
defined_options.
flat_map { |opt_name| [opt_name, self[opt_name]] }
flat_map { |opt_name| [opt_name, send(opt_name)] }
Hash[*hash_pairs]
end
......
......@@ -54,6 +54,8 @@ module HTTP
fail TooManyRedirectsError if too_many_hops?
fail EndlessRedirectError if endless_loop?
@response.flush
@request = redirect_to @response.headers[Headers::LOCATION]
@response = yield @request
end
......
......@@ -63,20 +63,24 @@ module HTTP
attr_reader :uri
attr_reader :proxy, :body, :version
# :nodoc:
def initialize(verb, uri, headers = {}, proxy = {}, body = nil, version = "1.1") # rubocop:disable ParameterLists
@verb = verb.to_s.downcase.to_sym
@uri = normalize_uri uri
@scheme = @uri.scheme && @uri.scheme.to_s.downcase.to_sym
# @option opts [String] :version
# @option opts [#to_s] :verb HTTP request method
# @option opts [HTTP::URI, #to_s] :uri
# @option opts [Hash] :headers
# @option opts [Hash] :proxy
# @option opts [String] :body
def initialize(opts)
@verb = opts.fetch(:verb).to_s.downcase.to_sym
@uri = normalize_uri(opts.fetch :uri)
@scheme = @uri.scheme.to_s.downcase.to_sym if @uri.scheme
fail(UnsupportedMethodError, "unknown method: #{verb}") unless METHODS.include?(@verb)
fail(UnsupportedSchemeError, "unknown scheme: #{scheme}") unless SCHEMES.include?(@scheme)
@proxy = proxy
@body = body
@version = version
@headers = HTTP::Headers.coerce(headers || {})
@proxy = opts[:proxy] || {}
@body = opts[:body]
@version = opts[:version] || "1.1"
@headers = HTTP::Headers.coerce(opts[:headers] || {})
@headers[Headers::HOST] ||= default_host_header_value
@headers[Headers::USER_AGENT] ||= USER_AGENT
......@@ -84,7 +88,15 @@ module HTTP
# Returns new Request with updated uri
def redirect(uri, verb = @verb)
req = self.class.new(verb, @uri.join(uri), headers, proxy, body, version)
req = self.class.new(
:verb => verb,
:uri => @uri.join(uri),
:headers => headers,
:proxy => proxy,
:body => body,
:version => version
)
req[Headers::HOST] = req.uri.host
req
end
......@@ -126,9 +138,6 @@ module HTTP
"#{verb.to_s.upcase} #{request_uri.omit :fragment} HTTP/#{version}"
end
# @deprecated Will be removed in 1.0.0
alias_method :request_header, :headline
# Compute HTTP request header SSL proxy connection
def proxy_connect_header
"CONNECT #{host}:#{port} HTTP/#{version}"
......
......@@ -18,7 +18,7 @@ module HTTP
# Types valid to be used as body source
VALID_BODY_TYPES = [String, NilClass, Enumerable]
def initialize(socket, body, headers, headline) # rubocop:disable ParameterLists
def initialize(socket, body, headers, headline)
@body = body
@socket = socket
@headers = headers
......@@ -36,8 +36,9 @@ module HTTP
# Stream the request to a socket
def stream
send_request_header
send_request_body
add_headers
add_body_type_headers
send_request
end
# Send headers needed to connect through proxy
......@@ -64,23 +65,27 @@ module HTTP
@request_header.join(CRLF) + (CRLF) * 2
end
def send_request_header
add_headers
add_body_type_headers
write(join_headers)
end
def send_request
headers = join_headers
# It's important to send the request in a single write call when
# possible in order to play nicely with Nagle's algorithm. Making
# two writes in a row triggers a pathological case where Nagle is
# expecting a third write that never happens.
case @body
when NilClass
write(headers)
when String
write(headers << @body)
when Enumerable
write(headers)
def send_request_body
if @body.is_a?(String)
write(@body)
elsif @body.is_a?(Enumerable)
@body.each do |chunk|
write(chunk.bytesize.to_s(16) << CRLF)
write(chunk << CRLF)
write(chunk.bytesize.to_s(16) << CRLF << chunk << CRLF)
end
write(CHUNKED_END)
else fail TypeError, "invalid body type: #{@body.class}"
end
end
......
......@@ -14,13 +14,6 @@ module HTTP
include HTTP::Headers::Mixin
# @deprecated Will be removed in 1.0.0
# Use Status::REASONS
STATUS_CODES = Status::REASONS
# @deprecated Will be removed in 1.0.0
SYMBOL_TO_STATUS_CODE = Hash[STATUS_CODES.map { |k, v| [v.downcase.gsub(/\s|-/, "_").to_sym, k] }].freeze
# @return [Status]
attr_reader :status
......@@ -30,12 +23,29 @@ module HTTP
# @return [URI, nil]
attr_reader :uri
def initialize(status, version, headers, body, uri = nil) # rubocop:disable ParameterLists
@version = version
@body = body
@uri = uri && HTTP::URI.parse(uri)
@status = HTTP::Response::Status.new status
@headers = HTTP::Headers.coerce(headers || {})
# Inits a new instance
#
# @option opts [Integer] :status Status code
# @option opts [String] :version HTTP version
# @option opts [Hash] :headers
# @option opts [HTTP::Connection] :connection
# @option opts [String] :encoding Encoding to use when reading body
# @option opts [String] :body
# @option opts [String] :uri
def initialize(opts)
@version = opts.fetch(:version)
@uri = HTTP::URI.parse(opts.fetch :uri) if opts.include? :uri
@status = HTTP::Response::Status.new(opts.fetch :status)
@headers = HTTP::Headers.coerce(opts[:headers] || {})
if opts.include?(:connection)
connection = opts.fetch(:connection)
encoding = opts[:encoding] || charset || Encoding::BINARY
@body = Response::Body.new(connection, encoding)
else
@body = opts.fetch(:body)
end
end
# @!method reason
......@@ -46,9 +56,6 @@ module HTTP
# @return (see HTTP::Response::Status#code)
def_delegator :status, :code
# @deprecated Will be removed in 1.0.0
alias_method :status_code, :code
# @!method to_s
# (see HTTP::Response::Body#to_s)
def_delegator :body, :to_s
......@@ -80,19 +87,15 @@ module HTTP
@content_type ||= ContentType.parse headers[Headers::CONTENT_TYPE]
end
# MIME type of response (if any)
#
# @return [String, nil]
def mime_type
@mime_type ||= content_type.mime_type
end
# @!method mime_type
# MIME type of response (if any)
# @return [String, nil]
def_delegator :content_type, :mime_type
# Charset of response (if any)
#
# @return [String, nil]
def charset
@charset ||= content_type.charset
end
# @!method charset
# Charset of response (if any)
# @return [String, nil]
def_delegator :content_type, :charset
def cookies
@cookies ||= headers.each_with_object CookieJar.new do |(k, v), jar|
......
......@@ -9,10 +9,11 @@ module HTTP
include Enumerable
def_delegator :to_s, :empty?
def initialize(client)
@client = client
@streaming = nil
@contents = nil
def initialize(client, encoding = Encoding::BINARY)
@client = client
@streaming = nil
@contents = nil
@encoding = encoding
end
# (see HTTP::Client#readpartial)
......@@ -36,9 +37,9 @@ module HTTP
begin
@streaming = false
@contents = "".force_encoding(Encoding::UTF_8)
@contents = "".force_encoding(@encoding)
while (chunk = @client.readpartial)
@contents << chunk.force_encoding(Encoding::ASCII_8BIT)
@contents << chunk.force_encoding(@encoding)
end
rescue
@contents = nil
......