💎HTTP Requests Client with a chainable REST API, built-in sessions and middlewares.
MIT License
HTTP Requests with a chainable REST API, built-in sessions and middleware written by Crystal. Inspired from the awesome Ruby's HTTP/RESTClient and Python's requests.
Build in Crystal version >= 1.0.0
, this document valid with latest commit.
Add this to your application's shard.yml
:
dependencies:
halite:
github: icyleaf/halite
require "halite"
Make a GET request:
# Direct get url
Halite.get("http://httpbin.org/get")
# Support NamedTuple as query params
Halite.get("http://httpbin.org/get", params: {
language: "crystal",
shard: "halite"
})
# Also support Hash as query params
Halite.get("http://httpbin.org/get", headers: {
"Private-Token" => "T0k3n"
}, params: {
"language" => "crystal",
"shard" => "halite"
})
# And support chainable
Halite.header(private_token: "T0k3n")
.get("http://httpbin.org/get", params: {
"language" => "crystal",
"shard" => "halite"
})
See also all chainable methods.
Many other HTTP methods are available as well:
get
head
post
put
delete
patch
options
Use the params
argument to add query string parameters to requests:
Halite.get("http://httpbin.org/get", params: { "firstname" => "Olen", "lastname" => "Rosenbaum" })
Use the form
argument to pass data serialized as form encoded:
Halite.post("http://httpbin.org/post", form: { "firstname" => "Olen", "lastname" => "Rosenbaum" })
To upload files as if form data, construct the form as follows:
Halite.post("http://httpbin.org/post", form: {
"username" => "Quincy",
"avatar" => File.open("/Users/icyleaf/quincy_avatar.png")
})
It is possible to upload multiple files:
Halite.post("http://httpbin.org/post", form: {
photos: [
File.open("/Users/icyleaf/photo1.png"),
File.open("/Users/icyleaf/photo2.png")
],
album_name: "samples"
})
Or pass the name with []
:
Halite.post("http://httpbin.org/post", form: {
"photos[]" => [
File.open("/Users/icyleaf/photo1.png"),
File.open("/Users/icyleaf/photo2.png")
],
"album_name" => "samples"
})
Multiple files can also be uploaded using both ways above, it depend on web server.
Use the json
argument to pass data serialized as body encoded:
Halite.post("http://httpbin.org/post", json: { "firstname" => "Olen", "lastname" => "Rosenbaum" })
Use the raw
argument to pass raw string as body and set the Content-Type
manually:
# Set content-type to "text/plain" by default
Halite.post("http://httpbin.org/post", raw: "name=Peter+Lee&address=%23123+Happy+Ave&language=C%2B%2B")
# Set content-type manually
Halite.post("http://httpbin.org/post",
headers: {
"content-type" => "application/json"
},
raw: %Q{{"name":"Peter Lee","address":"23123 Happy Ave","language":"C++"}}
)
Use the #basic_auth
method to perform HTTP Basic Authentication using a username and password:
Halite.basic_auth(user: "user", pass: "p@ss").get("http://httpbin.org/get")
# We can pass a raw authorization header using the auth method:
Halite.auth("Bearer dXNlcjpwQHNz").get("http://httpbin.org/get")
Use the #user_agent
method to overwrite default one:
Halite.user_agent("Crystal Client").get("http://httpbin.org/user-agent")
Here are two way to passing headers data:
#headers
methodHalite.headers(private_token: "T0k3n").get("http://httpbin.org/get")
# Also support Hash or NamedTuple
Halite.headers({ "private_token" => "T0k3n" }).get("http://httpbin.org/get")
# Or
Halite.headers({ private_token: "T0k3n" }).get("http://httpbin.org/get")
headers
argument in the available request method:Halite.get("http://httpbin.org/anything" , headers: { private_token: "T0k3n" })
Halite.post("http://httpbin.org/anything" , headers: { private_token: "T0k3n" })
The Halite.cookies
option can be used to configure cookies for a given request:
Halite.cookies(session_cookie: "6abaef100b77808ceb7fe26a3bcff1d0")
.get("http://httpbin.org/headers")
To obtain the cookies(cookie jar) for a given response, call the #cookies
method:
r = Halite.get("http://httpbin.org/cookies?set?session_cookie=6abaef100b77808ceb7fe26a3bcff1d0")
pp r.cookies
# => #<HTTP::Cookies:0x10dbed980 @cookies={"session_cookie" =>#<HTTP::Cookie:0x10ec20f00 @domain=nil, @expires=nil, @extension=nil, @http_only=false, @name="session_cookie", @path="/", @secure=false, @value="6abaef100b77808ceb7fe26a3bcff1d0">}>
The Halite.follow
method can be used for automatically following redirects(Max up to 5 times):
# Set the cookie and redirect to http://httpbin.org/cookies
Halite.follow
.get("http://httpbin.org/cookies/set/name/foo")
As above, set over 5 times, it will raise a Halite::TooManyRedirectsError
, but you can change less if you can:
Halite.follow(2)
.get("http://httpbin.org/relative-redirect/5")
It only redirects with GET
, HEAD
request and returns a 300
, 301
, 302
by default, otherwise it will raise a Halite::StateError
.
We can disable it to set :strict
to false
if we want any method(verb) requests, in which case the GET
method(verb) will be used for
that redirect:
Halite.follow(strict: false)
.post("http://httpbin.org/relative-redirect/5")
Response#history
property list contains the Response
objects that were created in order to complete the request.
The list is ordered from the oldest to most recent response.
r = Halite.follow
.get("http://httpbin.org/redirect/3")
r.uri
# => http://httpbin.org/get
r.status_code
# => 200
r.history
# => [
# #<Halite::Response HTTP/1.1 302 FOUND {"Location" => "/relative-redirect/2" ...>,
# #<Halite::Response HTTP/1.1 302 FOUND {"Location" => "/relative-redirect/1" ...>,
# #<Halite::Response HTTP/1.1 302 FOUND {"Location" => "/get" ...>,
# #<Halite::Response HTTP/1.1 200 OK {"Content-Type" => "application/json" ...>
# ]
NOTE: It contains the Response
object if you use history
and HTTP was not a 30x
, For example:
r = Halite.get("http://httpbin.org/get")
r.history.size # => 0
r = Halite.follow
.get("http://httpbin.org/get")
r.history.size # => 1
By default, the Halite does not enforce timeout on a request. We can enable per operation timeouts by configuring them through the chaining API.
The connect
timeout is the number of seconds Halite will wait for our client to establish a connection to a remote server call on the socket.
Once our client has connected to the server and sent the HTTP request,
the read
timeout is the number of seconds the client will wait for the server to send a response.
# Separate set connect and read timeout
Halite.timeout(connect: 3.0, read: 2.minutes)
.get("http://httpbin.org/anything")
# Boath set connect and read timeout
# The timeout value will be applied to both the connect and the read timeouts.
Halite.timeout(5)
.get("http://httpbin.org/anything")
The Halite supports HTTPS via Crystal's built-in OpenSSL module. All you have to do in order to use HTTPS is pass in an https://-prefixed URL.
To use client certificates, you can pass in a custom OpenSSL::SSL::Context::Client
object containing the certificates you wish to use:
tls = OpenSSL::SSL::Context::Client.new
tls.ca_certificates = File.expand_path("~/client.crt")
tls.private_key = File.expand_path("~/client.key")
Halite.get("https://httpbin.org/anything", tls: tls)
After an HTTP request, Halite::Response
object have several useful methods. (Also see the API documentation).
HTTP::Cookies
set by server.HTTP::Headers
of the response.Halite::HeaderLink
set from headers.#content_type
.Hash
of status code, response headers and body as a string.We can read the content of the server's response by call #body
:
r = Halite.get("http://httpbin.org/user-agent")
r.body
# => {"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"}
The gzip
and deflate
transfer-encodings are automatically decoded for you.
And requests will automatically decode content from the server. Most unicode charsets are seamlessly decoded.
There’s also a built-in a JSON adapter, in case you’re dealing with JSON data:
r = Halite.get("http://httpbin.org/user-agent")
r.parse("json")
r.parse # simplily by default
# => {
# => "user-agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Safari/537.36"
# => }
Halite::Response
has a MIME type adapter system that you can use a decoder to parse the content,
we can inherit Halite::MimeTypes::Adapter
to make our adapter:
# Define a MIME type adapter
class YAMLAdapter < Halite::MimeType::Adapter
def decode(string)
YAML.parse(string)
end
def encode(obj)
obj.to_yaml
end
end
# Register to Halite to invoke
Halite::MimeType.register YAMLAdapter.new, "application/x-yaml", "yaml", "yml"
# Test it!
r = Halite.get "https://raw.githubusercontent.com/icyleaf/halite/master/shard.yml"
r.parse("yaml") # or "yml"
# => {"name" => "halite", "version" => "0.4.0", "authors" => ["icyleaf <[email protected]>"], "crystal" => "0.25.0", "license" => "MIT"}
Store binary data (eg, application/octet-stream
) to file, you can use streaming requests:
Halite.get("https://github.com/icyleaf/halite/archive/master.zip") do |response|
filename = response.filename || "halite-master.zip"
File.open(filename, "w") do |file|
IO.copy(response.body_io, file)
end
end
Halite::Response
will be returned.Halite::TimeoutError
will be raised.Halite::TooManyRedirectsError
will raised.Halite::RequestError
will raised.Halite::ConnectionError
/Halite::UnsupportedMethodError
/Halite::UnsupportedSchemeError
will raised.If we made a bad request(a 4xx client error or a 5xx server error response), we can raise with Halite::Response.raise_for_status
.
But, since our status_code
was not 4xx
or 5xx
, it returns nil
when we call it:
urls = [
"https://httpbin.org/status/404",
"https://httpbin.org/status/500?foo=bar",
"https://httpbin.org/status/200",
]
urls.each do |url|
r = Halite.get url
begin
r.raise_for_status
p r.body
rescue ex : Halite::ClientError | Halite::ServerError
p "[#{ex.status_code}] #{ex.status_message} (#{ex.class})"
end
end
# => "[404] not found error with url: https://httpbin.org/status/404 (Halite::Exception::ClientError)"
# => "[500] internal server error error with url: https://httpbin.org/status/500?foo=bar (Halite::Exception::ServerError)"
# => ""
Halite now has middleware (a.k.a features) support providing a simple way to plug in intermediate custom logic in your HTTP client, allowing you to monitor outgoing requests, incoming responses, and use it as an interceptor.
Available features:
Let's implement simple middleware that prints each request:
class RequestMonister < Halite::Feature
@label : String
def initialize(**options)
@label = options.fetch(:label, "")
end
def request(request) : Halite::Request
puts @label
puts request.verb
puts request.uri
puts request.body
request
end
Halite.register_feature "request_monster", self
end
Then use it in Halite:
Halite.use("request_monster", label: "testing")
.post("http://httpbin.org/post", form: {name: "foo"})
# Or configure to client
client = Halite::Client.new do
use "request_monster", label: "testing"
end
client.post("http://httpbin.org/post", form: {name: "foo"})
# => testing
# => POST
# => http://httpbin.org/post
# => name=foo
Halite's killer feature is the interceptor, Use Halite::Feature::Chain
to process with two result:
next
: perform and run next interceptorreturn
: perform and returnSo, you can intercept and turn to the following registered features.
class AlwaysNotFound < Halite::Feature
def intercept(chain)
response = chain.perform
response = Halite::Response.new(chain.request.uri, 404, response.body, response.headers)
chain.next(response)
end
Halite.register_feature "404", self
end
class PoweredBy < Halite::Feature
def intercept(chain)
if response = chain.response
response.headers["X-Powered-By"] = "Halite"
chain.return(response)
else
chain
end
end
Halite.register_feature "powered_by", self
end
r = Halite.use("404").use("powered_by").get("http://httpbin.org/user-agent")
r.status_code # => 404
r.headers["X-Powered-By"] # => Halite
r.body # => {"user-agent":"Halite/0.6.0"}
For more implementation details about the feature layer, see the Feature class and examples and specs.
Halite provides a traditional way to instance client, and you can configure any chainable methods with block:
client = Halite::Client.new do
# Set basic auth
basic_auth "username", "password"
# Enable logging
logging true
# Set timeout
timeout 10.seconds
# Set user agent
headers user_agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36"
end
# You also can configure in this way
client.accept("application/json")
r = client.get("http://httpbin.org/get")
No more given endpoint per request, use endpoint
will make the request URI shorter, you can set it in flexible way:
client = Halite::Client.new do
endpoint "https://gitlab.org/api/v4"
user_agent "Halite"
end
client.get("users") # GET https://gitlab.org/api/v4/users
# You can override the path by using an absolute path
client.get("/users") # GET https://gitlab.org/users
As like requests.Session(), Halite built-in session by default.
Let's persist some cookies across requests:
client = Halite::Client.new
client.get("http://httpbin.org/cookies/set?private_token=6abaef100b77808ceb7fe26a3bcff1d0")
client.get("http://httpbin.org/cookies")
# => 2018-06-25 18:41:05 +08:00 | request | GET | http://httpbin.org/cookies/set?private_token=6abaef100b77808ceb7fe26a3bcff1d0
# => 2018-06-25 18:41:06 +08:00 | response | 302 | http://httpbin.org/cookies/set?private_token=6abaef100b77808ceb7fe26a3bcff1d0 | text/html
# => <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
# => <title>Redirecting...</title>
# => <h1>Redirecting...</h1>
# => <p>You should be redirected automatically to target URL: <a href="/cookies">/cookies</a>. If not click the link.
# => 2018-06-25 18:41:06 +08:00 | request | GET | http://httpbin.org/cookies
# => 2018-06-25 18:41:07 +08:00 | response | 200 | http://httpbin.org/cookies | application/json
# => {"cookies":{"private_token":"6abaef100b77808ceb7fe26a3bcff1d0"}}
All it support with chainable methods in the other examples list in requests.Session.
Note, however, that chainable methods will not be persisted across requests, even if using a session. This example will only send the cookies or headers with the first request, but not the second:
client = Halite::Client.new
r = client.cookies("username": "foobar").get("http://httpbin.org/cookies")
r.body # => {"cookies":{"username":"foobar"}}
r = client.get("http://httpbin.org/cookies")
r.body # => {"cookies":{}}
If you want to manually add cookies, headers (even features etc) to your session, use the methods start with with_
in Halite::Options
to manipulate them:
r = client.get("http://httpbin.org/cookies")
r.body # => {"cookies":{}}
client.options.with_cookie("username": "foobar")
r = client.get("http://httpbin.org/cookies")
r.body # => {"cookies":{"username":"foobar"}}
Similar to HTTP::Client usage with a block,
you can easily use same way, but Halite returns a Halite::Response
object:
r = Halite.get("http://httpbin.org/stream/5") do |response|
response.status_code # => 200
response.body_io.each_line do |line|
puts JSON.parse(line) # => {"url" => "http://httpbin.org/stream/5", "args" => {}, "headers" => {"Host" => "httpbin.org", "Connection" => "close", "User-Agent" => "Halite/0.8.0", "Accept" => "*/*", "Accept-Encoding" => "gzip, deflate"}, "id" => 0_i64}
end
end
Warning:
body_io
is avaiabled as anIO
and not reentrant safe. Might throws a "Nil assertion failed" exception if there is no data in theIO
(such likehead
requests). Calling this method multiple times causes some of the received data being lost.One more thing, use streaming requests the response will always enable redirect automatically.
Halite does not enable logging on each request and response too. We can enable per operation logging by configuring them through the chaining API.
By default, Halite will logging all outgoing HTTP requests and their responses(without binary stream) to STDOUT
on DEBUG level.
You can configuring the following options:
logging
: Instance your Halite::Logging::Abstract
, check Use the custom logging.format
: Output format, built-in common
and json
, you can write your own.file
: Write to file with path, works with format
.filemode
: Write file mode, works with format
, by default is a
. (append to bottom, create it if file is not exist)skip_request_body
: By default is false
.skip_response_body
: By default is false
.skip_benchmark
: Display elapsed time, by default is false
.colorize
: Enable colorize in terminal, only apply in common
format, by default is true
.NOTE:
format
(file
andfilemode
) andlogging
are conflict, you can not use both.
Let's try with it:
# Logging json request
Halite.logging
.get("http://httpbin.org/get", params: {name: "foobar"})
# => 2018-06-25 18:33:14 +08:00 | request | GET | http://httpbin.org/get?name=foobar
# => 2018-06-25 18:33:15 +08:00 | response | 200 | http://httpbin.org/get?name=foobar | 381.32ms | application/json
# => {"args":{"name":"foobar"},"headers":{"Accept":"*/*","Accept-Encoding":"gzip, deflate","Connection":"close","Host":"httpbin.org","User-Agent":"Halite/0.3.2"},"origin":"60.206.194.34","url":"http://httpbin.org/get?name=foobar"}
# Logging image request
Halite.logging
.get("http://httpbin.org/image/png")
# => 2018-06-25 18:34:15 +08:00 | request | GET | http://httpbin.org/image/png
# => 2018-06-25 18:34:15 +08:00 | response | 200 | http://httpbin.org/image/png | image/png
# Logging with options
Halite.logging(skip_request_body: true, skip_response_body: true)
.post("http://httpbin.org/get", form: {image: File.open("halite-logo.png")})
# => 2018-08-28 14:33:19 +08:00 | request | POST | http://httpbin.org/post
# => 2018-08-28 14:33:21 +08:00 | response | 200 | http://httpbin.org/post | 1.61s | application/json
It has JSON formatted for developer friendly logging.
Halite.logging(format: "json")
.get("http://httpbin.org/get", params: {name: "foobar"})
# Write plain text to a log file
Log.setup("halite.file", backend: Log::IOBackend.new(File.open("/tmp/halite.log", "a")))
Halite.logging(for: "halite.file", skip_benchmark: true, colorize: false)
.get("http://httpbin.org/get", params: {name: "foobar"})
# Write json data to a log file
Log.setup("halite.file", backend: Log::IOBackend.new(File.open("/tmp/halite.log", "a")))
Halite.logging(format: "json", for: "halite.file")
.get("http://httpbin.org/get", params: {name: "foobar"})
# Redirect *all* logging from Halite to a file:
Log.setup("halite", backend: Log::IOBackend.new(File.open("/tmp/halite.log", "a")))
Creating the custom logging by integration Halite::Logging::Abstract
abstract class.
Here has two methods must be implement: #request
and #response
.
class CustomLogging < Halite::Logging::Abstract
def request(request)
@logger.info { "| >> | %s | %s %s" % [request.verb, request.uri, request.body] }
end
def response(response)
@logger.info { "| << | %s | %s %s" % [response.status_code, response.uri, response.content_type] }
end
end
# Add to adapter list (optional)
Halite::Logging.register "custom", CustomLogging.new
Halite.logging(logging: CustomLogging.new)
.get("http://httpbin.org/get", params: {name: "foobar"})
# We can also call it use format name if you added it.
Halite.logging(format: "custom")
.get("http://httpbin.org/get", params: {name: "foobar"})
# => 2017-12-13 16:40:13 +08:00 | >> | GET | http://httpbin.org/get?name=foobar
# => 2017-12-13 16:40:15 +08:00 | << | 200 | http://httpbin.org/get?name=foobar application/json
Local cache feature is caching responses easily with Halite through an chainable method that is simple and elegant yet powerful. Its aim is to focus on the HTTP part of caching and do not worrying about how stuff stored, api rate limiting even works without network(offline).
It has the following options:
file
: Load cache from file. it conflict with path
and expires
.path
: The path of cache, default is "/tmp/halite/cache/"expires
: The expires time of cache, default is never expires.debug
: The debug mode of cache, default is true
With debug mode, cached response it always included some headers information:
X-Halite-Cached-From
: Cache source (cache or file)X-Halite-Cached-Key
: Cache key with verb, uri and body (return with cache, not file
passed)X-Halite-Cached-At
: Cache created timeX-Halite-Cached-Expires-At
: Cache expired time (return with cache, not file
passed)Halite.use("cache").get "http://httpbin.org/anything" # request a HTTP
r = Halite.use("cache").get "http://httpbin.org/anything" # request from local storage
r.headers # => {..., "X-Halite-Cached-At" => "2018-08-30 10:41:14 UTC", "X-Halite-Cached-By" => "Halite", "X-Halite-Cached-Expires-At" => "2018-08-30 10:41:19 UTC", "X-Halite-Cached-Key" => "2bb155e6c8c47627da3d91834eb4249a"}}
Many HTTP APIs feature Link headers. GitHub uses these for pagination in their API, for example:
r = Halite.get "https://api.github.com/users/icyleaf/repos?page=1&per_page=2"
r.links
# => {"next" =>
# => Halite::HeaderLink(
# => @params={},
# => @rel="next",
# => @target="https://api.github.com/user/17814/repos?page=2&per_page=2"),
# => "last" =>
# => Halite::HeaderLink(
# => @params={},
# => @rel="last",
# => @target="https://api.github.com/user/17814/repos?page=41&per_page=2")}
r.links["next"]
# => "https://api.github.com/user/17814/repos?page=2&per_page=2"
r.links["next"].params
# => {}
You can browse the API documents:
https://icyleaf.github.io/halite/
You can browse the all chainable methods:
https://icyleaf.github.io/halite/Halite/Chainable.html
You can browse the Changelog:
https://github.com/icyleaf/halite/blob/master/CHANGELOG.md
If you have found a bug, please create a issue here:
https://github.com/icyleaf/halite/issues/new
Your contributions are always welcome! Please submit a pull request or create an issue to add a new question, bug or feature to the list.
All Contributors are on the wall.
MIT License © icyleaf