HTTP and REST client for Crystal, inspired by the Ruby's RestClient gem.
Beloved features:
Hopefully, someday I can remove this shard though. Ideally, Crystal's standard library would do all this already.
Add this to your application's shard.yml
:
dependencies:
crest:
github: mamantoha/crest
require "crest"
Basic usage:
Crest.get(
"http://httpbin.org/get",
params: {:lang => "en"},
user_agent: "Mozilla/5.0"
)
# curl -L http://httpbin.org/get?lang=en -H 'User-Agent: Mozilla/5.0'
Crest.post(
"http://httpbin.org/post",
{:age => 27, :name => {:first => "Kurt", :last => "Cobain"}}
)
# curl -L --data "age=27&name[first]=Kurt&name[last]=Cobain" -X POST "http://httpbin.org/post"
Crest.post(
"http://httpbin.org/post",
{"file" => File.open("avatar.png"), "name" => "John"}
)
# curl -X POST http://httpbin.org/post -F 'file=@/path/to/avatar.png' -F 'name=John' -H 'Content-Type: multipart/form-data'
response = Crest.post(
"http://httpbin.org/post",
{:age => 27, :name => {:first => "Kurt", :last => "Cobain"}},
json: true
)
# curl -X POST http://httpbin.org/post -d '{"age":27,"name":{"first":"Kurt","last":"Cobain"}}' -H 'Content-Type: application/json'
Crest::Request
accept next parameters:
Mandatory parameters:
:method
- HTTP method (:get
. :post
, :put
, :patch
, :delete
, :options
, head
):url
- URL (e.g.: http://httpbin.org/ip
)Optional parameters:
:form
- a hash containing form data (or a raw string or IO or Bytes):headers
- a hash containing the request headers:cookies
- a hash containing the request cookies:params
- a hash that represent query params (or a raw string) - a string separated from the preceding part by a question mark (?
) and a sequence of attribute–value pairs separated by a delimiter (&
):params_encoder
params encoder (default to Crest::FlatParamsEncoder
):auth
- access authentication method basic
or digest
(default to basic
):user
and :password
- for authentication:tls
- client certificates, you can pass in a custom OpenSSL::SSL::Context::Client
(default to nil
):p_addr
, :p_port
, :p_user
, and :p_pass
- specify a per-request proxy by passing these parameters:json
- make a JSON request with the appropriate HTTP headers (default to false
):multipart
make a multipart request with the appropriate HTTP headers even if not sending a file (default to false
):user_agent
- set "User-Agent" HTTP header (default to Crest::USER_AGENT
):max_redirects
- maximum number of redirects (default to 10):logging
- enable logging (default to false
):logger
- set logger (default to Crest::CommonLogger
):handle_errors
- error handling (default to true
):close_connection
- close the connection after request is completed (default to true
):http_client
- instance of HTTP::Client
:read_timeout
- read timeout (default to nil
):write_timeout
- write timeout (default to nil
):connect_timeout
- connect timeout (default to nil
)More detailed examples:
request = Crest::Request.new(:post,
"http://httpbin.org/post",
headers: {"Content-Type" => "application/json"},
form: {:width => 640, "height" => "480"}
)
request.execute
# curl -L --data "width=640&height=480" --header "Content-Type: application/json" -X POST "http://httpbin.org/post"
Crest::Request.execute(:get,
"http://httpbin.org/get",
params: {:width => 640, "height" => "480"},
headers: {"Content-Type" => "application/json"}
)
# curl -L --header "Content-Type: application/json" "http://httpbin.org/get?width=640&height=480"
Crest::Request.new(:post, "http://httpbin.org/post", {:foo => "bar"}, json: true)
# curl -X POST http://httpbin.org/post -d '{\"foo\":\"bar\"}' -H 'Content-Type: application/json'"
Crest::Request.get(
"http://httpbin.org/get",
p_addr: "127.0.0.1",
p_port: 3128,
p_user: "admin",
p_pass: "1234"
)
# curl -L --proxy admin:[email protected]:3128 "http://httpbin.org/get"
A block can be passed to the Crest::Request
initializer.
This block will then be called with the Crest::Request
.
request = Crest::Request.new(:get, "http://httpbin.org/headers") do |request|
request.headers.add("foo", "bar")
end
request.execute
# curl -L --header "foo: bar" http://httpbin.org/headers
A Crest::Resource
class can be instantiated for access to a RESTful resource,
including authentication, proxy and logging.
Additionally, you can set default params
, headers
, and cookies
separately.
So you can use Crest::Resource
to share common params
, headers
, and cookies
.
The final parameters consist of:
get
, post
, etc)This is especially useful if you wish to define your site in one place and call it in multiple locations.
resource = Crest::Resource.new(
"http://httpbin.org",
params: {"key" => "value"},
headers: {"Content-Type" => "application/json"},
cookies: {"lang" => "uk"}
)
resource["/get"].get(
headers: {"Auth-Token" => "secret"}
)
resource["/post"].post(
{:height => 100, "width" => "100"},
params: {:secret => "secret"}
)
Use the []
syntax to allocate subresources:
site = Crest::Resource.new("http://httpbin.org")
site["/post"].post({:param1 => "value1", :param2 => "value2"})
# curl -L --data "param1=value1¶m2=value2" -X POST http://httpbin.org/post
You can pass suburl
through Request#http_verb
methods:
site = Crest::Resource.new("http://httpbin.org")
site.post("/post", {:param1 => "value1", :param2 => "value2"})
# curl -L --data "param1=value1¶m2=value2" -X POST http://httpbin.org/post
site.get("/get", params: {:status => "active"})
# curl -L http://httpbin.org/get?status=active
A block can be passed to the Crest::Resource
instance.
This block will then be called with the Crest::Resource
.
resource = Crest::Resource.new("http://httpbin.org") do |resource|
resource.headers.merge!({"foo" => "bar"})
end
resource["/headers"].get
With HTTP basic authentication:
resource = Crest::Resource.new(
"http://httpbin.org/basic-auth/user/passwd",
user: "user",
password: "passwd"
)
With Proxy:
resource = Crest::Resource.new(
"http://httpbin.org/get",
p_addr: "localhost",
p_port: 3128
)
The result of a Crest::Request
and Crest::Resource
is a Crest::Response
object.
Response objects have several useful methods:
Response#body
: The response body as a String
Response#body_io
: The response body as a IO
Response#status
: The response status as a HTTP::Status
Response#status_code
: The HTTP response codeResponse#headers
: A hash of HTTP response headersResponse#cookies
: A hash of HTTP cookies set by the serverResponse#request
: The Crest::Request
object used to make the requestResponse#http_client_res
: The HTTP::Client::Response
objectResponse#history
: A list of each response received in a redirection chain200
and 207
, a Crest::Response
will be returned301
, 302
, 303
or 307
, the redirection will be followed and the request transformed into a GET
Crest::RequestFailed
holding the Crest::Response
will be raised.response
on the exception to get the server's responseCrest.get("http://httpbin.org/status/404")
# => HTTP status code 404: Not Found (Crest::NotFound)
begin
Crest.get("http://httpbin.org/status/404")
rescue ex : Crest::NotFound
puts ex.response
end
To not raise exceptions but return the Crest::Response
you can set handle_errors
to false
.
response = Crest.get("http://httpbin.org/status/404", handle_errors: false) do |resp|
case resp
when .success?
puts resp.body_io.gets_to_end
when .client_error?
puts "Client error"
when .server_error?
puts "Server error"
end
end
# => Client error
response.status_code # => 404
But note that it may be more straightforward to use exceptions to handle different HTTP error response cases:
response = begin
Crest.get("http://httpbin.org/status/404")
rescue ex : Crest::NotFound
puts "Not found"
ex.response
rescue ex : Crest::InternalServerError
puts "Internal server error"
ex.response
end
# => Not found
response.status_code # => 404
Crest::ParamsEncoder
class is used to encode parameters.
The encoder affect both how crest
processes query strings and how it serializes POST bodies.
The default encoder is Crest::FlatParamsEncoder
.
It provides #encode
method, which converts the given params into a URI query string:
Crest::FlatParamsEncoder.encode({"a" => ["one", "two", "three"], "b" => true, "c" => "C", "d" => 1})
# => 'a[]=one&a[]=two&a[]=three&b=true&c=C&d=1'
You can build a custom params encoder.
The value of Crest params_encoder
can be any subclass of Crest::ParamsEncoder
that implement #encode(Hash) #=> String
Also Crest include other encoders.
Crest::NestedParamsEncoder
response = Crest.post(
"http://httpbin.org/post",
{"size" => "small", "topping" => ["bacon", "onion"]},
params_encoder: Crest::NestedParamsEncoder
)
# => curl -X POST http://httpbin.org/post -d 'size=small&topping=bacon&topping=onion' -H 'Content-Type: application/x-www-form-urlencoded'
Crest::EnumeratedFlatParamsEncoder
response = Crest.post(
"http://httpbin.org/post",
{"size" => "small", "topping" => ["bacon", "onion"]},
params_encoder: Crest::EnumeratedFlatParamsEncoder
)
# => curl -X POST http://httpbin.org/post -d 'size=small&topping[1]=bacon&topping[2]=onion' -H 'Content-Type: application/x-www-form-urlencoded'
Crest::ZeroEnumeratedFlatParamsEncoder
response = Crest.post(
"http://httpbin.org/post",
{"size" => "small", "topping" => ["bacon", "onion"]},
params_encoder: Crest::ZeroEnumeratedFlatParamsEncoder
)
# => curl -X POST http://httpbin.org/post -d 'size=small&topping[0]=bacon&topping[1]=onion' -H 'Content-Type: application/x-www-form-urlencoded'
Normally, when you use Crest
, Crest::Request
or Crest::Resource
methods to retrieve data, the entire response is buffered in memory and returned as the response to the call.
However, if you are retrieving a large amount of data, for example, an iso, or any other large file, you may want to stream the response directly to disk rather than loading it into memory. If you have a very large file, it may become impossible to load it into memory.
If you want to stream the data from the response to a file as it comes, rather than entirely in memory, you can pass a block to which you pass a additional logic, which you can use to stream directly to a file as each chunk is received.
With a block, an Crest::Response
body is returned and the response's body is available as an IO
by invoking Crest::Response#body_io
.
The following is an example:
Crest.get("https://github.com/crystal-lang/crystal/archive/1.0.0.zip") do |resp|
filename = resp.filename || "crystal.zip"
File.open(filename, "w") do |file|
IO.copy(resp.body_io, file)
end
end
This section covers some of crest
more advanced features.
Yeah, that's right! This does multipart sends for you!
file = File.open("#{__DIR__}/example.png")
Crest.post("http://httpbin.org/post", {:image => file})
file_content = "id,name\n1,test"
file = IO::Memory.new(file_content)
Crest.post("http://httpbin.org/post", {"data.csv" => file})
file = File.open("#{__DIR__}/example.png")
resource = Crest::Resource.new("https://httpbin.org")
response = resource["/post"].post({:image => file})
crest
speaks JSON natively by passing json: true
argument to crest
.
Crest.post("http://httpbin.org/post", {:foo => "bar"}, json: true)
As well you can serialize your form to a string by itself before passing it to crest
.
Crest.post(
"http://httpbin.org/post",
{:foo => "bar"}.to_json
headers: {"Accept" => "application/json", "Content-Type" => "application/json"},
)
Request headers can be set by passing a hash containing keys and values representing header names and values:
response = Crest.get(
"http://httpbin.org/bearer",
headers: {"Authorization" => "Bearer cT0febFoD5lxAlNAXHo6g"}
)
response.headers
# => {"Authorization" => ["Bearer cT0febFoD5lxAlNAXHo6g"]}
Request
and Response
objects know about HTTP cookies, and will automatically extract and set headers for them as needed:
response = Crest.get(
"http://httpbin.org/cookies/set",
params: {"k1" => "v1", "k2" => "v2"}
)
response.cookies
# => {"k1" => "v1", "k2" => "v2"}
response = Crest.get(
"http://httpbin.org/cookies",
cookies: {"k1" => "v1", "k2" => {"kk2" => "vv2"}}
)
response.cookies
# => {"k1" => "v1", "k2[kk2]" => "vv2"}
For basic access authentication for an HTTP user agent you should to provide a user
name and password
when making a request.
Crest.get(
"http://httpbin.org/basic-auth/user/passwd",
user: "user",
password: "passwd"
)
# curl -L --user user:passwd http://httpbin.org/basic-auth/user/passwd
For digest access authentication for an HTTP user agent you should to provide a user
name and password
when making a request.
Crest.get(
"https://httpbin.org/digest-auth/auth/user/passwd/MD5",
auth: "digest",
user: "user",
password: "passwd"
)
# curl -L --digest --user user:passwd https://httpbin.org/digest-auth/auth/user/passwd/MD5
If tls
is given it will be used:
Crest.get("https://expired.badssl.com", tls: OpenSSL::SSL::Context::Client.insecure)
If you need to use a proxy, you can configure individual requests with the proxy host and port arguments to any request method:
Crest.get(
"http://httpbin.org/ip",
p_addr: "localhost",
p_port: 3128
)
To use authentication with your proxy, use next syntax:
Crest.get(
"http://httpbin.org/ip",
p_addr: "localhost",
p_port: 3128,
p_user: "user",
p_pass: "qwerty"
)
Logger
class is completely taken from halite shard. Thanks icyleaf!
By default, the Crest
does not enable logging. You can enable it per request by setting logging: true
:
Crest.get("http://httpbin.org/get", logging: true)
resource = Crest::Request.get("http://httpbin.org/get", params: {api_key => "secret"}, logging: true) do |request|
request.logger.filter(/(api_key=)(\w+)/, "\\1[REMOVED]")
end
# => crest | 2018-07-04 14:49:49 | GET | http://httpbin.org/get?api_key=[REMOVED]
You can create the custom logger by integration Crest::Logger
abstract class.
Here has two methods must be implement: Crest::Logger.request
and Crest::Logger.response
.
class MyLogger < Crest::Logger
def request(request)
@logger.info { ">> | %s | %s" % [request.method, request.url] }
end
def response(response)
@logger.info { "<< | %s | %s" % [response.status_code, response.url] }
end
end
Crest.get("http://httpbin.org/get", logging: true, logger: MyLogger.new)
By default, crest
will follow HTTP 30x redirection requests.
To disable automatic redirection, set :max_redirects => 0
.
Crest::Request.execute(method: :get, url: "http://httpbin.org/redirect/1", max_redirects: 0)
# => Crest::Found: 302 Found
You can access HTTP::Client
via the http_client
instance method.
This is usually used to set additional options (e.g. read timeout, authorization header etc.)
client = HTTP::Client.new("httpbin.org")
client.read_timeout = 1.second
begin
Crest::Request.new(:get,
"http://httpbin.org/delay/10",
http_client: client
)
rescue IO::TimeoutError
puts "Timeout!"
end
client = HTTP::Client.new("httpbin.org")
client.read_timeout = 1.second
begin
resource = Crest::Resource.new("http://httpbin.org", http_client: client)
resource.get("/delay/10")
rescue IO::TimeoutError
puts "Timeout!"
end
Use to_curl
method on instance of Crest::Request
to convert request to cURL command.
request = Crest::Request.new(
:post,
"http://httpbin.org/post",
{"title" => "New Title", "author" => "admin"}
)
request.to_curl
# => curl -X POST http://httpbin.org/post -d 'title=New+Title&author=admin' -H 'Content-Type: application/x-www-form-urlencoded'
request = Crest::Request.new(
:get,
"http://httpbin.org/basic-auth/user/passwd",
user: "user",
password: "passwd"
)
request.to_curl
# => curl -X GET http://httpbin.org/basic-auth/user/passwd --user user:passwd
Also you can directly use Crest::Curlify
which accept instance of Crest::Request
request = Crest::Request.new(:get, "http://httpbin.org")
Crest::Curlify.new(request).to_curl
# => curl -X GET http://httpbin.org
Crest::ParamsDecoder
is a module for decoding query-string into parameters.
query = "size=small&topping[1]=bacon&topping[2]=onion"
Crest::ParamsDecoder.decode(query)
# => {"size" => "small", "topping" => ["bacon", "onion"]}
Install dependencies:
shards
To run test:
crystal spec
crystal play
open http://localhost:8080
Then select the Workbook -> Requests from the menu.
Copyright: 2017-2024 Anton Maminov ([email protected])
This library is distributed under the MIT license. Please see the LICENSE file.