We are going to build curl from scratch by accepting the coding challenge posted on Coding Challenges FYI.
Before moving ahead, you must need to know how tcp client server connections works.
You can read more about GeeksforGeeks.
On Server Side : -
On Client Side : -
We are doing to acheive in few steps: -
We will be using library for Clap - A simple-to-use, efficient, and full-featured library for parsing command line arguments and subcommands.
Tha clap library provides two different ways to build parse object. First is the Builder pattern(creational design pattern to create complex things by step by step process) and second Derive pattern in which library automatically generate code based on the macros.
We are using Builder pattern for our cli tool.
But you can implement Derive pattern at doc.rs/clap/_derive
'cli.rs'
use clap::{Arg, ArgMatches, Command};
pub fn get_arguments()-> ArgMatches{
Command::new("Ccurl - custom curl")
.about("It helps to make http methods")
.version("1.0")
.author("Praveen Chaudhary <[email protected]>")
.arg(Arg::new("url").index(1).required(true))
.arg(
Arg::new("x-method")
.help("Http method which you want to use")
.long("x-method")
.short('X'),
)
.arg(
Arg::new("data")
.help("Payload you want to send with the request")
.long("data")
.short('d'),
)
.arg(
Arg::new("headers")
.help("Request header")
.long("header")
.short('H')
.action(ArgAction::Append),
)
.arg(
Arg::new("verbose")
.help("verbose mode")
.long("verbose")
.short('v')
.action(clap::ArgAction::SetTrue),
)
.get_matches()
}
Firstly, we have define the basic info like about, author and version.
We have defined all the arguments need for our own curl. We have made one positional required argument url.
Clap makes it easier to match arguments.
For verbose, we have used action method .action(clap::ArgAction::SetTrue)
because it will not contain any subsequent value.
For headers, similarly we have used action method .action(ArgAction::Append)
, Append will append new values to the previous value if any value have already encountered.
For others, we have simply used get_one method to get the value.
let verbose_enabled = matches.contains_id("verbose") && matches.get_flag("verbose");
let url = matches.get_one::<String>("url").unwrap();
let data = matches.get_one::<String>("data");
let method = matches.get_one::<String>("x-method");
let headers: Vec<&str> = matches
.get_many::<String>("headers")
.unwrap_or_default()
.map(|s| s.as_str())
.collect();
We will be using the RFC9110 for HTTP 1.1 client.
we will start with empty string, and append all the information needed for the Request according to RFC.
fn populate_get_request(
protocol: &str,
host: &str,
path: &str,
data: Option<&String>,
method: Option<&String>,
headers: Vec<&str>,
) -> String {
let default_method = String::from("GET");
let method = method.unwrap_or(&default_method);
let mut res = String::new();
res += &format!("{} /{} {}\r\n", method, path, protocol);
res += &format!("Host: {}\r\n", host);
res += "Accept: */*\r\n";
res += "Connection: close\r\n";
....
....
....
res += "\r\n";
res
}
For PUT and POST, we need to add headers and data.
fn populate_get_request(
protocol: &str,
host: &str,
path: &str,
data: Option<&String>,
method: Option<&String>,
headers: Vec<&str>,
) -> String {
....
....
if method == "POST" || method == "PUT" {
if headers.len() > 0 {
for head in headers {
res += head;
}
res += "\r\n"
} else {
res += "Content-Type: application/json\r\n";
}
if let Some(data_str) = data {
let data_bytes = data_str.as_bytes();
res += &format!("Content-Length: {}\r\n\r\n", data_bytes.len());
res += data_str;
res += "\r\n";
}
}
....
res
}
According to RFC, for post or post we need to provide Content-Length and Content-Type header.
So now we have complete request string. Let's move to socket connection, sending this request string to the server.
fn populate_get_request(
protocol: &str,
host: &str,
path: &str,
data: Option<&String>,
method: Option<&String>,
headers: Vec<&str>,
) -> String {
let default_method = String::from("GET");
let method = method.unwrap_or(&default_method);
let mut res = String::new();
res += &format!("{} /{} {}\r\n", method, path, protocol);
res += &format!("Host: {}\r\n", host);
res += "Accept: */*\r\n";
res += "Connection: close\r\n";
if method == "POST" || method == "PUT" {
if headers.len() > 0 {
for head in headers {
res += head;
}
res += "\r\n"
} else {
res += "Content-Type: application/json\r\n";
}
if let Some(data_str) = data {
let data_bytes = data_str.as_bytes();
res += &format!("Content-Length: {}\r\n\r\n", data_bytes.len());
res += data_str;
res += "\r\n";
}
}
res += "\r\n";
res
}
We will using the standard rust network library for socket connection with the host server.
fn main() {
....
....
let tcp_socket = TcpStream::connect(socket_addr);
match tcp_socket {
Ok(mut stream) => {
....
....
}
Err(e) => {
eprintln!("Failed to establish connection: {}", e);
}
}
....
....
}
Once we are successfully connected, we can listen and send your own request to server.
fn main() {
....
....
match tcp_socket {
Ok(mut stream) => {
if verbose_enabled {
let lines = buffer_str.lines();
for line in lines {
println!("> {}", line)
}
}
stream
.write_all(buffer_str.as_bytes())
.expect("Failed to write data to stream");
// initialising the buffer, reads data from the stream and stores it in the buffer.
let mut buffer = [0; 1024];
stream
.read(&mut buffer)
.expect("Failed to read from response from host!");
// converts buffer data into a UTF-8 enccoded string (lossy ensures invalid data can be truncated).
let response = String::from_utf8_lossy(&buffer[..]);
// dividing the response headers and body
let (response_header, response_data) = parse_resp(&response);
if verbose_enabled {
let lines = response_header.split("\r\n");
for line in lines {
println!("< {}", line)
}
}
println!("{}", response_data);
}
Err(e) => {
eprintln!("Failed to establish connection: {}", e);
}
}
....
....
}
cli - cargo run -- http://eu.httpbin.org:80/get
response -
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "eu.httpbin.org",
"X-Amzn-Trace-Id": "Root=1-65fec214-25771a3e732101c433ce67a7"
},
"origin": "49.36.177.79",
"url": "http://eu.httpbin.org/get"
}
Similarly, you can test others.
Hurray!! We have able to make our own curl.