What JavaScript Developers Should Know About Curl

by Valeri Karpov@code_barbarianJune 16, 2020

Curl is a popular command line tool often used for making HTTP requests. Curl supports a wide variety of other protocols, but, as a Node.js developer, you'll most likely use it to make HTTP requests to RESTful APIs.

Unfortunately, the curl docs list 383 supported command-line flags, which makes it difficult to find what you're looking for. In this article, I'll list out the patterns that I find myself using most often with curl, using the excellent httpbin.org service as an example.

Making an HTTP GET Request

First, double-check that you have curl installed by running curl --version.

$ curl --version
curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
Release-Date: 2018-01-24
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

To make an HTTP GET request, you just need to run curl <url>. For example, to make an HTTP GET request to https://httpbin.org/get?answer=42, you can run curl https://httpbin.org/get?answer=42.

$ curl https://httpbin.org/get?answer=42
{
  "args": {
    "answer": "42"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0", 
    "X-Amzn-Trace-Id": "Root=1-5ee8d737-b39c6a466892725bbb52b916"
  }, 
  "origin": "69.84.111.39", 
  "url": "https://httpbin.org/get?answer=42"
}
$

When the HTTP request succeeds, curl prints out the HTTP response body. To make curl print the entire response, including response headers, use the -i flag.

$ curl -i https://httpbin.org/get?answer=42
HTTP/2 200 
date: Tue, 16 Jun 2020 14:30:57 GMT
content-type: application/json
content-length: 801
server: istio-envoy
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 2

{
  "args": {
    "answer": "42"
  }, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "0", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0", 
    "X-Amzn-Trace-Id": "Root=1-5ee8d7a1-cb3954c09299eb9e0dff70a6", 
    "X-B3-Parentspanid": "dffc55451e64b5fc", 
    "X-B3-Sampled": "0", 
    "X-B3-Spanid": "8e233a863fb18b6c", 
    "X-B3-Traceid": "45bd12a9067fb5c0dffc55451e64b5fc", 
    "X-Envoy-External-Address": "10.100.91.201", 
    "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/httpbin-istio/sa/httpbin;Hash=c1ff14671b3e24ee794f9a486570abf8ccc9d622846611d3f91a322db4d480cd;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
  }, 
  "origin": "69.84.111.39,10.100.91.201", 
  "url": "http://httpbin.org/get?answer=42"
}

The above output is a raw HTTP response. The response headers are the lines from date: to x-envoy-upstream-service-time:.

Downloading a File

Wget is a common tool for downloading files via the command line. Most Linux distros come with wget built in, but, conspicuously, OSX does not come with wget.

Practically speaking, wget url is equivalent to curl -OL url. The -O option is the --remote-name option, which tells curl to store the response body in a local file. The -L option tells curl to follow redirects.

For example, below is an image from Unsplash, the URL is https://images.unsplash.com/photo-1506812574058-fc75fa93fead.

To download this image using curl, you can run:

$ curl -OL https://images.unsplash.com/photo-1506812574058-fc75fa93fead
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12.1M  100 12.1M    0     0  3927k      0  0:00:03  0:00:03 --:--:-- 3927k
$

The -O option tells curl to use everything after the last / in the URL as the file name. So, in the above example, the image will be stored in the current directory with the file name photo-1506812574058-fc75fa93fead. To specify the file name, use the -o option (lowercase 'o').

$ curl -o miami-beach.jpg https://images.unsplash.com/photo-1506812574058-fc75fa93fead
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 12.1M  100 12.1M    0     0  6083k      0  0:00:02  0:00:02 --:--:-- 6083k
$ ls -l miami-beach-jpg
-rw-rw-r-- 1 val val 12788445 Jun 16 11:03 miami-beach.jpg
$

Making a Request with Authorization

The authorization header is where you usually need to put auth credentials when talking to a RESTful API. To set the authorization header manually, you need to use the -H flag, which sets a custom request header. For example, if your API key is my-secret-token, you can attach it to your curl HTTP request as shown below.

$ curl -H "Authorization: my-secret-token" https://httpbin.org/get
{
  "args": {}, 
  "headers": {
    "Accept": "*/*", 
    "Authorization": "my-secret-token", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0", 
    "X-Amzn-Trace-Id": "Root=1-5ee8e1a5-a3aa30e0765a7980b04ca4a0"
  }, 
  "origin": "69.84.111.39", 
  "url": "https://httpbin.org/get"
}
$

Note that httpbin.org/get sends back the HTTP request headers in the response body under the headers property.

Curl also supports basic auth using the -u flag. For example, below is how you send a request with username user and password pass:

$ curl -i -u "user:pass" https://httpbin.org/basic-auth/user/pass
HTTP/2 200 
date: Tue, 16 Jun 2020 15:18:45 GMT
content-type: application/json
content-length: 47
server: istio-envoy
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 1

{
  "authenticated": true, 
  "user": "user"
}
$

Here's what happens when you send the wrong username and password.

$ curl -i -u "user:wrongpass" https://httpbin.org/basic-auth/user/pass
HTTP/2 401 
date: Tue, 16 Jun 2020 15:18:51 GMT
content-length: 0
server: istio-envoy
www-authenticate: Basic realm="Fake Realm"
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 12

$

Making a POST Request With JSON

The -X flag tells curl what HTTP method to use: PUT, POST, etc. By default, curl uses GET, so you never need to do curl -X GET.

The -X flag is often used with the -d flag, which lets you set the request body. Below is how you can send a POST request with some JSON.

$ curl -X POST -d '{"answer":42}' https://httpbin.org/post
{
  "args": {}, 
  "data": "", 
  "files": {}, 
  "form": {
    "{\"answer\":42}": ""
  }, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "13", 
    "Content-Type": "application/x-www-form-urlencoded", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0", 
    "X-Amzn-Trace-Id": "Root=1-5ee8e3fd-8437029087be44707bd15320", 
    "X-B3-Parentspanid": "2a739cfc42d28236", 
    "X-B3-Sampled": "0", 
    "X-B3-Spanid": "8bdf030613bb9c8d", 
    "X-B3-Traceid": "75d84f317abad5232a739cfc42d28236", 
    "X-Envoy-External-Address": "10.100.91.201", 
    "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/httpbin-istio/sa/httpbin;Hash=ea8c3d70befa0d73aa0f07fdb74ec4700d42a72889a04630741193548f1a7ae1;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
  }, 
  "json": null, 
  "origin": "69.84.111.39,10.100.91.201", 
  "url": "http://httpbin.org/post"
}
$

Notice that, by default, curl sets the request's 'Content-Type' header to application/x-www-form-urlencoded. That's incorrect for JSON, so for JSON requests you also need to set the 'Content-Type' header using the -H flag.

$ curl -X POST -d '{"answer":42}' -H "Content-Type: application/json" https://httpbin.org/post
{
  "args": {}, 
  "data": "{\"answer\":42}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "13", 
    "Content-Type": "application/json", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0", 
    "X-Amzn-Trace-Id": "Root=1-5ee8e45e-ad875af4f83efd4379b86c34", 
    "X-B3-Parentspanid": "5f4f33d1c5ea13aa", 
    "X-B3-Sampled": "0", 
    "X-B3-Spanid": "a062c9bf2ebfd4bd", 
    "X-B3-Traceid": "44aa8d62412ae34d5f4f33d1c5ea13aa", 
    "X-Envoy-External-Address": "10.100.86.47", 
    "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/httpbin-istio/sa/httpbin;Hash=2f0b3331fe4d512975b4b82583a55dd5d1196023d0dfce9e0abed246991c5b67;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
  }, 
  "json": {
    "answer": 42
  }, 
  "origin": "69.84.111.39,10.100.86.47", 
  "url": "http://httpbin.org/post"
}

Making a PUT Request With a JSON File

The -d flag by itself is convenient for small requests, but it can get cumbersome to deal with huge JSON payloads. Thankfully, the -d flag supports pulling data from a local file.

For example, suppose you have a data.json file that has the below content:

{"answer":42}

To make an HTTP PUT request with that file as the request body, you can set the -d flag to '@data.json'. The @ prefix tells curl to load the request body from the 'data.json' file.

$ curl -X PUT -d '@data.json' -H "Content-Type: application/json" https://httpbin.org/put
{
  "args": {}, 
  "data": "{\"answer\":42}", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "13", 
    "Content-Type": "application/json", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/7.58.0", 
    "X-Amzn-Trace-Id": "Root=1-5ee8e745-37c4ef06326b7b4354a16b94", 
    "X-B3-Parentspanid": "a4f8f91f4f1b051e", 
    "X-B3-Sampled": "0", 
    "X-B3-Spanid": "a018b1a3fcebdc68", 
    "X-B3-Traceid": "7b48b01dc3f632eea4f8f91f4f1b051e", 
    "X-Envoy-External-Address": "10.100.91.201", 
    "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/httpbin-istio/sa/httpbin;Hash=6035260d9d551af6c1907270653214e8d3195abbdd19078c1c84fd9a4106f260;Subject=\"\";URI=spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
  }, 
  "json": {
    "answer": 42
  }, 
  "origin": "69.84.111.39,10.100.91.201", 
  "url": "http://httpbin.org/put"
}
$

Moving On

To recap, here are the curl options that I find most useful.

-X: set the HTTP method, for example curl -X POST url
-d: set the request body, as a string, for PUT and POST requests. Use @ to pull data from a file.
-H: set a request header, for example curl -H "Authorization: my-secret-token" url
-u: Auth credentials for basic auth
-O: store the response body in a file
-i: show the raw response, including response headers

Curl is a useful tool for tinkering with APIs from the command line, whether it's an API you signed up for or one you're developing locally. For quick tests, curl can be easier than writing an Axios request in Node.js or setting up a request in Postman, as long as you're familiar with the syntax.

Found a typo or error? Open up a pull request! This post is available as markdown on Github