Curl Cheat Sheet - Brian Cantoni

This is a quick introduction and cheat sheet for Curl – a very handy command-line tool for downloading pretty much anything from a URL.

The Curl website describes it as:

… a command line tool for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, kerberos…), file transfer resume, proxy tunneling and a busload of other useful tricks.

There are many things that Curl can do, and there is a voluminous man page that lists all of the details.

Here I want to boil down all those options into the most common and useful ones for web or webservices developers (using HTTP/HTTPS protocols). If you don’t already have Curl installed on your system (try running curl from a command prompt), see Getting Curl below.

Basic Usage

The basic form of all Curl commands is:

curl [options...] <url>

For example:

$ curl http://www.google.com/humans.txt
Google is built by a large team of engineers, designers, researchers, robots, and others in many different sites across the globe. It is updated continuously, and built with more tools and technologies than we can shake a stick at. If you'd like to help us out, see google.com/jobs.

Common Options

Options are the real power of of Curl. Here we’ll cover the most common ones that I’ve used for typical web and webservices development. (You can get the full set of options on your system with curl --help or curl --manual.)

-A / –user-agent AGENT: Set the HTTP User Agent string if you don’t want the default “curl” string
–compressed: Add the HTTP header to request compressed content, if the server can provide it
-d / –data DATA: Set data to be sent with a POST request
-D / –dump-header FILE: Save the response headers to a separate file
-H / –header HEADER: Set a custom HTTP header
-i / –include: Include the response headers in the output
-k / –insecure: Skip SSL certification verification
-o / –output FILE: Write output to a file rather than stdout
-s / –silent: Run silently (i.e., don’t show progress meter)
–trace-ascii FILE: Write request and response headers and data to local file
-x / –proxy HOST:PORT: Route data through the given proxy
-X / –request METHOD: Set custom HTTP method (GET, PUT, POST, DELETE)

Examples

Fetch and Save Web Page

$ curl --silent http://boston.com -o boston.html
$ head boston.html
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html lang="en">
<head>
<meta http-equiv="Refresh" content="900;url=?refresh=true">
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
<title>Boston.com - Boston, MA news, breaking news, sports, video</title>

Check Size Without Downloading

$ curl --head http://s3.amazonaws.com/hanselminutes/hanselminutes_0300.mp3
HTTP/1.1 200 OK
x-amz-request-id: 31D80700E3C2811E
Date: Wed, 11 Jan 2012 04:01:35 GMT
Last-Modified: Sat, 07 Jan 2012 02:49:49 GMT
ETag: "1d08609ab5434eea651e95af332ddb3a"
Accept-Ranges: bytes
Content-Type: audio/mpeg
Content-Length: 24474192
Server: AmazonS3

Getting Curl

If you’re running Mac OSX, Linux, FreeBSD, or similar systems, you’ve probably already got Curl installed. (Try curl --version to double-check your version.)

If you’re running Windows, you’ll need to download it yourself. Start at the Curl downloads page and find the Win32 section. I suggest the “Win32 – Generic binary, with SSL” option. You will also need the Windows OpenSSL libraries; I suggest using the “Win32 OpenSSL v1.x Light” installer. Make sure to put both Curl and OpenSSL libraries in the same location, and add that location to your path.