Command-line Basics: Network Requests

Making network requests from the command-line comes in a variety of shapes and sizes. Some folks swear by curl others love wget. For me, I love the lwp-request package which provides familiarly named GET, POST and HEAD commands. The third of the list, HEAD, is something I use quite regularly to troubleshoot the headers returned by servers.

Getting started

If you would like to follow along with this tutorial using a terminal in your browser, click the Launch an Interactive Terminal! button below. You’ll be able to run each of the commands right from your browser.

The commands we’re going to use are provided by a package called lwp-request.

The package in question is written in Perl and tends to come standard with quite a few Linux distributions as well as other Unix-like operating systems like macOS.

The lwp-request command could be invoked directly with a series of arguments, but we’ll be going over the helper scripts that it comes with for the sake of simplicity.

If you are lacking GET, POST and HEAD on your system, you most likely will be able to install the lwp-request package from your favorite package manager.

On Ubuntu and Debian derived systems, you can install it using the following commands:

On CentOS, Rocky Linux, Fedora, and other RedHat derived systems, you can install it with support for HTTPS URLs using the following command:

GET

Let’s say you loved the Alligator.io logo so much, you wanted to download it locally. It’s a pretty amazing logo, who wouldn’t want their own personal copy?

To GET the file, you can simply run:

Not so fast! All that does is show a bunch of SVG markup.

In true Unix Philosophy fashion, the GET command does one thing really well and that’s GETting a file.

This is great when you want to check a URL to see what’s being returned by the web server, but if you want to truly download that lovely logo, you’ll want to send the output to a file:

Now we have that fantastic Alligator.io logo downloaded to a local file!

POST

The GET command allows us to consume files from remote servers while POST allows us to send data to a server for processing as well as return it’s output.

At it’s very least, the syntax for POST is the same as GET:

This will then prompt you for the content you’d like to POST. The string expected should be in a query string format that looks something like this:

reptile=alligator&color=#008f68

When you’re done entering in your content, simply hit CTRL-D and the content will be POSTed. The service we’re posting to will mirror back the request:

{
  "args": {},
  "data": "",
  "files": {},
  "form": {
    "color": "#008f68\n",
    "reptile": "alligator"
  },
  "headers": {
    "Content-Length": "32",
    "Content-Type": "application/x-www-form-urlencoded",
    "Host": "httpbin.org",
    "User-Agent": "lwp-request/6.39 libwww-perl/6.39"
  },
  "json": null,
  "origin": "203.0.113.5",
  "url": "https://httpbin.org/post"
}

HEAD

As mentioned, HEAD is not only extremely useful for debugging and troubleshooting, but I’m pretty sure it’s in my top 5 of all-time favorite command-line utilities.

Similar to GET and POST, the syntax for HEAD is quite minimal:

This will return a 200 OK and information about the headers returned by the web service.

Unfortunately, this isn’t quite right, because we serve up Alligator.io over HTTPS like the good security-minded reptilian webizens that we are.

The HEAD command by default only gives you information about the last stop in the request chain. To see all of the requests, including the automatic 301 Moved Permanently, pass in the -S argument:

Which gives us a bit more insight:

Bonus

Monochromatic output got you down? If so, you could alias HTTPie’s http command to GET, POST and HEAD.

HTTPie can do everything the lwp-request library does, with similar syntax, with the added bonus of colorful output.

On Ubuntu and Debian derived systems you can install HTTPie with the following commands:

On Centos, Rocky Linux, Fedora, and RedHat derived distributions you can install HTTPie using the following commands, as long as you have the EPEL :

My local aliases look like this:

Conclusion

Next time you need to make a network request to an API or are troubleshooting the headers returned by a server, you can leave Postman and similar tools at the door.

You can omit your browser entirely, too!