The *nix commands
wget are useful for accessing URLs without resorting to a browser. Both commands allow you to transfer data from a network server, with
curl being the more robust of the two. You could use either of them to automate downloads from various servers.
The curl command
As mentioned, the
curl command allows you to transfer data from a network server, but it also enables you to move data to a network server. In addition to HTTP, you can use other protocols, including HTTPS, FTP, POP3, SMTP, and Telnet. Administrators commonly rely on
curl to interact with APIs using the DELETE, GET, POST, and PUT methods, as explained here.
The syntax for
curl is fairly straight-forward at first glance. Here is an example:
$ curl http://www.example.com/help.txt
You can supply various options to your command syntax:
curl [options] [url]
It is the options which make
curl so robust. The following are some of the available options used with
curl and examples of their use.
When uploading a file, this option allows you to append to the target file instead of overwriting it (FTP, SFTP).
$ curl --append file.txt ftp://ftp.example.com/file.txt
--connect-timeout option sets the maximum time in seconds that
curl can use to make its connection to the remote server. This option is handy to prevent the connection from terminating too quickly, and to minimize the amount of time you want the command to attempt the connection.
$ curl --connect-timeout 600 http://www.example.com/
This option allows you to list DNS servers
curl should use instead of the system default. This list can be handy when troubleshooting DNS issues or if you need to resolve an address against a specific nameserver.
$ curl --dns-servers 22.214.171.124 http://www.example.com/
You can specifically tell curl to use the http3 protocol to connect to the host and port provided with a https URL. --http2 and --http1.1 function in the same way and can be used to verify a webserver.
$ curl --http3 http://www.example.com:8080/
If you need to retrieve a file from a remote server via a URL,
--output is an easy way to save the file locally.
$ curl http://www.example.com/help.txt --output file.txt
This option displays the progress of the file transfer when combined with the
$ curl --progress-bar http://www.example.com/help.txt --output file.txt
Like with HTTP, you can specifically tell curl to use a specific SSL option for the command to connect to and in this case we are specifying version 2. --ssl specifies SSL needs to be used and --sslv3 specifies SSL version 3. Note: sslv2 and sslv3 are considered legacy by the maintainer though still available.
$ curl --sslv2 https://www.example.com/
--verbose option with
curl is useful for debugging and displaying what is going on during the call to the URL.
$ curl --verbose http://www.example.com
The wget command
wget command is solely for the retrieval of information from a remote server. By default, the information received is saved with the same name as in the provided URL.
Here is an example of the basic
$ wget http://www.example.com/help.txt
curl, you can supply various options to your
wget command syntax:
wget [option] [url]
You can specify one or more specific DNS servers to use when utilizing
wget to access a remote server. The syntax differs, however, if the option and nameserver addresses are joined with an =.
$ wget --dns-servers=126.96.36.199 http://www.example.com
To save a file with a new name when using
wget, utilize the
--output-document option, or more simply
$ wget http://www.example.com/help.txt -O file.txt
wget, you can supply a type (dot or bar) to determine the ASCII visual of the progress bar. If a type is not specified, it will default to dot.
$ wget --progress=dot http://www.example.com
wget commands can be very useful when added to scripts to automatically download RPM packages or other files. This post only touches some of the most common features of what these commands can do. Check the related man pages for a complete list of options available for both
[ Want to test your sysadmin skills? Take a skills assessment today. ]