What is Wget?

Wget is a free software package for retrieving files using HTTP, HTTPS, FTP and FTPS, the most widely used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.

How to install Wget?

# Install wget in Ubuntu \ Debian Linux
apt-get install wget

# Install wget on RHEL / CentOS / Fedora
yum install wget

# Install wget on OpenSUSE
zypper install wget

# Install wget on ArchLinux
pacman -Sy wget

# Install wget on FreeBSD
pkg install wget

# Install wget Using FreeBSD Ports Collection
portsnap fetch update
cd /usr/ports/ftp/wget
make install clean
rehash ## or hash -r for 'bash/sh/ksh'
Code language: PHP (php)

How to use Wget?

1. Now if we want do download a single file using HTTP we type:

# Download a single file
wget http://site-name.com/file-name.tar.gz
Code language: Bash (bash)

2. You can download a web page and save it to a folder of your desire:

# Download a website
wget -o index.html http://site-name.com/page-url

# Download a website into a different folder
wget --directory-prefix='./home/user/Downloads/site-name/' http://site-name.com/page-url
Code language: Bash (bash)

3. Here is how to download files from password protected websites:

# Download files from password protected websites
wget ‐‐http-user=username ‐‐http-password=password http://site-name.com/path-secret/file.tar.gz
Code language: Bash (bash)

4. Another way is to download specific type of files from the website:

# Download specific type of files from the website
# This will download all the mp3 files
$ wget --level=2 --recursive --accept mp3 http://site-name.com

# will download all jpeg files
$ wget ‐‐level=1 ‐‐recursive ‐‐no-parent ‐‐accept jpg,JPG http://site-name.com/
Code language: Bash (bash)

5. A cool options is to download multiple files with different protocols:

# Download multiple files with different protocols
wget http://site-name.com/file.tar.gz ftp://151.232.45.6/picture.jpg
Code language: Bash (bash)

6. It’s also possible to limit the bandwidth of a file you are downloading:

# Limit the bandwidth of a file you are downloading
wget --limit-rate=50k http://site-name.com/file.rar
Code language: Bash (bash)

7. You can download a entire website if you wish with all the files and folders inside:

# Mirror entire websites (all its pages and assets)
wget --mirror --no-parent --continue http://site-name.com
Code language: PHP (php)

8. You can put URLs in a file and then tell Wget to download all of the links in the file

# Download all of the links in the file 
wget ‐‐input filename.txt
Code language: PHP (php)

9. Here is how to resume a currently download file from where it was left

# Resume a currently download file from where it was left
wget -c http://site-name.com/file.zip
Code language: PHP (php)

10. Download in the Background Using wget -b

# Download in the Background Using wget -b
wget -b http://www.site-name.com/link/filename.tar.bz2
Code language: PHP (php)

11. Check the status of the download using tail -f

# Check the status of the download using tail -f
tail -f wget-log
Saving to: `filename.tar.bz2.4'

     0K .......... .......... .......... .......... ..........  1% 65.5K 57s
    50K .......... .......... .......... .......... ..........  2% 85.9K 49s
   100K .......... .......... .......... .......... ..........  3% 83.3K 47s
   150K .......... .......... .......... .......... ..........  5% 86.6K 45s
   200K .......... .......... .......... .......... ..........  6% 33.9K 56s
   250K .......... .......... .......... .......... ..........  7%  182M 46s
   300K .......... .......... .......... .......... ..........  9% 57.9K 47s
Code language: PHP (php)

12. A nice trick is to test Download URL using Wget –spider

# Test download
wget --spider http://site-name.com/link/file.tar.bz2
Code language: PHP (php)

13. And finally here is how to download only certain file types using wget -r -A

# Download only certain file types
wget -r -A.pdf http://site-name.com/files-folder/
Code language: PHP (php)

In this lesson, we learned some of the coolest ways to handle Wget. A tool without which no system administrator can work. For additional examples you can refer to the man page of the program.

Here is a quick video install on ubuntu and download a entire web page

We hope you enjoyed this article. if that is so please rate this page with the stars bellow and subscribe to our YouTube channel.