What is Wget?
Wget is a free software package for retrieving files using HTTP, HTTPS, FTP and FTPS, the most widely used Internet protocols. It is a non-interactive commandline tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.
How to install Wget?
# Install wget in Ubuntu \ Debian Linux
apt-get install wget
# Install wget on RHEL / CentOS / Fedora
yum install wget
# Install wget on OpenSUSE
zypper install wget
# Install wget on ArchLinux
pacman -Sy wget
# Install wget on FreeBSD
pkg install wget
# Install wget Using FreeBSD Ports Collection
portsnap fetch update
cd /usr/ports/ftp/wget
make install clean
rehash ## or hash -r for 'bash/sh/ksh'
Code language: PHP (php)
How to use Wget?
- Now if we want do download a single file using HTTP we type:
# Download a single file
wget http://site-name.com/file-name.tar.gz
Code language: Bash (bash)
2. You can download a web page and save it to a folder of your desire:
# Download a website
wget -o index.html http://site-name.com/page-url
# Download a website into a different folder
wget --directory-prefix='./home/user/Downloads/site-name/' http://site-name.com/page-url
Code language: Bash (bash)
3. Here is how to download files from password protected websites:
# Download files from password protected websites
wget ‐‐http-user=username ‐‐http-password=password http://site-name.com/path-secret/file.tar.gz
Code language: Bash (bash)
4. Another way is to download specific type of files from the website:
# Download specific type of files from the website
# This will download all the mp3 files
$ wget --level=2 --recursive --accept mp3 http://site-name.com
# will download all jpeg files
$ wget ‐‐level=1 ‐‐recursive ‐‐no-parent ‐‐accept jpg,JPG http://site-name.com/
Code language: Bash (bash)
5. A cool options is to download multiple files with different protocols:
# Download multiple files with different protocols
wget http://site-name.com/file.tar.gz ftp://151.232.45.6/picture.jpg
Code language: Bash (bash)
6. It’s also possible to limit the bandwidth of a file you are downloading:
# Limit the bandwidth of a file you are downloading
wget --limit-rate=50k http://site-name.com/file.rar
Code language: Bash (bash)
7. You can download a entire website if you wish with all the files and folders inside:
# Mirror entire websites (all its pages and assets)
wget --mirror --no-parent --continue http://site-name.com
Code language: PHP (php)
8. You can put URLs in a file and then tell Wget to download all of the links in the file
# Download all of the links in the file
wget ‐‐input filename.txt
Code language: PHP (php)
9. Here is how to resume a currently download file from where it was left
# Resume a currently download file from where it was left
wget -c http://site-name.com/file.zip
Code language: PHP (php)
10. Download in the Background Using wget -b
# Download in the Background Using wget -b
wget -b http://www.site-name.com/link/filename.tar.bz2
Code language: PHP (php)
11. Check the status of the download using tail -f
# Check the status of the download using tail -f
tail -f wget-log
Saving to: `filename.tar.bz2.4'
0K .......... .......... .......... .......... .......... 1% 65.5K 57s
50K .......... .......... .......... .......... .......... 2% 85.9K 49s
100K .......... .......... .......... .......... .......... 3% 83.3K 47s
150K .......... .......... .......... .......... .......... 5% 86.6K 45s
200K .......... .......... .......... .......... .......... 6% 33.9K 56s
250K .......... .......... .......... .......... .......... 7% 182M 46s
300K .......... .......... .......... .......... .......... 9% 57.9K 47s
Code language: PHP (php)
12. A nice trick is to test Download URL using Wget –spider
# Test download
wget --spider http://site-name.com/link/file.tar.bz2
Code language: PHP (php)
13. And finally here is how to download only certain file types using wget -r -A
# Download only certain file types
wget -r -A.pdf http://site-name.com/files-folder/
Code language: PHP (php)
In this lesson, we learned some of the coolest ways to handle Wget. A tool without which no system administrator can work. For additional examples you can refer to the man page of the program.