wget is a powerful command used on linux for retrievel remote, or local, objects using http and https protocol.
Well, we can do something
for j in `cat list.file` do.....
However, last command use 2 conditions: it’s sequential, it execute wget for each file one by one, sequentially and, second if the remote server limit download speed our wget command can take a long time for download each file.
So, what can we do?
Just using marvelous pipe *unix facilities we can launch one wget command for each file contained on your file list.
echo $URL_LIST | xargs -n 1 -P 10 wget -q
- $URL_LIST is a variable storing all files required to download
- -n= take on argument at time
- -P= Execute “n” parallel wget process
To consider, if you’re using a high parallel wget number your ip can be block for remote server, your bandwidth can be unusable and you computer can be blocked.
Enjoy this help!