wget is a powerful command used on linux for retrievel remote, or local, objects using http and https protocol.

Well, we can do something

for j in `cat list.file` do.....

However, last command use 2 conditions: it’s sequential, it execute wget for each file one by one, sequentially and, second if the remote server limit download speed our wget command can take a long time for download each file.

So, what can we do?

Just using marvelous pipe *unix facilities we can launch one wget command for each file contained on your file list.

echo $URL_LIST | xargs -n 1 -P 10 wget -q

Where

  • $URL_LIST is a variable storing all files required to download
  • -n= take on argument at time
  • -P= Execute “n” parallel wget process

To consider, if you’re using a high parallel wget number your ip can be block for remote server, your bandwidth can be unusable and you computer can be blocked.

Enjoy this help!

Deja un comentario

Tu dirección de correo electrónico no será publicada. Los campos obligatorios están marcados con *

Puedes usar las siguientes etiquetas HTML y atributos:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Este sitio usa Akismet para reducir el spam. Aprende cómo se procesan los datos de tus comentarios.