<pre><code>awk '{print "wget " $9 " -O " $1 "_" $2 ".fastq.gz"}' tmp 
</code></pre>

<p>will give you </p>

<pre><code>wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz
wget https://www.encodeproject.org/files/ENCFF001JPZ/@@download/ENCFF001JPZ.fastq.gz -O b_k.fastq.gz
wget https://www.encodeproject.org/files/ENCFF001JQB/@@download/ENCFF001JQB.fastq.gz -O c_k.fastq.gz
</code></pre>

<p>If you add a <code>| bash</code> to the command it will then execute all the <code>wget</code> commands.</p>

CC BY-SA 3.0
2017-12-13T19:37:45.747Z
This works great. Can you tell what |bash is doing ??
user3138373
CC BY-SA 3.0
2017-12-13T19:43:36.950Z
@user3138373 the output of the awk command is a set of shell commands, essentially a mini script. By piping it to bash with `| bash` you're telling it to execute each input line as a command. So it will run the `wget` calls and download the sequences.
terdon
CC BY-SA 3.0
2017-12-13T19:28:34.387Z
2017-12-13T19:46:31.503Z
2017-12-13T19:46:31.503Z
Bioathlete
<p>A variation of <a href="/a/3072/2042">Bioathlete’s answer</a> that doesn't suffer from escaping issues:</p>



<pre class="lang-sh prettyprint-override"><code>awk 'BEGIN{ FS="\t" } { printf("%s\t%s_%s.fastq.gz\n", $8, $1, $2) }' input |
while IFS=$'\t' read -r url filename; do
    wget -O "$filename" -- "$url"
done
</code></pre>

<p>Or entirely with shell parsing:</p>

<pre class="lang-sh prettyprint-override"><code>while IFS=$'\t' read -r c1 c2 c3 c4 c5 c6 c7 url; do
    wget -O "${c1}_${c2}.fastq.gz" -- "$url"
done &lt; input
</code></pre>

<p>Or entirely without shell parsing but with GNU <code>xargs</code>:</p>

<pre class="lang-sh prettyprint-override"><code>awk 'BEGIN{ FS="\t" } { printf("%s_%s.fastq.gz\n%s\n", $1, $2, $8) }' input |
xargs -r -d '\n' -n 2 -- wget -O
</code></pre>

<p>… you could even run multiple <code>wget</code> instances in parallel from GNU <code>xargs</code> with the <code>-P</code> option, though you should set the <code>-q</code> option on <code>wget</code> to disable progress bars overwriting each other on the same terminal. Then again it would be better to use a real command-line download manager like Aria or Axel for that task.</p>

CC BY-SA 3.0
2017-12-13T21:53:00.753Z
2017-12-13T23:05:17.623Z
2017-12-13T23:05:17.623Z
David Foerster
<p>I have a text file as follows</p>

<pre><code>a   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz
b   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JPZ/@@download/ENCFF001JPZ.fastq.gz
c   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JQB/@@download/ENCFF001JQB.fastq.gz
</code></pre>

<p>The file is tab delimited and has 8 columns with 8th column being the url link  I need to download the files one by one using wget and changing the name of the downloaded file with -O option in wget. The argument I want to give to the -O option is combination col1 and col2 separated by _ so that for first line I get the downloaded file as a_k.fastq.gz and so on. For example, the first line of the file above should give:</p>

<pre><code>wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz
</code></pre>

<p>Following the post from <a href="https://unix.stackexchange.com/questions/410710/splitting-a-line-into-array-in-bash-with-tab-as-delimiter?noredirect=1#comment735962_410710">here</a> I realized looping through the file is slow and sometimes difficult. How can I download these files using wget with awk or cut as the tool for fixed fields</p>

CC BY-SA 3.0
2017-12-13T19:20:06.200Z
Can you post up an example wget command?
Bioathlete
CC BY-SA 3.0
2017-12-13T19:23:19.647Z
wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz
user3138373
CC BY-SA 3.0
2017-12-13T19:23:43.013Z
how can I put it into code in the comment section??
user3138373
CC BY-SA 3.0
2017-12-13T19:25:52.240Z
edit the original post
Bioathlete
CC BY-SA 3.0
2017-12-15T10:35:25.247Z
@Llopis [we've decided](https://bioinformatics.meta.stackexchange.com/q/88/298) that basic things that bioinformaticians need to do in their daily work are on topic even when the issue could be expressed as a general text parsing question.
terdon
CC BY-SA 3.0
2017-12-15T11:21:30.533Z
Ok, I misjudged and thought it was about downloading data.
llrs
<awk><data-download>
Downloading URL using AWK for fixed fields
# Downloading URL using AWK for fixed fields

(Asked by: username_0 on 2017-12-13)

I have a text file as follows
a   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz
b   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JPZ/@@download/ENCFF001JPZ.fastq.gz
c   k   testis  adult   male    8 week  rRNA    https://www.encodeproject.org/files/ENCFF001JQB/@@download/ENCFF001JQB.fastq.gz

The file is tab delimited and has 8 columns with 8th column being the url link  I need to download the files one by one using wget and changing the name of the downloaded file with -O option in wget. The argument I want to give to the -O option is combination col1 and col2 separated by _ so that for first line I get the downloaded file as a_k.fastq.gz and so on. For example, the first line of the file above should give:
wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz

Following the post from here I realized looping through the file is slow and sometimes difficult. How can I download these files using wget with awk or cut as the tool for fixed fields

**Tags:** awk, data-download
> **Comments:**
> **username_1**: Can you post up an example wget command?  
> **username_0**: wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz  
> **username_0**: how can I put it into code in the comment section??  
> **username_1**: edit the original post  
> **username_2**: @Llopis [we've decided](https://bioinformatics.meta.stackexchange.com/q/88/298) that basic things that bioinformaticians need to do in their daily work are on topic even when the issue could be expressed as a general text parsing question.  
> **username_3**: Ok, I misjudged and thought it was about downloading data.  

---

## 2 answers

### Answer by username_1 on 2017-12-13. Score: 5

awk '{print "wget " $9 " -O " $1 "_" $2 ".fastq.gz"}' tmp 

will give you 
wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz
wget https://www.encodeproject.org/files/ENCFF001JPZ/@@download/ENCFF001JPZ.fastq.gz -O b_k.fastq.gz
wget https://www.encodeproject.org/files/ENCFF001JQB/@@download/ENCFF001JQB.fastq.gz -O c_k.fastq.gz

If you add a | bash to the command it will then execute all the wget commands.

> **Comments:**
> **username_0**: This works great. Can you tell what |bash is doing ??  
> **username_2**: @user3138373 the output of the awk command is a set of shell commands, essentially a mini script. By piping it to bash with `| bash` you're telling it to execute each input line as a command. So it will run the `wget` calls and download the sequences.  

---

### Answer by username_4 on 2017-12-13. Score: 4

A variation of Bioathlete’s answer that doesn't suffer from escaping issues:
awk 'BEGIN{ FS="\t" } { printf("%s\t%s_%s.fastq.gz\n", $8, $1, $2) }' input |
while IFS=$'\t' read -r url filename; do
    wget -O "$filename" -- "$url"
done

Or entirely with shell parsing:
while IFS=$'\t' read -r c1 c2 c3 c4 c5 c6 c7 url; do
    wget -O "${c1}_${c2}.fastq.gz" -- "$url"
done < input

Or entirely without shell parsing but with GNU xargs:
awk 'BEGIN{ FS="\t" } { printf("%s_%s.fastq.gz\n%s\n", $1, $2, $8) }' input |
xargs -r -d '\n' -n 2 -- wget -O

… you could even run multiple wget instances in parallel from GNU xargs with the -P option, though you should set the -q option on wget to disable progress bars overwriting each other on the same terminal. Then again it would be better to use a real command-line download manager like Aria or Axel for that task.


---

