<p>Since there doesn't seem to be an easy way to run this in parallel, you could instead break the job into sections. For example, separate each chromosome into its own gtf file, extract the sequences using that file and then cat them all together.</p>

<p>The commands would be something like this (using the human hg38 genome and the gff GENCODE annotations downloaded from <a href="https://www.gencodegenes.org/releases/current.html" rel="nofollow noreferrer">here</a>):</p>

<p>Split the file into one file per chromosome (or whatever else you have as the first field of your gff/gtf file):</p>

<pre><code>tmp=$(mktemp -d); ## create a temp directory
awk -vtmp="$tmp" '($1!~/^#/){print &gt; tmp"/"$1".gff"}' gencode.v27.annotation.gtf 
</code></pre>

<p>Extract the sequences described in each file, running each command in the background (<code>&amp;</code>):</p>

<pre><code>for f in $tmp/*; do 
    gffread -w "$f.fa" -g hg38.fa "$f" &amp; 
done
</code></pre>

<p>Concatenate them into one file:</p>

<pre><code>cat "$tmp/*fa" &gt; all.fa
</code></pre>

<p>You can combine all this into a single command, using <code>wait</code> to make sure you don't concatenate until all of the sub-commands have finished:</p>

<pre><code>tmp=$(mktemp -d); 
awk -vtmp="$tmp" '($1!~/^#/){print &gt; tmp"/"$1".gff"}' gencode.v27.annotation.gtf &amp;&amp; 
    for f in "$tmp"/*; do 
        gffread-0.9.12.Linux_x86_64/gffread -w $f.fa -g hg38.fa "$f" &amp; 
    done 
    wait 
    cat "$tmp"/*fa &gt; all.fa
</code></pre>

<p>Or, just save it as a script called pgffread.sh (or whatever):</p>

<pre class="lang-sh prettyprint-override"><code>#!/bin/sh
gtf="$1"
genome="$2"

tmp=$(mktemp -d);
awk -vtmp="$tmp" '($1!~/^#/){print &gt; tmp"/"$1".gff"}' "$gtf" &amp;&amp;
    for f in "$tmp"/*; do
      gffread -w "$f".fa -g "$genome" "$f" &amp;
    done
wait
cat "$tmp"/*fa 
rm -rf "$tmp"
</code></pre>

<p>And then run it giving the genome and annotation files as parameters:</p>

<pre><code>pgffread.sh gencode.v27.annotation.gtf hg38.fa &gt; all.fa
</code></pre>

<p>On the system I ran it on, which has 48 cores, the time difference was significant (although it's so fast anyway, I really don't know if it is worth the effort unless you do this very often):</p>

<pre><code>$ time gffread -w out.fa -g hg38.fa gencode.v27.annotation.gtf
real    1m52.072s
user    1m3.160s
sys     0m47.257s

$ time ./pgffread.sh gencode.v27.annotation.gtf hg38.fa &gt; all.fa
real    0m17.533s
user    1m5.228s
sys     0m43.039s
</code></pre>

CC BY-SA 3.0
2018-03-15T07:17:50.203Z
thanks! yes, I should have thought of breaking the reference into smaller bits.
aechchiki
CC BY-SA 3.0
2018-03-14T23:30:11.627Z
2018-03-14T23:30:11.627Z
terdon
<p>Is there any possibility to run <a href="http://ccb.jhu.edu/software/stringtie/gff.shtml" rel="noreferrer">gffread</a> in multi-thread mode? The answer seems to be 'no' from the <a href="http://manpages.ubuntu.com/manpages/trusty/man1/gffread.1.html" rel="noreferrer">manual</a> (or <code>gffread -h</code>), as no multi-thread option is mentioned.</p>

<p>I'm mostly using this utility to extract transcript sequences (FASTA) from annotation files (GTF). In single-thread mode, runtime is acceptable in most cases, but would like to speed up a bit in some cases (especially for big genomes).</p>

<p>Note that I already tried, as mentioned in the manual, to index the genome first (which, again, cuts the runtime in most but not all cases): </p>

<blockquote>
  <p>Note that the retrieval of the transcript sequences this way is going to be much faster if a fasta index file (genome.fa.fai in this example) is found in the same directory with the genomic fasta file. </p>
  
  <p>Such an index file can be created with the samtools utility prior to running gffread, like this: <code>samtools faidx genome.fa</code></p>
  
  <p>Then in subsequent runs using the -g option gffread will find this
  FASTA index and use it to speed up the extraction of transcript
  sequences.</p>
</blockquote>

<p>I'm also open to different alternatives than <code>gffread</code> to convert GTF -> FASTA. </p>

CC BY-SA 3.0
2018-03-14T16:43:58.860Z
I have used gffread a few times. My impression is it is fast. As I have just tried, for Ensembl human annotations, it gets all transcripts in 29 sec. This should be faster than many other operations on human genome. How often do you want to run gffread?
user172818
CC BY-SA 3.0
2018-03-14T21:40:52.613Z
Thank you. Yes it is usually quite fast. Actually I was adressing this question since it took me more than 8min to get the transcripts out of a putative genome size of 1.4Gb, but could not reproduce the slow runtime on another server (where it took me around 110sec). Weird.
aechchiki
<rna-seq><fasta><file-formats><gtf>
Run gffread in multi-thread mode
# Run gffread in multi-thread mode

(Asked by: username_0 on 2018-03-14)

Is there any possibility to run [gffread](http://ccb.jhu.edu/software/stringtie/gff.shtml) in multi-thread mode? The answer seems to be 'no' from the [manual](http://manpages.ubuntu.com/manpages/trusty/man1/gffread.1.html) (or `gffread -h`), as no multi-thread option is mentioned.

I'm mostly using this utility to extract transcript sequences (FASTA) from annotation files (GTF). In single-thread mode, runtime is acceptable in most cases, but would like to speed up a bit in some cases (especially for big genomes).

Note that I already tried, as mentioned in the manual, to index the genome first (which, again, cuts the runtime in most but not all cases):

> Note that the retrieval of the transcript sequences this way is going to be much faster if a fasta index file (genome.fa.fai in this example) is found in the same directory with the genomic fasta file.
>
> Such an index file can be created with the samtools utility prior to running gffread, like this: `samtools faidx genome.fa`
>
> Then in subsequent runs using the -g option gffread will find this
> FASTA index and use it to speed up the extraction of transcript
> sequences.

I'm also open to different alternatives than `gffread` to convert GTF -> FASTA.

**Tags:** rna-seq, fasta, file-formats, gtf
> **Comments:**
> **username_1**: I have used gffread a few times. My impression is it is fast. As I have just tried, for Ensembl human annotations, it gets all transcripts in 29 sec. This should be faster than many other operations on human genome. How often do you want to run gffread?  
> **username_0**: Thank you. Yes it is usually quite fast. Actually I was adressing this question since it took me more than 8min to get the transcripts out of a putative genome size of 1.4Gb, but could not reproduce the slow runtime on another server (where it took me around 110sec). Weird.  

---

## 1 answers

### Answer by username_2 on 2018-03-14. Score: 3 | Accepted answer

Since there doesn't seem to be an easy way to run this in parallel, you could instead break the job into sections. For example, separate each chromosome into its own gtf file, extract the sequences using that file and then cat them all together.
The commands would be something like this (using the human hg38 genome and the gff GENCODE annotations downloaded from here):
Split the file into one file per chromosome (or whatever else you have as the first field of your gff/gtf file):
tmp=$(mktemp -d); ## create a temp directory
awk -vtmp="$tmp" '($1!~/^#/){print > tmp"/"$1".gff"}' gencode.v27.annotation.gtf 

Extract the sequences described in each file, running each command in the background (&):
for f in $tmp/*; do 
    gffread -w "$f.fa" -g hg38.fa "$f" & 
done

Concatenate them into one file:
cat "$tmp/*fa" > all.fa

You can combine all this into a single command, using wait to make sure you don't concatenate until all of the sub-commands have finished:
tmp=$(mktemp -d); 
awk -vtmp="$tmp" '($1!~/^#/){print > tmp"/"$1".gff"}' gencode.v27.annotation.gtf && 
    for f in "$tmp"/*; do 
        gffread-0.9.12.Linux_x86_64/gffread -w $f.fa -g hg38.fa "$f" & 
    done 
    wait 
    cat "$tmp"/*fa > all.fa

Or, just save it as a script called pgffread.sh (or whatever):
#!/bin/sh
gtf="$1"
genome="$2"

tmp=$(mktemp -d);
awk -vtmp="$tmp" '($1!~/^#/){print > tmp"/"$1".gff"}' "$gtf" &&
    for f in "$tmp"/*; do
      gffread -w "$f".fa -g "$genome" "$f" &
    done
wait
cat "$tmp"/*fa 
rm -rf "$tmp"

And then run it giving the genome and annotation files as parameters:
pgffread.sh gencode.v27.annotation.gtf hg38.fa > all.fa

On the system I ran it on, which has 48 cores, the time difference was significant (although it's so fast anyway, I really don't know if it is worth the effort unless you do this very often):
$ time gffread -w out.fa -g hg38.fa gencode.v27.annotation.gtf
real    1m52.072s
user    1m3.160s
sys     0m47.257s

$ time ./pgffread.sh gencode.v27.annotation.gtf hg38.fa > all.fa
real    0m17.533s
user    1m5.228s
sys     0m43.039s

> **Comments:**
> **username_0**: thanks! yes, I should have thought of breaking the reference into smaller bits.  

---

