<p>The obvious answer is to break up the file size and upload each separately,
First I'd check the file size,</p>

<pre><code>wc infile.fa
</code></pre>

<p>Output</p>

<blockquote>
  <p>XXXXX      YYYY     ZZZZZ infile.fa</p>
</blockquote>

<p>The number you want is XXXX - the number of lines in the file.</p>

<p>You'd then decide how many files you wanted to break it up into ... in your case divide XXXXX by 3. Then .. for example</p>

<pre><code>sed -n '1,5000000p' infile.fa &gt; outfile.fa
sed -n '5000001,2000000p' infile.fa &gt; outfile2.fa
sed -n '2000001,4000000p' infile.fa &gt; outfile3.fa
</code></pre>

<hr>

<p>I'm not certain this would be cool however. An miRNA expert could answer this question very quickly. I think is you should check your data for sequence duplication. Simply piping the fasta file through a simple distance matrix and screening for "0" values would confirm this. </p>

<p>Creating a frequency table of duplicate RNA must be a feature of Bioconductor, but in any case screening for duplication (technically duplicate records) is a formal SQL database command. </p>

<p>It can be done a Perl/Python but its a bit clunky for a data set this size. I'd do it by setting the sequence as the key within a hash, the inefficiency has been solved (but its a weird way to do it). </p>

<p>You would have a database for "headers" for a given sequence, which would take up space, but I would assess the potential level of duplication first.</p>

CC BY-SA 4.0
2019-02-09T07:42:19.267Z
2019-02-09T07:42:19.267Z
M__
<p>The tool you want to use accepts compressed files, so the first thing you should do is compress your file. I created a 111M file of random, 51bp sequences, each with a random, 52 character header and a random 51 nt sequence:</p>

<pre><code>$ ls -lh file.fa
-rw-r--r-- 1 terdon terdon 111M Feb  9 12:21 file.fa

$ head file.fa
&gt;YZF8SGLVBPCKMY18GJ94XTRCS61BB4OZKUZ61D 1J6R7H3 EJQA5 length=51
ACGAAGGGCATTTGGCAACGAACCTATACTATATCACGAGCAATTAGGCAG
&gt;SPR0INGWKMILAXZZPF73BKRV73ELL1ROZSHYO1E9JX6P5X6MUZ20 length=51
TTAAGATTCCGAGAGAGTCTGTGGTTGGGTCCCTCTCCATAATGCTTTCAT
&gt;U CB82L2M3BWJ9ZZZG3V6DL0Y0FTW8BXIFTU0TK7SL VN5QE42J6 length=51
TGGCTGGGAGGGATAGTAGCGAGATTATCCAGTCCGGTGTTCACAAGCCGG
&gt;TGH62IM2TJK2QJTIVWGZR IVKOW94X7D52KGM7DJP61KKF81KPYA length=51
GCCGCAATACCAAAAAGATGGTGTGATTAGGAAATAAATAGGATCCAACAA
&gt;L5VS9AVEF3UTEO3341STET092YEZ43ESVH32W7GI6Q27OIR NTXF length=51
CTATAGACGACTACGAACGACTGCAGGTGTGCCAGTTCTCGGTGAAACGGA
</code></pre>

<p>If we now compress that file:</p>

<pre><code><span class="math-container">$ tar cvzf file.fa.tar.gz file.fa
file.fa
$</span> ls -lh file.fa.tar.gz 
-rw-r--r-- 1 terdon terdon 56M Feb  9 12:22 file.tar.gz
</code></pre>

<p>You will likely get a significantly better compression ratio since your headers won't be random and will likely be very repetitive. </p>

<p>If that isn't enough, you should try simplifying your headers. For instance, just use an increasing number:</p>

<pre><code><span class="math-container">$ perl -pe 's/^&gt;.*/"&gt;" . ++$</span>i/e' file.fa &gt; file.short.fa
$ ls -lh file.short.fa 
-rw-r--r-- 1 terdon terdon 58M Feb  9 12:23 file.short.fa
</code></pre>

<p>See what a huge difference that one change made? And if we now compress the file:</p>

<pre><code><span class="math-container">$ tar cvzf file.short.tar.gz file.short.fa
file.short.fa
$</span> ls -lh file.short.tar.gz
-rw-r--r-- 1 terdon terdon 19M Feb  9 12:24 file.short.tar.gz
</code></pre>

<p>Well below the threshold! </p>

<hr>

<p>If compression isn't enough, you should make sure you have no duplicate sequences in the file. Use the <code>FastaToTbl</code> and <code>TblToFasta</code> scripts I have posted <a href="https://bioinformatics.stackexchange.com/a/2651/298">here</a> and:</p>

<pre><code>FastaToTbl file.fa | sort | uniq | TblToFasta &gt; file.uniq.fa
</code></pre>

<p>Finally, if none of these helps enough, split your file into multiple smaller ones:</p>

<pre><code>split -l 100000 file.fa
</code></pre>

<p>That will create multiple files of 100000 lines (so 50000 sequences, assuming all your sequences are one line) each. </p>

CC BY-SA 4.0
2019-02-09T12:30:23.060Z
2019-02-09T12:30:23.060Z
terdon
<p>I should get and analyse isomiR (isoform of miRNA) profile from RNA-seq with FASTA format. The best tool for this is <a href="https://mcg.ustc.edu.cn/bsc/deanniso/upload.html" rel="nofollow noreferrer">"DeAnnIso"</a>. But here is a problem with this tool, it only allows you to upload FASTA format files  with a maximum size of 50MB. However, the size of my FASTA file is more than 1000MB. I tried to compress the file in different ways and uploading, but it doesn't work.
Does anybody know any way to solving this problem?</p>

<p>Or Does anybody know <strong>any other easy way to get isomiR profile from RNA-seq with FASTA format?</strong></p>

<p>A sample of RNA-seq like this: </p>

<pre><code>&gt;RR42357.95441.1 I-D00105:274:H7MDXX:1:112:8124:865 length=51
TGGAATTCTCGGGTGCCAAGGAACTCCAGTCACCTTGTAATCTCGTATGCC
&gt;RR42357.95442.1 I-D00105:274:H7MDXX:1:112:8133:866 length=51
AAAGTGCTACTACTTTTGAGTCTTGGAATTCTCGGGTGCCAAGGAACTCCA
&gt;RR42357.95443.1 I-D00105:274:H7MDXX:1:112:8144:8637 length=51
</code></pre>

<p>Thank you very very much!</p>

CC BY-SA 4.0
2019-02-08T19:23:57.330Z
Have you tried to compress your file into `.tar.gz`? What about the way to create a "tag" file like recommended [here](https://mcg.ustc.edu.cn/bsc/deanniso/tools.html)?
finswimmer
CC BY-SA 4.0
2019-02-08T22:32:47.250Z
Have you tried cutting the sequence names down to integers instead of all the embedded information and then compressing them?
Bioathlete
CC BY-SA 4.0
2019-02-08T23:01:07.553Z
In fact it looks like the header lines are longer than the sequence lines so I would expect a significant reduction in the file size.
Bioathlete
CC BY-SA 4.0
2019-02-08T23:14:55.340Z
^ this is funny.. but its true. microRNA isn't huge in length. I'm not exactly sure how 50 bp of sequence can take up 100 MB of data.
M__
<rna-seq><fasta>
Get & Annotation of isomiR (isoform of miRNA) profile from RNA-seq
# Get & Annotation of isomiR (isoform of miRNA) profile from RNA-seq

(Asked by: username_0 on 2019-02-08)

I should get and analyse isomiR (isoform of miRNA) profile from RNA-seq with FASTA format. The best tool for this is "DeAnnIso". But here is a problem with this tool, it only allows you to upload FASTA format files  with a maximum size of 50MB. However, the size of my FASTA file is more than 1000MB. I tried to compress the file in different ways and uploading, but it doesn't work.
Does anybody know any way to solving this problem?
Or Does anybody know any other easy way to get isomiR profile from RNA-seq with FASTA format?
A sample of RNA-seq like this: 
>RR42357.95441.1 I-D00105:274:H7MDXX:1:112:8124:865 length=51
TGGAATTCTCGGGTGCCAAGGAACTCCAGTCACCTTGTAATCTCGTATGCC
>RR42357.95442.1 I-D00105:274:H7MDXX:1:112:8133:866 length=51
AAAGTGCTACTACTTTTGAGTCTTGGAATTCTCGGGTGCCAAGGAACTCCA
>RR42357.95443.1 I-D00105:274:H7MDXX:1:112:8144:8637 length=51

Thank you very very much!

**Tags:** rna-seq, fasta
> **Comments:**
> **username_1**: Have you tried to compress your file into `.tar.gz`? What about the way to create a "tag" file like recommended [here](https://mcg.ustc.edu.cn/bsc/deanniso/tools.html)?  
> **username_2**: Have you tried cutting the sequence names down to integers instead of all the embedded information and then compressing them?  
> **username_2**: In fact it looks like the header lines are longer than the sequence lines so I would expect a significant reduction in the file size.  
> **username_3**: ^ this is funny.. but its true. microRNA isn't huge in length. I'm not exactly sure how 50 bp of sequence can take up 100 MB of data.  

---

## 2 answers

### Answer by username_3 on 2019-02-09. Score: 1

The obvious answer is to break up the file size and upload each separately,
First I'd check the file size,
wc infile.fa

Output

XXXXX      YYYY     ZZZZZ infile.fa

The number you want is XXXX - the number of lines in the file.
You'd then decide how many files you wanted to break it up into ... in your case divide XXXXX by 3. Then .. for example
sed -n '1,5000000p' infile.fa > outfile.fa
sed -n '5000001,2000000p' infile.fa > outfile2.fa
sed -n '2000001,4000000p' infile.fa > outfile3.fa


I'm not certain this would be cool however. An miRNA expert could answer this question very quickly. I think is you should check your data for sequence duplication. Simply piping the fasta file through a simple distance matrix and screening for "0" values would confirm this. 
Creating a frequency table of duplicate RNA must be a feature of Bioconductor, but in any case screening for duplication (technically duplicate records) is a formal SQL database command. 
It can be done a Perl/Python but its a bit clunky for a data set this size. I'd do it by setting the sequence as the key within a hash, the inefficiency has been solved (but its a weird way to do it). 
You would have a database for "headers" for a given sequence, which would take up space, but I would assess the potential level of duplication first.


---

### Answer by username_4 on 2019-02-09. Score: 1

The tool you want to use accepts compressed files, so the first thing you should do is compress your file. I created a 111M file of random, 51bp sequences, each with a random, 52 character header and a random 51 nt sequence:
$ ls -lh file.fa
-rw-r--r-- 1 terdon terdon 111M Feb  9 12:21 file.fa

$ head file.fa
>YZF8SGLVBPCKMY18GJ94XTRCS61BB4OZKUZ61D 1J6R7H3 EJQA5 length=51
ACGAAGGGCATTTGGCAACGAACCTATACTATATCACGAGCAATTAGGCAG
>SPR0INGWKMILAXZZPF73BKRV73ELL1ROZSHYO1E9JX6P5X6MUZ20 length=51
TTAAGATTCCGAGAGAGTCTGTGGTTGGGTCCCTCTCCATAATGCTTTCAT
>U CB82L2M3BWJ9ZZZG3V6DL0Y0FTW8BXIFTU0TK7SL VN5QE42J6 length=51
TGGCTGGGAGGGATAGTAGCGAGATTATCCAGTCCGGTGTTCACAAGCCGG
>TGH62IM2TJK2QJTIVWGZR IVKOW94X7D52KGM7DJP61KKF81KPYA length=51
GCCGCAATACCAAAAAGATGGTGTGATTAGGAAATAAATAGGATCCAACAA
>L5VS9AVEF3UTEO3341STET092YEZ43ESVH32W7GI6Q27OIR NTXF length=51
CTATAGACGACTACGAACGACTGCAGGTGTGCCAGTTCTCGGTGAAACGGA

If we now compress that file:
$ tar cvzf file.fa.tar.gz file.fa
file.fa
$ ls -lh file.fa.tar.gz 
-rw-r--r-- 1 terdon terdon 56M Feb  9 12:22 file.tar.gz

You will likely get a significantly better compression ratio since your headers won't be random and will likely be very repetitive. 
If that isn't enough, you should try simplifying your headers. For instance, just use an increasing number:
$ perl -pe 's/^>.*/">" . ++$i/e' file.fa > file.short.fa
$ ls -lh file.short.fa 
-rw-r--r-- 1 terdon terdon 58M Feb  9 12:23 file.short.fa

See what a huge difference that one change made? And if we now compress the file:
$ tar cvzf file.short.tar.gz file.short.fa
file.short.fa
$ ls -lh file.short.tar.gz
-rw-r--r-- 1 terdon terdon 19M Feb  9 12:24 file.short.tar.gz

Well below the threshold! 

If compression isn't enough, you should make sure you have no duplicate sequences in the file. Use the FastaToTbl and TblToFasta scripts I have posted here and:
FastaToTbl file.fa | sort | uniq | TblToFasta > file.uniq.fa

Finally, if none of these helps enough, split your file into multiple smaller ones:
split -l 100000 file.fa

That will create multiple files of 100000 lines (so 50000 sequences, assuming all your sequences are one line) each.


---

