<p>Using more threads solves the problem.</p>
<h1>Run FASTQC with increased memory allocation</h1>
<pre><code>fastqc --outdir=&quot;<span class="math-container">$OUTPUT_QC" -t 10 "$</span>OUTPUT_PC&quot;
</code></pre>

CC BY-SA 4.0
2024-08-28T11:28:42.447Z
2024-08-29T09:06:39.393Z
2024-08-29T09:06:39.393Z
dim
<p>I'm running a bioinformatics pipeline to analyze large nanopore sequencing files using FastQC. However, I keep running into an <code>OutOfMemoryError</code>. Here's what happens:</p>
<pre><code>Unknown option: memory
Started analysis of SQK-NBD114-24_barcode16.fastq
Approx 5% complete for SQK-NBD114-24_barcode16.fastq
...
Approx 40% complete for SQK-NBD114-24_barcode16.fastq
Exception in thread &quot;Thread-1&quot; java.lang.OutOfMemoryError: Java heap space
        at uk.ac.babraham.FastQC.Utilities.QualityCount.&lt;init&gt;(QualityCount.java:33)
        at uk.ac.babraham.FastQC.Modules.PerBaseQualityScores.processSequence(PerBaseQualityScores.java:141)
        at uk.ac.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:89)
        at java.base/java.lang.Thread.run(Thread.java:829)
^CError: FastQC failed for /home/jupyter/bioinfo_pipeline/demuxed_2/SQK-NBD114-24_barcode16.fastq. Exiting pipeline.
</code></pre>
<p>I attempted to allocate more memory by setting <code>FASTQC_MEMORY=&quot;4g&quot;</code> and running FastQC with the following script:</p>
<pre class="lang-bash prettyprint-override"><code># Allocate more memory to FastQC (e.g., 4GB)
FASTQC_MEMORY=&quot;4g&quot;

# Loop over all .fastq files in the demuxed directory
for filename in &quot;<span class="math-container">$DEMUX_DIR"/*.fastq; do
    echo -e "\nProcessing $</span>filename...&quot;
    
    #-= STEP 3 - SECOND quality control (FASTQC and NANOQC) =- 
    echo -e &quot;\n~~~~~~-= Step 3, QC =-~~~~~~\n&quot;
    OUTPUT_PC=&quot;$filename&quot;  # Assuming OUTPUT_PC refers to the current .fastq file
    
    # Run FASTQC with increased memory allocation
    fastqc --outdir=&quot;<span class="math-container">$OUTPUT_QC" --memory="$</span>FASTQC_MEMORY&quot; &quot;$OUTPUT_PC&quot;
    
    # Check if FastQC failed due to memory issues or other errors
    if [ <span class="math-container">$? -ne 0 ]; then
    echo "Error: FastQC failed for $</span>filename. Exiting pipeline.&quot;
        exit 1
    fi
done
</code></pre>
<p>But it seems like 4GB isn't enough, as the process still crashes. The file I'm working with is quite large (Nanopore sequencing data):</p>
<pre><code>-rw-r--r-- 1 jupyter jupyter 442198080 Aug 27 14:49 SQK-NBD114-24_barcode16.fastq
</code></pre>
<p>According to <code>free -h</code>, I have 22 GiB of available memory. How much memory should I allocate to FastQC to prevent these errors? Is there a rule of thumb or specific calculation I should use when working with large <code>.fastq</code> files?</p>

CC BY-SA 4.0
2024-08-27T15:45:09.573Z
For now I would say give it like 15-20 GB and see if you run into the same error. Which version of FastQC are you using?
Karthik Nair
CC BY-SA 4.0
2024-08-27T16:13:09.720Z
Where did you get this `--memory` option? That doesn't seem to exist. Note the first line of your error output: `Unknown option: memory`. On my fastqc version 0.12.1, the `--memory` option is known, but only takes values in megabytes. Check that your version i) has a `--memory` option and ii) it can accept the string `4g` as a valid value.
terdon
CC BY-SA 4.0
2024-08-28T10:02:02.343Z
@Karthik Nair ```FastQC v0.11.9```
dim
<nanopore><fastqc>
How much memory should I allocate to FastQC for analyzing large nanopore sequencing files?
# How much memory should I allocate to FastQC for analyzing large nanopore sequencing files?

(Asked by: username_0 on 2024-08-27)

I'm running a bioinformatics pipeline to analyze large nanopore sequencing files using FastQC. However, I keep running into an OutOfMemoryError. Here's what happens:
Unknown option: memory
Started analysis of SQK-NBD114-24_barcode16.fastq
Approx 5% complete for SQK-NBD114-24_barcode16.fastq
...
Approx 40% complete for SQK-NBD114-24_barcode16.fastq
Exception in thread "Thread-1" java.lang.OutOfMemoryError: Java heap space
        at uk.ac.babraham.FastQC.Utilities.QualityCount.(QualityCount.java:33)
        at uk.ac.babraham.FastQC.Modules.PerBaseQualityScores.processSequence(PerBaseQualityScores.java:141)
        at uk.ac.babraham.FastQC.Analysis.AnalysisRunner.run(AnalysisRunner.java:89)
        at java.base/java.lang.Thread.run(Thread.java:829)
^CError: FastQC failed for /home/jupyter/bioinfo_pipeline/demuxed_2/SQK-NBD114-24_barcode16.fastq. Exiting pipeline.

I attempted to allocate more memory by setting FASTQC_MEMORY="4g" and running FastQC with the following script:
# Allocate more memory to FastQC (e.g., 4GB)
FASTQC_MEMORY="4g"

# Loop over all .fastq files in the demuxed directory
for filename in "$DEMUX_DIR"/*.fastq; do
    echo -e "\nProcessing $filename..."
    
    #-= STEP 3 - SECOND quality control (FASTQC and NANOQC) =- 
    echo -e "\n~~~~~~-= Step 3, QC =-~~~~~~\n"
    OUTPUT_PC="$filename"  # Assuming OUTPUT_PC refers to the current .fastq file
    
    # Run FASTQC with increased memory allocation
    fastqc --outdir="$OUTPUT_QC" --memory="$FASTQC_MEMORY" "$OUTPUT_PC"
    
    # Check if FastQC failed due to memory issues or other errors
    if [ $? -ne 0 ]; then
    echo "Error: FastQC failed for $filename. Exiting pipeline."
        exit 1
    fi
done

But it seems like 4GB isn't enough, as the process still crashes. The file I'm working with is quite large (Nanopore sequencing data):
-rw-r--r-- 1 jupyter jupyter 442198080 Aug 27 14:49 SQK-NBD114-24_barcode16.fastq

According to free -h, I have 22 GiB of available memory. How much memory should I allocate to FastQC to prevent these errors? Is there a rule of thumb or specific calculation I should use when working with large .fastq files?

**Tags:** nanopore, fastqc
> **Comments:**
> **username_1**: For now I would say give it like 15-20 GB and see if you run into the same error. Which version of FastQC are you using?  
> **username_2**: Where did you get this `--memory` option? That doesn't seem to exist. Note the first line of your error output: `Unknown option: memory`. On my fastqc version 0.12.1, the `--memory` option is known, but only takes values in megabytes. Check that your version i) has a `--memory` option and ii) it can accept the string `4g` as a valid value.  
> **username_0**: @Karthik Nair ```FastQC v0.11.9```  

---

## 1 answers

### Answer by username_0 on 2024-08-28. Score: 0

Using more threads solves the problem.
Run FASTQC with increased memory allocation
fastqc --outdir="$OUTPUT_QC" -t 10 "$OUTPUT_PC"


---

