<p>The following workaround seems to work but ideally I don’t want to have to change the <code>script</code> portion of my process so that it works with Docker.</p>

<pre><code>params.ref = 'data/reference/GRCh38.fa'

ref = file(params.ref)
ref_index_dir = ref.parent
fastq_channel = Channel.fromFilePairs('data/fastq/*_R{1,2}.fastq.gz')

process map_reads {
    input:
    set sample, '?.fastq.gz' from fastq_channel
    file ref
    file ref_index_dir

    output:
    file("${sample}.sam") into sam_channel

    script:
    """
    echo 'bwa mem -Y "reference/<span class="math-container">$ref" 1.fastq.gz 2.fastq.gz &gt;$</span>{sample}.sam'
    stat -t reference/<span class="math-container">${ref}.bwt &amp;&amp; touch $</span>{sample}.sam
    """
}
</code></pre>

<p>Here, <code>reference</code> inside the script section refers to the parent directory name of the reference, and needs to be adjusted accordingly. Having to refer to <code>reference/$ref</code> is annoying but at least it works — both inside a Docker container and outside of it.</p>

CC BY-SA 4.0
2019-06-06T12:38:55.310Z
2019-06-06T12:38:55.310Z
Konrad Rudolph
<p>Process input files need to be declared properly to allow nextflow to create the required docker mounts accordingly. Therefore the <code>reference</code> file should be declared in the process input block, eh: </p>

<pre><code>input:
 set sample, file(fastq) from fastq_channel
 file reference 
</code></pre>

CC BY-SA 4.0
2019-06-10T11:08:24.530Z
The issue is that BWA for instance doesn’t just use the “reference” file, otherwise there’d be no issue. But it additionally derives various other files from the reference filename opaquely (i.e. these names are in principle implementation details and not known, except that they all have the reference filename as a prefix).
Konrad Rudolph
CC BY-SA 4.0
2019-06-07T10:08:32.027Z
2019-06-07T10:08:32.027Z
pditommaso
<p>I’m trying to run a Nextflow workflow in a custom Docker container. Without Docker, the workflow succeeds. But running it inside the container leads to an error because a dependent file cannot be found.</p>

<p>Here’s a minimal script to illustrate the issue (similar to the <a href="https://www.nextflow.io/docs/latest/basic.html" rel="nofollow noreferrer">basic Nextflow script example</a>):</p>

<pre><code>params.reference = "/data/reference/GRCh38.fa"

reference = file(params.reference)
fastq_channel = Channel.fromFilePairs('/data/fastq/*_R{1,2}.fastq.gz')

process map_reads {
    input:
    set sample, '?.fastq.gz' from fastq_channel

    output:
    file("${sample}.sam") into sam_channel

    script:
    """
    echo 'bwa mem -Y "<span class="math-container">$reference" 1.fastq.gz 2.fastq.gz &gt;$</span>{sample}.sam'
    stat -t "${reference}.bwt"
    """
}
</code></pre>

<p>Running this inside a container will cause the path of the query, <code>/some/fastq/</code>, to be mounted inside the container. However, the path to the database, <code>/data/reference</code>, will <em>not</em> be mounted. Consequently, I get the following execution error:</p>

<blockquote>
  <p>stat: cannot stat 'GRCh38-chr21.fa.bwt': No such file or directory</p>
</blockquote>

<p>Without the container, the command succeeds.</p>

<hr>

<p>I understand <em>why</em> the error occurs: Nextflow doesn’t track the dependency on <code>$reference</code> inside the command, and thus doesn’t mount its path in the container.</p>

<p>My question is, <strong>how do I fix this?</strong> The documentation is silent on this regard, and none of the workflows I can find online address the problem. For example, the <a href="https://github.com/IARCbioinfo" rel="nofollow noreferrer">GATK4 IARC workflows</a> would presumably have the same issue, and aren’t configured for use with Docker.</p>

<p>Just adding the <code>reference</code> as a process input is insufficient, since BWA requires various other index files. Am I supposed to add these all manually as inputs?</p>

<docker><nextflow>
Run Nextflow with file dependencies inside a Docker container?
# Run Nextflow with file dependencies inside a Docker container?

(Asked by: username_0 on 2019-06-06)

I’m trying to run a Nextflow workflow in a custom Docker container. Without Docker, the workflow succeeds. But running it inside the container leads to an error because a dependent file cannot be found.
Here’s a minimal script to illustrate the issue (similar to the basic Nextflow script example):
params.reference = "/data/reference/GRCh38.fa"

reference = file(params.reference)
fastq_channel = Channel.fromFilePairs('/data/fastq/*_R{1,2}.fastq.gz')

process map_reads {
    input:
    set sample, '?.fastq.gz' from fastq_channel

    output:
    file("${sample}.sam") into sam_channel

    script:
    """
    echo 'bwa mem -Y "$reference" 1.fastq.gz 2.fastq.gz >${sample}.sam'
    stat -t "${reference}.bwt"
    """
}

Running this inside a container will cause the path of the query, /some/fastq/, to be mounted inside the container. However, the path to the database, /data/reference, will not be mounted. Consequently, I get the following execution error:

stat: cannot stat 'GRCh38-chr21.fa.bwt': No such file or directory

Without the container, the command succeeds.

I understand why the error occurs: Nextflow doesn’t track the dependency on $reference inside the command, and thus doesn’t mount its path in the container.
My question is, how do I fix this? The documentation is silent on this regard, and none of the workflows I can find online address the problem. For example, the GATK4 IARC workflows would presumably have the same issue, and aren’t configured for use with Docker.
Just adding the reference as a process input is insufficient, since BWA requires various other index files. Am I supposed to add these all manually as inputs?

**Tags:** docker, nextflow

---

## 2 answers

### Answer by username_0 on 2019-06-06. Score: 3

The following workaround seems to work but ideally I don’t want to have to change the script portion of my process so that it works with Docker.
params.ref = 'data/reference/GRCh38.fa'

ref = file(params.ref)
ref_index_dir = ref.parent
fastq_channel = Channel.fromFilePairs('data/fastq/*_R{1,2}.fastq.gz')

process map_reads {
    input:
    set sample, '?.fastq.gz' from fastq_channel
    file ref
    file ref_index_dir

    output:
    file("${sample}.sam") into sam_channel

    script:
    """
    echo 'bwa mem -Y "reference/$ref" 1.fastq.gz 2.fastq.gz >${sample}.sam'
    stat -t reference/${ref}.bwt && touch ${sample}.sam
    """
}

Here, reference inside the script section refers to the parent directory name of the reference, and needs to be adjusted accordingly. Having to refer to reference/$ref is annoying but at least it works — both inside a Docker container and outside of it.


---

### Answer by username_1 on 2019-06-07. Score: 1

Process input files need to be declared properly to allow nextflow to create the required docker mounts accordingly. Therefore the reference file should be declared in the process input block, eh: 
input:
 set sample, file(fastq) from fastq_channel
 file reference

> **Comments:**
> **username_0**: The issue is that BWA for instance doesn’t just use the “reference” file, otherwise there’d be no issue. But it additionally derives various other files from the reference filename opaquely (i.e. these names are in principle implementation details and not known, except that they all have the reference filename as a prefix).  

---

