<p>It's a good idea to pass around structured metadata, but together with data/files, as tuples: tutorial <a href="https://training.nextflow.io/advanced/metadata/" rel="nofollow noreferrer">here</a></p>
<p>So, like this:</p>
<pre><code>process bamToFastq {
    input:
        tuple val(meta), path(bam)

    container 'quay.io/biocontainers/samtools:1.8--4'
    cpus 4
    disk '50 GB'

    output:
        tuple val(meta), file('r0.fastq'), emit: read_other
        tuple val(meta), file('r{1,2}.fastq'), emit: read_paired
    &quot;&quot;&quot;
    samtools fastq -0 r0.fastq -1 r1.fastq -2 r2.fastq -@ <span class="math-container">${task.cpus} $</span>bam
    &quot;&quot;&quot;
}
</code></pre>
<p>If you need to update/carry over URLs/other metadata, I recommend doing this outside of processes - using <code>map</code> constructs on channel outputs.</p>

CC BY-SA 4.0
2024-08-26T13:41:07.483Z
2024-08-26T13:41:07.483Z
bricoletc
<p>I have a number of samples (from the 1000 genomes) that I want to analyse using Nextflow. However, I need to keep track of their metadata, fields like sample ID, analysis type etc. So I have a basic metadata object like this for each sample that I then pipe into a channel:</p>

<pre><code>            metadata = [
                sample: split[0],
                mapped: split[1],
                platform : split[2],
                alignment: split[3],
                tsi: split[4],
                type: split[5],
                date: split[6],
                filetype: split[7..-1],
                url: file(bam)
            ]
</code></pre>

<p>Then, in the first process, I attempt to use this metadata object:</p>

<pre><code>process bamToFastq {
    input:
        val bam

    container 'quay.io/biocontainers/samtools:1.8--4'
    cpus 4
    disk '50 GB'

    output:
        file('r*.fastq') into fastq

    """
    samtools fastq -0 r0.fastq -1 r1.fastq -2 r2.fastq -@ <span class="math-container">${task.cpus} $</span>bam.url
    """
}
</code></pre>

<p>However, I already run into some issues:</p>

<ul>
<li>Nextflow doesn't realise that <code>metadata.url</code> is a file, since I declared the whole object using <code>val</code>, so I'm guessing it won't be correctly localized (based on <a href="https://www.nextflow.io/docs/latest/process.html#input-of-files" rel="nofollow noreferrer">the documentation</a>).</li>
<li>I want pass down a new metadata object for each of my generated FASTQs, that's the same as the input one, but I want to add in the FASTQ as a URL, and the read number of each FASTQ. Where/how can I do this? Nextflow doesn't provide any section of the <code>process</code> for structuring the output data. If this were Python I would do the following:</li>
</ul>

<pre class="lang-py prettyprint-override"><code>    metadata = bam.copy()
    metadata['fastq'] = file
    metadata['read_number'] = file.name[1]
</code></pre>

<p>Is there a good way to pass around structured metadata like this in nextflow? Will this solve either of my problems?</p>

<nextflow>
Passing around complex metadata in Nextflow
# Passing around complex metadata in Nextflow

(Asked by: username_0 on 2019-07-30)

I have a number of samples (from the 1000 genomes) that I want to analyse using Nextflow. However, I need to keep track of their metadata, fields like sample ID, analysis type etc. So I have a basic metadata object like this for each sample that I then pipe into a channel:
            metadata = [
                sample: split[0],
                mapped: split[1],
                platform : split[2],
                alignment: split[3],
                tsi: split[4],
                type: split[5],
                date: split[6],
                filetype: split[7..-1],
                url: file(bam)
            ]

Then, in the first process, I attempt to use this metadata object:
process bamToFastq {
    input:
        val bam

    container 'quay.io/biocontainers/samtools:1.8--4'
    cpus 4
    disk '50 GB'

    output:
        file('r*.fastq') into fastq

    """
    samtools fastq -0 r0.fastq -1 r1.fastq -2 r2.fastq -@ ${task.cpus} $bam.url
    """
}

However, I already run into some issues:

Nextflow doesn't realise that metadata.url is a file, since I declared the whole object using val, so I'm guessing it won't be correctly localized (based on the documentation).
I want pass down a new metadata object for each of my generated FASTQs, that's the same as the input one, but I want to add in the FASTQ as a URL, and the read number of each FASTQ. Where/how can I do this? Nextflow doesn't provide any section of the process for structuring the output data. If this were Python I would do the following:

    metadata = bam.copy()
    metadata['fastq'] = file
    metadata['read_number'] = file.name[1]

Is there a good way to pass around structured metadata like this in nextflow? Will this solve either of my problems?

**Tags:** nextflow

---

## 1 answers

### Answer by username_1 on 2024-08-26. Score: 0

It's a good idea to pass around structured metadata, but together with data/files, as tuples: tutorial here
So, like this:
process bamToFastq {
    input:
        tuple val(meta), path(bam)

    container 'quay.io/biocontainers/samtools:1.8--4'
    cpus 4
    disk '50 GB'

    output:
        tuple val(meta), file('r0.fastq'), emit: read_other
        tuple val(meta), file('r{1,2}.fastq'), emit: read_paired
    """
    samtools fastq -0 r0.fastq -1 r1.fastq -2 r2.fastq -@ ${task.cpus} $bam
    """
}

If you need to update/carry over URLs/other metadata, I recommend doing this outside of processes - using map constructs on channel outputs.


---

