<p>What does this program do in the expected use case, presumably where intermediate output files are generated? From that, can you get any information about how files are updated? For example, are the creation times on the files identical (as would be expected if they're written at the same time), or are they different from each other?</p>
<p>I could imagine a situation where the initial cat test works because the two file writes are running concurrently and are non-blocking, whereas the second test doesn't work because it's waiting for input on one of the two files.</p>

CC BY-SA 4.0
2024-06-17T22:13:56.290Z
2024-06-18T13:18:51.890Z
2024-06-18T13:18:51.890Z
gringer
<p>I am trying to use a named pipe to pass data from the Trimmer utility from the <a href="https://www.agilent.com/cs/library/software/Public/AGeNT%20ReadMe.pdf" rel="nofollow noreferrer">AGeNT</a> toolbox from Agilent, through bwa-mem2. The normal behavior is that the Trimmer utility writes trimmed fastq files to disk. I would like to use a named pipe to pass the data to bwa-mem2 for a few reasons:</p>
<ol>
<li>speed</li>
<li>to see if I can.</li>
</ol>
<p>I first verified that the the output of Trimmer can in principle be passed through a named pipe. The output of Trimmer was piped to cat whose output was redirected into a text file. The text of this file was compared to a corresponding fastq files that I had previously produced by running Trimmer normally. The text of the files were found to be identical, indicating that output of trimmer was successfully piped.</p>
<pre><code>#!/usr/bin/bash
set -e
set -x
set -o pipefail
pipe=True
ID=345705 
# untrimmmmed fastq files
R1=&quot;/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/<span class="math-container">${ID}_S9_R1_001.fastq.gz"
R2="/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/$</span>{ID}_S9_R2_001.fastq.gz&quot;

# where and how to name the output files
out_dir=&quot;/mounts/pipes/&quot;
mkdir -p <span class="math-container">${out_dir}
out_prefix="$</span>{out_dir}<span class="math-container">${ID}_Cut"
#for file in /mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/*.fastq.gz; do gunzip -c "$</span>file&quot; &gt; &quot;/path/to/output/directory/<span class="math-container">$(basename "$</span>{file%.gz}&quot;)&quot;; done
# software paths
jarfile=&quot;/mounts/CSF_CFDNA-development/resources/software/agent3.0/agent3.0/lib/trimmer-3.0.3.jar&quot;
mem2_executable=&quot;/mounts/CSF_CFDNA-development/containers/bwa-mem2-2.2.1_x64-linux/bwa-mem2.avx512bw&quot;
ref_genome=&quot;/mounts/CSF_CFDNA-development/resources/hg38/v0/Homo_sapiens_assembly38.fasta&quot;

# make output pipes
R1_PIPE=<span class="math-container">${out_prefix}_R1.fastq.gz
R2_PIPE=$</span>{out_prefix}_R2.fastq.gz
mkfifo <span class="math-container">${R1_PIPE} $</span>{R2_PIPE}
# start writer process trimmer in background
java -Xmx15G -jar <span class="math-container">${jarfile}\
        -fq1 $</span>{R1} \
        -fq2 <span class="math-container">${R2} \
    -v2 \
    -out $</span>{out_prefix} &amp;

trimmerPid=<span class="math-container">$!
# start writer processes
cat $</span>{R1_PIPE} &gt; <span class="math-container">${out_dir}$</span>{ID}&quot;R1.txt&quot;&amp;
cat <span class="math-container">${R2_PIPE} &gt; $</span>{out_dir}${ID}&quot;R2.txt&quot;

wait &quot;<span class="math-container">$trimmerPid"
# compare the generated text files to the actualtrimmed files calculated on a previous run of trimmer
# if they agree then the output was successfully piped
trimdir="/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/TRIMMED_FASTQ"
diff -q "$</span>{trimdir}/<span class="math-container">${ID}_Cut_R1.fastq.gz" $</span>{out_dir}<span class="math-container">${ID}"R1.txt"
if [ $</span>? -eq 0 ]
then
    echo &quot;The R1 files are the same.&quot;
else
    echo &quot;The R1 files are different.&quot;
fi

diff -q &quot;<span class="math-container">${trimdir}/$</span>{ID}_Cut_R2.fastq.gz&quot; <span class="math-container">${out_dir}$</span>{ID}&quot;R2.txt&quot;
if [ $? -eq 0 ]
then
    echo &quot;The R2 files are the same.&quot;
else
    echo &quot;The R2 files are different.&quot;
fi
</code></pre>
<p>Agilent advised me that in principle it was not possible to catch the output of trimmer in a named pipe. However, given the result of the first experiment in my post. I think their advice was not correct.</p>
<p>However when I try to pipe instead to bwa-mem2 I have problems. bwa-mem2 starts, loads the reference genome and writes the header of the SAM file, but no reads. Then hangs for eternity.</p>
<pre><code>#!/usr/bin/bash
set -e
set -x
set -o pipefail
pipe=True
ID=345705 
# untrimmmmed fastq files
R1=&quot;/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/<span class="math-container">${ID}_S9_R1_001.fastq.gz"
R2="/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/$</span>{ID}_S9_R2_001.fastq.gz&quot;

# where and how to name the output files
out_dir=&quot;/mounts/pipes/&quot;
mkdir -p <span class="math-container">${out_dir}
out_prefix="$</span>{out_dir}${ID}_Cut&quot;

# software paths
jarfile=&quot;/mounts/CSF_CFDNA-development/resources/software/agent3.0/agent3.0/lib/trimmer-3.0.3.jar&quot;
mem2_executable=&quot;/mounts/CSF_CFDNA-development/containers/bwa-mem2-2.2.1_x64-linux/bwa-mem2.avx512bw&quot;
ref_genome=&quot;/mounts/CSF_CFDNA-development/resources/hg38/v0/Homo_sapiens_assembly38.fasta&quot;

# make output pipes
R1_PIPE=<span class="math-container">${out_prefix}_R1.fastq.gz
R2_PIPE=$</span>{out_prefix}_R2.fastq.gz
mkfifo <span class="math-container">${R1_PIPE} $</span>{R2_PIPE}
#cat &lt;(zcat sample_1.fq.gz | head -n 4) &lt;(zcat sample_2.fq.gz | head -n 4) | bwa-mem2 mem -t 4 -p ref.fa - &gt; out.mem2.sam
# start writer process trimmer in background
java -Xmx15G -jar <span class="math-container">${jarfile}\
    -fq1 $</span>{R1} \
        -fq2 <span class="math-container">${R2} \
    -v2 \
    -out $</span>{out_prefix} &amp;

trimmerPid=<span class="math-container">$!
$</span>{mem2_executable} mem -t 7 -C -R &quot;@RG\tID:<span class="math-container">${ID}\tPL:ILLUMINA\tLB:SureSelectXTHSv2\tSM:$</span>{ID}&quot;\
         <span class="math-container">${ref_genome} $</span>{R1_PIPE} <span class="math-container">${R2_PIPE} &gt; $</span>{out_dir}${ID}.sam

wait &quot;$trimmerPid&quot;
</code></pre>
<p>Any help you can give is much appreciated.</p>
<p><strong>Update</strong></p>
<p>It looks like the problem may be that bwa-mem2 cannot read from the named pipe. The output of strace indicates a failed call to lseek immediately after opening the the pipes.</p>
<p>openat(AT_FDCWD, &quot;/mounts/pipes/345705_Cut_R1.fastq.gz&quot;, O_RDONLY) = 4</p>
<p>lseek(4, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)</p>

<phylogenetics><linux><shell>
Passing data from the Agilent Trimmer utility to bwa-mem2 via a named pipe
# Passing data from the Agilent Trimmer utility to bwa-mem2 via a named pipe

(Asked by: username_0 on 2024-06-17)

I am trying to use a named pipe to pass data from the Trimmer utility from the AGeNT toolbox from Agilent, through bwa-mem2. The normal behavior is that the Trimmer utility writes trimmed fastq files to disk. I would like to use a named pipe to pass the data to bwa-mem2 for a few reasons:

speed
to see if I can.

I first verified that the the output of Trimmer can in principle be passed through a named pipe. The output of Trimmer was piped to cat whose output was redirected into a text file. The text of this file was compared to a corresponding fastq files that I had previously produced by running Trimmer normally. The text of the files were found to be identical, indicating that output of trimmer was successfully piped.
#!/usr/bin/bash
set -e
set -x
set -o pipefail
pipe=True
ID=345705 
# untrimmmmed fastq files
R1="/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/${ID}_S9_R1_001.fastq.gz"
R2="/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/${ID}_S9_R2_001.fastq.gz"

# where and how to name the output files
out_dir="/mounts/pipes/"
mkdir -p ${out_dir}
out_prefix="${out_dir}${ID}_Cut"
#for file in /mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/*.fastq.gz; do gunzip -c "$file" > "/path/to/output/directory/$(basename "${file%.gz}")"; done
# software paths
jarfile="/mounts/CSF_CFDNA-development/resources/software/agent3.0/agent3.0/lib/trimmer-3.0.3.jar"
mem2_executable="/mounts/CSF_CFDNA-development/containers/bwa-mem2-2.2.1_x64-linux/bwa-mem2.avx512bw"
ref_genome="/mounts/CSF_CFDNA-development/resources/hg38/v0/Homo_sapiens_assembly38.fasta"

# make output pipes
R1_PIPE=${out_prefix}_R1.fastq.gz
R2_PIPE=${out_prefix}_R2.fastq.gz
mkfifo ${R1_PIPE} ${R2_PIPE}
# start writer process trimmer in background
java -Xmx15G -jar ${jarfile}\
        -fq1 ${R1} \
        -fq2 ${R2} \
    -v2 \
    -out ${out_prefix} &

trimmerPid=$!
# start writer processes
cat ${R1_PIPE} > ${out_dir}${ID}"R1.txt"&
cat ${R2_PIPE} > ${out_dir}${ID}"R2.txt"

wait "$trimmerPid"
# compare the generated text files to the actualtrimmed files calculated on a previous run of trimmer
# if they agree then the output was successfully piped
trimdir="/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/TRIMMED_FASTQ"
diff -q "${trimdir}/${ID}_Cut_R1.fastq.gz" ${out_dir}${ID}"R1.txt"
if [ $? -eq 0 ]
then
    echo "The R1 files are the same."
else
    echo "The R1 files are different."
fi

diff -q "${trimdir}/${ID}_Cut_R2.fastq.gz" ${out_dir}${ID}"R2.txt"
if [ $? -eq 0 ]
then
    echo "The R2 files are the same."
else
    echo "The R2 files are different."
fi

Agilent advised me that in principle it was not possible to catch the output of trimmer in a named pipe. However, given the result of the first experiment in my post. I think their advice was not correct.
However when I try to pipe instead to bwa-mem2 I have problems. bwa-mem2 starts, loads the reference genome and writes the header of the SAM file, but no reads. Then hangs for eternity.
#!/usr/bin/bash
set -e
set -x
set -o pipefail
pipe=True
ID=345705 
# untrimmmmed fastq files
R1="/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/${ID}_S9_R1_001.fastq.gz"
R2="/mounts/CSF_CFDNA-development/workflows/preprocessing/workflow/results/L1283/UNTRIMMED_VERY_DOWNSAMPLED_FASTQ/${ID}_S9_R2_001.fastq.gz"

# where and how to name the output files
out_dir="/mounts/pipes/"
mkdir -p ${out_dir}
out_prefix="${out_dir}${ID}_Cut"

# software paths
jarfile="/mounts/CSF_CFDNA-development/resources/software/agent3.0/agent3.0/lib/trimmer-3.0.3.jar"
mem2_executable="/mounts/CSF_CFDNA-development/containers/bwa-mem2-2.2.1_x64-linux/bwa-mem2.avx512bw"
ref_genome="/mounts/CSF_CFDNA-development/resources/hg38/v0/Homo_sapiens_assembly38.fasta"

# make output pipes
R1_PIPE=${out_prefix}_R1.fastq.gz
R2_PIPE=${out_prefix}_R2.fastq.gz
mkfifo ${R1_PIPE} ${R2_PIPE}
#cat <(zcat sample_1.fq.gz | head -n 4) <(zcat sample_2.fq.gz | head -n 4) | bwa-mem2 mem -t 4 -p ref.fa - > out.mem2.sam
# start writer process trimmer in background
java -Xmx15G -jar ${jarfile}\
    -fq1 ${R1} \
        -fq2 ${R2} \
    -v2 \
    -out ${out_prefix} &

trimmerPid=$!
${mem2_executable} mem -t 7 -C -R "@RG\tID:${ID}\tPL:ILLUMINA\tLB:SureSelectXTHSv2\tSM:${ID}"\
         ${ref_genome} ${R1_PIPE} ${R2_PIPE} > ${out_dir}${ID}.sam

wait "$trimmerPid"

Any help you can give is much appreciated.
Update
It looks like the problem may be that bwa-mem2 cannot read from the named pipe. The output of strace indicates a failed call to lseek immediately after opening the the pipes.
openat(AT_FDCWD, "/mounts/pipes/345705_Cut_R1.fastq.gz", O_RDONLY) = 4
lseek(4, 0, SEEK_CUR)                   = -1 ESPIPE (Illegal seek)

**Tags:** phylogenetics, linux, shell

---

## 1 answers

### Answer by username_1 on 2024-06-17. Score: 2

What does this program do in the expected use case, presumably where intermediate output files are generated? From that, can you get any information about how files are updated? For example, are the creation times on the files identical (as would be expected if they're written at the same time), or are they different from each other?

I could imagine a situation where the initial cat test works because the two file writes are running concurrently and are non-blocking, whereas the second test doesn't work because it's waiting for input on one of the two files.


---

