<blockquote>
<p>The first two digits of vcf file names defines which sub-folder they are located.</p>
</blockquote>
<p>You could use this information to edit temp.txt so it contains the full path:</p>
<pre class="lang-bash prettyprint-override"><code>grep -v &quot;.tbi$&quot; temp.txt | sed 's/^\(.\)\(.\)/\1\2\/\1\2/' &gt; new_temp.txt
</code></pre>
<p>I'm assuming here that you don't actually want to run <code>dx run swiss-army-knife</code> on the vcf indexes.</p>
<p>This will turn the example of temp.txt you gave into this:</p>
<pre><code>32/3295094_24053_0_0.dragen.hard-filtered.vcf.gz
33/3309510_24053_0_0.dragen.hard-filtered.vcf.gz
34/3468518_24053_0_0.dragen.hard-filtered.vcf.gz
34/3471960_24053_0_0.dragen.hard-filtered.vcf.gz
</code></pre>
<p>You can then simply run a single for loop over that, instead of the nested one you are using now.</p>
<p>eg.</p>
<pre class="lang-bash prettyprint-override"><code>while read vcf
  do
    dx run swiss-army-knife [...]
  done &lt;  new_temp.txt
</code></pre>

CC BY-SA 4.0
2024-02-23T12:50:59.920Z
2024-02-23T14:39:17.773Z
2024-02-23T14:39:17.773Z
DavyCats
<p>Please somebody debug my code</p>
<p>In sub-folders there are <code>500000 vcf files</code> from which I only need <code>500 vcf files</code> listed in <code>temp.txt</code> which randomly distributed in sub-folders 10 to 60. The code reads each of 500000 vcf files one by one, if a file was in temp.txt, the code runs</p>
<pre><code>dx run swiss-army-knife -icmd=&quot;tabix -h <span class="math-container">$vcf -R $</span>REGIONS_FILE &gt; <span class="math-container">$output_file" -iin="$</span>filename_root&quot; -iin=&quot;<span class="math-container">$vcf.*" -iin="/Bulk/DRAGEN\ WGS/Whole\ genome\ variant\ call\ files\ (VCFs)\ (DRAGEN)\ [500k release]/file.bed" --destination "$</span>OUTPUT_FOLDER&quot; --detach --priority high --yes
    done &lt; &quot;$VCF_LIST_FILE&quot;
</code></pre>
<p>on that if not, the code goes to next file such a process takes years. I want this code only locates files listed in temp.txt and run the main code on them by iteration.</p>
<pre><code>#!/bin/bash

# Set the working directory to the desired location within DNAnexus

# Number of concurrent processes
N=4

# Iterate from 10 to 60
for folder_number in {10..60}; do
  (
    # Define the input files and parameters for this iteration
    folder_path=&quot;/Bulk/DRAGEN\ WGS/Whole\ genome\ variant\ call\ files\ (VCFs)\ (DRAGEN)\ [500k release]/$folder_number&quot;
    VCF_LIST_FILE=&quot;temp.txt&quot;
    REGIONS_FILE='file.bed'
    OUTPUT_FOLDER='/out'  # Replace with your desired output folder

    # Set the working directory to the folder for this iteration
    dx cd &quot;$folder_path&quot;

    # Read each VCF file from the list
    while read -r vcf; do
      filename_root=<span class="math-container">$(basename "$</span>vcf&quot;)
      output_file=&quot;${filename_root}_extracted.vcf.gz&quot;

      # Use dx run swiss-army-knife to execute tabix on the VCF file
      dx run swiss-army-knife -icmd=&quot;tabix -h <span class="math-container">$vcf -R $</span>REGIONS_FILE &gt; <span class="math-container">$output_file" -iin="$</span>filename_root&quot; -iin=&quot;<span class="math-container">$vcf.*" -iin="/Bulk/DRAGEN\ WGS/Whole\ genome\ variant\ call\ files\ (VCFs)\ (DRAGEN)\ [500k release]/file.bed" --destination "$</span>OUTPUT_FOLDER&quot; --detach --priority high --yes
    done &lt; &quot;$VCF_LIST_FILE&quot;
  ) &amp;

  # Limit the number of concurrent processes
  ((i=i%N)); ((i++==0)) &amp;&amp; wait
done

# Wait for all background jobs to finish
wait

# Finished message
echo &quot;++ Finished extracting genomic coordinates from VCF files&quot;
</code></pre>
<p>This is how temp.txt looks. The first two digits of vcf file names defines which sub-folder they are located. For instance, the below vcf files distributed in sub-folders 32, 33 and 34</p>
<pre><code>head temp_new.txt
head: temp_new.txt: No such file or directory
izadia@LV19Y7325V Downloads % head new_temp.txt
11/1195100_24053_0_0.dragen.hard-filtered.vcf.gz
12/1204363_24053_0_0.dragen.hard-filtered.vcf.gz
12/1246629_24053_0_0.dragen.hard-filtered.vcf.gz
13/1311312_24053_0_0.dragen.hard-filtered.vcf.gz
14/1427973_24053_0_0.dragen.hard-filtered.vcf.gz
14/1435043_24053_0_0.dragen.hard-filtered.vcf.gz
15/1558824_24053_0_0.dragen.hard-filtered.vcf.gz
16/1683316_24053_0_0.dragen.hard-filtered.vcf.gz
16/1692707_24053_0_0.dragen.hard-filtered.vcf.gz
17/1749825_24053_0_0.dragen.hard-filtered.vcf.gz
izadia@LV19Y7325V Downloads % tail new_temp.txt
52/5230856_24053_0_0.dragen.hard-filtered.vcf.gz
52/5257892_24053_0_0.dragen.hard-filtered.vcf.gz
54/5453459_24053_0_0.dragen.hard-filtered.vcf.gz
55/5514349_24053_0_0.dragen.hard-filtered.vcf.gz
55/5526010_24053_0_0.dragen.hard-filtered.vcf.gz
57/5718723_24053_0_0.dragen.hard-filtered.vcf.gz
58/5868836_24053_0_0.dragen.hard-filtered.vcf.gz
58/5875882_24053_0_0.dragen.hard-filtered.vcf.gz
58/5889611_24053_0_0.dragen.hard-filtered.vcf.gz
59/5917345_24053_0_0.dragen.hard-filtered.vcf.gz
izadia@LV19Y7325V Downloads % 
</code></pre>
<p>After running the script I get this error</p>
<pre><code>dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1195100_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1204363_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1246629_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1311312_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1427973_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1435043_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1558824_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1683316_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1692707_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field &quot;in&quot; could not be parsed as file: could not resolve &quot;1749825_24053_0_0.dragen.hard-filtered.vcf.gz&quot; to a name or ID
</code></pre>
<p>because the codes starts searching from sub-folder 10 for each of files in new_temp.txt one by one</p>

CC BY-SA 4.0
2024-02-23T11:28:37.363Z
So what do you want to do? If you don't want to check if the files exist, then don't check if the files exist. Are there any vcf files in your various folders that should not be processed? What is the point of `temp,txt`? And what does it contain? Full paths? FIle names?
terdon
CC BY-SA 4.0
2024-02-23T11:29:22.553Z
Oh, and your code never checks if a file exists. Just [edit] your question, explain what you really want to do, and maybe we can help.
terdon
CC BY-SA 4.0
2024-02-23T11:42:01.897Z
I edited my main post
Zizogolu
CC BY-SA 4.0
2024-02-23T11:54:32.927Z
And what does `temp.txt` look like? Please show us an example. We need to know if it is one file name per line, many per line, if it is just the names or the full paths etc.
terdon
CC BY-SA 4.0
2024-02-23T12:45:38.500Z
While you can do it in bash often switching to ie `python` gives you more debugging options, cleaner code, etc.
darked89
CC BY-SA 4.0
2024-02-23T17:00:39.647Z
Please do not edit your posts as follow up to answers without making the edits extremely clear to follow. Right now, we need to read the answers and your edit history (not question) side by side to understand what's going on and that's a huge pain.
Ram RS
CC BY-SA 4.0
2024-02-26T17:00:46.457Z
Please don't forget to mark the answer below as "accepted" if this addresses your question (it does look like it).
M__
CC BY-SA 4.0
2024-02-27T10:31:55.977Z
Thanks for the answer but the problem have not been solved yet so the code still stats checking from sub-folder 10 for the listed files one by one which takes forever
Zizogolu
<vcf><bash><linux><wgs><for-loop>
How to debug a bash script
# How to debug a bash script

(Asked by: username_0 on 2024-02-23)

Please somebody debug my code
In sub-folders there are 500000 vcf files from which I only need 500 vcf files listed in temp.txt which randomly distributed in sub-folders 10 to 60. The code reads each of 500000 vcf files one by one, if a file was in temp.txt, the code runs
dx run swiss-army-knife -icmd="tabix -h $vcf -R $REGIONS_FILE > $output_file" -iin="$filename_root" -iin="$vcf.*" -iin="/Bulk/DRAGEN\ WGS/Whole\ genome\ variant\ call\ files\ (VCFs)\ (DRAGEN)\ [500k release]/file.bed" --destination "$OUTPUT_FOLDER" --detach --priority high --yes
    done < "$VCF_LIST_FILE"

on that if not, the code goes to next file such a process takes years. I want this code only locates files listed in temp.txt and run the main code on them by iteration.
#!/bin/bash

# Set the working directory to the desired location within DNAnexus

# Number of concurrent processes
N=4

# Iterate from 10 to 60
for folder_number in {10..60}; do
  (
    # Define the input files and parameters for this iteration
    folder_path="/Bulk/DRAGEN\ WGS/Whole\ genome\ variant\ call\ files\ (VCFs)\ (DRAGEN)\ [500k release]/$folder_number"
    VCF_LIST_FILE="temp.txt"
    REGIONS_FILE='file.bed'
    OUTPUT_FOLDER='/out'  # Replace with your desired output folder

    # Set the working directory to the folder for this iteration
    dx cd "$folder_path"

    # Read each VCF file from the list
    while read -r vcf; do
      filename_root=$(basename "$vcf")
      output_file="${filename_root}_extracted.vcf.gz"

      # Use dx run swiss-army-knife to execute tabix on the VCF file
      dx run swiss-army-knife -icmd="tabix -h $vcf -R $REGIONS_FILE > $output_file" -iin="$filename_root" -iin="$vcf.*" -iin="/Bulk/DRAGEN\ WGS/Whole\ genome\ variant\ call\ files\ (VCFs)\ (DRAGEN)\ [500k release]/file.bed" --destination "$OUTPUT_FOLDER" --detach --priority high --yes
    done < "$VCF_LIST_FILE"
  ) &

  # Limit the number of concurrent processes
  ((i=i%N)); ((i++==0)) && wait
done

# Wait for all background jobs to finish
wait

# Finished message
echo "++ Finished extracting genomic coordinates from VCF files"

This is how temp.txt looks. The first two digits of vcf file names defines which sub-folder they are located. For instance, the below vcf files distributed in sub-folders 32, 33 and 34
head temp_new.txt
head: temp_new.txt: No such file or directory
izadia@LV19Y7325V Downloads % head new_temp.txt
11/1195100_24053_0_0.dragen.hard-filtered.vcf.gz
12/1204363_24053_0_0.dragen.hard-filtered.vcf.gz
12/1246629_24053_0_0.dragen.hard-filtered.vcf.gz
13/1311312_24053_0_0.dragen.hard-filtered.vcf.gz
14/1427973_24053_0_0.dragen.hard-filtered.vcf.gz
14/1435043_24053_0_0.dragen.hard-filtered.vcf.gz
15/1558824_24053_0_0.dragen.hard-filtered.vcf.gz
16/1683316_24053_0_0.dragen.hard-filtered.vcf.gz
16/1692707_24053_0_0.dragen.hard-filtered.vcf.gz
17/1749825_24053_0_0.dragen.hard-filtered.vcf.gz
izadia@LV19Y7325V Downloads % tail new_temp.txt
52/5230856_24053_0_0.dragen.hard-filtered.vcf.gz
52/5257892_24053_0_0.dragen.hard-filtered.vcf.gz
54/5453459_24053_0_0.dragen.hard-filtered.vcf.gz
55/5514349_24053_0_0.dragen.hard-filtered.vcf.gz
55/5526010_24053_0_0.dragen.hard-filtered.vcf.gz
57/5718723_24053_0_0.dragen.hard-filtered.vcf.gz
58/5868836_24053_0_0.dragen.hard-filtered.vcf.gz
58/5875882_24053_0_0.dragen.hard-filtered.vcf.gz
58/5889611_24053_0_0.dragen.hard-filtered.vcf.gz
59/5917345_24053_0_0.dragen.hard-filtered.vcf.gz
izadia@LV19Y7325V Downloads % 

After running the script I get this error
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1195100_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1204363_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1246629_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1311312_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1427973_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1435043_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1558824_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1683316_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1692707_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID
dxpy.exceptions.DXCLIError: Value provided for input field "in" could not be parsed as file: could not resolve "1749825_24053_0_0.dragen.hard-filtered.vcf.gz" to a name or ID

because the codes starts searching from sub-folder 10 for each of files in new_temp.txt one by one

**Tags:** vcf, bash, linux, wgs, for-loop
> **Comments:**
> **username_1**: So what do you want to do? If you don't want to check if the files exist, then don't check if the files exist. Are there any vcf files in your various folders that should not be processed? What is the point of `temp,txt`? And what does it contain? Full paths? FIle names?  
> **username_1**: Oh, and your code never checks if a file exists. Just [edit] your question, explain what you really want to do, and maybe we can help.  
> **username_0**: I edited my main post  
> **username_1**: And what does `temp.txt` look like? Please show us an example. We need to know if it is one file name per line, many per line, if it is just the names or the full paths etc.  
> **username_2**: While you can do it in bash often switching to ie `python` gives you more debugging options, cleaner code, etc.  
> **username_3**: Please do not edit your posts as follow up to answers without making the edits extremely clear to follow. Right now, we need to read the answers and your edit history (not question) side by side to understand what's going on and that's a huge pain.  
> **username_4**: Please don't forget to mark the answer below as "accepted" if this addresses your question (it does look like it).  
> **username_0**: Thanks for the answer but the problem have not been solved yet so the code still stats checking from sub-folder 10 for the listed files one by one which takes forever  

---

## 1 answers

### Answer by username_5 on 2024-02-23. Score: 4

The first two digits of vcf file names defines which sub-folder they are located.

You could use this information to edit temp.txt so it contains the full path:
grep -v ".tbi$" temp.txt | sed 's/^\(.\)\(.\)/\1\2\/\1\2/' > new_temp.txt

I'm assuming here that you don't actually want to run dx run swiss-army-knife on the vcf indexes.
This will turn the example of temp.txt you gave into this:
32/3295094_24053_0_0.dragen.hard-filtered.vcf.gz
33/3309510_24053_0_0.dragen.hard-filtered.vcf.gz
34/3468518_24053_0_0.dragen.hard-filtered.vcf.gz
34/3471960_24053_0_0.dragen.hard-filtered.vcf.gz

You can then simply run a single for loop over that, instead of the nested one you are using now.
eg.
while read vcf
  do
    dx run swiss-army-knife [...]
  done <  new_temp.txt


---

