<p>[![example csv file][1]][1]</p>
<p>First I want to start by parsing and doing some operation on a text/csv file. How could I read a csv/text file in nextflow?</p>
<p>Suppose, I have above csv file with 3 columns. Lets say I want to create a channel which emits an array from minimum of column 2 (7.9e5) to maximum of column 2 (8.8e5) with an increment of 1e4. How could I create the following channel and later access it in the script? Any help? The</p>
<pre><code>array=([1]=&quot;7.9e5 8.0e5&quot; [2]=&quot;8.0e5 8.1e5&quot; [3]=&quot;8.1e5 8.2e5&quot; [4]=&quot;8.2e5 8.3e5&quot; [5]=&quot;8.3e5 8.4e5&quot; [6]=&quot;8.4e5 8.5e5&quot; [7]=&quot;8.5e5 8.6e5&quot; [8]=&quot;8.6e5 8.7e5&quot; [9]=&quot;8.7e5 8.8e5&quot; [10]=&quot;8.8e5 8.9e5&quot;)
</code></pre>
<p>I need to do some operations with this array in the script which will generate multiple files for each element in the array. These multiple files need to be merged in a single file.
Sorry, I could not provide any nextflow script that I tried, because I not sure how create an array with <code>splitCsv</code></p>
<p>I have tried in python:</p>
<pre><code>df1=pd.read_csv('data1/omni-ec-eur-fin-pass-qc.bim',sep='\t',header=None)
arr1=np.arange(df1.iloc[:,3].min()-1,df1.iloc[:,3].max()+1,3000000)
diffs = []
for x in zip(arr1[0::],arr1[1::]):
    diffs.append(x)
arr2=np.arange(1,len(diffs)+1,1)
dict(zip(arr2,diffs))
</code></pre>
<p>Which gave me:</p>
<pre><code>{1: (12343, 3012343),
 2: (3012343, 6012343),
 3: (6012343, 9012343),
 4: (9012343, 12012343),...
</code></pre>
<p>But I need something like this:</p>
<pre><code>([1]=&quot;12343 3012343&quot; [2]=&quot;3012343 6012343&quot; [3]=&quot;6012343   9012343&quot; [4]=&quot;9012343 12012343&quot;....)
</code></pre>
<p>It could be arrays (22 arrays for 22 chromosome) or a csv file containing arrays for each chromosome which could be used as an input channel. There are 22 bim files (one for each chromosome) stored in the output channel which would be used to create the above mentioned array.
The original data inside the bim looks like this:</p>
<pre><code>    0   1   2   3   4   5
0   1   rs11240777  0   798959  A   G
1   1   rs4475691   0   846808  A   G
2   1   rs7537756   0   854250  G   A
3   1   rs13302982  0   861808  A   G
4   1   rs1110052   0   873558  C   A
</code></pre>
<p>The 4th column would be used to create the arrays.
[1]: <a href="https://i.sstatic.net/cxqyl.png" rel="nofollow noreferrer">https://i.sstatic.net/cxqyl.png</a></p>

CC BY-SA 4.0
2021-04-09T13:00:32.177Z
PLease [edit] your question and give us an example file and minimal nextflow script to work with.
terdon
CC BY-SA 4.0
2021-04-10T20:22:21.050Z
Is there any book to learn nextflow operations? I was trying to find any book to learn nextflow but failed. At least if there is any example of creating array in nextflow would be great.
zillur rahman
CC BY-SA 4.0
2021-04-11T00:08:05.720Z
Have you read the nextflow documentation? I don't know much about it, but the whole idea is to use it for control flow; it runs other programs and they can have arrays. But next flow isn't a programing language per se. You write your script in Python or peel or whatever and then run the pipeline if scripts with nextflow.
terdon
CC BY-SA 4.0
2021-04-11T10:04:32.877Z
@zillurrahman Do you really need an array? Or do you mean a channel? Your last paragraph suggests the latter would be a better fit. Please edit your question to clarify exactly what is needed.
Steve
CC BY-SA 4.0
2021-04-11T10:05:58.497Z
@zillurrahman Generally, it's better to off-load all work to one or more processes. Writing a process to create a range of values, like in your example, is trivial in Perl/Python. You can use your favorite language in a process' script block. Please see: https://www.nextflow.io/docs/latest/process.html#scripts-a-la-carte
Steve
CC BY-SA 4.0
2021-04-11T10:11:34.923Z
@terdon It's an extension of the Groovy programming language :-)
Steve
CC BY-SA 4.0
2021-04-11T12:02:55.620Z
@steve sure, but it isn't designed to be used as a regular programing language but as a process control system to build and manage pipelines. It seems far more reasonable to manage the data in another script and then have nextflow manage that script instead of trying to use nextflow as your scripting language directly. Which is exacy what you suggested too.
terdon
CC BY-SA 4.0
2021-04-11T12:08:48.280Z
Thanks @Steve and @terdon, I edited the above question. Hope, it would clarify my necessity.  However, I am also trying to create such array in `bash` . But, I think probably a nextflow channel would be more appropriate in this case.
zillur rahman
CC BY-SA 4.0
2021-04-11T15:22:16.280Z
@terdon wasn't disagreeing with you on that...
Steve
CC BY-SA 4.0
2021-04-11T15:39:41.857Z
@zillurrahman Is there some reason for choosing Bash over Python for example? Bash isn't a good choice given your requirements.
Steve
CC BY-SA 4.0
2021-04-11T16:39:44.140Z
`@Steve` No reason to choose bash. I tried in python and got an array. `np.arange(df1.iloc[:,3].min()-1,df1.iloc[:,3].max()+1,3000000)` which gives me: `array([    12343,   3012343,   6012343,   9012343,  12012343, ....` . How could I get something like: `([1]="12343 3012343" [2]="3012343 6012343" [3]="6012343   9012343" [4]="9012343 12012343"....)`
zillur rahman
CC BY-SA 4.0
2021-04-11T20:30:35.480Z
So far, I have done: `diffs = []
for x in zip(arr1[0::],arr1[1::]):
    diffs.append(x)
diffs=np.array(diffs)
arr2=np.arange(1,len(diffs)+1,1)
arr3=np.column_stack((arr2,diffs))
arr4=arr3.tolist()` which gave me `[[1, 12343, 3012343],
 [2, 3012343, 6012343],
 [3, 6012343, 9012343],` . Should I convert it to string and use regex to get `([1]="12343 3012343" [2]="3012343 6012343" [3]="6012343   9012343" [4]="9012343 12012343"....)` or there are better alternatives?
zillur rahman
CC BY-SA 4.0
2021-04-11T23:19:01.390Z
@zillurrahman Do you really need to create the array string? If all you eventually want to do is create a channel of tuples, your first process should write a simple CSV which can then be split using the [splitCsv operator](https://www.nextflow.io/docs/latest/operator.html#splitcsv).
Steve
CC BY-SA 4.0
2021-04-11T23:30:24.850Z
@zillurrahman Also, your comments above should be edits to your question. Small snippets of unformatted code isn't especially helpful. Please include a small minimal example and the code that reproduces the issue.
Steve
CC BY-SA 4.0
2021-04-12T01:24:39.037Z
Thanks @`Steve`. I have edited my above question. Hope it makes sense now.
zillur rahman
CC BY-SA 4.0
2021-04-12T02:44:45.310Z
@zillurrahman Like terdon mentioned earlier, please include some example input (i.e. `cat data1/omni-ec-eur-fin-pass-qc.bim`). Please format it in a code block.
Steve
CC BY-SA 4.0
2021-04-12T03:03:03.050Z
`@Steve` . The first few rows of the input data is added to the above question.
zillur rahman
CC BY-SA 4.0
2021-04-14T09:16:21.150Z
Hi, any help/suggestion for the above problem? Did I clarify enough the issue?
zillur rahman
<nextflow>
parse csv/text file to create a channel with array in nextflow
# parse csv/text file to create a channel with array in nextflow

(Asked by: username_0 on 2021-04-08)

[![example csv file][1]][1]
First I want to start by parsing and doing some operation on a text/csv file. How could I read a csv/text file in nextflow?
Suppose, I have above csv file with 3 columns. Lets say I want to create a channel which emits an array from minimum of column 2 (7.9e5) to maximum of column 2 (8.8e5) with an increment of 1e4. How could I create the following channel and later access it in the script? Any help? The
array=([1]="7.9e5 8.0e5" [2]="8.0e5 8.1e5" [3]="8.1e5 8.2e5" [4]="8.2e5 8.3e5" [5]="8.3e5 8.4e5" [6]="8.4e5 8.5e5" [7]="8.5e5 8.6e5" [8]="8.6e5 8.7e5" [9]="8.7e5 8.8e5" [10]="8.8e5 8.9e5")

I need to do some operations with this array in the script which will generate multiple files for each element in the array. These multiple files need to be merged in a single file.
Sorry, I could not provide any nextflow script that I tried, because I not sure how create an array with splitCsv
I have tried in python:
df1=pd.read_csv('data1/omni-ec-eur-fin-pass-qc.bim',sep='\t',header=None)
arr1=np.arange(df1.iloc[:,3].min()-1,df1.iloc[:,3].max()+1,3000000)
diffs = []
for x in zip(arr1[0::],arr1[1::]):
    diffs.append(x)
arr2=np.arange(1,len(diffs)+1,1)
dict(zip(arr2,diffs))

Which gave me:
{1: (12343, 3012343),
 2: (3012343, 6012343),
 3: (6012343, 9012343),
 4: (9012343, 12012343),...

But I need something like this:
([1]="12343 3012343" [2]="3012343 6012343" [3]="6012343   9012343" [4]="9012343 12012343"....)

It could be arrays (22 arrays for 22 chromosome) or a csv file containing arrays for each chromosome which could be used as an input channel. There are 22 bim files (one for each chromosome) stored in the output channel which would be used to create the above mentioned array.
The original data inside the bim looks like this:
    0   1   2   3   4   5
0   1   rs11240777  0   798959  A   G
1   1   rs4475691   0   846808  A   G
2   1   rs7537756   0   854250  G   A
3   1   rs13302982  0   861808  A   G
4   1   rs1110052   0   873558  C   A

The 4th column would be used to create the arrays.
[1]: https://i.sstatic.net/cxqyl.png

**Tags:** nextflow
> **Comments:**
> **username_1**: PLease [edit] your question and give us an example file and minimal nextflow script to work with.  
> **username_0**: Is there any book to learn nextflow operations? I was trying to find any book to learn nextflow but failed. At least if there is any example of creating array in nextflow would be great.  
> **username_1**: Have you read the nextflow documentation? I don't know much about it, but the whole idea is to use it for control flow; it runs other programs and they can have arrays. But next flow isn't a programing language per se. You write your script in Python or peel or whatever and then run the pipeline if scripts with nextflow.  
> **username_2**: @zillurrahman Do you really need an array? Or do you mean a channel? Your last paragraph suggests the latter would be a better fit. Please edit your question to clarify exactly what is needed.  
> **username_2**: @zillurrahman Generally, it's better to off-load all work to one or more processes. Writing a process to create a range of values, like in your example, is trivial in Perl/Python. You can use your favorite language in a process' script block. Please see: https://www.nextflow.io/docs/latest/process.html#scripts-a-la-carte  
> **username_2**: @terdon It's an extension of the Groovy programming language :-)  
> **username_1**: @steve sure, but it isn't designed to be used as a regular programing language but as a process control system to build and manage pipelines. It seems far more reasonable to manage the data in another script and then have nextflow manage that script instead of trying to use nextflow as your scripting language directly. Which is exacy what you suggested too.  
> **username_0**: Thanks @Steve and @terdon, I edited the above question. Hope, it would clarify my necessity.  However, I am also trying to create such array in `bash` . But, I think probably a nextflow channel would be more appropriate in this case.  
> **username_2**: @terdon wasn't disagreeing with you on that...  
> **username_2**: @zillurrahman Is there some reason for choosing Bash over Python for example? Bash isn't a good choice given your requirements.  
> **username_0**: `@Steve` No reason to choose bash. I tried in python and got an array. `np.arange(df1.iloc[:,3].min()-1,df1.iloc[:,3].max()+1,3000000)` which gives me: `array([    12343,   3012343,   6012343,   9012343,  12012343, ....` . How could I get something like: `([1]="12343 3012343" [2]="3012343 6012343" [3]="6012343   9012343" [4]="9012343 12012343"....)`  
> **username_0**: So far, I have done: `diffs = [] for x in zip(arr1[0::],arr1[1::]):     diffs.append(x) diffs=np.array(diffs) arr2=np.arange(1,len(diffs)+1,1) arr3=np.column_stack((arr2,diffs)) arr4=arr3.tolist()` which gave me `[[1, 12343, 3012343],  [2, 3012343, 6012343],  [3, 6012343, 9012343],` . Should I convert it to string and use regex to get `([1]="12343 3012343" [2]="3012343 6012343" [3]="6012343   9012343" [4]="9012343 12012343"....)` or there are better alternatives?  
> **username_2**: @zillurrahman Do you really need to create the array string? If all you eventually want to do is create a channel of tuples, your first process should write a simple CSV which can then be split using the [splitCsv operator](https://www.nextflow.io/docs/latest/operator.html#splitcsv).  
> **username_2**: @zillurrahman Also, your comments above should be edits to your question. Small snippets of unformatted code isn't especially helpful. Please include a small minimal example and the code that reproduces the issue.  
> **username_0**: Thanks @`Steve`. I have edited my above question. Hope it makes sense now.  
> **username_2**: @zillurrahman Like terdon mentioned earlier, please include some example input (i.e. `cat data1/omni-ec-eur-fin-pass-qc.bim`). Please format it in a code block.  
> **username_0**: `@Steve` . The first few rows of the input data is added to the above question.  
> **username_0**: Hi, any help/suggestion for the above problem? Did I clarify enough the issue?  

---

## 0 answers

