<p>This is a bash script (#!/bin/bash) and you try to run python in it.
You should save your python script (e.g. in &quot;myscript.py&quot;) and call it within your bash script.
Maybe first try to see if you are able to run it in your terminal and then run it on your server with SLURM.</p>
<p>EDIT1: Also I don't think you need 3 nodes to run such script and this will probably not be even possible here. I think you need some specific modules to parallelize a python script with MPI (see <a href="https://stackoverflow.com/questions/25792357/how-to-distribute-a-python-script-across-multiple-nodes">https://stackoverflow.com/questions/25792357/how-to-distribute-a-python-script-across-multiple-nodes</a>), but again I think this would be an overkill. Try first with <code>#SBATCH -c 1</code> instead.</p>
<p>Edit2: Finally it might be possible to run jupyter-notebook on a node allocated by SLRUM instead of putting this into a python script by using <code>jupyter-notebook --no-browser --port=${port} --ip=${node}</code> command in your bash slurm script. Maybe you'll find this doc interesting: <a href="https://docs.ycrc.yale.edu/clusters-at-yale/guides/jupyter/" rel="nofollow noreferrer">https://docs.ycrc.yale.edu/clusters-at-yale/guides/jupyter/</a> .</p>

CC BY-SA 4.0
2020-09-24T13:01:42.673Z
thanks I am running it as python script after loading jupyter and python module but still shows an error? I am going to update the script and the error.
bioinfonext
CC BY-SA 4.0
2020-09-24T13:08:18.523Z
`%matplotlib inline` is a jupyter notebook command. Try to comment this line.
jeremy
CC BY-SA 4.0
2020-09-24T13:10:13.063Z
Again error. it is updated now.
bioinfonext
CC BY-SA 4.0
2020-09-24T13:19:06.577Z
Those are again some **jupyter notebook** writings to use bash commands within a notebook. You should try with `subprocess` module instead or build a bash script that does those steps and then call your python script.
I think you should think by yourself now and googling until you are really stuck, this is a good training to write real python script ;)
jeremy
CC BY-SA 4.0
2020-09-24T13:24:10.537Z
Hi @bioinfonext please don't forget to upvote this post and if you feel it answered your question to mark as 'resolved' if resolved.
M__
CC BY-SA 4.0
2020-09-24T13:29:16.483Z
You look to be running Jupyter notebook interactively and if run from PC/laptop/Mac via a jupyter-notebook command it should connect interactively. Pointing this onto your server I'm not certain however. Normally jupyter- notebook is run on one of the big cloud HPC platforms such as AWS or GCS.
M__
CC BY-SA 4.0
2020-09-24T13:34:00.593Z
I think now it is a bit confusing because the post has been edited a lot. But I think @bioinfonext wrote a jupyter notebook to plot his/here data and then wanted to do the same task but by running it on a cluster using SLURM job scheduling manager.
jeremy
CC BY-SA 4.0
2020-09-24T14:44:12.347Z
should I use below line of code instead "a = !unzip $qza" ;  import subprocess


a = subprocess.Popen(["unzip'', qza])
bioinfonext
CC BY-SA 4.0
2020-09-24T14:57:01.203Z
unzip is your command, qza is a python variable, if you look for "give variable to subprocess call" you find: https://stackoverflow.com/questions/4795190/passing-variables-to-a-subprocess-call
jeremy
CC BY-SA 4.0
2020-09-25T12:28:18.370Z
Thank you so much. Now I am trying to run this on server terminal after loading the modules and with some changes, but still getting error? I have updated the code and error.
bioinfonext
CC BY-SA 4.0
2020-09-24T12:47:05.750Z
2020-09-24T14:45:43.693Z
2020-09-24T14:45:43.693Z
jeremy
<p>I'll make a contribution and keep editing it rather than use 'comments', the error is about</p>
<pre><code>import numpy as np
</code></pre>
<p>At the moment this isn't about Jupyter, its about getting Python working on your server. Is that a good assessment? With Jupyter notebook you write a little bit of code, check it, write a bit more. That's how it is designed to operate. The other way in is to use Python in your shebang statement otherwise it will default to <code>sh</code> or <code>bash</code>.</p>
<p>The code you supplied doesn't explicitly use <code>numpy</code>. Pandas does use numpy and <code>np</code> is the standard shorthand, <code>Multiarray</code> is probably something pandas would use. Did you get this code working on your local machine?</p>
<p>I am wondering whether the numpy module on your server is up to date? Did you creat a specific Python environment, use <code>pip</code> or <code>conda</code>? Checking the Python version on your server and whether <code>numpy</code> needs updating is perhaps where I might start.</p>
<p>It is not pretty getting a HPC to run with your stuff and can take real work.</p>
<hr />
<p>I read your new post, I would simply reinstall Python from <code>Anaconda3</code> and run the script for now from shell pointing the shebang into Anaconda3 (I can't remember if you need to do this, I can check if needed). This a more professional way of using a HPC than using the resident Python. In good old days of Perl, when everything was nice and simple, <code>perlbrew</code> was essentially the Perl equivalent and no-one would use the system Perl.</p>
<p>The reason why I'm saying  this is I get this feeling it will be an endless cycle of import errors. Anaconda3 will give you control of your Python system without admin privelidge.</p>
<p>Just to explain if you are not calling <code>np</code> explicitly (and looking at your code I don't think you are) then explicitly stating <code>import numpy</code> isn't needed. The fact you wrote it and it then jumped onto a different error is a bit worrying. I would just use your own version of Python. In Jupyter I would guess, but don't know they use their own Python ... but from a data science point of view Jupyter is the front end to AWS and GCS because those guys want some control of what you are doing, not running riot. In DNN jupyter is big. If you want an absolute diagnosis then Stackoverflow might help, but I think you'll just another bug and another etc... One thing is that <code>pandas</code> changes alot and the version you are using might be ahead of the curve, thats my guess</p>
<p>Final point is if you are just plotting the <code>ggplot2</code> is really good, but that is <code>R</code>. <code>seaborne</code> is really good, <code>matplotlib</code> but hmmmm, but nothing is as clear or explicit as <code>ggplot2</code>. Python is good at lots of things, pandas is fabulous, but plotting isn't on par with <code>ggplot2</code></p>
<hr />
<p>What is 100% clear is that you are loading the HPC version of Python and this is out of date (3.4) should be 3.7 (or even 3.8?). The thing I don't understand is why is it loads Python 2 modules?? This is a complete mystery to me, I thought Python 2 and Python 3 were incompatible, this is suggesting backwards compatibility or else the reason it is falling over. I have no idea, but I wouldn't use the resident Python.</p>

CC BY-SA 4.0
2020-09-25T15:14:11.980Z
script and error updated after importing numpy.
bioinfonext
CC BY-SA 4.0
2020-09-25T15:42:24.743Z
@bioinfonext commented below
M__
CC BY-SA 4.0
2020-09-26T06:10:12.133Z
Yes, thanks, this is all very confusing, R is much easier to adopt for potting.But I could not able to find any good R code for rarefaction plot that can plot it based on metadata column.
bioinfonext
CC BY-SA 4.0
2020-09-26T12:09:20.237Z
@bioinfonext answered above
M__
CC BY-SA 4.0
2020-09-26T12:09:20.737Z
@bioinfonext answered above
M__
CC BY-SA 4.0
2020-09-25T14:36:15.357Z
2020-09-26T12:13:51.617Z
2020-09-26T12:13:51.617Z
M__
<p>Now jupyter installed on the server and I am using below code to plot the rarefaction plot but I am still getting some error. could you please suggest how I can get rid of it?</p>
<pre><code>    #Run it in jupyter lab or notebook


import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import os

#%matplotlib inline 

qza = 'alpha-rarefaction-20000.qzv' # path to rarefaction file from qiime2
os.system(&quot;unzip qza&quot;)

digest = a[1].split('/')[0].replace('  inflating: ','')
inf = digest + '/data/observed_otus.csv'
otus = pd.read_csv(inf,index_col=0, sep=',')
os.system('rm -r $digest')

cols = [col for col in otus.columns if 'iter' not in col]
mean,data = otus[cols],pd.DataFrame(columns=cols)
depths = [col.split('_')[0] for col in otus.columns if 'depth' in col]
otus = otus.drop(cols,axis=1)
otus.columns = depths
for depth in depths:
    mean['ASV'] = otus[depth].mean(axis=1)
    mean['depth']= depth.split('-')[-1]
    data = pd.concat([data,mean])
# here provide colors for each item that will be plotted
pal={'Irrigated':'brown','Rainfed':'red'} 
fig,ax = plt.subplots(figsize=(2.5,2),dpi=600,tight_layout=True)
sns.set(style='ticks',rc={&quot;lines.linewidth&quot;:0.7,&quot;axes.linewidth&quot;:0.5})
# use your column name to plot here
sns.lineplot(x='depth',y='ASV',data=data,palette=pal,hue='Season',sort=False,err_style='bars',\
             dashes=True,style='Season',ci=67)
ax.set_xlabel('Sequencing depth',fontsize=8)
ax.set_ylabel('Observed ASVs',fontsize=8)
ax.tick_params(axis='x', labelrotation=90)
ax.tick_params(axis='both',which='major',length=2,pad=0.5,labelsize=6)
handles, labels = ax.get_legend_handles_labels()
ax.legend(handles=handles[1:],labels=labels[1:],fontsize=5,frameon=False,numpoints=4,borderaxespad=0,handletextpad=0.2,loc=2,)
plt.savefig('Infected_observed_ASVs.png', bbox_inches='tight')
</code></pre>
<p>Error;</p>
<pre><code> $ python3 myscript.rarefaction.py
Traceback (most recent call last):
  File &quot;myscript.rarefaction.py&quot;, line 4, in &lt;module&gt;
    import seaborn as sns
  File &quot;/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/__init__.py&quot;, line 2, in &lt;module&gt;
    from .rcmod import *  # noqa: F401,F403
  File &quot;/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/rcmod.py&quot;, line 7, in &lt;module&gt;
    from . import palettes
  File &quot;/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/palettes.py&quot;, line 9, in &lt;module&gt;
    from .utils import desaturate, get_color_cycle
  File &quot;/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/utils.py&quot;, line 11, in &lt;module&gt;
    import pandas as pd
  File &quot;/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/pandas/__init__.py&quot;, line 11, in &lt;module&gt;
    __import__(dependency)
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 983, in _find_and_load
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 967, in _find_and_load_unlocked
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 668, in _load_unlocked
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 638, in _load_backward_compatible
  File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py&quot;, line 1101, in &lt;module&gt;
  File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/lazy.py&quot;, line 101, in _lazy
  File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py&quot;, line 1098, in &lt;genexpr&gt;
  File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py&quot;, line 114, in resource_exists
  File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py&quot;, line 102, in open_resource
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 983, in _find_and_load
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 967, in _find_and_load_unlocked
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 668, in _load_unlocked
  File &quot;&lt;frozen importlib._bootstrap&gt;&quot;, line 638, in _load_backward_compatible
  File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 2927, in &lt;module&gt;
  File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 2913, in _call_aside
  File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 2952, in _initialize_master_working_set
  File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 956, in subscribe
  File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 2952, in &lt;lambda&gt;
  File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 2515, in activate
  File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 2097, in declare_namespace
  File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 2047, in _handle_ns
  File &quot;/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py&quot;, line 2066, in _rebuild_mod_path
AttributeError: '_NamespacePath' object has no attribute 'sort'

         
     
    
    metadata file ;
    #SampleID       Villages        Region  Treatment       Batch   Season
    PN0086D.1.S1    Kholabari       Mymensingh      T1      batch1  Irrigated
    PN0086D.2.S2    Bogajan Mymensingh      T1      batch1  Irrigated
    PN0086D.3.S3    Bonkua  Mymensingh      T1      batch1  Irrigated
</code></pre>

CC BY-SA 4.0
2020-09-28T14:36:00.760Z
Just ask your cluster admin / manager if it's possible to run notebooks via the queueing system / workload manager, if it is not, you won't get it running anyhow.
Pallie
<python><bash><hpc><jupyter>
How to run Jupyter script on Slurm HPC
# How to run Jupyter script on Slurm HPC

(Asked by: username_0 on 2020-09-24)

Now jupyter installed on the server and I am using below code to plot the rarefaction plot but I am still getting some error. could you please suggest how I can get rid of it?
    #Run it in jupyter lab or notebook


import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import os

#%matplotlib inline 

qza = 'alpha-rarefaction-20000.qzv' # path to rarefaction file from qiime2
os.system("unzip qza")

digest = a[1].split('/')[0].replace('  inflating: ','')
inf = digest + '/data/observed_otus.csv'
otus = pd.read_csv(inf,index_col=0, sep=',')
os.system('rm -r $digest')

cols = [col for col in otus.columns if 'iter' not in col]
mean,data = otus[cols],pd.DataFrame(columns=cols)
depths = [col.split('_')[0] for col in otus.columns if 'depth' in col]
otus = otus.drop(cols,axis=1)
otus.columns = depths
for depth in depths:
    mean['ASV'] = otus[depth].mean(axis=1)
    mean['depth']= depth.split('-')[-1]
    data = pd.concat([data,mean])
# here provide colors for each item that will be plotted
pal={'Irrigated':'brown','Rainfed':'red'} 
fig,ax = plt.subplots(figsize=(2.5,2),dpi=600,tight_layout=True)
sns.set(style='ticks',rc={"lines.linewidth":0.7,"axes.linewidth":0.5})
# use your column name to plot here
sns.lineplot(x='depth',y='ASV',data=data,palette=pal,hue='Season',sort=False,err_style='bars',\
             dashes=True,style='Season',ci=67)
ax.set_xlabel('Sequencing depth',fontsize=8)
ax.set_ylabel('Observed ASVs',fontsize=8)
ax.tick_params(axis='x', labelrotation=90)
ax.tick_params(axis='both',which='major',length=2,pad=0.5,labelsize=6)
handles, labels = ax.get_legend_handles_labels()
ax.legend(handles=handles[1:],labels=labels[1:],fontsize=5,frameon=False,numpoints=4,borderaxespad=0,handletextpad=0.2,loc=2,)
plt.savefig('Infected_observed_ASVs.png', bbox_inches='tight')

Error;
 $ python3 myscript.rarefaction.py
Traceback (most recent call last):
  File "myscript.rarefaction.py", line 4, in 
    import seaborn as sns
  File "/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/__init__.py", line 2, in 
    from .rcmod import *  # noqa: F401,F403
  File "/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/rcmod.py", line 7, in 
    from . import palettes
  File "/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/palettes.py", line 9, in 
    from .utils import desaturate, get_color_cycle
  File "/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/utils.py", line 11, in 
    import pandas as pd
  File "/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/pandas/__init__.py", line 11, in 
    __import__(dependency)
  File "", line 983, in _find_and_load
  File "", line 967, in _find_and_load_unlocked
  File "", line 668, in _load_unlocked
  File "", line 638, in _load_backward_compatible
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py", line 1101, in 
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/lazy.py", line 101, in _lazy
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py", line 1098, in 
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py", line 114, in resource_exists
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py", line 102, in open_resource
  File "", line 983, in _find_and_load
  File "", line 967, in _find_and_load_unlocked
  File "", line 668, in _load_unlocked
  File "", line 638, in _load_backward_compatible
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2927, in 
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2913, in _call_aside
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2952, in _initialize_master_working_set
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 956, in subscribe
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2952, in 
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2515, in activate
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2097, in declare_namespace
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2047, in _handle_ns
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2066, in _rebuild_mod_path
AttributeError: '_NamespacePath' object has no attribute 'sort'

         
     
    
    metadata file ;
    #SampleID       Villages        Region  Treatment       Batch   Season
    PN0086D.1.S1    Kholabari       Mymensingh      T1      batch1  Irrigated
    PN0086D.2.S2    Bogajan Mymensingh      T1      batch1  Irrigated
    PN0086D.3.S3    Bonkua  Mymensingh      T1      batch1  Irrigated

**Tags:** python, bash, hpc, jupyter
> **Comments:**
> **username_1**: Just ask your cluster admin / manager if it's possible to run notebooks via the queueing system / workload manager, if it is not, you won't get it running anyhow.  

---

## 2 answers

### Answer by username_2 on 2020-09-24. Score: 2

This is a bash script (#!/bin/bash) and you try to run python in it.
You should save your python script (e.g. in "myscript.py") and call it within your bash script.
Maybe first try to see if you are able to run it in your terminal and then run it on your server with SLURM.

EDIT1: Also I don't think you need 3 nodes to run such script and this will probably not be even possible here. I think you need some specific modules to parallelize a python script with MPI (see <https://stackoverflow.com/questions/25792357/how-to-distribute-a-python-script-across-multiple-nodes>), but again I think this would be an overkill. Try first with `#SBATCH -c 1` instead.

Edit2: Finally it might be possible to run jupyter-notebook on a node allocated by SLRUM instead of putting this into a python script by using `jupyter-notebook --no-browser --port=${port} --ip=${node}` command in your bash slurm script. Maybe you'll find this doc interesting: <https://docs.ycrc.yale.edu/clusters-at-yale/guides/jupyter/> .

> **Comments:**
> **username_0**: thanks I am running it as python script after loading jupyter and python module but still shows an error? I am going to update the script and the error.  
> **username_2**: `%matplotlib inline` is a jupyter notebook command. Try to comment this line.  
> **username_0**: Again error. it is updated now.  
> **username_2**: Those are again some **jupyter notebook** writings to use bash commands within a notebook. You should try with `subprocess` module instead or build a bash script that does those steps and then call your python script. I think you should think by yourself now and googling until you are really stuck, this is a good training to write real python script ;)  
> **username_3**: Hi @bioinfonext please don't forget to upvote this post and if you feel it answered your question to mark as 'resolved' if resolved.  
> **username_3**: You look to be running Jupyter notebook interactively and if run from PC/laptop/Mac via a jupyter-notebook command it should connect interactively. Pointing this onto your server I'm not certain however. Normally jupyter- notebook is run on one of the big cloud HPC platforms such as AWS or GCS.  
> **username_2**: I think now it is a bit confusing because the post has been edited a lot. But I think @bioinfonext wrote a jupyter notebook to plot his/here data and then wanted to do the same task but by running it on a cluster using SLURM job scheduling manager.  
> **username_0**: should I use below line of code instead "a = !unzip $qza" ;  import subprocess   a = subprocess.Popen(["unzip'', qza])  
> **username_2**: unzip is your command, qza is a python variable, if you look for "give variable to subprocess call" you find: https://stackoverflow.com/questions/4795190/passing-variables-to-a-subprocess-call  
> **username_0**: Thank you so much. Now I am trying to run this on server terminal after loading the modules and with some changes, but still getting error? I have updated the code and error.  

---

### Answer by username_3 on 2020-09-25. Score: 0

I'll make a contribution and keep editing it rather than use 'comments', the error is about
import numpy as np

At the moment this isn't about Jupyter, its about getting Python working on your server. Is that a good assessment? With Jupyter notebook you write a little bit of code, check it, write a bit more. That's how it is designed to operate. The other way in is to use Python in your shebang statement otherwise it will default to sh or bash.
The code you supplied doesn't explicitly use numpy. Pandas does use numpy and np is the standard shorthand, Multiarray is probably something pandas would use. Did you get this code working on your local machine?
I am wondering whether the numpy module on your server is up to date? Did you creat a specific Python environment, use pip or conda? Checking the Python version on your server and whether numpy needs updating is perhaps where I might start.
It is not pretty getting a HPC to run with your stuff and can take real work.

I read your new post, I would simply reinstall Python from Anaconda3 and run the script for now from shell pointing the shebang into Anaconda3 (I can't remember if you need to do this, I can check if needed). This a more professional way of using a HPC than using the resident Python. In good old days of Perl, when everything was nice and simple, perlbrew was essentially the Perl equivalent and no-one would use the system Perl.
The reason why I'm saying  this is I get this feeling it will be an endless cycle of import errors. Anaconda3 will give you control of your Python system without admin privelidge.
Just to explain if you are not calling np explicitly (and looking at your code I don't think you are) then explicitly stating import numpy isn't needed. The fact you wrote it and it then jumped onto a different error is a bit worrying. I would just use your own version of Python. In Jupyter I would guess, but don't know they use their own Python ... but from a data science point of view Jupyter is the front end to AWS and GCS because those guys want some control of what you are doing, not running riot. In DNN jupyter is big. If you want an absolute diagnosis then Stackoverflow might help, but I think you'll just another bug and another etc... One thing is that pandas changes alot and the version you are using might be ahead of the curve, thats my guess
Final point is if you are just plotting the ggplot2 is really good, but that is R. seaborne is really good, matplotlib but hmmmm, but nothing is as clear or explicit as ggplot2. Python is good at lots of things, pandas is fabulous, but plotting isn't on par with ggplot2

What is 100% clear is that you are loading the HPC version of Python and this is out of date (3.4) should be 3.7 (or even 3.8?). The thing I don't understand is why is it loads Python 2 modules?? This is a complete mystery to me, I thought Python 2 and Python 3 were incompatible, this is suggesting backwards compatibility or else the reason it is falling over. I have no idea, but I wouldn't use the resident Python.

> **Comments:**
> **username_0**: script and error updated after importing numpy.  
> **username_3**: @bioinfonext commented below  
> **username_0**: Yes, thanks, this is all very confusing, R is much easier to adopt for potting.But I could not able to find any good R code for rarefaction plot that can plot it based on metadata column.  
> **username_3**: @bioinfonext answered above  
> **username_3**: @bioinfonext answered above  

---

