Difference between revisions of "Running Jupyter Notebook"

(On VDI)
 
(18 intermediate revisions by 4 users not shown)
Line 1: Line 1:
=On VDI=
 
  
Currently, the easiest way to run IPython Notebook is on NCI's [[VDI|Virtual Desktop Infrastructure (VDI)]]. For a guide to set up and use VDI, click [https://opus.nci.org.au/display/Help/VDI+User+Guide here].
 
  
Within VDI, open a Linux terminal ('''Applications menu''' -> '''System Tools''' -> '''Terminal''').
 
Inside the Terminal, load the conda environment
 
<syntaxhighlight lang=bash>
 
module use /g/data/hh5/public/modules
 
module load conda
 
  
</syntaxhighlight>
 
You should then be able to start the notebook with
 
<syntaxhighlight lang=bash>
 
jupyter notebook
 
</syntaxhighlight>
 
  
==Connect your local browser==
 
  
You can connect the browser on your computer to the jupyter notebook running on vdi once it's started.
+
= On ARE =
 +
 
 +
Currently, the easiest way to run a Jupyter Notebook is on NCI's [https://opus.nci.org.au/display/Help/ARE+User+Guide Australian Research Environment]&nbsp;a Open-On-Demand service.&nbsp; For a guide to set up and use the OOD, click [https://opus.nci.org.au/display/Help/3.1+Starting+JupyterLab+App here]. You can access all the gadi filesystems from the ARE environment and select different queues and cpus/memory options to suit your workflow. You can also select our hh5 conda environments.
 +
 
 +
Please follow the instructions to request the best sized cluster for your work.
 +
 
 +
&nbsp;
  
You still need to connect to VDI in the usual way. Once the session is started, open a terminal. You need the name of the computer that is running your VDI instance. It is of the form '''vdi-nXX''' where XX is a number. It is possible that this is already displayed in your command line prompt there, otherwise you can run the command:
+
= On Gadi =
  
<syntaxhighlight lang=bash>
+
<span style="color:#c0392b;">NB as ARE has been released there is no advantage&nbsp;in using&nbsp;gadi_jupyter script. So we have stopped supporting it!</span>
hostname
 
</syntaxhighlight>
 
  
and it will give you the hostname. Remember it.
+
You can also run a notebook from Gadi's compute nodes using the [https://github.com/coecms/nci_scripts/blob/master/gadi_jupyter gadi_jupyter] script.
  
Next, you load the conda module as above, but tell jupyter to not start a browser:
+
=== Pre-requisite ===
  
<syntaxhighlight lang=bash>
+
For this method to work, you must become part of the '''hh5''' project at NCI using [https://my.nci.org.au/mancini/project/hh5 my.nci.org.au].
module use /g/data/hh5/public/modules
 
module load conda
 
jupyter notebook --no-browser
 
</syntaxhighlight>
 
  
You wait for this line:
+
This project will give you access to [[Conda|conda environments]] for Python managed by the CMS team.
  
<syntaxhighlight>
+
=== Setting Up SSH ===
http://localhost:8888/?token=...
 
</syntaxhighlight>
 
  
The 4 digits after the colon (often just 8888, but not always) are the port. We use this number and the hostname to create a new ssh connection to VDI. So in a new terminal session on your computer, run the command:
+
You will need to set up SSH keys to use gadi_jupyter
  
<syntaxhighlight lang=bash>
+
Create the file ~/.ssh/config with something like the following lines in:
ssh userXX@vdi-nXX.nci.org.au -L 8888:localhost:8888
+
<syntaxhighlight lang="text">
 +
Host *.nci.org.au
 +
User abc123 # Your NCI username here
 
</syntaxhighlight>
 
</syntaxhighlight>
  
Of course, you need to replace '''userXX''' with your NCI username, '''vdi-nXX''' with the hostname, and ''both'' instances of '''8888''' with the actual port number.
+
Create a SSH key by running 'ssh-keygen'. It will ask for a 'passphrase', make sure you enter one, it doesn't need to be the same as your NCI password. Use the default key file name.
  
Once you've done that, you can copy the full line that you got from jupyter notebook (the one '''<nowiki>http://localhost:8888/?token=...</nowiki>''' (including the full token) into your browser of choice.
+
Copy your SSH key to gadi. Some computers have a command that can do this automatically - 'ssh-copy-id gadi.nci.org.au'. If not, you need to add the contents of the file '~/.ssh/id_rsa.pub' on your local computer to the end of the file '~/.ssh/authorized_keys' on Gadi.
  
=On Gadi=
+
Enable your SSH key by running the command 'ssh-add', it will ask for the SSH passphrase you entered when you made the key (which may not be your NCI password). If it says 'Could not open a connection to your SSH agent', or '‘could not connect to authentication agent', run 'ssh-agent bash' and then 'ssh-add'.
  
You can also run a notebook from Gadi's compute nodes using the [https://github.com/coecms/nci_scripts/blob/master/gadi_jupyter gadi_jupyter] script.
+
You can now run './gadi_jupyter'
  
==Notes==
+
=== Notes ===
  
* You need to download this script to your own computer and execute there. The script handles the connection.
+
*You need to download this script to your own computer and execute there. The script handles the connection.  
* This job requires an implementation of bash. This is default on MacOS and Linux, but not Windows.
+
*This job requires an implementation of bash. This is default on MacOS and Linux, but not Windows.  
* This will cost SU depending on the queue and the resources requested (default: 1 cores, express queue)
+
*This will cost SU depending on the queue and the resources requested (default: 1 cores, express queue)  
* You also won't have access to the internet to, for example, download data.
+
*You also won't have access to the internet to, for example, download data.  
  
==Running==
+
=== Running ===
  
 
You can download the script from the link above directly, or by running
 
You can download the script from the link above directly, or by running
 
+
<syntaxhighlight lang="bash">
<syntaxhighlight lang='bash'>
 
 
git clone https://github.com/coecms/nci_scripts.git
 
git clone https://github.com/coecms/nci_scripts.git
 
</syntaxhighlight>
 
</syntaxhighlight>
Line 73: Line 59:
  
 
There are several options that you can use when running the script:
 
There are several options that you can use when running the script:
 
+
<syntaxhighlight lang="unknown">
<syntaxhighlight>
 
 
General Options:
 
General Options:
 
     -h:        Print help
 
     -h:        Print help
Line 89: Line 74:
 
     -P PROJ:    Submit job under project PROJ
 
     -P PROJ:    Submit job under project PROJ
 
</syntaxhighlight>
 
</syntaxhighlight>
 +
 +
=== Windows Desktops ===
 +
 +
To run gadi_jupyter on Windows you'll need Bash and some form of SSH available. This could be from
 +
 +
* git bash - [https://gitforwindows.org/ https://gitforwindows.org/]
 +
* cygwin terminal - [https://www.cygwin.com/ https://www.cygwin.com/]
 +
 +
Install one of these, then within the terminal follow the 'Setting up SSH' instructions before running 'gadi_jupyter'. (The terminal should start in your 'My Documents' directory if you need to find the downloaded script)
  
 
[[Category:Python]]
 
[[Category:Python]]

Latest revision as of 20:05, 22 February 2023



On ARE

Currently, the easiest way to run a Jupyter Notebook is on NCI's Australian Research Environment a Open-On-Demand service.  For a guide to set up and use the OOD, click here. You can access all the gadi filesystems from the ARE environment and select different queues and cpus/memory options to suit your workflow. You can also select our hh5 conda environments.

Please follow the instructions to request the best sized cluster for your work.

 

On Gadi

NB as ARE has been released there is no advantage in using gadi_jupyter script. So we have stopped supporting it!

You can also run a notebook from Gadi's compute nodes using the gadi_jupyter script.

Pre-requisite

For this method to work, you must become part of the hh5 project at NCI using my.nci.org.au.

This project will give you access to conda environments for Python managed by the CMS team.

Setting Up SSH

You will need to set up SSH keys to use gadi_jupyter

Create the file ~/.ssh/config with something like the following lines in:

Host *.nci.org.au
User abc123 # Your NCI username here

Create a SSH key by running 'ssh-keygen'. It will ask for a 'passphrase', make sure you enter one, it doesn't need to be the same as your NCI password. Use the default key file name.

Copy your SSH key to gadi. Some computers have a command that can do this automatically - 'ssh-copy-id gadi.nci.org.au'. If not, you need to add the contents of the file '~/.ssh/id_rsa.pub' on your local computer to the end of the file '~/.ssh/authorized_keys' on Gadi.

Enable your SSH key by running the command 'ssh-add', it will ask for the SSH passphrase you entered when you made the key (which may not be your NCI password). If it says 'Could not open a connection to your SSH agent', or '‘could not connect to authentication agent', run 'ssh-agent bash' and then 'ssh-add'.

You can now run './gadi_jupyter'

Notes

  • You need to download this script to your own computer and execute there. The script handles the connection.
  • This job requires an implementation of bash. This is default on MacOS and Linux, but not Windows.
  • This will cost SU depending on the queue and the resources requested (default: 1 cores, express queue)
  • You also won't have access to the internet to, for example, download data.

Running

You can download the script from the link above directly, or by running

git clone https://github.com/coecms/nci_scripts.git

which will download the script into a directory called nci_scripts.

There are several options that you can use when running the script:

General Options:
    -h:         Print help
    -l:         NCI username
    -L:         NCI login node (default 'gadi.nci.org.au')
    -e:         Conda environment

Queue Options:
    -q QUEUE:   Queue name
    -n NCPU:    Use NCPU cpus
    -m MEM:     Memory allocation (default 4*NCPU GB)
    -t TIME:    Walltime limit (default 1 hour)
    -J JOBFS:   Jobfs allocation (default 100 GB)
    -P PROJ:    Submit job under project PROJ

Windows Desktops

To run gadi_jupyter on Windows you'll need Bash and some form of SSH available. This could be from

* git bash - https://gitforwindows.org/
* cygwin terminal - https://www.cygwin.com/

Install one of these, then within the terminal follow the 'Setting up SSH' instructions before running 'gadi_jupyter'. (The terminal should start in your 'My Documents' directory if you need to find the downloaded script)