Difference between revisions of "Getting started on Windows"

(VSCode & Power Shell SSH)
(NCI Virtual Desktop (VDI))
 
(15 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
[[Category:Setting up]]
 
Windows computers work a little different to Linux and Mac, which both share a common UNIX ancestry. While you can achieve similar results using Power Shell in Windows to Bash in Linux/Mac, the way it works is completely different so you have to learn a new way of doing things when connecting to facilities like NCI. For that reason we recommend setting up a Linux-like environment on your Windows computer, so that everything works in a similar way.
 
Windows computers work a little different to Linux and Mac, which both share a common UNIX ancestry. While you can achieve similar results using Power Shell in Windows to Bash in Linux/Mac, the way it works is completely different so you have to learn a new way of doing things when connecting to facilities like NCI. For that reason we recommend setting up a Linux-like environment on your Windows computer, so that everything works in a similar way.
  
Line 8: Line 9:
 
For more information on how to use the terminal and Bash, check out the [http://swcarpentry.github.io/shell-novice/01-intro/index.html Software Carpentry Bash course]
 
For more information on how to use the terminal and Bash, check out the [http://swcarpentry.github.io/shell-novice/01-intro/index.html Software Carpentry Bash course]
  
= NCI Virtual Desktop (VDI) =
+
'''[https://youtu.be/gnldlGztqpU Video Introduction]'''
  
NCI's virtual desktops (VDI) are an easy way to connect to NCI facilities from desktop computers. VDI provides a Linux desktop in a window, running on NCI's cloud
+
== NCI Open OnDemand Service(OOD) ==
 +
NCI's Open OnDemand Service (OOD) is an easy way to connect to NCI facilities from desktop computers. OOD provides a Linux desktop in a browser and access to JupyterLab, running on NCI's cloud
  
Check NCI's VDI documentation for the current installation and setup instructions https://vdi.nci.org.au/help
+
Check NCI's OOD documentation for the current installation and setup instructions [https://opus.nci.org.au/display/OOD/Open+OnDemand+%28OOD%29+Service
  
 
'''Advantages:'''
 
'''Advantages:'''
 
* Access to NCI's /g/data filesystem
 
* Access to NCI's /g/data filesystem
* [[Conda]] and Matlab (see NCI documentation) available
+
* Conda and Matlab (see NCI documentation) available
* No compute time/SU charge when running analyses on VDI
 
 
* Doesn't require a powerful home computer
 
* Doesn't require a powerful home computer
 +
* When viewing graphical windows, connection can be faster than via X11 on slow networks
  
 
'''Disadvantages:'''
 
'''Disadvantages:'''
* VDI servers are shared between multiple users, can run out of memory
+
* Can not access /scratch
 
* Requires network connection - can't use VDI on a plane etc.
 
* Requires network connection - can't use VDI on a plane etc.
  
Line 31: Line 33:
  
 
Once you have WSL and a Linux environment set up you can use Linux commands to install further programs, e.g. for Ubuntu
 
Once you have WSL and a Linux environment set up you can use Linux commands to install further programs, e.g. for Ubuntu
{code}
+
<syntaxhighlight>
 
sudo apt install vim
 
sudo apt install vim
{code}
+
</syntaxhighlight>
 
will install the vim text editor.
 
will install the vim text editor.
 +
 +
With WSL installed you can then follow the [[Getting started on Linux]] instructions to set up SSH etc.
  
 
'''Advantages:'''
 
'''Advantages:'''
Line 62: Line 66:
  
 
There are a few different X11 server programs available for Windows, including:
 
There are a few different X11 server programs available for Windows, including:
* https://sourceforge.net/projects/vcxsrv/
+
* https://sourceforge.net/projects/vcxsrv/
 +
* https://github.com/microsoft/wslg
 +
 
 +
If you encounter problems with WSL 2 and VcXsrv, check that:
 +
* On your machine the environment variable DISPLAY is set using <syntaxhighlight>export DISPLAY=$(ip route list default | awk '{print $3}'):0.0</syntaxhighlight>
 +
* Incomming connections to VcXsrv are allowed through your firewall. In the start menu, search for 'Windows Defender Firewall with Advanced Security'. In the list 'Inbound Rules' find all entries for 'VcXsrv windows xserver' and make sure they have a green check. If they don't double click the entry and change the 'Action' to 'Allow the connection'.
  
 
== VSCode & Power Shell SSH ==
 
== VSCode & Power Shell SSH ==
Line 72: Line 81:
 
ssh-keygen -t ecdsa -b 521 -f .ssh/id_ecdsa
 
ssh-keygen -t ecdsa -b 521 -f .ssh/id_ecdsa
 
</syntaxhighlight>
 
</syntaxhighlight>
Power shell doesn't have the ssh-copy-id script, so you'll need to add the contents of .ssh/id_ecdsa.pub to the ~/.ssh/authorized_keys file on Gadi manually.
+
Power shell doesn't have the ssh-copy-id script, so you'll need to add the contents of the new public key file .ssh/id_ecdsa.pub to the ~/.ssh/authorized_keys file on Gadi manually.
  
 
The Power Shell SSH config file is stored in your user directory, e.g. C:\Users\Scott\.ssh\config. Settings work just like Linux, e.g.:
 
The Power Shell SSH config file is stored in your user directory, e.g. C:\Users\Scott\.ssh\config. Settings work just like Linux, e.g.:
Line 86: Line 95:
 
https://code.visualstudio.com/
 
https://code.visualstudio.com/
  
= Anaconda Python / Rstudio =
+
= Software =
 +
== NCI Jupyter ==
 +
 
 +
[https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.
 +
 
 +
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.
 +
 
 +
You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'.
 +
 
 +
On Windows, the jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).
 +
 
 +
== Anaconda Python / Rstudio ==
  
 
We recommend using [https://www.anaconda.com/products/individual Anaconda] if you're working with Python on your own computer. This makes it easy to install packages like numpy and netCDF4, which depend on binary libraries. Anaconda uses the same system we use for the [[Conda]] environments at NCI. Anaconda also supports Rstudio if you work with R.
 
We recommend using [https://www.anaconda.com/products/individual Anaconda] if you're working with Python on your own computer. This makes it easy to install packages like numpy and netCDF4, which depend on binary libraries. Anaconda uses the same system we use for the [[Conda]] environments at NCI. Anaconda also supports Rstudio if you work with R.
Line 92: Line 112:
 
You can install the Windows native version, or the Linux version in WSL. Normally more packages are available for Linux, but commonly used ones will be available for either. We recommend enabling the 'conda-forge' package repository, you will find a wider variety of packages there and they are generally well supported.
 
You can install the Windows native version, or the Linux version in WSL. Normally more packages are available for Linux, but commonly used ones will be available for either. We recommend enabling the 'conda-forge' package repository, you will find a wider variety of packages there and they are generally well supported.
  
= Matlab =
+
== Matlab ==
  
Check with your local IT for information on how to use Matlab, they will have information on how to connect to their Matlab license server.
+
Check with your local IT for information on how to use Matlab, they will have information on how to connect to their Matlab license server from your own computer.
  
You can also use Matlab on NCI's systems via VDI, however you need to use your university license.
+
You can also use Matlab on NCI's systems via VDI, however you need to use your university license. See the [https://opus.nci.org.au/display/Help/3.+VDI+software VDI docs] or contact help@nci.org.au for more information.
  
 
= Other Alternatives =
 
= Other Alternatives =
Line 111: Line 131:
  
 
https://www.cygwin.com/
 
https://www.cygwin.com/
 +
 +
Git for windows includes a Bash terminal based on Cygwin as well
 +
 +
https://gitforwindows.org/
 +
 +
== File transfers ==
 +
It is sometimes useful to transfer files between NCI servers and your local computer. You can do so via the command line with programs like rsync which is readily available. If you prefer a graphical interface, NCI advises you can install a program like [https://cyberduck.io/?l=en Cyberduck]. You can then use a SFTP connection. To access files on Gadi, it is preferable to use the data mover nodes: gadi-dm.nci.org.au for the server's name.

Latest revision as of 17:48, 16 August 2021

Windows computers work a little different to Linux and Mac, which both share a common UNIX ancestry. While you can achieve similar results using Power Shell in Windows to Bash in Linux/Mac, the way it works is completely different so you have to learn a new way of doing things when connecting to facilities like NCI. For that reason we recommend setting up a Linux-like environment on your Windows computer, so that everything works in a similar way.

You should be able to:

  • Connect to NCI to access models and data
  • Develop, test and run Python, Fortran etc. programs locally
  • Use a common interface on your local computer and NCI

For more information on how to use the terminal and Bash, check out the Software Carpentry Bash course

Video Introduction

NCI Open OnDemand Service(OOD)

NCI's Open OnDemand Service (OOD) is an easy way to connect to NCI facilities from desktop computers. OOD provides a Linux desktop in a browser and access to JupyterLab, running on NCI's cloud

Check NCI's OOD documentation for the current installation and setup instructions [https://opus.nci.org.au/display/OOD/Open+OnDemand+%28OOD%29+Service

Advantages:

  • Access to NCI's /g/data filesystem
  • Conda and Matlab (see NCI documentation) available
  • Doesn't require a powerful home computer
  • When viewing graphical windows, connection can be faster than via X11 on slow networks

Disadvantages:

  • Can not access /scratch
  • Requires network connection - can't use VDI on a plane etc.

Windows Subsystem for Linux (WSL)

Windows subsystem for Linux (WSL) is a way to have a Linux environment on your Windows computer, without having to install and switch between different operating systems.

For up-to-date WSL installation instructions, see Microsoft's documentation at https://docs.microsoft.com/en-us/windows/wsl/install-win10. There are a few different Linux variants you can install, a good starting point is Ubuntu if you can't decide.

Once you have WSL and a Linux environment set up you can use Linux commands to install further programs, e.g. for Ubuntu

sudo apt install vim

will install the vim text editor.

With WSL installed you can then follow the Getting started on Linux instructions to set up SSH etc.

Advantages:

  • Runs on your own computer
  • Install whatever programs you need

Disadvantages:

  • Large size (~2GB for base install, another GB or so for Anaconda)
  • Remote X11 windows can be slow if you have a bad connection

Windows Terminal

Windows Terminal is a nicer command line interface for Windows than the default Power Shell program, with more customisation options. It integrates with WSL, letting you use it to start Bash terminal sessions

Windows terminal can be installed through the Windows app store, further details at https://docs.microsoft.com/en-us/windows/terminal/

X11

In order to show remote windows from Gadi on your own screen you'll need an X11 server program running on your computer.

If X11 isn't running, you will get errors when starting programs that bring up a new window, e.g:

saw562@gadi-login-04 ~ $ xclock
Error: Can't open display:

There are a few different X11 server programs available for Windows, including:

If you encounter problems with WSL 2 and VcXsrv, check that:

  • On your machine the environment variable DISPLAY is set using
    export DISPLAY=$(ip route list default | awk '{print $3}'):0.0
  • Incomming connections to VcXsrv are allowed through your firewall. In the start menu, search for 'Windows Defender Firewall with Advanced Security'. In the list 'Inbound Rules' find all entries for 'VcXsrv windows xserver' and make sure they have a green check. If they don't double click the entry and change the 'Action' to 'Allow the connection'.

VSCode & Power Shell SSH

VS Code is a nice code editor that can run on Windows, Mac and Linux. You can edit code locally, or use plugins to remotely connect to servers over SSH. It also integrates with WSL allowing you to edit code in the Linux environment, and with Anaconda, letting you run Python programs in different environments.

Windows includes its own version of SSH that you can use directly from Power Shell. Visual Studio Code can use this to connect to Gadi from your Windows computer. You will need some special options when creating a SSH key with Power Shell SSH:

ssh-keygen -t ecdsa -b 521 -f .ssh/id_ecdsa

Power shell doesn't have the ssh-copy-id script, so you'll need to add the contents of the new public key file .ssh/id_ecdsa.pub to the ~/.ssh/authorized_keys file on Gadi manually.

The Power Shell SSH config file is stored in your user directory, e.g. C:\Users\Scott\.ssh\config. Settings work just like Linux, e.g.:

Host gadi
  HostName gadi.nci.org.au
  User saw562 # Replace with your own username
  ForwardAgent yes

You can also use the Power Shell SSH to connect to Gadi, bypassing WSL, but that may have problems running X11 windows

https://code.visualstudio.com/

Software

NCI Jupyter

Jupyter provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.

The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.

You can also run Jupyter directly on VDI, by loading the Conda environment and running 'jupyter lab'.

On Windows, the jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).

Anaconda Python / Rstudio

We recommend using Anaconda if you're working with Python on your own computer. This makes it easy to install packages like numpy and netCDF4, which depend on binary libraries. Anaconda uses the same system we use for the Conda environments at NCI. Anaconda also supports Rstudio if you work with R.

You can install the Windows native version, or the Linux version in WSL. Normally more packages are available for Linux, but commonly used ones will be available for either. We recommend enabling the 'conda-forge' package repository, you will find a wider variety of packages there and they are generally well supported.

Matlab

Check with your local IT for information on how to use Matlab, they will have information on how to connect to their Matlab license server from your own computer.

You can also use Matlab on NCI's systems via VDI, however you need to use your university license. See the VDI docs or contact help@nci.org.au for more information.

Other Alternatives

Putty

Putty is an alternative SSH implementation for Windows

https://www.putty.org/

Cygwin

Cygwin works similarly to WSL, it lets you run Linux programs on a Windows computer

https://www.cygwin.com/

Git for windows includes a Bash terminal based on Cygwin as well

https://gitforwindows.org/

File transfers

It is sometimes useful to transfer files between NCI servers and your local computer. You can do so via the command line with programs like rsync which is readily available. If you prefer a graphical interface, NCI advises you can install a program like Cyberduck. You can then use a SFTP connection. To access files on Gadi, it is preferable to use the data mover nodes: gadi-dm.nci.org.au for the server's name.