Difference between revisions of "Getting started on Windows"
(→VSCode & Power Shell SSH) |
(→NCI Virtual Desktop (VDI)) |
||
(15 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
+ | [[Category:Setting up]] | ||
Windows computers work a little different to Linux and Mac, which both share a common UNIX ancestry. While you can achieve similar results using Power Shell in Windows to Bash in Linux/Mac, the way it works is completely different so you have to learn a new way of doing things when connecting to facilities like NCI. For that reason we recommend setting up a Linux-like environment on your Windows computer, so that everything works in a similar way. | Windows computers work a little different to Linux and Mac, which both share a common UNIX ancestry. While you can achieve similar results using Power Shell in Windows to Bash in Linux/Mac, the way it works is completely different so you have to learn a new way of doing things when connecting to facilities like NCI. For that reason we recommend setting up a Linux-like environment on your Windows computer, so that everything works in a similar way. | ||
Line 8: | Line 9: | ||
For more information on how to use the terminal and Bash, check out the [http://swcarpentry.github.io/shell-novice/01-intro/index.html Software Carpentry Bash course] | For more information on how to use the terminal and Bash, check out the [http://swcarpentry.github.io/shell-novice/01-intro/index.html Software Carpentry Bash course] | ||
− | + | '''[https://youtu.be/gnldlGztqpU Video Introduction]''' | |
− | NCI's | + | == NCI Open OnDemand Service(OOD) == |
+ | NCI's Open OnDemand Service (OOD) is an easy way to connect to NCI facilities from desktop computers. OOD provides a Linux desktop in a browser and access to JupyterLab, running on NCI's cloud | ||
− | Check NCI's | + | Check NCI's OOD documentation for the current installation and setup instructions [https://opus.nci.org.au/display/OOD/Open+OnDemand+%28OOD%29+Service |
'''Advantages:''' | '''Advantages:''' | ||
* Access to NCI's /g/data filesystem | * Access to NCI's /g/data filesystem | ||
− | * | + | * Conda and Matlab (see NCI documentation) available |
− | |||
* Doesn't require a powerful home computer | * Doesn't require a powerful home computer | ||
+ | * When viewing graphical windows, connection can be faster than via X11 on slow networks | ||
'''Disadvantages:''' | '''Disadvantages:''' | ||
− | * | + | * Can not access /scratch |
* Requires network connection - can't use VDI on a plane etc. | * Requires network connection - can't use VDI on a plane etc. | ||
Line 31: | Line 33: | ||
Once you have WSL and a Linux environment set up you can use Linux commands to install further programs, e.g. for Ubuntu | Once you have WSL and a Linux environment set up you can use Linux commands to install further programs, e.g. for Ubuntu | ||
− | + | <syntaxhighlight> | |
sudo apt install vim | sudo apt install vim | ||
− | + | </syntaxhighlight> | |
will install the vim text editor. | will install the vim text editor. | ||
+ | |||
+ | With WSL installed you can then follow the [[Getting started on Linux]] instructions to set up SSH etc. | ||
'''Advantages:''' | '''Advantages:''' | ||
Line 62: | Line 66: | ||
There are a few different X11 server programs available for Windows, including: | There are a few different X11 server programs available for Windows, including: | ||
− | + | * https://sourceforge.net/projects/vcxsrv/ | |
+ | * https://github.com/microsoft/wslg | ||
+ | |||
+ | If you encounter problems with WSL 2 and VcXsrv, check that: | ||
+ | * On your machine the environment variable DISPLAY is set using <syntaxhighlight>export DISPLAY=$(ip route list default | awk '{print $3}'):0.0</syntaxhighlight> | ||
+ | * Incomming connections to VcXsrv are allowed through your firewall. In the start menu, search for 'Windows Defender Firewall with Advanced Security'. In the list 'Inbound Rules' find all entries for 'VcXsrv windows xserver' and make sure they have a green check. If they don't double click the entry and change the 'Action' to 'Allow the connection'. | ||
== VSCode & Power Shell SSH == | == VSCode & Power Shell SSH == | ||
Line 72: | Line 81: | ||
ssh-keygen -t ecdsa -b 521 -f .ssh/id_ecdsa | ssh-keygen -t ecdsa -b 521 -f .ssh/id_ecdsa | ||
</syntaxhighlight> | </syntaxhighlight> | ||
− | Power shell doesn't have the ssh-copy-id script, so you'll need to add the contents of .ssh/id_ecdsa.pub to the ~/.ssh/authorized_keys file on Gadi manually. | + | Power shell doesn't have the ssh-copy-id script, so you'll need to add the contents of the new public key file .ssh/id_ecdsa.pub to the ~/.ssh/authorized_keys file on Gadi manually. |
The Power Shell SSH config file is stored in your user directory, e.g. C:\Users\Scott\.ssh\config. Settings work just like Linux, e.g.: | The Power Shell SSH config file is stored in your user directory, e.g. C:\Users\Scott\.ssh\config. Settings work just like Linux, e.g.: | ||
Line 86: | Line 95: | ||
https://code.visualstudio.com/ | https://code.visualstudio.com/ | ||
− | = Anaconda Python / Rstudio = | + | = Software = |
+ | == NCI Jupyter == | ||
+ | |||
+ | [https://jupyter.org/ Jupyter] provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface. | ||
+ | |||
+ | The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage. | ||
+ | |||
+ | You can also run Jupyter directly on VDI, by loading the [[Conda]] environment and running 'jupyter lab'. | ||
+ | |||
+ | On Windows, the jupyter scripts must be run through a Bash terminal (From WSL or Cygwin). | ||
+ | |||
+ | == Anaconda Python / Rstudio == | ||
We recommend using [https://www.anaconda.com/products/individual Anaconda] if you're working with Python on your own computer. This makes it easy to install packages like numpy and netCDF4, which depend on binary libraries. Anaconda uses the same system we use for the [[Conda]] environments at NCI. Anaconda also supports Rstudio if you work with R. | We recommend using [https://www.anaconda.com/products/individual Anaconda] if you're working with Python on your own computer. This makes it easy to install packages like numpy and netCDF4, which depend on binary libraries. Anaconda uses the same system we use for the [[Conda]] environments at NCI. Anaconda also supports Rstudio if you work with R. | ||
Line 92: | Line 112: | ||
You can install the Windows native version, or the Linux version in WSL. Normally more packages are available for Linux, but commonly used ones will be available for either. We recommend enabling the 'conda-forge' package repository, you will find a wider variety of packages there and they are generally well supported. | You can install the Windows native version, or the Linux version in WSL. Normally more packages are available for Linux, but commonly used ones will be available for either. We recommend enabling the 'conda-forge' package repository, you will find a wider variety of packages there and they are generally well supported. | ||
− | = Matlab = | + | == Matlab == |
− | Check with your local IT for information on how to use Matlab, they will have information on how to connect to their Matlab license server. | + | Check with your local IT for information on how to use Matlab, they will have information on how to connect to their Matlab license server from your own computer. |
− | You can also use Matlab on NCI's systems via VDI, however you need to use your university license. | + | You can also use Matlab on NCI's systems via VDI, however you need to use your university license. See the [https://opus.nci.org.au/display/Help/3.+VDI+software VDI docs] or contact help@nci.org.au for more information. |
= Other Alternatives = | = Other Alternatives = | ||
Line 111: | Line 131: | ||
https://www.cygwin.com/ | https://www.cygwin.com/ | ||
+ | |||
+ | Git for windows includes a Bash terminal based on Cygwin as well | ||
+ | |||
+ | https://gitforwindows.org/ | ||
+ | |||
+ | == File transfers == | ||
+ | It is sometimes useful to transfer files between NCI servers and your local computer. You can do so via the command line with programs like rsync which is readily available. If you prefer a graphical interface, NCI advises you can install a program like [https://cyberduck.io/?l=en Cyberduck]. You can then use a SFTP connection. To access files on Gadi, it is preferable to use the data mover nodes: gadi-dm.nci.org.au for the server's name. |
Latest revision as of 17:48, 16 August 2021
Windows computers work a little different to Linux and Mac, which both share a common UNIX ancestry. While you can achieve similar results using Power Shell in Windows to Bash in Linux/Mac, the way it works is completely different so you have to learn a new way of doing things when connecting to facilities like NCI. For that reason we recommend setting up a Linux-like environment on your Windows computer, so that everything works in a similar way.
You should be able to:
- Connect to NCI to access models and data
- Develop, test and run Python, Fortran etc. programs locally
- Use a common interface on your local computer and NCI
For more information on how to use the terminal and Bash, check out the Software Carpentry Bash course
Contents
NCI Open OnDemand Service(OOD)
NCI's Open OnDemand Service (OOD) is an easy way to connect to NCI facilities from desktop computers. OOD provides a Linux desktop in a browser and access to JupyterLab, running on NCI's cloud
Check NCI's OOD documentation for the current installation and setup instructions [https://opus.nci.org.au/display/OOD/Open+OnDemand+%28OOD%29+Service
Advantages:
- Access to NCI's /g/data filesystem
- Conda and Matlab (see NCI documentation) available
- Doesn't require a powerful home computer
- When viewing graphical windows, connection can be faster than via X11 on slow networks
Disadvantages:
- Can not access /scratch
- Requires network connection - can't use VDI on a plane etc.
Windows Subsystem for Linux (WSL)
Windows subsystem for Linux (WSL) is a way to have a Linux environment on your Windows computer, without having to install and switch between different operating systems.
For up-to-date WSL installation instructions, see Microsoft's documentation at https://docs.microsoft.com/en-us/windows/wsl/install-win10. There are a few different Linux variants you can install, a good starting point is Ubuntu if you can't decide.
Once you have WSL and a Linux environment set up you can use Linux commands to install further programs, e.g. for Ubuntu
sudo apt install vim
will install the vim text editor.
With WSL installed you can then follow the Getting started on Linux instructions to set up SSH etc.
Advantages:
- Runs on your own computer
- Install whatever programs you need
Disadvantages:
- Large size (~2GB for base install, another GB or so for Anaconda)
- Remote X11 windows can be slow if you have a bad connection
Windows Terminal
Windows Terminal is a nicer command line interface for Windows than the default Power Shell program, with more customisation options. It integrates with WSL, letting you use it to start Bash terminal sessions
Windows terminal can be installed through the Windows app store, further details at https://docs.microsoft.com/en-us/windows/terminal/
X11
In order to show remote windows from Gadi on your own screen you'll need an X11 server program running on your computer.
If X11 isn't running, you will get errors when starting programs that bring up a new window, e.g:
saw562@gadi-login-04 ~ $ xclock
Error: Can't open display:
There are a few different X11 server programs available for Windows, including:
If you encounter problems with WSL 2 and VcXsrv, check that:
- On your machine the environment variable DISPLAY is set using
export DISPLAY=$(ip route list default | awk '{print $3}'):0.0
- Incomming connections to VcXsrv are allowed through your firewall. In the start menu, search for 'Windows Defender Firewall with Advanced Security'. In the list 'Inbound Rules' find all entries for 'VcXsrv windows xserver' and make sure they have a green check. If they don't double click the entry and change the 'Action' to 'Allow the connection'.
VSCode & Power Shell SSH
VS Code is a nice code editor that can run on Windows, Mac and Linux. You can edit code locally, or use plugins to remotely connect to servers over SSH. It also integrates with WSL allowing you to edit code in the Linux environment, and with Anaconda, letting you run Python programs in different environments.
Windows includes its own version of SSH that you can use directly from Power Shell. Visual Studio Code can use this to connect to Gadi from your Windows computer. You will need some special options when creating a SSH key with Power Shell SSH:
ssh-keygen -t ecdsa -b 521 -f .ssh/id_ecdsa
Power shell doesn't have the ssh-copy-id script, so you'll need to add the contents of the new public key file .ssh/id_ecdsa.pub to the ~/.ssh/authorized_keys file on Gadi manually.
The Power Shell SSH config file is stored in your user directory, e.g. C:\Users\Scott\.ssh\config. Settings work just like Linux, e.g.:
Host gadi
HostName gadi.nci.org.au
User saw562 # Replace with your own username
ForwardAgent yes
You can also use the Power Shell SSH to connect to Gadi, bypassing WSL, but that may have problems running X11 windows
https://code.visualstudio.com/
Software
NCI Jupyter
Jupyter provides a 'notebook' interface for working with Python - you can combine Python code, text, latex equations and plots in a web interface.
The Centre has developed scripts that can run Jupyter on NCI facilities - VDI or Gadi - and display the notebook interface on your local computer. These scripts are available at https://github.com/coecms/nci_scripts, see the instructions there for usage.
You can also run Jupyter directly on VDI, by loading the Conda environment and running 'jupyter lab'.
On Windows, the jupyter scripts must be run through a Bash terminal (From WSL or Cygwin).
Anaconda Python / Rstudio
We recommend using Anaconda if you're working with Python on your own computer. This makes it easy to install packages like numpy and netCDF4, which depend on binary libraries. Anaconda uses the same system we use for the Conda environments at NCI. Anaconda also supports Rstudio if you work with R.
You can install the Windows native version, or the Linux version in WSL. Normally more packages are available for Linux, but commonly used ones will be available for either. We recommend enabling the 'conda-forge' package repository, you will find a wider variety of packages there and they are generally well supported.
Matlab
Check with your local IT for information on how to use Matlab, they will have information on how to connect to their Matlab license server from your own computer.
You can also use Matlab on NCI's systems via VDI, however you need to use your university license. See the VDI docs or contact help@nci.org.au for more information.
Other Alternatives
Putty
Putty is an alternative SSH implementation for Windows
Cygwin
Cygwin works similarly to WSL, it lets you run Linux programs on a Windows computer
Git for windows includes a Bash terminal based on Cygwin as well
File transfers
It is sometimes useful to transfer files between NCI servers and your local computer. You can do so via the command line with programs like rsync which is readily available. If you prefer a graphical interface, NCI advises you can install a program like Cyberduck. You can then use a SFTP connection. To access files on Gadi, it is preferable to use the data mover nodes: gadi-dm.nci.org.au for the server's name.