Difference between revisions of "Migrate experiment projects"

m
 
(6 intermediate revisions by 2 users not shown)
Line 1: Line 1:
  
 
This page contains information on what to do if you want to change project to work at NCI.
 
This page contains information on what to do if you want to change project to work at NCI.
 +
 +
 
  
 
= Setting your default project =
 
= Setting your default project =
Line 7: Line 9:
  
 
If you also run models from accessdev you can change the default project on that computer by editing the file '~/.rashrc'
 
If you also run models from accessdev you can change the default project on that computer by editing the file '~/.rashrc'
 +
 +
If the project listed in those files is either w35 or w48, please edit the file and list w97 or w40 as appropriate.
  
 
In either case you will need to log out and back in again for the change to take effect
 
In either case you will need to log out and back in again for the change to take effect
 +
 +
= Setting your working space =
 +
 +
You will need to create a new working directory for you under the new project if you weren't already a member. You can create the directory under /g/data/<PROJECT> as needed. Please make sure to use your NCI login id as your top directory name.
 +
 +
All data copied from w35 will be stored under: /g/data/w97/W35_GDATA_MOVED and /scratch/w97/W35_SCRATCH_MOVED. Make sure to move your data into your proper working directory under /g/data and /scratch before working with it.
  
 
= Climate model experiments =
 
= Climate model experiments =
Line 18: Line 28:
  
 
WRF does not have a standard experiment setup so we can't give specific instructions for this model. But the compilation script will work without modifications after the project merge.
 
WRF does not have a standard experiment setup so we can't give specific instructions for this model. But the compilation script will work without modifications after the project merge.
 +
 +
&nbsp;
  
 
= Tips for other scripts =
 
= Tips for other scripts =
Line 29: Line 41:
 
It is difficult to give a list of specific steps that would work for anyone. Below are a few tips that might help you.
 
It is difficult to give a list of specific steps that would work for anyone. Below are a few tips that might help you.
  
 +
It is important for you to make sure you know what changes you are doing, that's why we do not provide a magic <code>sed</code> one-liner command to change all the project IDs in all your files at once.
  
 
== Find files with the project listed in ==
 
== Find files with the project listed in ==
Line 38: Line 51:
  
 
At the opposite, if you only want to search through files with a specific extension, you could use <code>--include=GLOB</code>. Replace <samp>GLOB</samp> by the extension you want to search. For example the following command will search for <samp>w35</samp> in all the Python Notebooks and only these:
 
At the opposite, if you only want to search through files with a specific extension, you could use <code>--include=GLOB</code>. Replace <samp>GLOB</samp> by the extension you want to search. For example the following command will search for <samp>w35</samp> in all the Python Notebooks and only these:
<pre>grep --include=*.ipynb -irlI w35 <directory_name> > ipynb_to_update.txt</pre>
+
<pre>grep --include=*.ipynb -irlI w35 <directory_name> > ipynb_to_update.txt
 +
</pre>
 +
 
 +
== Backup before modifications ==
 +
 
 +
You will want to create some sort of backup of your text&nbsp;files before changing them. <span style="color:#c0392b">DO NOT create copies. </span>We do not have heaps of free storage around to allow for it. Our recommended method is to use Git. If your files are not under the Git version control yet, here is a simple workflow that allows to record the state of your files before change. And some tips on comparing the modified and initial versions of the files.
 +
 
 +
Let's say, you want to change some files under a directory called <tt>my_scripts</tt>. You will first need to create a Git repository from that directory, record the current state of your files, do the changes you want to do, test the modified files work:
 +
<pre># Create repository. Go in your my_scripts directory
 +
git init
 +
 
 +
# Record current state
 +
git add .
 +
git commit -m "State at time of merge"
 +
 
 +
# Change your files as you require, test the new files</pre>
 +
 
 +
If something does not work, you can then use the following commands depending on what you want to do:
 +
<pre># Difference between modified file and previous version
 +
git diff <filename>
 +
 
 +
# Re-instate the previous version and delete the modified file
 +
git checkout <filename></pre>
  
 
== Find all and replace all ==
 
== Find all and replace all ==
  
All text file editors come with the ability to find all occurences of a pattern and to replace all these occurences. Make sure you know how to use this functionality for your favourite text editor.
+
All text file editors come with the ability to find all occurences of a pattern and to replace all these occurences. Make sure you know how to use this functionality, including reviewing the changes before making the change, for your favourite text editor.
 +
 
 +
= MDSS data migration =
 +
 
 +
After w35 and w48 are deleted any data stored on the [[How_to_use_MDSS_tape_storage_at_NCI|MDSS tape storage system]] would be inaccessible. To allow continued access all data has been migrated to a directory in&nbsp;the corresponding project (w35 to w97, w48 to w40).
 +
<pre style="background-color: rgb(238, 238, 238); border: 1px solid rgb(204, 204, 204); padding: 5px 10px;">$ mdss -P w40 ls -ld W48_MASSDATA_MOVED
 +
drwxrws--- 22 root w40 411 Mar  2 10:33 W48_MASSDATA_MOVED
 +
$ mdss -P w97 ls -ld W35_MASSDATA_MOVED
 +
drwxrws--- 27 root w97 143360 Mar  2 11:47 W35_MASSDATA_MOVED</pre>

Latest revision as of 18:10, 28 June 2022

This page contains information on what to do if you want to change project to work at NCI.

 

Setting your default project

To change the default project that jobs get submitted to when you run 'qsub', edit the file on Gadi '~/.config/gadi-login.conf'

If you also run models from accessdev you can change the default project on that computer by editing the file '~/.rashrc'

If the project listed in those files is either w35 or w48, please edit the file and list w97 or w40 as appropriate.

In either case you will need to log out and back in again for the change to take effect

Setting your working space

You will need to create a new working directory for you under the new project if you weren't already a member. You can create the directory under /g/data/<PROJECT> as needed. Please make sure to use your NCI login id as your top directory name.

All data copied from w35 will be stored under: /g/data/w97/W35_GDATA_MOVED and /scratch/w97/W35_SCRATCH_MOVED. Make sure to move your data into your proper working directory under /g/data and /scratch before working with it.

Climate model experiments

WRF does not have a standard experiment setup so we can't give specific instructions for this model. But the compilation script will work without modifications after the project merge.

 

Tips for other scripts

In your script, you may have used w35 or w48 in the following lines:

  • PBS directive -P to specify the project
  • PBS directive -l storage to load the storage space for these projects.
  • Full paths of input or output data

It is difficult to give a list of specific steps that would work for anyone. Below are a few tips that might help you.

It is important for you to make sure you know what changes you are doing, that's why we do not provide a magic sed one-liner command to change all the project IDs in all your files at once.

Find files with the project listed in

To find all the files that contain the mention of w35, you can use the following command:

grep --exclude=*.nc -irlI w35 <directory_name> > files_to_update.txt

files_to_update.txt will contain a list of files where w35 appear in at least one line. The command will not search in binary files and netCDF files and you can exclude more files from the search (e.g. figures) with the --exclude= option. Replace <directory_name> with the directory you want to search through.

At the opposite, if you only want to search through files with a specific extension, you could use --include=GLOB. Replace GLOB by the extension you want to search. For example the following command will search for w35 in all the Python Notebooks and only these:

grep --include=*.ipynb -irlI w35 <directory_name> > ipynb_to_update.txt

Backup before modifications

You will want to create some sort of backup of your text files before changing them. DO NOT create copies. We do not have heaps of free storage around to allow for it. Our recommended method is to use Git. If your files are not under the Git version control yet, here is a simple workflow that allows to record the state of your files before change. And some tips on comparing the modified and initial versions of the files.

Let's say, you want to change some files under a directory called my_scripts. You will first need to create a Git repository from that directory, record the current state of your files, do the changes you want to do, test the modified files work:

# Create repository. Go in your my_scripts directory
git init

# Record current state
git add .
git commit -m "State at time of merge"

# Change your files as you require, test the new files

If something does not work, you can then use the following commands depending on what you want to do:

# Difference between modified file and previous version
git diff <filename>

# Re-instate the previous version and delete the modified file
git checkout <filename>

Find all and replace all

All text file editors come with the ability to find all occurences of a pattern and to replace all these occurences. Make sure you know how to use this functionality, including reviewing the changes before making the change, for your favourite text editor.

MDSS data migration

After w35 and w48 are deleted any data stored on the MDSS tape storage system would be inaccessible. To allow continued access all data has been migrated to a directory in the corresponding project (w35 to w97, w48 to w40).

$ mdss -P w40 ls -ld W48_MASSDATA_MOVED
drwxrws--- 22 root w40 411 Mar  2 10:33 W48_MASSDATA_MOVED
$ mdss -P w97 ls -ld W35_MASSDATA_MOVED
drwxrws--- 27 root w97 143360 Mar  2 11:47 W35_MASSDATA_MOVED