|
|
(9 intermediate revisions by 2 users not shown) |
Line 1: |
Line 1: |
− | {{color box |lightsteelblue|[[:Template:Stub]] This is a stub page and needs expansion}}
| + | This page provides information on how to create files with group read permissions at NCI. |
− | This page provides information on working with shared CLEx storage resources at NCI. | |
− | == NCI projects ==
| |
− | The base unit for the accounting of computational and storage resources is the project, except for your home directory which has limited personal storage. There are 2 types of projects at NCI:
| |
− | * data projects: for published and shared datasets
| |
− | * computational projects: for running programs
| |
− | To have a valid login at NCI, you need to be a member of at least one computational project.
| |
| | | |
− | This means when working on NCI machine, you will mainly work with shared resources. It is essential you learn how to use these resources fairly and efficiently.
| + | By default, when you create a file under /short of /g/data, the file will be readable by you and the group owning the file. So you don't need to change anything then. But there are a few cases where the default behaviour is different and you may want to change it. |
− | === CLEx projects ===
| |
− | CLEx is managing several computational and data projects at NCI via the CMS team. We will not list the data projects since you do not have write access to any of those. To find datasets published at NCI, please see the information on [[Data_Access|this page]]
| |
| | | |
− | The computational projects are split by Research Program (RP):
| + | == PBS logs == |
− | {| class="wikitable"
| + | By default, all log files from PBS are only readable by the user owning the file. |
− | |-
| + | You can use a PBS option to change this behaviour so your logs are readable by the user and the group owning the file: |
− | ! Project !! RP !! Lead CI
| + | <syntaxhighlight lang=text> |
− | |-
| + | #PBS -W umask=0022 |
− | | v45 || Extra-tropical variability || [mailto:andy.hogg@anu.edu.au Andy Hogg]
| + | </syntaxhighlight> |
− | |-
| + | You need to add this option either to the script you submit to the queue (with qsub) or directly to the qsub command line. |
− | | w35 || Drought || [mailto:c.carouge@unsw.edu.au Claire Carouge]
| |
− | |-
| |
− | | w40 || Extreme Rainfall || [mailto:martin.jucker@unsw.edu.au Martin Jucker]
| |
− | |-
| |
− | | w42 || Extreme Rainfall || [mailto:abhnil.prasad@unsw.edu.au Abhnil Prasad]
| |
− | |-
| |
− | | w48 || Tropical Variability || [mailto:dietmar.dommenget@monash.edu Dietmar Dommenget]
| |
− | |-
| |
− | | w97 || Heatwaves || [mailto:jason.evans@unsw.edu.au Jason Evans]
| |
− | |}
| |
− | Some of the following information applies to all projects at NCI but keep in mind, this page focuses on the CLEx projects, as such some information might only apply to CLEx projects. See the Lead CIs of your other projects for information on those.
| |
− | == Accounting ==
| |
− | NCI applies 2 types of quota on storage:
| |
− | * size
| |
− | * number of files
| |
− | If a project exceeds one of these quota, all jobs from this project are put on hold until the storage is under quota again.
| |
| | | |
− | NCI and CMS provide a number of tools to check quota and help find the biggest users per project.
| + | == Files created by programs == |
− | === Tools for accounting === | |
− | ==== lquota ====
| |
− | ------
| |
− | '''Description:''' an instantaneous view of usage and quota of all projects you are a member of for disk filesystems and of your home directory. Provided by NCI.
| |
| | | |
− | '''Output example:''' | + | Some programs might impose a different default of the files they create which might not be what you'd like. |
− | $ lquota
| |
− | -----------------------------------------------------------------------
| |
− | fs Usage Quota Limit iUsage iQuota iLimit
| |
− | -----------------------------------------------------------------------
| |
− | ccc561 home 1.23GB 2.0GB 2.5GB 11235 80000 100000
| |
− | c25 short 59.05GB 356.0GB 712.0GB 16530 314000 628000
| |
− | e14 short 28.25TB 33.95TB 67.89TB 2203538 3109000 6218000
| |
− | k10 short 10.9TB 11.91TB 23.83TB 369385 5775000 11550000
| |
− | y03 short 3.13TB 5.0TB 9.99TB 1803428 3779000 7558000
| |
− | ua6 short 310.29GB 1.0MB 1.0MB 3337 1025 1025 Over size limit Over inode limit
| |
− | '''Update frequency:''' instantaneous
| |
| | | |
− | '''More information:''' [https://opus.nci.org.au/display/Help/How+to+check+your+disk+quota+and+usage NCI help page]
| + | === You have access to the source code === |
− | ==== ncimonitor ==== | + | If you have access to the source code, it is usually possible to modify the default behaviour. What needs to be modified varies depending on the language. Please contact [mailto:cws_help@nci.org.au the CMS team] if you need help. |
− | ------
| |
− | '''Description:''' time-series of computational and storage usage per user. Several options available. Provided by CMS.
| |
| | | |
− | '''Output example:''' | + | === You don't have access to the source code === |
− | $module use /g/data/hh5/public/modules | + | You then need to add a post-processing step, for example in a bash script calling your program. You can change the permissions on the files created by your program, using the <code>chmod</code> command: |
− | $module load conda
| + | chmod g+r <filename> |
− | $ncimonitor -P w35 | + | This command can be run recursively through a directory tree: |
− | [[File:ncimonitor_example.png||800px]]
| + | chmod -R g+r <directory_name> |
| | | |
− | '''Update frequency:''' Daily (at night) | + | === Files owned by the wrong group === |
| + | If you are using compute time from several projects, you may end up with files owned by different projects in the same area. This might not be the ideal situation for you and it is possible to change the group owning the file. '''Note:''' NCI's storage accounting depends only on the group owning the files and not the project space under which the files are stored. '''Before changing the group owning files, please make sure the new group has enough free allocation and you will not cause that group to get over quota.''' |
| | | |
− | '''More information:''' https://github.com/coecms/ncimonitor
| + | To change the group ownership of a file: |
− | ==== nci_account -P <project> ====
| + | chgrp <newgroup> <filename> |
− | ------
| + | Or recursively through a directory tree: |
− | '''Description:''' usage and quota information for the specific project required on all systems. Several options available. Provided by NCI.
| + | chgrp -R <newgroup> <directory_name> |
| | | |
− | '''Output example:'''
| + | [[Category:Data]] |
− | $nci_account -P w35
| |
− | Usage Report: Project=w35 Storage Period=2019.7 (01/07/2019-30/09/2019)
| |
− | =======================================================================
| |
− | -------------------------------------------------------------------------------------------------
| |
− | System StoragePt Grant Usage Avail iGrant iUsage iAvail
| |
− | -------------------------------------------------------------------------------------------------
| |
− | dmf massdata 63.95TB 28.26TB 35.70TB 4729.00K 7.88K 4721.12K
| |
− | global gdata1a 66.00TB 37.17TB 28.83TB 3883.00K 1503.29K 2379.71K
| |
− | raijin short 15.00TB 12.45TB 2.55TB 3280.00K 1983.78K 1296.22K
| |
− | -------------------------------------------------------------------------------------------------
| |
− | Total 144.95TB 77.88TB 67.08TB 11.89M 3.49M 8.40M
| |
− | '''Update frequency:''' About 15 minutes
| |
− | | |
− | '''More information:''' [https://opus.nci.org.au/display/Help/How+to+check+your+disk+quota+and+usage NCI help page]
| |
− | ==== short_files_report and gdata''N''_files_report ====
| |
− | ------
| |
− | '''Description:''' storage usage on the specified filesystem split by user. 2 available options <code>-G <project></code> and <code>-P <project></code>
| |
− | | |
− | '''Output example:'''
| |
− | ''-G option'' gives usage for all files owned by the group specified (not necessarily located in that group directory)
| |
− | $ short_files_report -G w35
| |
− | Usage details for project w35:
| |
− | ---------------------------------------------------
| |
− | folder user size inodes scan date
| |
− | ---------------------------------------------------
| |
− | z00 hxw599 5996kB 32 2019-07-17
| |
− | n81 gdv561 54.3GB 111044 2019-07-17
| |
− | p66 jxs599 218MB 12 2019-07-17
| |
− | p66 saw562 8943MB 20696 2019-07-17
| |
− | p66 kxl561 28.0kB 7 2019-07-17
| |
− | ''-P option'' gives usage for all files located in the directory specified (not necessarily owned by the same group)
| |
− | $gdata1a_files_report -P w35
| |
− | Usage details for /g/data1a/w35:
| |
− | ---------------------------------------------------
| |
− | group user size inodes scan date
| |
− | ---------------------------------------------------
| |
− | c25 dei561 409MB 56 2019-07-17
| |
− | k10 hxw599 4690MB 7 2019-07-17
| |
− | n81 jpe561 340kB 31 2019-07-17
| |
− | ma3 nw3057 737GB 959 2019-07-17
| |
− | w35 aph502 3810MB 11 2019-07-17
| |
− | | |
− | [[Category:Stub]] | |
This page provides information on how to create files with group read permissions at NCI.
By default, when you create a file under /short of /g/data, the file will be readable by you and the group owning the file. So you don't need to change anything then. But there are a few cases where the default behaviour is different and you may want to change it.
PBS logs
By default, all log files from PBS are only readable by the user owning the file.
You can use a PBS option to change this behaviour so your logs are readable by the user and the group owning the file:
You need to add this option either to the script you submit to the queue (with qsub) or directly to the qsub command line.
Files created by programs
Some programs might impose a different default of the files they create which might not be what you'd like.
You have access to the source code
If you have access to the source code, it is usually possible to modify the default behaviour. What needs to be modified varies depending on the language. Please contact the CMS team if you need help.
You don't have access to the source code
You then need to add a post-processing step, for example in a bash script calling your program. You can change the permissions on the files created by your program, using the chmod
command:
chmod g+r <filename>
This command can be run recursively through a directory tree:
chmod -R g+r <directory_name>
Files owned by the wrong group
If you are using compute time from several projects, you may end up with files owned by different projects in the same area. This might not be the ideal situation for you and it is possible to change the group owning the file. Note: NCI's storage accounting depends only on the group owning the files and not the project space under which the files are stored. Before changing the group owning files, please make sure the new group has enough free allocation and you will not cause that group to get over quota.
To change the group ownership of a file:
chgrp <newgroup> <filename>
Or recursively through a directory tree:
chgrp -R <newgroup> <directory_name>