|
|
| Line 1: |
Line 1: |
| | {{color box |lightsteelblue|[[:Template:Stub]] This is a stub page and needs expansion}} | | {{color box |lightsteelblue|[[:Template:Stub]] This is a stub page and needs expansion}} |
| − | This page provides information on working with shared CLEx storage resources at NCI. | + | This page provides information on how to create files with group read permissions. |
| − | == NCI projects ==
| |
| − | The base unit for the accounting of computational and storage resources is the project, except for your home directory which has limited personal storage. There are 2 types of projects at NCI:
| |
| − | * data projects: for published and shared datasets
| |
| − | * computational projects: for running programs
| |
| − | To have a valid login at NCI, you need to be a member of at least one computational project.
| |
| − | | |
| − | This means when working on NCI machine, you will mainly work with shared resources. It is essential you learn how to use these resources fairly and efficiently.
| |
| − | === CLEx projects ===
| |
| − | CLEx is managing several computational and data projects at NCI via the CMS team. We will not list the data projects since you do not have write access to any of those. To find datasets published at NCI, please see the information on [[Data_Access|this page]]
| |
| − | | |
| − | The computational projects are split by Research Program (RP):
| |
| − | {| class="wikitable"
| |
| − | |-
| |
| − | ! Project !! RP !! Lead CI
| |
| − | |-
| |
| − | | v45 || Extra-tropical variability || [mailto:andy.hogg@anu.edu.au Andy Hogg]
| |
| − | |-
| |
| − | | w35 || Drought || [mailto:c.carouge@unsw.edu.au Claire Carouge]
| |
| − | |-
| |
| − | | w40 || Extreme Rainfall || [mailto:martin.jucker@unsw.edu.au Martin Jucker]
| |
| − | |-
| |
| − | | w42 || Extreme Rainfall || [mailto:abhnil.prasad@unsw.edu.au Abhnil Prasad]
| |
| − | |-
| |
| − | | w48 || Tropical Variability || [mailto:dietmar.dommenget@monash.edu Dietmar Dommenget]
| |
| − | |-
| |
| − | | w97 || Heatwaves || [mailto:jason.evans@unsw.edu.au Jason Evans]
| |
| − | |}
| |
| − | Some of the following information applies to all projects at NCI but keep in mind, this page focuses on the CLEx projects, as such some information might only apply to CLEx projects. See the Lead CIs of your other projects for information on those.
| |
| − | == Accounting ==
| |
| − | NCI applies 2 types of quota on storage:
| |
| − | * size
| |
| − | * number of files
| |
| − | If a project exceeds one of these quota, all jobs from this project are put on hold until the storage is under quota again.
| |
| − | | |
| − | NCI and CMS provide a number of tools to check quota and help find the biggest users per project.
| |
| − | === Tools for accounting ===
| |
| − | ==== lquota ====
| |
| − | ------
| |
| − | '''Description:''' an instantaneous view of usage and quota of all projects you are a member of for disk filesystems and of your home directory. Provided by NCI.
| |
| − | | |
| − | '''Output example:'''
| |
| − | $ lquota
| |
| − | -----------------------------------------------------------------------
| |
| − | fs Usage Quota Limit iUsage iQuota iLimit
| |
| − | -----------------------------------------------------------------------
| |
| − | ccc561 home 1.23GB 2.0GB 2.5GB 11235 80000 100000
| |
| − | c25 short 59.05GB 356.0GB 712.0GB 16530 314000 628000
| |
| − | e14 short 28.25TB 33.95TB 67.89TB 2203538 3109000 6218000
| |
| − | k10 short 10.9TB 11.91TB 23.83TB 369385 5775000 11550000
| |
| − | y03 short 3.13TB 5.0TB 9.99TB 1803428 3779000 7558000
| |
| − | ua6 short 310.29GB 1.0MB 1.0MB 3337 1025 1025 Over size limit Over inode limit
| |
| − | '''Update frequency:''' instantaneous
| |
| − | | |
| − | '''More information:''' [https://opus.nci.org.au/display/Help/How+to+check+your+disk+quota+and+usage NCI help page]
| |
| − | ==== ncimonitor ====
| |
| − | ------
| |
| − | '''Description:''' time-series of computational and storage usage per user. Several options available. Provided by CMS.
| |
| − | | |
| − | '''Output example:'''
| |
| − | $module use /g/data/hh5/public/modules
| |
| − | $module load conda
| |
| − | $ncimonitor -P w35
| |
| − | [[File:ncimonitor_example.png||800px]]
| |
| − | | |
| − | '''Update frequency:''' Daily (at night)
| |
| − | | |
| − | '''More information:''' https://github.com/coecms/ncimonitor
| |
| − | ==== nci_account -P <project> ====
| |
| − | ------
| |
| − | '''Description:''' usage and quota information for the specific project required on all systems. Several options available. Provided by NCI.
| |
| − | | |
| − | '''Output example:'''
| |
| − | $nci_account -P w35
| |
| − | Usage Report: Project=w35 Storage Period=2019.7 (01/07/2019-30/09/2019)
| |
| − | =======================================================================
| |
| − | -------------------------------------------------------------------------------------------------
| |
| − | System StoragePt Grant Usage Avail iGrant iUsage iAvail
| |
| − | -------------------------------------------------------------------------------------------------
| |
| − | dmf massdata 63.95TB 28.26TB 35.70TB 4729.00K 7.88K 4721.12K
| |
| − | global gdata1a 66.00TB 37.17TB 28.83TB 3883.00K 1503.29K 2379.71K
| |
| − | raijin short 15.00TB 12.45TB 2.55TB 3280.00K 1983.78K 1296.22K
| |
| − | -------------------------------------------------------------------------------------------------
| |
| − | Total 144.95TB 77.88TB 67.08TB 11.89M 3.49M 8.40M
| |
| − | '''Update frequency:''' About 15 minutes
| |
| − | | |
| − | '''More information:''' [https://opus.nci.org.au/display/Help/How+to+check+your+disk+quota+and+usage NCI help page]
| |
| − | ==== short_files_report and gdata''N''_files_report ====
| |
| − | ------
| |
| − | '''Description:''' storage usage on the specified filesystem split by user. 2 available options <code>-G <project></code> and <code>-P <project></code>
| |
| − | | |
| − | '''Output example:'''</br>
| |
| − | ''-G option'' gives usage for all files owned by the group specified (not necessarily located in that group directory)
| |
| − | $ short_files_report -G w35
| |
| − | Usage details for project w35:
| |
| − | ---------------------------------------------------
| |
| − | folder user size inodes scan date
| |
| − | ---------------------------------------------------
| |
| − | z00 hxw599 5996kB 32 2019-07-17
| |
| − | n81 gdv561 54.3GB 111044 2019-07-17
| |
| − | p66 jxs599 218MB 12 2019-07-17
| |
| − | p66 saw562 8943MB 20696 2019-07-17
| |
| − | p66 kxl561 28.0kB 7 2019-07-17
| |
| − | ''-P option'' gives usage for all files located in the directory specified (not necessarily owned by the same group)
| |
| − | $gdata1a_files_report -P w35
| |
| − | Usage details for /g/data1a/w35:
| |
| − | ---------------------------------------------------
| |
| − | group user size inodes scan date
| |
| − | ---------------------------------------------------
| |
| − | c25 dei561 409MB 56 2019-07-17
| |
| − | k10 hxw599 4690MB 7 2019-07-17
| |
| − | n81 jpe561 340kB 31 2019-07-17
| |
| − | ma3 nw3057 737GB 959 2019-07-17
| |
| − | w35 aph502 3810MB 11 2019-07-17
| |
| | | | |
| | + | == PBS logs == |
| | + | By default, all log files from PBS are only open to the user owning of the file. |
| | + | You can use a PBS option to change this behaviour so your logs are readable by the user and the group owning the file: |
| | + | #PBS -W umask=0022 |
| | [[Category:Stub]] | | [[Category:Stub]] |
