Difference between revisions of "Tips: Custom file permissions at creation"

Line 1: Line 1:
 
{{color box |lightsteelblue|[[:Template:Stub]] This is a stub page and needs expansion}}
 
{{color box |lightsteelblue|[[:Template:Stub]] This is a stub page and needs expansion}}
 +
This page provides information on working with shared CLEx storage resources at NCI.
 
== NCI projects ==
 
== NCI projects ==
 
The base unit for the accounting of computational and storage resources is the project, except for your home directory which has limited personal storage. There are 2 types of projects at NCI:
 
The base unit for the accounting of computational and storage resources is the project, except for your home directory which has limited personal storage. There are 2 types of projects at NCI:
Line 6: Line 7:
 
To have a valid login at NCI, you need to be a member of at least one computational project.
 
To have a valid login at NCI, you need to be a member of at least one computational project.
  
This means when working on NCI machine, you will mainly work with shared resources. It is essential you learn how to use these resources fairly and efficiently.
+
This means when working on NCI machine, you will mainly work with shared resources. It is essential you learn how to use these resources fairly and efficiently.  
 
=== CLEx projects ===
 
=== CLEx projects ===
 
CLEx is managing several computational and data projects at NCI via the CMS team. We will not list the data projects since you do not have write access to any of those. To find datasets published at NCI, please see the information on [[Data_Access|this page]]
 
CLEx is managing several computational and data projects at NCI via the CMS team. We will not list the data projects since you do not have write access to any of those. To find datasets published at NCI, please see the information on [[Data_Access|this page]]
Line 28: Line 29:
 
|}
 
|}
 
Some of the following information applies to all projects at NCI but keep in mind, this page focuses on the CLEx projects, as such some information might only apply to CLEx projects. See the Lead CIs of your other projects for information on those.
 
Some of the following information applies to all projects at NCI but keep in mind, this page focuses on the CLEx projects, as such some information might only apply to CLEx projects. See the Lead CIs of your other projects for information on those.
 +
== Accounting ==
 +
NCI applies 2 types of quota on storage:
 +
* size
 +
* number of files
 +
If a project exceeds one of these quota, all jobs from this project are put on hold until the storage is under quota again.
  
 +
NCI and CMS provide a number of tools to check quota and help find the biggest users per project.
 +
=== Tools for accounting ===
 +
==== lquota ====
 +
------
 +
'''Description:''' an instantaneous view of usage and quota of all projects you are a member of for disk filesystems and of your home directory. Provided by NCI.
 +
 +
'''Output example:'''
 +
$ lquota
 +
-----------------------------------------------------------------------
 +
            fs      Usage    Quota    Limit  iUsage  iQuota  iLimit
 +
-----------------------------------------------------------------------
 +
ccc561    home    1.23GB    2.0GB    2.5GB    11235    80000  100000
 +
    c25  short  59.05GB  356.0GB  712.0GB    16530  314000  628000
 +
    e14  short  28.25TB  33.95TB  67.89TB  2203538  3109000  6218000
 +
    k10  short    10.9TB  11.91TB  23.83TB  369385  5775000 11550000
 +
    y03  short    3.13TB    5.0TB    9.99TB  1803428  3779000  7558000
 +
    ua6  short  310.29GB    1.0MB    1.0MB    3337    1025    1025 Over size limit Over inode limit
 +
'''Update frequency:''' instantaneous
 +
 +
'''More information:''' [https://opus.nci.org.au/display/Help/How+to+check+your+disk+quota+and+usage NCI help page]
 +
==== ncimonitor ====
 +
------
 +
'''Description:''' time-series of computational and storage usage per user. Several options available. Provided by CMS.
 +
 +
'''Output example:'''
 +
$module use /g/data/hh5/public/modules
 +
$module load conda
 +
$ncimonitor -P w35
 +
[[File:ncimonitor_example.png||800px]]
 +
 +
'''Update frequency:''' Daily (at night)
 +
 +
'''More information:''' https://github.com/coecms/ncimonitor
 +
==== nci_account -P <project> ====
 +
------
 +
'''Description:''' usage and quota information for the specific project required on all systems. Several options available. Provided by NCI.
 +
 +
'''Output example:'''
 +
$nci_account -P w35
 +
Usage Report: Project=w35 Storage Period=2019.7 (01/07/2019-30/09/2019)
 +
=======================================================================
 +
-------------------------------------------------------------------------------------------------
 +
System    StoragePt            Grant      Usage      Avail      iGrant      iUsage      iAvail
 +
-------------------------------------------------------------------------------------------------
 +
dmf      massdata            63.95TB    28.26TB    35.70TB    4729.00K      7.88K    4721.12K
 +
global    gdata1a            66.00TB    37.17TB    28.83TB    3883.00K    1503.29K    2379.71K
 +
raijin    short              15.00TB    12.45TB      2.55TB    3280.00K    1983.78K    1296.22K
 +
-------------------------------------------------------------------------------------------------
 +
Total                        144.95TB    77.88TB    67.08TB      11.89M      3.49M      8.40M
 +
'''Update frequency:''' About 15 minutes
 +
 +
'''More information:''' [https://opus.nci.org.au/display/Help/How+to+check+your+disk+quota+and+usage NCI help page]
 
[[Category:Stub]]
 
[[Category:Stub]]

Revision as of 01:15, 17 July 2019

Template:Stub This is a stub page and needs expansion

This page provides information on working with shared CLEx storage resources at NCI.

NCI projects

The base unit for the accounting of computational and storage resources is the project, except for your home directory which has limited personal storage. There are 2 types of projects at NCI:

  • data projects: for published and shared datasets
  • computational projects: for running programs

To have a valid login at NCI, you need to be a member of at least one computational project.

This means when working on NCI machine, you will mainly work with shared resources. It is essential you learn how to use these resources fairly and efficiently.

CLEx projects

CLEx is managing several computational and data projects at NCI via the CMS team. We will not list the data projects since you do not have write access to any of those. To find datasets published at NCI, please see the information on this page

The computational projects are split by Research Program (RP):

Project RP Lead CI
v45 Extra-tropical variability Andy Hogg
w35 Drought Claire Carouge
w40 Extreme Rainfall Martin Jucker
w42 Extreme Rainfall Abhnil Prasad
w48 Tropical Variability Dietmar Dommenget
w97 Heatwaves Jason Evans

Some of the following information applies to all projects at NCI but keep in mind, this page focuses on the CLEx projects, as such some information might only apply to CLEx projects. See the Lead CIs of your other projects for information on those.

Accounting

NCI applies 2 types of quota on storage:

  • size
  • number of files

If a project exceeds one of these quota, all jobs from this project are put on hold until the storage is under quota again.

NCI and CMS provide a number of tools to check quota and help find the biggest users per project.

Tools for accounting

lquota


Description: an instantaneous view of usage and quota of all projects you are a member of for disk filesystems and of your home directory. Provided by NCI.

Output example:

$ lquota
-----------------------------------------------------------------------
           fs      Usage     Quota     Limit   iUsage   iQuota   iLimit
-----------------------------------------------------------------------
ccc561    home    1.23GB     2.0GB     2.5GB    11235    80000   100000
   c25   short   59.05GB   356.0GB   712.0GB    16530   314000   628000
   e14   short   28.25TB   33.95TB   67.89TB  2203538  3109000  6218000
   k10   short    10.9TB   11.91TB   23.83TB   369385  5775000 11550000
   y03   short    3.13TB     5.0TB    9.99TB  1803428  3779000  7558000
   ua6   short  310.29GB     1.0MB     1.0MB     3337     1025     1025 Over size limit Over inode limit

Update frequency: instantaneous

More information: NCI help page

ncimonitor


Description: time-series of computational and storage usage per user. Several options available. Provided by CMS.

Output example:

$module use /g/data/hh5/public/modules
$module load conda
$ncimonitor -P w35

Ncimonitor example.png

Update frequency: Daily (at night)

More information: https://github.com/coecms/ncimonitor

nci_account -P <project>


Description: usage and quota information for the specific project required on all systems. Several options available. Provided by NCI.

Output example:

$nci_account -P w35
Usage Report: Project=w35 Storage Period=2019.7 (01/07/2019-30/09/2019)
=======================================================================
-------------------------------------------------------------------------------------------------
System    StoragePt             Grant       Usage       Avail      iGrant      iUsage      iAvail
-------------------------------------------------------------------------------------------------
dmf       massdata            63.95TB     28.26TB     35.70TB    4729.00K       7.88K    4721.12K
global    gdata1a             66.00TB     37.17TB     28.83TB    3883.00K    1503.29K    2379.71K
raijin    short               15.00TB     12.45TB      2.55TB    3280.00K    1983.78K    1296.22K
-------------------------------------------------------------------------------------------------
Total                        144.95TB     77.88TB     67.08TB      11.89M       3.49M       8.40M

Update frequency: About 15 minutes

More information: NCI help page