Difference between revisions of "Tips: Custom file permissions at creation"

(Replaced content with "{{color box |lightsteelblue|Template:Stub This is a stub page and needs expansion}} This page provides information on how to create files with group read permissions....")
(Tag: Replaced)
Line 1: Line 1:
 
{{color box |lightsteelblue|[[:Template:Stub]] This is a stub page and needs expansion}}
 
{{color box |lightsteelblue|[[:Template:Stub]] This is a stub page and needs expansion}}
This page provides information on working with shared CLEx storage resources at NCI.
+
This page provides information on how to create files with group read permissions.
== NCI projects ==
 
The base unit for the accounting of computational and storage resources is the project, except for your home directory which has limited personal storage. There are 2 types of projects at NCI:
 
* data projects: for published and shared datasets
 
* computational projects: for running programs
 
To have a valid login at NCI, you need to be a member of at least one computational project.
 
 
 
This means when working on NCI machine, you will mainly work with shared resources. It is essential you learn how to use these resources fairly and efficiently.
 
=== CLEx projects ===
 
CLEx is managing several computational and data projects at NCI via the CMS team. We will not list the data projects since you do not have write access to any of those. To find datasets published at NCI, please see the information on [[Data_Access|this page]]
 
 
 
The computational projects are split by Research Program (RP):
 
{| class="wikitable"
 
|-
 
! Project !! RP !! Lead CI
 
|-
 
| v45 || Extra-tropical variability || [mailto:andy.hogg@anu.edu.au Andy Hogg]
 
|-
 
| w35 || Drought || [mailto:c.carouge@unsw.edu.au Claire Carouge]
 
|-
 
| w40 || Extreme Rainfall || [mailto:martin.jucker@unsw.edu.au Martin Jucker]
 
|-
 
| w42 || Extreme Rainfall || [mailto:abhnil.prasad@unsw.edu.au Abhnil Prasad]
 
|-
 
| w48 || Tropical Variability || [mailto:dietmar.dommenget@monash.edu Dietmar Dommenget]
 
|-
 
| w97 || Heatwaves || [mailto:jason.evans@unsw.edu.au Jason Evans]
 
|}
 
Some of the following information applies to all projects at NCI but keep in mind, this page focuses on the CLEx projects, as such some information might only apply to CLEx projects. See the Lead CIs of your other projects for information on those.
 
== Accounting ==
 
NCI applies 2 types of quota on storage:
 
* size
 
* number of files
 
If a project exceeds one of these quota, all jobs from this project are put on hold until the storage is under quota again.
 
 
 
NCI and CMS provide a number of tools to check quota and help find the biggest users per project.
 
=== Tools for accounting ===
 
==== lquota ====
 
------
 
'''Description:''' an instantaneous view of usage and quota of all projects you are a member of for disk filesystems and of your home directory. Provided by NCI.
 
 
 
'''Output example:'''
 
$ lquota
 
-----------------------------------------------------------------------
 
            fs      Usage    Quota    Limit  iUsage  iQuota  iLimit
 
-----------------------------------------------------------------------
 
ccc561    home    1.23GB    2.0GB    2.5GB    11235    80000  100000
 
    c25  short  59.05GB  356.0GB  712.0GB    16530  314000  628000
 
    e14  short  28.25TB  33.95TB  67.89TB  2203538  3109000  6218000
 
    k10  short    10.9TB  11.91TB  23.83TB  369385  5775000 11550000
 
    y03  short    3.13TB    5.0TB    9.99TB  1803428  3779000  7558000
 
    ua6  short  310.29GB    1.0MB    1.0MB    3337    1025    1025 Over size limit Over inode limit
 
'''Update frequency:''' instantaneous
 
 
 
'''More information:''' [https://opus.nci.org.au/display/Help/How+to+check+your+disk+quota+and+usage NCI help page]
 
==== ncimonitor ====
 
------
 
'''Description:''' time-series of computational and storage usage per user. Several options available. Provided by CMS.
 
 
 
'''Output example:'''
 
$module use /g/data/hh5/public/modules
 
$module load conda
 
$ncimonitor -P w35
 
[[File:ncimonitor_example.png||800px]]
 
 
 
'''Update frequency:''' Daily (at night)
 
 
 
'''More information:''' https://github.com/coecms/ncimonitor
 
==== nci_account -P <project> ====
 
------
 
'''Description:''' usage and quota information for the specific project required on all systems. Several options available. Provided by NCI.
 
 
 
'''Output example:'''
 
$nci_account -P w35
 
Usage Report: Project=w35 Storage Period=2019.7 (01/07/2019-30/09/2019)
 
=======================================================================
 
-------------------------------------------------------------------------------------------------
 
System    StoragePt            Grant      Usage      Avail      iGrant      iUsage      iAvail
 
-------------------------------------------------------------------------------------------------
 
dmf      massdata            63.95TB    28.26TB    35.70TB    4729.00K      7.88K    4721.12K
 
global    gdata1a            66.00TB    37.17TB    28.83TB    3883.00K    1503.29K    2379.71K
 
raijin    short              15.00TB    12.45TB      2.55TB    3280.00K    1983.78K    1296.22K
 
-------------------------------------------------------------------------------------------------
 
Total                        144.95TB    77.88TB    67.08TB      11.89M      3.49M      8.40M
 
'''Update frequency:''' About 15 minutes
 
 
 
'''More information:''' [https://opus.nci.org.au/display/Help/How+to+check+your+disk+quota+and+usage NCI help page]
 
==== short_files_report and gdata''N''_files_report ====
 
------
 
'''Description:''' storage usage on the specified filesystem split by user. 2 available options <code>-G <project></code> and <code>-P <project></code>
 
 
 
'''Output example:'''</br>
 
''-G option'' gives usage for all files owned by the group specified (not necessarily located in that group directory)
 
$ short_files_report -G w35
 
Usage details for project w35:
 
---------------------------------------------------
 
          folder  user  size    inodes  scan date
 
---------------------------------------------------
 
            z00 hxw599 5996kB        32 2019-07-17
 
            n81 gdv561 54.3GB    111044 2019-07-17
 
            p66 jxs599  218MB        12 2019-07-17
 
            p66 saw562 8943MB    20696 2019-07-17
 
            p66 kxl561 28.0kB        7 2019-07-17
 
''-P option'' gives usage for all files located in the directory specified (not necessarily owned by the same group)
 
$gdata1a_files_report -P w35
 
Usage details for /g/data1a/w35:
 
---------------------------------------------------
 
          group  user  size    inodes  scan date
 
---------------------------------------------------
 
            c25 dei561  409MB        56 2019-07-17
 
            k10 hxw599 4690MB        7 2019-07-17
 
            n81 jpe561  340kB        31 2019-07-17
 
            ma3 nw3057  737GB      959 2019-07-17
 
            w35 aph502 3810MB        11 2019-07-17
 
  
 +
== PBS logs ==
 +
By default, all log files from PBS are only open to the user owning of the file.
 +
You can use a PBS option to change this behaviour so your logs are readable by the user and the group owning the file:
 +
#PBS -W umask=0022
 
[[Category:Stub]]
 
[[Category:Stub]]

Revision as of 02:09, 8 August 2019

Template:Stub This is a stub page and needs expansion

This page provides information on how to create files with group read permissions.

PBS logs

By default, all log files from PBS are only open to the user owning of the file. You can use a PBS option to change this behaviour so your logs are readable by the user and the group owning the file:

  1. PBS -W umask=0022