Difference between revisions of "Tips: Custom file permissions at creation"

(9 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{color box |lightsteelblue|[[:Template:Stub]] This is a stub page and needs expansion}}
+
This page provides information on how to create files with group read permissions at NCI.
This page provides information on working with shared CLEx storage resources at NCI.
 
== NCI projects ==
 
The base unit for the accounting of computational and storage resources is the project, except for your home directory which has limited personal storage. There are 2 types of projects at NCI:
 
* data projects: for published and shared datasets
 
* computational projects: for running programs
 
To have a valid login at NCI, you need to be a member of at least one computational project.
 
  
This means when working on NCI machine, you will mainly work with shared resources. It is essential you learn how to use these resources fairly and efficiently.  
+
By default, when you create a file under /short of /g/data, the file will be readable by you and the group owning the file. So you don't need to change anything then. But there are a few cases where the default behaviour is different and you may want to change it.
=== CLEx projects ===
 
CLEx is managing several computational and data projects at NCI via the CMS team. We will not list the data projects since you do not have write access to any of those. To find datasets published at NCI, please see the information on [[Data_Access|this page]]
 
  
The computational projects are split by Research Program (RP):
+
== PBS logs ==
{| class="wikitable"
+
By default, all log files from PBS are only readable by the user owning the file.
|-
+
You can use a PBS option to change this behaviour so your logs are readable by the user and the group owning the file:
! Project !! RP !! Lead CI
+
<syntaxhighlight lang=text>
|-
+
#PBS -W umask=0022
| v45 || Extra-tropical variability || [mailto:andy.hogg@anu.edu.au Andy Hogg]
+
</syntaxhighlight>
|-
+
You need to add this option either to the script you submit to the queue (with qsub) or directly to the qsub command line.
| w35 || Drought || [mailto:c.carouge@unsw.edu.au Claire Carouge]
 
|-
 
| w40 || Extreme Rainfall || [mailto:martin.jucker@unsw.edu.au Martin Jucker]
 
|-
 
| w42 || Extreme Rainfall || [mailto:abhnil.prasad@unsw.edu.au Abhnil Prasad]
 
|-
 
| w48 || Tropical Variability || [mailto:dietmar.dommenget@monash.edu Dietmar Dommenget]
 
|-
 
| w97 || Heatwaves || [mailto:jason.evans@unsw.edu.au Jason Evans]
 
|}
 
Some of the following information applies to all projects at NCI but keep in mind, this page focuses on the CLEx projects, as such some information might only apply to CLEx projects. See the Lead CIs of your other projects for information on those.
 
== Accounting ==
 
NCI applies 2 types of quota on storage:
 
* size
 
* number of files
 
If a project exceeds one of these quota, all jobs from this project are put on hold until the storage is under quota again.
 
  
NCI and CMS provide a number of tools to check quota and help find the biggest users per project.
+
== Files created by programs ==
=== Tools for accounting ===
 
==== lquota ====
 
------
 
'''Description:''' an instantaneous view of usage and quota of all projects you are a member of for disk filesystems and of your home directory. Provided by NCI.
 
  
'''Output example:'''
+
Some programs might impose a different default of the files they create which might not be what you'd like.
$ lquota
 
-----------------------------------------------------------------------
 
            fs      Usage    Quota    Limit  iUsage  iQuota  iLimit
 
-----------------------------------------------------------------------
 
ccc561    home    1.23GB    2.0GB    2.5GB    11235    80000  100000
 
    c25  short  59.05GB  356.0GB  712.0GB    16530  314000  628000
 
    e14  short  28.25TB  33.95TB  67.89TB  2203538  3109000  6218000
 
    k10  short    10.9TB  11.91TB  23.83TB  369385  5775000 11550000
 
    y03  short    3.13TB    5.0TB    9.99TB  1803428  3779000  7558000
 
    ua6  short  310.29GB    1.0MB    1.0MB    3337    1025    1025 Over size limit Over inode limit
 
'''Update frequency:''' instantaneous
 
  
'''More information:''' [https://opus.nci.org.au/display/Help/How+to+check+your+disk+quota+and+usage NCI help page]
+
=== You have access to the source code ===
==== ncimonitor ====
+
If you have access to the source code, it is usually possible to modify the default behaviour. What needs to be modified varies depending on the language. Please contact [mailto:cws_help@nci.org.au the CMS team] if you need help.
------
 
'''Description:''' time-series of computational and storage usage per user. Several options available. Provided by CMS.
 
  
'''Output example:'''
+
=== You don't have access to the source code ===
  $module use /g/data/hh5/public/modules
+
You then need to add a post-processing step, for example in a bash script calling your program. You can change the permissions on the files created by your program, using the <code>chmod</code> command:
$module load conda
+
  chmod g+r <filename>
  $ncimonitor -P w35
+
This command can be run recursively through a directory tree:
[[File:ncimonitor_example.png||800px]]
+
  chmod -R g+r <directory_name>
  
'''Update frequency:''' Daily (at night)
+
=== Files owned by the wrong group ===
 +
If you are using compute time from several projects, you may end up with files owned by different projects in the same area. This might not be the ideal situation for you and it is possible to change the group owning the file. '''Note:''' NCI's storage accounting depends only on the group owning the files and not the project space under which the files are stored. '''Before changing the group owning files, please make sure the new group has enough free allocation and you will not cause that group to get over quota.'''
  
'''More information:''' https://github.com/coecms/ncimonitor
+
To change the group ownership of a file:
==== nci_account -P <project> ====
+
chgrp <newgroup> <filename>
------
+
Or recursively through a directory tree:
'''Description:''' usage and quota information for the specific project required on all systems. Several options available. Provided by NCI.
+
chgrp -R <newgroup> <directory_name>
  
'''Output example:'''
+
[[Category:Data]]
$nci_account -P w35
 
Usage Report: Project=w35 Storage Period=2019.7 (01/07/2019-30/09/2019)
 
=======================================================================
 
-------------------------------------------------------------------------------------------------
 
System    StoragePt            Grant      Usage      Avail      iGrant      iUsage      iAvail
 
-------------------------------------------------------------------------------------------------
 
dmf      massdata            63.95TB    28.26TB    35.70TB    4729.00K      7.88K    4721.12K
 
global    gdata1a            66.00TB    37.17TB    28.83TB    3883.00K    1503.29K    2379.71K
 
raijin    short              15.00TB    12.45TB      2.55TB    3280.00K    1983.78K    1296.22K
 
-------------------------------------------------------------------------------------------------
 
Total                        144.95TB    77.88TB    67.08TB      11.89M      3.49M      8.40M
 
'''Update frequency:''' About 15 minutes
 
 
 
'''More information:''' [https://opus.nci.org.au/display/Help/How+to+check+your+disk+quota+and+usage NCI help page]
 
==== short_files_report and gdata''N''_files_report ====
 
------
 
'''Description:''' storage usage on the specified filesystem split by user. 2 available options <code>-G <project></code> and <code>-P <project></code>
 
 
 
'''Output example:'''
 
''-G option'' gives usage for all files owned by the group specified (not necessarily located in that group directory)
 
$ short_files_report -G w35
 
Usage details for project w35:
 
---------------------------------------------------
 
          folder  user  size    inodes  scan date
 
---------------------------------------------------
 
            z00 hxw599 5996kB        32 2019-07-17
 
            n81 gdv561 54.3GB    111044 2019-07-17
 
            p66 jxs599  218MB        12 2019-07-17
 
            p66 saw562 8943MB    20696 2019-07-17
 
            p66 kxl561 28.0kB        7 2019-07-17
 
''-P option'' gives usage for all files located in the directory specified (not necessarily owned by the same group)
 
$gdata1a_files_report -P w35
 
Usage details for /g/data1a/w35:
 
---------------------------------------------------
 
          group  user  size    inodes  scan date
 
---------------------------------------------------
 
            c25 dei561  409MB        56 2019-07-17
 
            k10 hxw599 4690MB        7 2019-07-17
 
            n81 jpe561  340kB        31 2019-07-17
 
            ma3 nw3057  737GB      959 2019-07-17
 
            w35 aph502 3810MB        11 2019-07-17
 
 
 
[[Category:Stub]]
 

Revision as of 00:44, 12 December 2019

This page provides information on how to create files with group read permissions at NCI.

By default, when you create a file under /short of /g/data, the file will be readable by you and the group owning the file. So you don't need to change anything then. But there are a few cases where the default behaviour is different and you may want to change it.

PBS logs

By default, all log files from PBS are only readable by the user owning the file. You can use a PBS option to change this behaviour so your logs are readable by the user and the group owning the file:

 #PBS -W umask=0022

You need to add this option either to the script you submit to the queue (with qsub) or directly to the qsub command line.

Files created by programs

Some programs might impose a different default of the files they create which might not be what you'd like.

You have access to the source code

If you have access to the source code, it is usually possible to modify the default behaviour. What needs to be modified varies depending on the language. Please contact the CMS team if you need help.

You don't have access to the source code

You then need to add a post-processing step, for example in a bash script calling your program. You can change the permissions on the files created by your program, using the chmod command:

chmod g+r <filename>

This command can be run recursively through a directory tree:

chmod -R g+r <directory_name>

Files owned by the wrong group

If you are using compute time from several projects, you may end up with files owned by different projects in the same area. This might not be the ideal situation for you and it is possible to change the group owning the file. Note: NCI's storage accounting depends only on the group owning the files and not the project space under which the files are stored. Before changing the group owning files, please make sure the new group has enough free allocation and you will not cause that group to get over quota.

To change the group ownership of a file:

chgrp <newgroup> <filename>

Or recursively through a directory tree:

chgrp -R <newgroup> <directory_name>