Difference between revisions of "Tips: Custom file permissions at creation"

 
(12 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{color box |lightsteelblue|[[:Template:Stub]] This is a stub page and needs expansion}}
 
This page provides information on working with shared CLEx storage resources at NCI.
 
== NCI projects ==
 
The base unit for the accounting of computational and storage resources is the project, except for your home directory which has limited personal storage. There are 2 types of projects at NCI:
 
* data projects: for published and shared datasets
 
* computational projects: for running programs
 
To have a valid login at NCI, you need to be a member of at least one computational project.
 
  
This means when working on NCI machine, you will mainly work with shared resources. It is essential you learn how to use these resources fairly and efficiently.
+
This page provides information on how to create files with group read permissions at NCI.
=== CLEx projects ===
 
CLEx is managing several computational and data projects at NCI via the CMS team. We will not list the data projects since you do not have write access to any of those. To find datasets published at NCI, please see the information on [[Data_Access|this page]]
 
  
The computational projects are split by Research Program (RP):
+
By default, when you create a file under /short of /g/data, the file will be readable by you and the group owning the file. So you don't need to change anything then. But there are a few cases where the default behaviour is different and you may want to change it.
{| class="wikitable"
 
|-
 
! Project !! RP !! Lead CI
 
|-
 
| v45 || Extra-tropical variability || [mailto:andy.hogg@anu.edu.au Andy Hogg]
 
|-
 
| w35 || Drought || [mailto:c.carouge@unsw.edu.au Claire Carouge]
 
|-
 
| w40 || Extreme Rainfall || [mailto:martin.jucker@unsw.edu.au Martin Jucker]
 
|-
 
| w42 || Extreme Rainfall || [mailto:abhnil.prasad@unsw.edu.au Abhnil Prasad]
 
|-
 
| w48 || Tropical Variability || [mailto:dietmar.dommenget@monash.edu Dietmar Dommenget]
 
|-
 
| w97 || Heatwaves || [mailto:jason.evans@unsw.edu.au Jason Evans]
 
|}
 
Some of the following information applies to all projects at NCI but keep in mind, this page focuses on the CLEx projects, as such some information might only apply to CLEx projects. See the Lead CIs of your other projects for information on those.
 
== Accounting ==
 
NCI applies 2 types of quota on storage:
 
* size
 
* number of files
 
If a project exceeds one of these quota, all jobs from this project are put on hold until the storage is under quota again.
 
  
NCI and CMS provide a number of tools to check quota and help find the biggest users per project.
+
== PBS logs ==
=== Tools for accounting ===
 
==== lquota ====
 
------
 
'''Description:''' an instantaneous view of usage and quota of all projects you are a member of for disk filesystems and of your home directory. Provided by NCI.
 
  
'''Output example:'''
+
By default, all log files from PBS are only readable by the user owning the file. You can use a PBS option to change this behaviour so your logs are readable by the user and the group owning the file:
$ lquota
+
<syntaxhighlight lang="text">#PBS -W umask=0022
-----------------------------------------------------------------------
+
</syntaxhighlight>
            fs      Usage    Quota    Limit  iUsage  iQuota  iLimit
 
-----------------------------------------------------------------------
 
ccc561    home    1.23GB    2.0GB    2.5GB    11235    80000  100000
 
    c25  short  59.05GB  356.0GB  712.0GB    16530  314000  628000
 
    e14  short  28.25TB  33.95TB  67.89TB  2203538  3109000  6218000
 
    k10  short    10.9TB  11.91TB  23.83TB  369385  5775000 11550000
 
    y03  short    3.13TB    5.0TB    9.99TB  1803428  3779000  7558000
 
    ua6  short  310.29GB    1.0MB    1.0MB    3337    1025    1025 Over size limit Over inode limit
 
'''Update frequency:''' instantaneous
 
  
'''More information:''' [https://opus.nci.org.au/display/Help/How+to+check+your+disk+quota+and+usage NCI help page]
+
You need to add this option either to the script you submit to the queue (with qsub) or directly to the qsub command line.
==== ncimonitor ====
 
------
 
'''Description:''' time-series of computational and storage usage per user. Several options available. Provided by CMS.
 
  
'''Output example:'''
+
For PBS log files that have already been created you will need to do the following:
$module use /g/data/hh5/public/modules
+
<pre>chmod g+r <filename></pre>
$module load conda
 
$ncimonitor -P w35
 
[[File:ncimonitor_example.png||800px]]
 
  
'''Update frequency:''' Daily (at night)
+
== HOME directory permissions ==
  
'''More information:''' https://github.com/coecms/ncimonitor
+
By default your home directory is not accessible to anyone but you. If you need others to access files in your home directory you will need to give them access. If they are members of the same group. To find your home directory permissions and group:
==== nci_account -P <project> ====
+
<syntaxhighlight lang="text">$ ls -ld $HOME
------
+
drwx------ 36 usr123 g00 20480 May 26 09:17 /home/001/usr123
'''Description:''' usage and quota information for the specific project required on all systems. Several options available. Provided by NCI.
+
</syntaxhighlight>
  
'''Output example:'''
+
If the other user you wish to give permissions to is also a member of the same group (in this case g00) then use this command:
$nci_account -P w35
+
<syntaxhighlight lang="text">$ chmod g+rx $HOME
Usage Report: Project=w35 Storage Period=2019.7 (01/07/2019-30/09/2019)
+
</syntaxhighlight>
=======================================================================
 
-------------------------------------------------------------------------------------------------
 
System    StoragePt            Grant      Usage      Avail      iGrant      iUsage      iAvail
 
-------------------------------------------------------------------------------------------------
 
dmf      massdata            63.95TB    28.26TB    35.70TB    4729.00K      7.88K    4721.12K
 
global    gdata1a            66.00TB    37.17TB    28.83TB    3883.00K    1503.29K    2379.71K
 
raijin    short              15.00TB    12.45TB      2.55TB    3280.00K    1983.78K    1296.22K
 
-------------------------------------------------------------------------------------------------
 
Total                        144.95TB    77.88TB    67.08TB      11.89M      3.49M      8.40M
 
'''Update frequency:''' About 15 minutes
 
  
'''More information:''' [https://opus.nci.org.au/display/Help/How+to+check+your+disk+quota+and+usage NCI help page]
+
Then the permissions will reflect the change:
==== short_files_report and gdata''N''_files_report ====
+
<syntaxhighlight lang="text">$ ls -ld $HOME
------
+
drwxr-x--- 36 usr123 g00 20480 May 26 09:17 /home/001/usr123
'''Description:''' storage usage on the specified filesystem split by user. 2 available options <code>-G <project></code> and <code>-P <project></code>
+
</syntaxhighlight>
  
'''Output example:'''
+
If the other user is not a member of the correct group, they can either join the project in question, or you can use [[Useful_resources|Access Control Lists (ACLs)]].
''-G option'' gives usage for all files owned by the group specified (not necessarily located in that group directory)
 
$ short_files_report -G w35
 
Usage details for project w35:
 
---------------------------------------------------
 
          folder  user  size    inodes  scan date
 
---------------------------------------------------
 
            z00 hxw599 5996kB        32 2019-07-17
 
            n81 gdv561 54.3GB    111044 2019-07-17
 
            p66 jxs599  218MB        12 2019-07-17
 
            p66 saw562 8943MB    20696 2019-07-17
 
            p66 kxl561 28.0kB        7 2019-07-17
 
''-P option'' gives usage for all files located in the directory specified (not necessarily owned by the same group)
 
$ gdata1a_files_report -P w35
 
Usage details for /g/data1a/w35:
 
---------------------------------------------------
 
          group  user  size    inodes  scan date
 
---------------------------------------------------
 
            c25 dei561  409MB        56 2019-07-17
 
            k10 hxw599 4690MB        7 2019-07-17
 
            n81 jpe561  340kB        31 2019-07-17
 
            ma3 nw3057  737GB      959 2019-07-17
 
            w35 aph502 3810MB        11 2019-07-17
 
  
[[Category:Stub]]
+
== Files created by programs ==
 +
 
 +
Some programs might impose a different default of the files they create which might not be what you'd like.
 +
 
 +
=== You have access to the source code ===
 +
 
 +
If you have access to the source code, it is usually possible to modify the default behaviour. What needs to be modified varies depending on the language. Please contact [mailto:cws_help@nci.org.au the CMS team] if you need help.
 +
 
 +
=== You don't have access to the source code ===
 +
 
 +
You then need to add a post-processing step, for example in a bash script calling your program. You can change the permissions on the files created by your program, using the <code>chmod</code> command:
 +
 
 +
chmod g+r <filename>
 +
 
 +
This command can be run recursively through a directory tree:
 +
 
 +
chmod -R g+r <directory_name>
 +
 
 +
=== Files owned by the wrong group ===
 +
 
 +
If you are using compute time from several projects, you may end up with files owned by different projects in the same area. This might not be the ideal situation for you and it is possible to change the group owning the file. '''Note:''' NCI's storage accounting depends only on the group owning the files and not the project space under which the files are stored. '''Before changing the group owning files, please make sure the new group has enough free allocation and you will not cause that group to get over quota.'''
 +
 
 +
To change the group ownership of a file:
 +
 
 +
chgrp <newgroup> <filename>
 +
 
 +
Or recursively through a directory tree:
 +
 
 +
chgrp -R <newgroup> <directory_name>
 +
 
 +
[[Category:Data]]

Latest revision as of 00:14, 31 May 2022

This page provides information on how to create files with group read permissions at NCI.

By default, when you create a file under /short of /g/data, the file will be readable by you and the group owning the file. So you don't need to change anything then. But there are a few cases where the default behaviour is different and you may want to change it.

PBS logs

By default, all log files from PBS are only readable by the user owning the file. You can use a PBS option to change this behaviour so your logs are readable by the user and the group owning the file:

#PBS -W umask=0022

You need to add this option either to the script you submit to the queue (with qsub) or directly to the qsub command line.

For PBS log files that have already been created you will need to do the following:

chmod g+r <filename>

HOME directory permissions

By default your home directory is not accessible to anyone but you. If you need others to access files in your home directory you will need to give them access. If they are members of the same group. To find your home directory permissions and group:

$ ls -ld $HOME
drwx------ 36 usr123 g00 20480 May 26 09:17 /home/001/usr123

If the other user you wish to give permissions to is also a member of the same group (in this case g00) then use this command:

$ chmod g+rx $HOME

Then the permissions will reflect the change:

$ ls -ld $HOME
drwxr-x--- 36 usr123 g00 20480 May 26 09:17 /home/001/usr123

If the other user is not a member of the correct group, they can either join the project in question, or you can use Access Control Lists (ACLs).

Files created by programs

Some programs might impose a different default of the files they create which might not be what you'd like.

You have access to the source code

If you have access to the source code, it is usually possible to modify the default behaviour. What needs to be modified varies depending on the language. Please contact the CMS team if you need help.

You don't have access to the source code

You then need to add a post-processing step, for example in a bash script calling your program. You can change the permissions on the files created by your program, using the chmod command:

chmod g+r <filename>

This command can be run recursively through a directory tree:

chmod -R g+r <directory_name>

Files owned by the wrong group

If you are using compute time from several projects, you may end up with files owned by different projects in the same area. This might not be the ideal situation for you and it is possible to change the group owning the file. Note: NCI's storage accounting depends only on the group owning the files and not the project space under which the files are stored. Before changing the group owning files, please make sure the new group has enough free allocation and you will not cause that group to get over quota.

To change the group ownership of a file:

chgrp <newgroup> <filename>

Or recursively through a directory tree:

chgrp -R <newgroup> <directory_name>