Accounting at NCI - Revision history

A.heerdegen at 23:52, 4 May 2022

2022-05-04T23:52:48Z

C.carouge at 20:31, 6 October 2021

2021-10-06T20:31:24Z

C.carouge at 20:29, 6 October 2021

2021-10-06T20:29:50Z

C.carouge at 20:29, 6 October 2021

2021-10-06T20:29:22Z

A.heerdegen: Added in ncdu and other info from deleted NCI Guidelines Monitoring page

2021-10-06T00:35:02Z

Added in ncdu and other info from deleted NCI Guidelines Monitoring page

A.heerdegen at 06:12, 13 April 2021

2021-04-13T06:12:36Z

A.heerdegen: Added cost

2021-04-13T05:58:20Z

Added cost

C.carouge: Created page with "{{Stub}} Category:NCI This page describes the different tools available for the accounting of computing and storage resources at NCI. Those tools have been developed by N..."

2020-04-02T04:10:19Z

Created page with "{{Stub}} Category:NCI This page describes the different tools available for the accounting of computing and storage resources at NCI. Those tools have been developed by N..."

New page

{{Stub}}
[[Category:NCI]]

This page describes the different tools available for the accounting of computing and storage resources at NCI. Those tools have been developed by NCI and the CMS team.

= Grafana =
CMS has put in place a Grafana server for visualising a range of accounting statistics for CLEx.
You can access this server using your NCI credentials at: https://accessdev.nci.org.au/grafana/login

Unfortunately, we haven't yet put in place the collection of all the statistics for Gadi. Currently, you can see timeseries of your storage and SU usage on the [https://accessdev.nci.org.au/grafana/d/toeLAYDWz/user-report?orgId=1 User Report].

We will update this section as more statistics become available.

= Computing resources =
== nci_account ==
You can see the current allocation and the usage so far in the quarter for any computing project you are a member of. Using the <code>-v</code> option, you can see the usage per user:
$nci_account -v

== nqstat ===
<code>qstat</code> allows you to monitor your jobs at NCI. <code>nqstat</code> allows you to see the jobs currently submitted:
* for a given project
* for a given queue and project
* for a given user and project

== uqstat ==
The information output by <code>qstat</code> and <code>nqstat</code> isn't always enough. CMS has developed <code>uqstat</code> to output more information by default such as the job efficiency (cpu%), the queueing time, the walltime, the cost in SU etc.

To use this command, please load the <code>nci_scripts</code> module:
module use /g/data/hh5/public/modules
module load nci_scripts

Contrary to what we usually recommend, it should be safe to load this module in your .bashrc file so this module is loaded by default.

$uqstat

The <code>-x</code> option allows you to see the same information on current jobs and jobs finished up to 24 hours previously.

= Storage resources =
== lquota ==
This command will list the quota and usage for all the projects you are a member of on both /scratch and /g/data. It will give usage and quota for both space and number of files.

== nci-files-report ==
This command will list the usage in both space and number of files along different dimensions depending on options.
=== Usage owned by a project per user ===
If you want to know who owns files in your project or where those files are, you should use:
$nci-files-report --group w35 --filesystem gdata

This is probably the most useful options for the project's managers to find who has the most data owned by a project.

=== Usage in a project directory per user ===
If you want to know who owns files in the main directory of one of your projects, you should use:
$nci-files-report --project w35 --filesystem gdata

This is probably the least useful option for this function.

=== Usage per user ===
If you want to know your data footprint across all your projects in a filesystem, you should use:
$nci-files-report --user --filesystem gdata

'''Why are the totals different in <code>nci-files-report</code> and Grafana?'''

CMS is collecting the data for Grafana and we can only scan data that has group read permissions. We ask everyone to put those permissions on if there are no restrictions on your scientific project.

@@ Line 103: / Line 103: @@
 '''Why are the totals different in <code>nci-files-report</code> and Grafana?'''
-CMS is collecting the data for Grafana and we can only scan data that has group read permissions. We ask everyone to put those permissions on if there are no restrictions on your scientific project.
+CMS is collecting the data for Grafana and we can only scan data that has group read permissions. We ask everyone to put those permissions on if there are no restrictions on your scientific project
+== ncfind and nccompres ==
 These utilities are maintained by the CMS team. These allow you to find uncompressed netcdf files and to easily compress them. You'll find a complete documentation on [[NetCDF_Compression_Tools|this wiki page]]
+== du, find and wc ==
 These commands can help you identify where in your area are the large files or which directories contain lots of files. These are Unix commands and all have man pages for information.
@@ Line 131: / Line 129: @@
 Note: the Internet is the perfect source for commands and options variations to find this type of information on your files and directories.
-= ncdu =
+== ncdu ==
 [https://dev.yorhel.nl/ncdu ncdu] is a disk usage analyzer with an ncurses interface. It is designed to find space hogs and aims to be fast, simple and easy to use.

@@ Line 109: / Line 109: @@
 These utilities are maintained by the CMS team. These allow you to find uncompressed netcdf files and to easily compress them. You'll find a complete documentation on [[NetCDF_Compression_Tools|this wiki page]]
 = du, find and wc =
 These commands can help you identify where in your area are the large files or which directories contain lots of files. These are Unix commands and all have man pages for information.

@@ Line 105: / Line 105: @@
 CMS is collecting the data for Grafana and we can only scan data that has group read permissions. We ask everyone to put those permissions on if there are no restrictions on your scientific project.
-----
+= ncfind and nccompres =
 These utilities are maintained by the CMS team. These allow you to find uncompressed netcdf files and to easily compress them. You'll find a complete documentation on [[NetCDF_Compression_Tools|this wiki page]]

@@ Line 42: / Line 42: @@
 The <code>-x</code> option allows you to see the same information on current jobs and jobs finished up to 24 hours previously.
 == qcost ==
@@ Line 63: / Line 64: @@
    e.g. qcost -q normal -n 4 -m 60GB -t 3:00:00
-Note that if no time is specified it defaults to 1 hour (1:00:00). Walltime can be specified as <tt>H:M</tt> or <tt>H:M:S</tt>. Memory must be specified in units of bytes (B), e.g. <tt>160GB</tt>, <code>2000MB</code>.&nbsp;For example:
+Note that if no time is specified it defaults to 1 hour (1:00:00). Walltime can be specified as <tt>H:M</tt> or <tt>H:M:S</tt>. Memory must be specified in units of bytes (B), e.g. <tt>160GB</tt>, <tt>2000MB</tt>.&nbsp;For example:
 <pre>  $ qcost -q normal -n 4 -m 60GB -t 3:00:00
 &nbsp; 90 SU

@@ Line 11: / Line 11: @@
 = Computing resources =
 == nci_account ==
-You can see the current allocation and the usage so far in the quarter for any computing project you are a member of. Using the <code>-v</code> option, you can see the usage per user:
+You can see the current allocation and the usage so far in the quarter for any computing project you are a member of. If the project has a <tt>/g/data</tt> quota it&nbsp;will also report how much of this disk space is utilised.&nbsp;Using the <code>-v</code> option, you can see the usage per user:
   nci_account -v

@@ Line 1: / Line 1: @@
 This page describes the different tools available for the accounting of computing and storage resources at NCI. Those tools have been developed by NCI and the CMS team.
@@ Line 18: / Line 16: @@
 You can see the current allocation and the usage so far in the quarter for any computing project you are a member of. Using the <code>-v</code> option, you can see the usage per user:
-  $nci_account -v
+  nci_account -v
 == nqstat ==
@@ Line 39: / Line 37: @@
 Contrary to what we usually recommend, it should be safe to load this module in your .bashrc file so this module is loaded by default.
-  $uqstat
+  uqstat
 The <code>-x</code> option allows you to see the same information on current jobs and jobs finished up to 24 hours previously.
 == qcost ==
@@ Line 50: / Line 47: @@
 To use this command, please load the <code>nci-scripts</code> module (see above). Usage:
-   $ qcost -h
+   qcost -h
   &nbsp;usage: qcost [-h] -q QUEUE -n NCPUS -m MEM [-t TIME]
@@ Line 65: / Line 62: @@
 Note that if no time is specified it defaults to 1 hour (1:00:00). Walltime can be specified as <tt>H:M</tt> or <tt>H:M:S</tt>. Memory must be specified in units of bytes (B), e.g. <tt>160GB</tt>, <tt>2000MB</tt>.&nbsp;For example:
-<pre>  $ qcost -q normal -n 4 -m 60GB -t 3:00:00
+<pre>   qcost -q normal -n 4 -m 60GB -t 3:00:00
-&nbsp; 90 SU
+SU
-&nbsp; $ qcost -q express -n 4 -m 60GB -t 3:00:00
 SU
 </pre>
@@ Line 85: / Line 83: @@
 If you want to know who owns files in your project or where those files are, you should use:
-  $nci-files-report --group w35 --filesystem gdata
+  nci-files-report --group w35 --filesystem gdata
 This is probably the most useful options for the project's managers to find who has the most data owned by a project.
@@ Line 93: / Line 91: @@
 If you want to know who owns files in the main directory of one of your projects, you should use:
-  $nci-files-report --project w35 --filesystem gdata
+  nci-files-report --project w35 --filesystem gdata
 This is probably the least useful option for this function.
@@ Line 101: / Line 99: @@
 If you want to know your data footprint across all your projects in a filesystem, you should use:
-  $nci-files-report --user --filesystem gdata
+  nci-files-report --user --filesystem gdata
 '''Why are the totals different in <code>nci-files-report</code> and Grafana?'''
 CMS is collecting the data for Grafana and we can only scan data that has group read permissions. We ask everyone to put those permissions on if there are no restrictions on your scientific project.
 [[Category:NCI]]

@@ Line 1: / Line 1: @@
 {{Stub}}
 This page describes the different tools available for the accounting of computing and storage resources at NCI. Those tools have been developed by NCI and the CMS team.
 = Grafana =
 CMS has put in place a Grafana server for visualising a range of accounting statistics for CLEx.
-You can access this server using your NCI credentials at: https://accessdev.nci.org.au/grafana/login
+CMS has put in place a Grafana server for visualising a range of accounting statistics for CLEx. You can access this server using your NCI credentials at: [https://accessdev.nci.org.au/grafana/login https://accessdev.nci.org.au/grafana/login]
 Unfortunately, we haven't yet put in place the collection of all the statistics for Gadi. Currently, you can see timeseries of your storage and SU usage on the [https://accessdev.nci.org.au/grafana/d/toeLAYDWz/user-report?orgId=1 User Report].
@@ Line 13: / Line 13: @@
 = Computing resources =
 == nci_account ==
 You can see the current allocation and the usage so far in the quarter for any computing project you are a member of. Using the <code>-v</code> option, you can see the usage per user:
   $nci_account -v
-== nqstat ===
+== nqstat ==
 <code>qstat</code> allows you to monitor your jobs at NCI. <code>nqstat</code> allows you to see the jobs currently submitted:
 * for a given project
-* for a given queue and project
+*for a given project
-* for a given user and project
+*for a given queue and project
 == uqstat ==
 The information output by <code>qstat</code> and <code>nqstat</code> isn't always enough. CMS has developed <code>uqstat</code> to output more information by default such as the job efficiency (cpu%), the queueing time, the walltime, the cost in SU etc.
-To use this command, please load the <code>nci_scripts</code> module:
+To use this command, please load the <code>nci-scripts</code> module:
   module use /g/data/hh5/public/modules
-  module load nci_scripts
+  module load nci-scripts
 Contrary to what we usually recommend, it should be safe to load this module in your .bashrc file so this module is loaded by default.
@@ Line 34: / Line 41: @@
   $uqstat
 The <code>-x</code> option allows you to see the same information on current jobs and jobs finished up to 24 hours previously.
 = Storage resources =
 == lquota ==
 This command will list the quota and usage for all the projects you are a member of on both /scratch and /g/data. It will give usage and quota for both space and number of files.
 == nci-files-report ==
 This command will list the usage in both space and number of files along different dimensions depending on options.
 === Usage owned by a project per user ===
 If you want to know who owns files in your project or where those files are, you should use:
   $nci-files-report --group w35 --filesystem gdata
@@ Line 49: / Line 89: @@
 === Usage in a project directory per user ===
 If you want to know who owns files in the main directory of one of your projects, you should use:
   $nci-files-report --project w35 --filesystem gdata
@@ Line 55: / Line 97: @@
 === Usage per user ===
 If you want to know your data footprint across all your projects in a filesystem, you should use:
   $nci-files-report --user --filesystem gdata
 '''Why are the totals different in <code>nci-files-report</code> and Grafana?'''
 CMS is collecting the data for Grafana and we can only scan data that has group read permissions. We ask everyone to put those permissions on if there are no restrictions on your scientific project.