Difference between revisions of "Storage-training"

(Test of storage training first page based on)
 
 
(4 intermediate revisions by one other user not shown)
Line 1: Line 1:
  
{{Expand}}
+
{{Stub}}
  
 
How to manage storage efficiently
 
How to manage storage efficiently
  
 
The introduction to NCI training presents the basics of the different filesystems at NCI and the organisation along projects used at NCI. This training provides you with the Centre's expectations on your responsibilities in managing your storage footprint in this shared environment. As well as existing tools to help you with this task.
 
The introduction to NCI training presents the basics of the different filesystems at NCI and the organisation along projects used at NCI. This training provides you with the Centre's expectations on your responsibilities in managing your storage footprint in this shared environment. As well as existing tools to help you with this task.
 +
 +
=== Setup ===
 +
 +
*Pre-requisite: [https://github.com/coecms/training/blob/master/Introduction_to_NCI.md Using projects at NCI]
 +
*Pre-requisite: [https://github.com/coecms/training/blob/master/Introduction_to_NCI.md Filesystems at NCI]
 +
*Pre-requisire: [https://github.com/coecms/training/blob/master/Introduction_to_NCI.md Project accounting at NCI]
 +
*Find and join the appropriate CLEx project
 +
*Know the resource management processes for this project
 +
*Determining resources required, e.g. running model, analysing data
 +
**Compute resources (KSU)
 +
**Storage resources (GB): short term (<code>/short</code>), medium term (<code>/gdata</code>) and archive (<code>mdss</code>) 
 +
*Contact project coordinator with resource request as required&nbsp;
 +
 +
=== Monitor resource use ===
 +
 +
*Review of the tools available to you
 +
**NCI tools
 +
**CLEx tools
 +
**bash commands 
 +
 +
=== Managing storage ===
 +
 +
*Pre-requisite:&nbsp;[https://github.com/coecms/training/blob/master/Introduction_to_NCI.md Filesystems at NCI]
 +
*More on correct use of NCI filesystems
 +
*Managing file ownership and permissions
 +
*File compression
 +
*Prepare data for safe keeping over medium term or longer:
 +
**what can be deleted
 +
**documentation
 +
**tarring 
 +
*Transfer the data:
 +
**across filesystems
 +
**from/to remote server
 +
**from/to tape 
  
 
=== Skill set #1: Analysing your storage usage ===
 
=== Skill set #1: Analysing your storage usage ===
Line 20: Line 54:
 
=== Skill set #1: ===
 
=== Skill set #1: ===
  
*Review of the filesystems and their characteristics:  
+
*Review of the filesystems and their characteristics:&nbsp;[[Storage|Storage]]
 
**/home  
 
**/home  
 
**/short  
 
**/short  
Line 33: Line 67:
 
**gdataX_files_report -P/-G   
 
**gdataX_files_report -P/-G   
  
[https://github.com/coecms/training/blob/master/??? Link to training]
+
&nbsp;
  
 
=== Skill set #2 ===
 
=== Skill set #2 ===
  
 
*ncdu  
 
*ncdu  
*nccompress  
+
*nccompress:&nbsp;[https://github.com/aidanheerdegen/nccompress nccompress]&nbsp;
*tar  
+
*tar&nbsp;&nbsp;[https://docs.google.com/document/d/1do9dtyzTQ5VY3yFC6YEj-fhAW7OnkA5jFzol1vjRu4w/edit?usp=sharing TAR cheat sheet]
 
*chmod / chgrp  
 
*chmod / chgrp  
 
*mdssdiff  
 
*mdssdiff  
Line 45: Line 79:
 
*ln  
 
*ln  
 
*&nbsp;
 
*&nbsp;
 +
 +
[[Category: Training]]

Latest revision as of 00:42, 5 December 2019

Template:Stub This is a stub page and needs expansion

How to manage storage efficiently

The introduction to NCI training presents the basics of the different filesystems at NCI and the organisation along projects used at NCI. This training provides you with the Centre's expectations on your responsibilities in managing your storage footprint in this shared environment. As well as existing tools to help you with this task.

Setup

  • Pre-requisite: Using projects at NCI
  • Pre-requisite: Filesystems at NCI
  • Pre-requisire: Project accounting at NCI
  • Find and join the appropriate CLEx project
  • Know the resource management processes for this project
  • Determining resources required, e.g. running model, analysing data
    • Compute resources (KSU)
    • Storage resources (GB): short term (/short), medium term (/gdata) and archive (mdss)
  • Contact project coordinator with resource request as required 

Monitor resource use

  • Review of the tools available to you
    • NCI tools
    • CLEx tools
    • bash commands

Managing storage

  • Pre-requisite: Filesystems at NCI
  • More on correct use of NCI filesystems
  • Managing file ownership and permissions
  • File compression
  • Prepare data for safe keeping over medium term or longer:
    • what can be deleted
    • documentation
    • tarring
  • Transfer the data:
    • across filesystems
    • from/to remote server
    • from/to tape

Skill set #1: Analysing your storage usage

  • Differences between the filesystems /home, /short, /g/data, /jobfs, mdss
  • Monitoring your usage.

Skill set #2: Tools to help tidy your space

  • Pre-requisite: Shell
  • Linux commands and other tools existing to find the data and tidy the data.

Details of the training

Skill set #1:

  • Review of the filesystems and their characteristics: Storage
    • /home
    • /short
    • /g/data
    • /jobfs
    • /mdss
  • Accounting commands:
    • nci_account -P/-G
    • lquota
    • ncimonitor
    • short_files_report -P/-G
    • gdataX_files_report -P/-G

 

Skill set #2