Difference between revisions of "Storage-training"
P.petrelli (talk | contribs) (Test of storage training first page based on) |
|||
(4 intermediate revisions by one other user not shown) | |||
Line 1: | Line 1: | ||
− | {{ | + | {{Stub}} |
How to manage storage efficiently | How to manage storage efficiently | ||
The introduction to NCI training presents the basics of the different filesystems at NCI and the organisation along projects used at NCI. This training provides you with the Centre's expectations on your responsibilities in managing your storage footprint in this shared environment. As well as existing tools to help you with this task. | The introduction to NCI training presents the basics of the different filesystems at NCI and the organisation along projects used at NCI. This training provides you with the Centre's expectations on your responsibilities in managing your storage footprint in this shared environment. As well as existing tools to help you with this task. | ||
+ | |||
+ | === Setup === | ||
+ | |||
+ | *Pre-requisite: [https://github.com/coecms/training/blob/master/Introduction_to_NCI.md Using projects at NCI] | ||
+ | *Pre-requisite: [https://github.com/coecms/training/blob/master/Introduction_to_NCI.md Filesystems at NCI] | ||
+ | *Pre-requisire: [https://github.com/coecms/training/blob/master/Introduction_to_NCI.md Project accounting at NCI] | ||
+ | *Find and join the appropriate CLEx project | ||
+ | *Know the resource management processes for this project | ||
+ | *Determining resources required, e.g. running model, analysing data | ||
+ | **Compute resources (KSU) | ||
+ | **Storage resources (GB): short term (<code>/short</code>), medium term (<code>/gdata</code>) and archive (<code>mdss</code>) | ||
+ | *Contact project coordinator with resource request as required | ||
+ | |||
+ | === Monitor resource use === | ||
+ | |||
+ | *Review of the tools available to you | ||
+ | **NCI tools | ||
+ | **CLEx tools | ||
+ | **bash commands | ||
+ | |||
+ | === Managing storage === | ||
+ | |||
+ | *Pre-requisite: [https://github.com/coecms/training/blob/master/Introduction_to_NCI.md Filesystems at NCI] | ||
+ | *More on correct use of NCI filesystems | ||
+ | *Managing file ownership and permissions | ||
+ | *File compression | ||
+ | *Prepare data for safe keeping over medium term or longer: | ||
+ | **what can be deleted | ||
+ | **documentation | ||
+ | **tarring | ||
+ | *Transfer the data: | ||
+ | **across filesystems | ||
+ | **from/to remote server | ||
+ | **from/to tape | ||
=== Skill set #1: Analysing your storage usage === | === Skill set #1: Analysing your storage usage === | ||
Line 20: | Line 54: | ||
=== Skill set #1: === | === Skill set #1: === | ||
− | *Review of the filesystems and their characteristics: | + | *Review of the filesystems and their characteristics: [[Storage|Storage]] |
**/home | **/home | ||
**/short | **/short | ||
Line 33: | Line 67: | ||
**gdataX_files_report -P/-G | **gdataX_files_report -P/-G | ||
− | + | | |
=== Skill set #2 === | === Skill set #2 === | ||
*ncdu | *ncdu | ||
− | *nccompress | + | *nccompress: [https://github.com/aidanheerdegen/nccompress nccompress] |
− | *tar | + | *tar [https://docs.google.com/document/d/1do9dtyzTQ5VY3yFC6YEj-fhAW7OnkA5jFzol1vjRu4w/edit?usp=sharing TAR cheat sheet] |
*chmod / chgrp | *chmod / chgrp | ||
*mdssdiff | *mdssdiff | ||
Line 45: | Line 79: | ||
*ln | *ln | ||
* | * | ||
+ | |||
+ | [[Category: Training]] |
Latest revision as of 00:42, 5 December 2019
Template:Stub This is a stub page and needs expansion
How to manage storage efficiently
The introduction to NCI training presents the basics of the different filesystems at NCI and the organisation along projects used at NCI. This training provides you with the Centre's expectations on your responsibilities in managing your storage footprint in this shared environment. As well as existing tools to help you with this task.
Contents
Setup
- Pre-requisite: Using projects at NCI
- Pre-requisite: Filesystems at NCI
- Pre-requisire: Project accounting at NCI
- Find and join the appropriate CLEx project
- Know the resource management processes for this project
- Determining resources required, e.g. running model, analysing data
- Compute resources (KSU)
- Storage resources (GB): short term (
/short
), medium term (/gdata
) and archive (mdss
)
- Contact project coordinator with resource request as required
Monitor resource use
- Review of the tools available to you
- NCI tools
- CLEx tools
- bash commands
Managing storage
- Pre-requisite: Filesystems at NCI
- More on correct use of NCI filesystems
- Managing file ownership and permissions
- File compression
- Prepare data for safe keeping over medium term or longer:
- what can be deleted
- documentation
- tarring
- Transfer the data:
- across filesystems
- from/to remote server
- from/to tape
Skill set #1: Analysing your storage usage
- Differences between the filesystems /home, /short, /g/data, /jobfs, mdss
- Monitoring your usage.
Skill set #2: Tools to help tidy your space
- Pre-requisite: Shell
- Linux commands and other tools existing to find the data and tidy the data.
Skill set #1:
- Review of the filesystems and their characteristics: Storage
- /home
- /short
- /g/data
- /jobfs
- /mdss
- Accounting commands:
- nci_account -P/-G
- lquota
- ncimonitor
- short_files_report -P/-G
- gdataX_files_report -P/-G
Skill set #2
- ncdu
- nccompress: nccompress
- tar TAR cheat sheet
- chmod / chgrp
- mdssdiff
- rsync
- ln