Storage-training

Revision as of 23:58, 14 March 2019 by P.petrelli (talk | contribs)
Template:Stub This is a stub page and needs expansion

How to manage storage efficiently

The introduction to NCI training presents the basics of the different filesystems at NCI and the organisation along projects used at NCI. This training provides you with the Centre's expectations on your responsibilities in managing your storage footprint in this shared environment. As well as existing tools to help you with this task.

Setup

  • Pre-requisite: Using projects at NCI
  • Pre-requisite: Filesystems at NCI
  • Pre-requisire: Project accounting at NCI
  • Find and join the appropriate CLEx project
  • Know the resource management processes for this project
  • Determining resources required, e.g. running model, analysing data
    • Compute resources (KSU)
    • Storage resources (GB): short term (/short), medium term (/gdata) and archive (mdss)
  • Contact project coordinator with resource request as required 

Monitor resource use

  • Review of the tools available to you
    • NCI tools
    • CLEx tools
    • bash commands

Managing storage

  • Pre-requisite: Filesystems at NCI
  • More on correct use of NCI filesystems
  • Managing file ownership and permissions
  • File compression
  • Prepare data for safe keeping over medium term or longer:
    • what can be deleted
    • documentation
    • tarring
  • Transfer the data:
    • across filesystems
    • from/to remote server
    • from/to tape

Skill set #1: Analysing your storage usage

  • Differences between the filesystems /home, /short, /g/data, /jobfs, mdss
  • Monitoring your usage.

Skill set #2: Tools to help tidy your space

  • Pre-requisite: Shell
  • Linux commands and other tools existing to find the data and tidy the data.

Details of the training

Skill set #1:

  • Review of the filesystems and their characteristics: Storage
    • /home
    • /short
    • /g/data
    • /jobfs
    • /mdss
  • Accounting commands:
    • nci_account -P/-G
    • lquota
    • ncimonitor
    • short_files_report -P/-G
    • gdataX_files_report -P/-G

 

Skill set #2