/scratch file expiry

Revision as of 02:52, 26 April 2022 by C.carouge (talk | contribs) (Created page with "On 17th May 2022, NCI is introducing an automatic process to delete files from /scratch that have not been accessed in a long time.   In preparation for this, we share he...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

On 17th May 2022, NCI is introducing an automatic process to delete files from /scratch that have not been accessed in a long time.   In preparation for this, we share here some relevant information.

Preparation

Your best course of action is to prepare before the 17th May so you avoid files you want to keep being placed into quarantine. Below are steps we recommend for you to take:

  • Read the information provided by NCI: [1]
  • Clean up /g/data: delete what you can, archive to tape or outside NCI what you can.
  • Clean up /scratch: delete what you can, move to /g/data or tape or outside NCI if you need long term storage without accessing it.
  • Run “nci-file-expiry list-warnings -p <project> > expiry_warning_<project>.txt” for all the projects you are a member.
  • Check the output of nci-file-expiry. If you identify anything here that is important and at risk of deletion, decide if it should be put on /g/data or tape instead.
  • Run “nci-file-expiry list-warnings” : this will catch any file you own in any project you may have forgotten about.
  • Check the output of nci-file-expiry. If you identify anything here that is important and at risk of deletion, decide if it should be put on /g/data or tape instead.
  • Rethink your data pipeline: do not leave data you are not using anymore in /scratch. Decide what you need to do with it when you stop using it: delete, move to /g/data or outside NCI or archive to tape.
  • Manage your data under /g/data regularly: review your data, archive to tape or outside NCI as necessary.

Additional resources

Archiving

Blog on creating a sustainable data workflow

Description of filesystems at NCI