How to use MDSS tape storage at NCI - Revision history

A.heerdegen at 01:43, 3 May 2022

2022-05-03T01:43:02Z

A.heerdegen: A.heerdegen moved page Archiving data to How to use MDSS tape storage at NCI: Better title, reflects usage better

2022-05-03T01:33:12Z

A.heerdegen moved page Archiving data to How to use MDSS tape storage at NCI: Better title, reflects usage better

C.carouge at 01:58, 26 April 2022

2022-04-26T01:58:28Z

Show changes

P.petrelli at 02:16, 4 August 2021

2021-08-04T02:16:33Z

P.petrelli at 02:11, 4 August 2021

2021-08-04T02:11:31Z

P.petrelli at 05:59, 30 July 2021

2021-07-30T05:59:32Z

P.petrelli at 05:53, 30 July 2021

2021-07-30T05:53:01Z

P.petrelli: P.petrelli moved page NCI Guidelines MDSS to Archiving data

2021-07-30T05:34:47Z

P.petrelli moved page NCI Guidelines MDSS to Archiving data

P.petrelli at 05:33, 30 July 2021

2021-07-30T05:33:32Z

P.petrelli at 05:30, 30 July 2021

2021-07-30T05:30:01Z

@@ Line 1: / Line 1: @@
-<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">All universities have their own archives where you can store your data, for more information contact your Library or IT department. Here we will focus on archiving data using the NCI archive storage, for all cases where using a university archive is not applicable.</span></span>
+<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Massdata (Mass Data Storage System, MDSS for short) is the tape storage available at NCI. This kind of storage is intended for long term storage of large files. It is possible to retrieve data from MDSS, so this is a good place to store data that will not be required for some time, for&nbsp;backup,&nbsp;or for&nbsp;archiving. Each project has a directory on the MDSS, the amount of storage allocated depends on the project allocation.</span></span>
 = <span style="font-family:Arial,Helvetica,sans-serif">MDSS proper usage</span> =
-<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">MDSS is designed for medium to long-term archive of large files, this means it is optimised for storing big amounts&nbsp;of data, but not for retrieving this data frequently. This means it is most suitable for:</span></span>
+<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">MDSS is designed for medium to long-term storage of large files, this means it is optimised for storing big amounts&nbsp;of data. This means it is most suitable for:</span></span>
 *<span style="font-size:medium"><span style="font-family:Arial,Helvetica,sans-serif">Files you are required to keep for a long term, like data underlining published datasets, publications, PhD thesis etc.</span></span>

@@ Line 56: / Line 56: @@
   </syntaxhighlight>
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Please note mdss commands work only interactively or with ‘copyq’</span></span>
+<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Please note mdss commands work only interactively or with ‘copyq’. To use it with copyq remember to set the storage flag as</span></span>
 <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">'''Monitoring MDSS&nbsp;usage'''</span></span>

@@ Line 1: / Line 1: @@
 <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">All universities have their own archives where you can store your data, for more information contact your Library or IT department. Here we will focus on archiving data using the NCI archive storage, for all cases where using a university archive is not applicable.</span></span>
@@ Line 14: / Line 12: @@
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">MDSS is suitable for&nbsp;backup of big data projects, like model output which could not be backed up elsewhere.</span></span>
 === '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Preparing your data for mdss</span></span>''' ===

@@ Line 13: / Line 13: @@
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Files that&nbsp;you or someone else are likely to reuse or analyse again in the future but not in the next few months. For example restart files or other model output you are not immediately using should be moved from disk to massdata&nbsp;as soon as possible.</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">MDSS is suitable for&nbsp;backup of big data projects, like model output which could not be backed up elsewhere.</span></span>
 === '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Preparing your data for mdss</span></span>''' ===
@@ Line 25: / Line 26: @@
 <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Useful tools:</span></span>
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;[https://docs.google.com/document/d/1do9dtyzTQ5VY3yFC6YEj-fhAW7OnkA5jFzol1vjRu4w/edit?usp=sharing TAR - to create archives]&nbsp;[[TAR_guidelines|TAR- to create archives]] cheetsheet</span></span>
+<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;[[TAR_guidelines|TAR- to create archives]] cheatsheet</span></span>
 <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;[[NetCDF_Compression_Tools|Compressing tools]]</span></span>

@@ Line 25: / Line 25: @@
 <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Useful tools:</span></span>
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;[https://docs.google.com/document/d/1do9dtyzTQ5VY3yFC6YEj-fhAW7OnkA5jFzol1vjRu4w/edit?usp=sharing TAR - to create archives]</span></span>
+<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;[https://docs.google.com/document/d/1do9dtyzTQ5VY3yFC6YEj-fhAW7OnkA5jFzol1vjRu4w/edit?usp=sharing TAR - to create archives]&nbsp;[[TAR_guidelines|TAR- to create archives]] cheetsheet</span></span>
 <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;[[NetCDF_Compression_Tools|Compressing tools]]</span></span>

← Older revision	Revision as of 01:33, 3 May 2022
(No difference)

← Older revision	Revision as of 05:34, 30 July 2021
(No difference)

← Older revision		Revision as of 05:33, 30 July 2021
Line 1:		Line 1:
		+
		+	==   ==
		+
		+	<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">All universities have their own archives where you can store your data, for more information contact your Library or IT department. Here we will focus on archiving data using the NCI archive storage, for all cases where using a university archive is not applicable.</span></span>

	<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Massdata (Mass Data Storage System, MDSS for short) is the tape storage available at NCI. This kind of storage is intended for long term archiving of large files. Each project has a directory on the MDSS, the amount of storage allocated depends on the project allocation and can be checked using the [[Accounting_at_NCI\|nci_account command.]]</span></span>		<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Massdata (Mass Data Storage System, MDSS for short) is the tape storage available at NCI. This kind of storage is intended for long term archiving of large files. Each project has a directory on the MDSS, the amount of storage allocated depends on the project allocation and can be checked using the [[Accounting_at_NCI\|nci_account command.]]</span></span>

@@ Line 1: / Line 1: @@
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Massdata (Mass Data Storage System, MDSS for short) is the tape storage available at NCI. This kind of storage is intended for long term archiving of large files. Each project has a directory on the MDSS, the amount of storage allocated depends on the project allocation and can be checked using the nci_account command.</span></span>
+<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Massdata (Mass Data Storage System, MDSS for short) is the tape storage available at NCI. This kind of storage is intended for long term archiving of large files. Each project has a directory on the MDSS, the amount of storage allocated depends on the project allocation and can be checked using the [[Accounting_at_NCI|nci_account command.]]</span></span>
 === <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">'''MDSS proper usage'''</span></span> ===
@@ Line 56: / Line 56: @@
 <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">'''Monitoring MDSS&nbsp;usage'''</span></span>
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Unfortunately, there is also not a command to check quickly usage by user-id as for /g/data and /short. The only way to get this information currently is to ask help@nci.org.au, administrators can access this information for any CI of the group.</span></span>
+<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Unfortunately, there is also not a command to check quickly usage by user-id as for /g/data and /short. The only way to get this information currently is to ask help<at>nci.org.au, administrators can access this information for any CI of the group.</span></span>
 === <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">'''Transferring data to and from MDSS'''</span></span> ===
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">NCI supports different commands to work with MDSS as it is explained on their User Guide. The CMS team has also developed a utility called mdssdiff. This utility allows users to compare the contents of the local directory and a directory under /massdata. It will also recursively update the content on the massdata directory to copy the local directory or vice versa.</span></span>
+<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">NCI also supports the netmv and netcp commands to work with MDSS. These commands create a copyq job to transfer multiple files. Files can be automatically tarred and compressed as part of the copy process.</span></span>
 === <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">'''Modifications to MDSS datasets'''</span></span> ===
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Contact NCI at help@nci.org.au if large metadata operations are needed on massdata, as changing ownership, project code, permissions etc. of existing datasets</span></span>
+<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Contact NCI at help<at>nci.org.au if large metadata operations are needed on massdata, as changing ownership, project code, permissions etc. of existing datasets</span></span>
 [[Category:NCI Guidelines]] [[Category:Data induction]]