Best practices for directories and files - Revision history

P.petrelli at 01:58, 13 September 2021

2021-09-13T01:58:25Z

P.petrelli at 01:57, 13 September 2021

2021-09-13T01:57:44Z

P.petrelli at 00:32, 9 July 2021

2021-07-09T00:32:09Z

P.petrelli at 00:31, 9 July 2021

2021-07-09T00:31:47Z

P.petrelli at 00:31, 9 July 2021

2021-07-09T00:31:18Z

P.petrelli at 00:15, 9 July 2021

2021-07-09T00:15:10Z

P.petrelli at 23:52, 8 July 2021

2021-07-08T23:52:53Z

P.petrelli at 23:51, 8 July 2021

2021-07-08T23:51:12Z

P.petrelli at 02:07, 1 July 2021

2021-07-01T02:07:51Z

P.petrelli at 07:30, 30 June 2021

2021-06-30T07:30:08Z

@@ Line 45: / Line 45: @@
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">avoid having a&nbsp;large&nbsp;number of files in a single directory&nbsp;</span></span><span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">… but also an excessive number of directories with one file each</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">always include file extension, some software can recognise files from their header, but this is not always the case</span></span>
 === '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Online Resources</span></span>''' ===
@@ Line 50: / Line 51: @@
 <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">We partially based this page on&nbsp;the resources listed below, we recommend to check them for more insight and advice.</span></span>
-*<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">[https://www.earthdatascience.org/courses/intro-to-earth-data-science/open-reproducible-science/get-started-open-reproducible-science/best-practices-for-organizing-open-reproducible-science/ Best practice to organise your data] - part of an Open reproducible scoenc course from the University of Colorado&nbsp;</span></span>
+*<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">[https://www.earthdatascience.org/courses/intro-to-earth-data-science/open-reproducible-science/get-started-open-reproducible-science/best-practices-for-organizing-open-reproducible-science/ Best practice to organise your data] - part of an Open reproducible science course from the University of Colorado&nbsp;</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">[https://youtu.be/3MEJ38BO6Mo Software Carpentry video covering DRS best practices]</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">[https://stanford.box.com/shared/static/yl5a04udc7hff6a61rc0egmed8xol5yd.pdf Best file naming practice handout (pdf) from Standford University]&nbsp;</span></span>
 [[Category:Data induction]]

@@ Line 35: / Line 35: @@
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">review at regular intervals what you are keeping, what needs to be removed and how things are organised</span></span>
-=== '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Tips to for machine-readable files</span></span>''' ===
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">avoid special characters: ~&nbsp;! @ # $&nbsp;% ^ & * ( ) `&nbsp;; < >&nbsp;? , [ ] { } ‘ “</span></span>
@@ Line 43: / Line 44: @@
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">try to avoid overly long names, for a single&nbsp;file, directory&nbsp;keep it under&nbsp;255 characters, for paths 30000.</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">avoid having a&nbsp;large&nbsp;number of files in a single directory&nbsp;</span></span><span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">… but also an excessive number of directories with one file each</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">always include file extension, some software can recognise files from their header, but this is not always the case</span></span>
 === '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Online Resources</span></span>''' ===

@@ Line 24: / Line 24: @@
 === '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">DRS</span></span>''' ===
-[[File:Example of directory structure.png|800px]]
+[[File:Example of directory structure.png|800px|Example of directory structure.png]]
 <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">The figure above shows an example of an organised working directory for a model output, things to consider:</span></span>
@@ Line 44: / Line 44: @@
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">avoid having a&nbsp;large&nbsp;number of files in a single directory&nbsp;</span></span><span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">… but also an excessive number of directories with one file each</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">always include file extension, some software can recognise files from their header, but this is not always the case</span></span>
 === '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Online Resources</span></span>''' ===

@@ Line 44: / Line 44: @@
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">avoid having a&nbsp;large&nbsp;number of files in a single directory&nbsp;</span></span><span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">… but also an excessive number of directories with one file each</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">always include file extension, some software can recognise files from their header, but this is not always the case</span></span>
 === '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Online Resources</span></span>''' ===
@@ Line 49: / Line 50: @@
 <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">We partially based this page on&nbsp;the resources listed below, we recommend to check them for more insight and advice.</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">[https://www.earthdatascience.org/courses/intro-to-earth-data-science/open-reproducible-science/get-started-open-reproducible-science/best-practices-for-organizing-open-reproducible-science/ Best practice to organise your data] - part of an Open reproducible scoenc course from the University of Colorado&nbsp;</span></span>
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">[https://www.earthdatascience.org/courses/intro-to-earth-data-science/open-reproducible-science/get-started-open-reproducible-science/best-practices-for-organizing-open-reproducible-science/ Best practice to organise your data] - part of an Open reproducible scoenc course from the University of Colorado&nbsp;</span></span>
+*<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">[https://youtu.be/3MEJ38BO6Mo Software Carpentry video covering DRS best practices]</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">[https://stanford.box.com/shared/static/yl5a04udc7hff6a61rc0egmed8xol5yd.pdf Best file naming practice handout (pdf) from Standford University]&nbsp;</span></span>
 [[Category:Data induction]]

@@ Line 1: / Line 1: @@
 <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">The names you choose for files and directories and generally the way you organise your data, i.e. your directory structure (DRS) can help navigating the data and provide extra information, avoid confusion and the user ending up accessing the wrong data. As for many other cases the best file organisation will depend on the specific research project and the actual server where the data is stored.&nbsp;Here we are just listing a few guidelines and tips to help you decide.&nbsp;&nbsp;</span></span>
@@ Line 26: / Line 24: @@
 === '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">DRS</span></span>''' ===
-&nbsp;
+[[File:Example of directory structure.png|800px]]
 <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">The figure above shows an example of an organised working directory for a model output, things to consider:</span></span>
@@ Line 34: / Line 32: @@
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">think of the way you would access these directories in a code, as an example having the variable directories using exactly the same name as the actual variable</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">make sure your code is separate from your data, you want to be able to use something like git to version control it and possibly GitHub to back it up easily</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">review at regular intervals what you are keeping, what needs to be removed and how things are organised</span></span>
@@ Line 48: / Line 47: @@
 === '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Online Resources</span></span>''' ===
-*[https://www.earthdatascience.org/courses/intro-to-earth-data-science/open-reproducible-science/get-started-open-reproducible-science/best-practices-for-organizing-open-reproducible-science/ Best practice to organise your data] - part of an Open reproducible scoenc course from the University of Colorado&nbsp;
+<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">We partially based this page on&nbsp;the resources listed below, we recommend to check them for more insight and advice.</span></span>
-*&nbsp;
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">[https://libguides.princeton.edu/c.php?g=102546&p=930626 https://libguides.princeton.edu/c.php?g=102546&p=930626] includes link to video from software carpentry data management&nbsp;&nbsp;</span></span>
 *&nbsp;
 &nbsp;
 [[Category:Data induction]]

@@ Line 26: / Line 26: @@
 === '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">DRS</span></span>''' ===
-[[File:Example of directory structure.png|800px|Example of a DRS for a model project]]
+&nbsp;
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Above there is an example of an organised working directory for a model output, things to consider:</span></span>
+<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">The figure above shows an example of an organised working directory for a model output, things to consider:</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">try to organise files in directories&nbsp;based on type and how you process them</span></span>
@@ Line 48: / Line 48: @@
 === '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Online Resources</span></span>''' ===
+*[https://www.earthdatascience.org/courses/intro-to-earth-data-science/open-reproducible-science/get-started-open-reproducible-science/best-practices-for-organizing-open-reproducible-science/ Best practice to organise your data] - part of an Open reproducible scoenc course from the University of Colorado&nbsp;
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">&nbsp; [https://www.earthdatascience.org/courses/intro-to-earth-data-science/open-reproducible-science/get-started-open-reproducible-science/best-practices-for-organizing-open-reproducible-science/ https://www.earthdatascience.org/courses/intro-to-earth-data-science/open-reproducible-science/get-started-open-reproducible-science/best-practices-for-organizing-open-reproducible-science/] &nbsp;</span></span>
+*&nbsp;
+<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">[https://libguides.princeton.edu/c.php?g=102546&p=930626 https://libguides.princeton.edu/c.php?g=102546&p=930626] includes link to video from software carpentry data management&nbsp;&nbsp;</span></span>
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">[https://libguides.princeton.edu/c.php?g=102546&p=930626 https://libguides.princeton.edu/c.php?g=102546&p=930626] incldues video &nbsp;</span></span>
 <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">[https://library.stanford.edu/research/data-management-services/data-best-practices/best-practices-file-naming https://library.stanford.edu/research/data-management-services/data-best-practices/best-practices-file-naming] - you can download the same as a pdf</span></span>
 &nbsp;
 [[Category:Data induction]]

@@ Line 15: / Line 15: @@
 <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">You can use your filenames to include information here is some you can consider:</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">project, simulation and/or&nbsp;experiment acronyms, you might have to use a combination&nbsp;of them.</span></span>
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">project, simulation and/or&nbsp;experiment acronyms, you might have to use a combination&nbsp;of them.</span></span>
+*<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">spatial coverage: the region or coordinates range covered by the data, could also be a specific domain for climate model data, i.e. ocean, land etc.</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">grid: could be either a grid label or spatial resolution</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">temporal coverage: a specific year/date or a temporal range</span></span>
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">spatial coverage: the region or coordinates range covered by the data, could also be a specific domain for climate model data, i.e. ocean, land etc.</span></span>
+*<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">temporal frequency: monthly, daily etc</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">type of data: again this depends on context, if the same directory contains data from different instrumentations it is important to specify the instrument in the name.&nbsp;For coupled model output this could be the model component, if you are using one file per variable&nbsp;the variable name.</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">version: this is really important if you are sharing the data even if only 1 version exists at the time</span></span>
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">grid: could be either a grid label or spatial resolution</span></span>
+*<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">correct file extension</span></span>
 === '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">DRS</span></span>''' ===
-[[File:Example of directory structure.png|Example of a DRS for a model project]]
+[[File:Example of directory structure.png|800px|Example of a DRS for a model project]]
 <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Above there is an example of an organised working directory for a model output, things to consider:</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">try to organise files in directories&nbsp;based on type and how you process them</span></span>
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">try to organise files in directories&nbsp;based on type and how you process them</span></span>
+*<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">for the final output think also about&nbsp;how other might use them:&nbsp;are they going to be used for analysis or they could be used as forcing or restart files for a model?&nbsp;</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">think of the way you would access these directories in a code, as an example having the variable directories using exactly the same name as the actual variable</span></span>
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">make sure your code is separate from your data, you want to be able to use something like git to version control it and possibly GitHub to back it up easily</span></span>
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">for the final output think also about&nbsp;how other might use them:&nbsp;are they going to be used for analysis or they could be used as forcing or restart files for a model?&nbsp;</span></span>
+*<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">review at regular intervals what you are keeping, what needs to be removed and how things are organised</span></span>
 === '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Tips to for machine-readable files</span></span>''' ===
@@ Line 72: / Line 46: @@
 *<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">always include file extension, some software can recognise files from their header, but this is not always the case</span></span>
-=== '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Resources</span></span>''' ===
+=== '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Online Resources</span></span>''' ===
 &nbsp;
 [[Category:Data induction]]

@@ Line 2: / Line 2: @@
 {{Template:Working_on}}
-<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">The names you choose for files and directories and generally the way you organise your data, ie your directory structure (DRS) can help navigating the data and provide extra information, avoid confusion and the user ending up accessing the wrong data.</span></span> <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">As for many other cases the best file organisation will depend on the your specific research project and the actual server where the data is stored.</span></span><span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">&nbsp;here we are just listing a few guidelines and tips to help you decide.</span></span> &nbsp;
+<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">The names you choose for files and directories and generally the way you organise your data, i.e. your directory structure (DRS) can help navigating the data and provide extra information, avoid confusion and the user ending up accessing the wrong data. As for many other cases the best file organisation will depend on the specific research project and the actual server where the data is stored.&nbsp;Here we are just listing a few guidelines and tips to help you decide.&nbsp;&nbsp;</span></span>
 === '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">General considerations</span></span>''' ===
@@ Line 8: / Line 8: @@
 #&nbsp;<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Familiarise yourself with the [[Storage|storage system]], make sure you are storing the files in the most&nbsp;appropriate place, get to know if the storage is backed up or not, what is your allocation, and also&nbsp;what&nbsp;rules or best practices apply.</span></span>
 #<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Take&nbsp;into account how yourself or others might want to use the data, this is particularly important when deciding the DRS but also how to divide data across files for big dataset as model output. Doing so at the start of the project will spare you lots of time you might otherwise spend re-processing all your files.</span></span>
-#<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Be consistent, this applies both to the organisation and the naming, consistency is essential for the data to be machine-readable, ie. data which is easy to access by coding. In fact use community standards and/or controlled vocabularies wherever possible.</span></span>
+#<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Be consistent, this applies both to the organisation and the naming, consistency is essential for the data to be machine-readable, i.e. data which is easy to access by coding. In fact, use community standards and/or controlled vocabularies wherever possible.</span></span>
 #<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Consider adding a [[Dataset_readme_file_template|readme file]] in the main directory (we always do that for data we publish), including an explanation of the DRS and the naming conventions, abbreviation and/or codes you used. If you used standards and controlled vocabularies&nbsp;all you have to do is to include a link to them. &nbsp;</span></span> &nbsp;
@@ Line 15: / Line 15: @@
 <span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">You can use your filenames to include information here is some you can consider:</span></span>
-*project, simulation and/or&nbsp;experiment acronyms, you might have to use a combinations of them.
-*spatial coverage: the region or coordinates range covered by the data, could also be a specific domain for climate model data, ie.e ocean, land etc.
+<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">project, simulation and/or&nbsp;experiment acronyms, you might have to use a combination&nbsp;of them.</span></span>
 *grid: could be either a grid label or spatial resolution
 *temporal coverage: a specific year/date or a temporal range
-*temporal frequency: monthly, daily etc
+<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">spatial coverage: the region or coordinates range covered by the data, could also be a specific domain for climate model data, i.e. ocean, land etc.</span></span>
-*type of data: again this depends on context, if the same directory contains data from different instrumentations it is important to specify the instrument in the name, for coupled model output again this could be the model component, if you are using one file per variable then you should have the variable name
-*version: this is really important if you are sharing the data even if only 1 verison exists at the time
-*correct file extension
+<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">grid: could be either a grid label or spatial resolution</span></span>
 === '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">DRS</span></span>''' ===
-&nbsp;
+[[File:Example of directory structure.png|Example of a DRS for a model project]]
 === '''<span style="font-size:medium;"><span style="font-family:Arial,Helvetica,sans-serif;">Tips to for machine-readable files</span></span>''' ===

← Older revision		Revision as of 02:07, 1 July 2021
Line 46:		Line 46:
	*[https://library.stanford.edu/research/data-management-services/data-best-practices/best-practices-file-naming https://library.stanford.edu/research/data-management-services/data-best-practices/best-practices-file-naming] - you can download the same as a pdf		*[https://library.stanford.edu/research/data-management-services/data-best-practices/best-practices-file-naming https://library.stanford.edu/research/data-management-services/data-best-practices/best-practices-file-naming] - you can download the same as a pdf

−	~~[[Category:Data]]~~[[Category:Data induction]]	+
		+
		+	[[Category:Data induction]]