Managing and storing data

Keeping your files organised is often easier said than done. A simple file naming convention can help you quickly know what your files contain without having to open them. Common strategies for naming files:

• Use a consistent structure for sorting that makes sense for your project
• Names should be descriptive, but not too long
• Don’t embed important information for files in folders (e.g. sample01test04.txt, NOT sample01/test04.txt)
• Use numerical dates in YYYYMMDD or YYMMDD order
• Avoid spaces, full stops, or special characters in file names.

For more strategies and examples check out Edinburgh’s advice, Stanford’s case study, and Broman & Woo 2017 Data Organization in Spreadsheets.

Managing research inevitably means managing versions of files. Often software has version control built into it. If not, then the simplest strategy is to use the file name to indicate the version. Whole numbers (1, 2, 3) can be used for major changes and decimals for minor changes (1.1, 2.1, 3.2). For collaborative documents you can also include a log at the beginning of the file to record who and what changes have been made to the document and when. Take a look at an example of version control.

For software and code development, GitHub is a popular free option aimed at open collaboration and sharing of code. Surrey also has its own GitLab if you need the same functionality with additional privacy or security. Learn a bit more about getting started with GitHub.

One of the best ways to keep track of versions is by deleting files. If you want to keep some earlier versions of work, then only keep major changes and delete any minor change versions. Deleting unnecessary files is some of the best data management you can do.

For more information, check out Software Carpentry’s video on data management and version control and Edinburgh’s online module. Have evolving datasets with many versions? Datastorr for R might be able to help you develop a workflow for maintaining and distributing successive versions of datasets.

Before beginning it’s worth thinking about what file formats you will be creating and using during your research. You may want to consider:

• The quality, size, and compression needed
• Whether it is a widely used format in your field
• How easily the format can be shared, transformed, or exported
• The long-term viability of the format.

If possible, try to use open and non-proprietary formats or widely used formats to make your data more openly accessible and reduce their risk of becoming obsolete. York has good advice on how to future proof your file formats.

Data repositories often have recommended formats for deposit. If you have a repository in mind, check if they have any deposit requirements. For example, you can see the UK Data Service’s recommended formats for deposit.

If you have collaborators and are concerned about having stronger access controls to your files, then SharePoint can afford you more protection and control over files than OneDrive.

Surrey Drop-off can be used for temporary one-off transfers of large files between you and your collaborators.

If you have sensitive and personally identifiable data, contact IT Services (itservicedesk@surrey.ac.uk) about appropriate storage options.

Local hard drives, portable storage devices, laptops, and tablets are NOT recommended for research data storage. Don’t use third party cloud storage (e.g. Dropbox, Google Drive, etc.) – especially if you have sensitive data. These are not as secure or as protected as University storage.

If you don’t have reliable access to the internet during your research, then regularly sync your working copy with a master copy on university storage. Consider building in a schedule for syncing your data as part of your research workflow.

If using an approved third-party online data collection tool (such as Qualtrics or Gorilla), be sure to move the data to University storage and delete it from the tool as soon as possible, i.e. at the close of collection or end of project.

For more check out data protection pages on privacy, security, and information management.

If you have special storage or computing requirements, IT Services can work with you to meet any technical or security requirements and help you cost it into bids.