Using the Starfish Storage Management Tool to Automate Research Data Management

Deb McCaffrey, Suhail Ghafoor, Ian Schaeffer

Starfish software is a type of unstructured data management tool designed for PB size storage systems. It creates a database of files with metadata to allow for easier searching and management. The tool is built with storage administrators in mind, so the default metadata collected are things like file size, file owner, etc. At Arizona State University, it was originally purchased to manage the file system on the supercomputer to help find users using the most space and older files that could be deleted. However, it is a versatile tool with many capabilities. It can extract standard metadata in files such as GPS coordinates in images or read custom metadata from sidecar files (think a README file). We are piloting extending Starfish access to research data managers to help automate management tasks. The first pilot user is the data manager for the Center for Evolution and Medicine. Starfish has made day-to-day tasks easier, like optimizing storage costs by archiving datasets that haven’t been used in a while, looking for specific files that are referenced in the ~500TB of data, and estimating data generation per day for budgeting on proposals. Automated metadata extraction from files like DICOM images has also saved time and effort for individual projects.