HPC-Unity Project

Overview

The objective of this research is to design and evaluate a new distributed data storage paradigm that unifies the traditionally distinct application views of memory- and file-based data storage into a single scalable and resilient data environment.  As we move into the exascale computing era, four challenges drive the need for a unified data environment: 

1. exascale computing systems will contain unprecedented concurrency, both across all nodes of a system and within individual computing nodes due to the emergence of many-core processors and accelerators,
   
2. exascale computing environments will leverage multi-tier memory and storage hierarchies that dramatically increase the complexity of managing and efficiently accessing data,
3. exascale computing environments will have limited power budgets that require efficient, power-aware data movement, and
4. exascale computing systems will experience a nearly constant stream of hardware faults due the sheer number of components in the systems. 

Together, these exascale challenges demand a radically new data environment that has two key qualities: (1) explicit application-level semantics for data sharing and persistence among large collections of concurrent data users, and (2) intelligent management of data placement, movement, and durability within multi-tier memory and storage systems.  A unified data environment frees applications from the complexity of directly placing and moving data within multi-tier storage hierarchies, while still meeting application-prescribed requirements for data access performance, efficient data sharing within and between applications (e.g., coupled applications, analytics and co-visualization), and data durability.

Funding for the HPC-Unity Project is provided by a grant from
the U.S. Department of Energy Office of Science.