HPC-Unity Project

Goals

The objective of this research is to design and evaluate a new distributed data storage and analysis paradigm that unifies the traditionally distinct application data views of memory and file storage into a single scalable environment for sharing datasets composed of resilient data objects. This permits hiding the complexity of moving and placing data within multi-tier storage hierarchies. It enables efficient data sharing within and between applications (e.g., coupled applications and co-visualization). It explicitly incorporates semantics for data resiliency and durability. The unified approach helps to expose design tradeoffs inherent in data movement and simplifies data contracts.

Our research hypothesis: A unified interface that removes the dichotomy of memory and file-based storage provides improved resilience, improved sharing semantics, and reduced software development complexity for extreme-scale environments.

Our questions to be addressed include:

• What data abstractions are necessary to support the various data decomposition approaches currently used in scientific simulations, analysis, and visualization workloads? How do these abstractions map into existing I/O middleware such as ADIOS and HDF5?
• Can novel data sharing and resilience semantics be leveraged by individual and coupled applications to simplify software development while still meeting expectations for performance and data durability?
• What are the benefits and costs of hiding data management within multi-tier storage from applications?
• What are the potential benefits to checkpoint-restart?

Funding for the HPC-Unity Project is provided by a grant from
the U.S. Department of Energy Office of Science.