Back to search

Georgia Institute of Technology

Optimizing HPC I/O performance in New Memory and Storage Hierarchies: A Data-Driven Approach

Abstract

dc:description.abstract

Multi-component HPC workflows face exponential data growth, creating bottlenecks in data and metadata I/O. Achieving optimal end-to-end I/O performance is challenging due to the complex interplay of workload characteristics and technology-specific factors. New I/O stacks have distinct semantics that complicate optimization, requiring empirical evaluation of trade-offs. Our studies provide insights into placement, scheduling, and appropriate I/O APIs for maximizing performance. This thesis contributes data-driven methodologies for managing workflow I/O in next-generation HPC systems. First, we evaluate 18 workflow configurations on a system with heterogeneous memories and demonstrate that scheduling and data allocation can impact end-to-end performance by up to 1.6x. By characterizing different workflow and technology parameters, we identify factors with dominant performance impact and reduce the complexity of workflow scheduler design. Our findings provide a template for data-driven schedulers to maximize benefits of emerging memory technologies for in situ workflows. Second, we focus on HPC metadata I/O management, increasingly important at large scales. We demonstrate using WarpX and ADIOS that at 4,000 nodes, metadata I/O consumes 25% of total I/O time. We investigate improving metadata I/O through DAOS, a storage system designed for complex memory/storage hierarchies. Its multiple APIs can yield up to 25x differences in I/O time. We develop a new ADIOS engine for DAOS metadata I/O achieving 2.3x improvement over DAOS POSIX, with careful configuration enabling elimination of metadata scaling bottlenecks. Third, we develop MetaBench, a lightweight benchmarking suite identifying optimal DAOS metadata I/O strategies for given configurations without expensive full-scale executions. MetaBench simulates metadata workloads using configurable sizes and consumer multiplier values reflecting application reuse patterns. By reducing the configuration space to promising options, MetaBench achieves 10x speedup over brute-force profiling while maintaining practical accuracy. Validation across scales, from ORNL (768 ranks) to Aurora (4,096 ranks), shows optimal strategies vary by system configuration and application characteristics. Different engines are optimal on different systems, highlighting the need for configuration-aware optimization. These studies demonstrate the critical importance of data-driven configuration methods for workflow I/O in next-generation systems. Integration of DAOS engines into ADIOS enables wider community adoption. While focused on ADIOS, the findings apply broadly to HPC I/O middleware like HDF5 and PnetCDF.

Degree

thesis:*
Name thesis:degree_name
Computer Science, PhD
Department thesis:degree_department
Computer Science
Grantor
Georgia Institute of Technology
Year dc:date.issued
2026

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Sarpangala Venkatesh, Ranjan
Advisor dc:contributor.advisor
  • Gavrilovska, Ada
Committee members dc:contributor.committeemember
  • Eisenhauer, Greg
  • Vuduc, Richard
  • Pande, Santosh
  • Klasky, Scott
ORCID person:identifier.orcid

Rights

Language dc:language.iso
English

Identifiers

dc:identifier.*
Handle dc:identifier.uri
https://hdl.handle.net/1853/81680

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
related terms
citation

Sarpangala Venkatesh, Ranjan. Optimizing HPC I/O performance in New Memory and Storage Hierarchies: A Data-Driven Approach. Georgia Institute of Technology, 2026. https://hdl.handle.net/1853/81680