Back to search

Georgia Institute of Technology

Extreme scale data management in high performance computing

Abstract

dc:description.abstract

Extreme scale data management in high performance computing requires consideration of the end-to-end scientific workflow process. Of particular importance for runtime performance, the write-read cycle must be addressed as a complete unit. Any optimization made to enhance writing performance must consider the subsequent impact on reading performance. Only by addressing the full write-read cycle can scientific productivity be enhanced. The ADIOS middleware developed as part of this thesis provides an API nearly as simple as the standard POSIX interface, but with the flexibilty to choose what transport mechanism(s) to employ at or during runtime. The accompanying BP file format is designed for high performance parallel output with limited coordination overheads while incorporating features to accelerate subsequent use of the output for reading operations. This pair of optimizations of the output mechanism and the output format are done such that they either do not negatively impact or greatly improve subsequent reading performance when compared to popular self-describing file formats. This end-to-end advantage of the ADIOS architecture is further enhanced through techniques to better enable asychronous data transports affording the incorporation of 'in flight' data processing operations and pseudo-transport mechanisms that can trigger workflows or other operations.

Degree

thesis:*
Department dc:contributor.department
Computing
Grantor dc:publisher
Georgia Institute of Technology
Year dc:date.issued
2010

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Lofstead, Gerald Fredrick
Advisor dc:contributor.advisor
  • Schwan, Karsten
Committee members dc:contributor.committeemember
  • Liu, Ling
  • Wolf, Matthew
  • Oldfield, Ron
  • Klasky, Scott

Subjects

dc:subject × 8

Identifiers

dc:identifier.*
Handle dc:identifier.uri
http://hdl.handle.net/1853/37232

Chain of custody

source
Harvested from
Georgia Tech
Base URL
repository.gatech.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Lofstead, Gerald Fredrick. Extreme scale data management in high performance computing. Georgia Institute of Technology, 2010. http://hdl.handle.net/1853/37232