Back to search

Central Florida

Concentric Layout, A New Scientific Data Layout For Matrix Data Set In Hadoop File System

Abstract

dc:description.abstract

The data generated by scientific simulation, sensor, monitor or optical telescope has increased with dramatic speed. In order to analyze the raw data speed and space efficiently, data preprocess operation is needed to achieve better performance in data analysis phase. Current research shows an increasing tread of adopting MapReduce framework for large scale data processing. However, the data access patterns which generally applied to scientific data set are not supported by current MapReduce framework directly. The gap between the requirement from analytics application and the property of MapReduce framework motivates us to provide support for these data access patterns in MapReduce framework. In our work, we studied the data access patterns in matrix files and proposed a new concentric data layout solution to facilitate matrix data access and analysis in MapReduce framework. Concentric data layout is a data layout which maintains the dimensional property in chunk level. Contrary to the continuous data layout which adopted in current Hadoop framework by default, concentric data layout stores the data from the same sub-matrix into one chunk. This matches well with the matrix operations like computation. The concentric data layout preprocesses the data beforehand, and optimizes the afterward run of MapReduce application. The experiments indicate that the concentric data layout improves the overall performance, reduces the execution time by 38% when the file size is 16 GB, also it relieves the data overhead phenomenon and increases the effective data retrieval rate by 32% on average.

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Cheng, Lu
Contributors dc:contributor
  • Wang, Jun

Subjects

dc:subject × 8

Rights

Language dc:language
English

Identifiers

dc:identifier.*
Identifier
CFE0003537

Chain of custody

source
Harvested from
Central Florida
Base URL
stars.library.ucf.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Cheng, Lu. Concentric Layout, A New Scientific Data Layout For Matrix Data Set In Hadoop File System. https://stars.library.ucf.edu/etd/1601