Back to search

Purdue University

Optimal "Big Data" Aggregation Systems - From Theory to Practical Application

Abstract

dc:description.abstract

The integration of computers into many facets of our lives has made the collection and storage of staggering amounts of data feasible. However, the data on its own is not so useful to us as the analysis and manipulation which allows manageable descriptive information to be extracted. New tools to extract this information from ever growing repositories of data are required. Some of these analyses can take the form of a two phase problem which is easily distributed to take advantage of available computing power. The first phase involves computing some descriptive partial result from some subset of the original data, and the second phase involves aggregating all the partial results to create a combined output. We formalize this compute-aggregate model for a rigorous performance analysis in an effort to minimize the latency of the aggregation phase with minimal intrusive analysis or modification. Based on our model we find an aggregation overlay attribute which highly affects aggregation latency and its dependence on an easily findable trait of aggregation. We rigorously prove the dependence and find optimal overlays for aggregation. We use the proven optima to create simple heuristics and build a system, NOAH, to take advantage of the findings. NOAH can be used by big data analysis systems. We also study an individual problem, top-k matching, to explore the effects of optimizing the computation phase separately from aggregation and create a complete distributed system to fulfill an economically relevant task.

Degree

thesis:*
Name thesis:degree_name
Doctor of Philosophy (PhD)
Level thesis:degree_level
Dissertation
Discipline thesis:degree_discipline
Philosophy

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Culhane, William J, IV
Contributors dc:contributor
  • Patrick Eugster
  • Charles Killian
  • Sonia Fahmy
  • Dongyan Xu

Subjects

dc:subject × 10

Identifiers

dc:identifier.*

Chain of custody

source
Harvested from
Purdue University
Base URL
docs.lib.purdue.edu/do/oai/
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Culhane, William J, IV. Optimal "Big Data" Aggregation Systems - From Theory to Practical Application. Dissertation thesis, https://docs.lib.purdue.edu/open_access_dissertations/216