Back to search

Carleton University

Improving Automatic Tuning of Hadoop and Spark by Analysing Container Performance Metrics

Abstract

dc:description.abstract

This research introduces novel container performance metrics and proves that these metrics are beneficial in the development of automatic tuning systems. Hadoop and Spark show different patterns in the static and dynamic values of container creation rate, container completion rate, container average response time and relative standard deviation of response-time(RSD). By applying five kinds of machine learning algorithms, container creation rate was found to be the most sensitive metric to identify and classify the workload type at an average accuracy of 83%. RSD can be used to detect workload transitions with an average accuracy of 74%. Our research results will decrease tuning overhead and promote the development of automatic tuning systems.

Degree

thesis:*
Name thesis:degree_name
Master of Computer Science (M.C.S.)
Level thesis:degree_level
Master's
Discipline thesis:degree_discipline
Computer Science
Grantor dc:publisher
Carleton University
Year dc:date.issued
2019

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Zhou, Siyu

Subjects

dc:subject × 1

Rights

dc:rights
Statement dc:rights
  • Copyright © 2019 the author(s). Theses may be used for non-commercial research, educational, or related academic purposes only. Such uses include personal study, research, scholarship, and teaching. Theses may only be shared by linking to Carleton University Institutional Repository and no part may be used without proper attribution to the author. No part may be used for commercial purposes directly or indirectly via a for-profit platform; no adaptation or derivative works are permitted without consent from the copyright owner.
Language dc:language.iso
en

Identifiers

dc:identifier.*

Chain of custody

source
Harvested from
Carleton University
Base URL
carleton.scholaris.ca/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Zhou, Siyu. Improving Automatic Tuning of Hadoop and Spark by Analysing Container Performance Metrics. Master's thesis, Carleton University, 2019. https://hdl.handle.net/20.500.14718/40643