Carleton University
Improving Automatic Tuning of Hadoop and Spark by Analysing Container Performance Metrics
Abstract
dc:description.abstractThis research introduces novel container performance metrics and proves that these metrics are beneficial in the development of automatic tuning systems. Hadoop and Spark show different patterns in the static and dynamic values of container creation rate, container completion rate, container average response time and relative standard deviation of response-time(RSD). By applying five kinds of machine learning algorithms, container creation rate was found to be the most sensitive metric to identify and classify the workload type at an average accuracy of 83%. RSD can be used to detect workload transitions with an average accuracy of 74%. Our research results will decrease tuning overhead and promote the development of automatic tuning systems.
Degree
thesis:*- Name thesis:degree_name
- Master of Computer Science (M.C.S.)
- Level thesis:degree_level
- Master's
- Discipline thesis:degree_discipline
- Computer Science
- Grantor dc:publisher
- Carleton University
- Year dc:date.issued
- 2019
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Zhou, Siyu
Subjects
dc:subject × 1Rights
dc:rights- Statement dc:rights
-
- Copyright © 2019 the author(s). Theses may be used for non-commercial research, educational, or related academic purposes only. Such uses include personal study, research, scholarship, and teaching. Theses may only be shared by linking to Carleton University Institutional Repository and no part may be used without proper attribution to the author. No part may be used for commercial purposes directly or indirectly via a for-profit platform; no adaptation or derivative works are permitted without consent from the copyright owner.
- Language dc:language.iso
- en