University of Nevada - Reno
Toward Terabit-per-second Networks: Developing Data Transfer Solutions for Next-Generation Research Networks
Abstract
dc:description.abstractResearch networks provide high-speed wide-area network connectivity between research and education institutions to facilitate large-scale data transfers. However, scalability issues of legacy transfer applications (e.g., scp and FTP) and their extensions (e.g., GridFTP and rsync) hinder the effective utilization of these networks. In this dissertation, we propose online optimization algorithms to tune the degree of parallelism for file transfers to maximize transfer throughput while keeping system overhead at a minimum. First, we introduce Falcon that utilizes game theory-inspired novel utility function to evaluate the performance of various parallelism levels such that competing transfers are guaranteed to converge to a fair and stable solution. We assessed the performance of Falcon in isolated and production high-speed networks and found that it can discover optimal transfer parallelism in as little as 20 seconds and outperform the state-of-the-art solutions by more than 2x. On the other hand, Falcon uses the same level of parallelism for network and I/O operations which may result in increased system overhead and unfair resource allocation. To address this issue, we developed modular file transfer architecture, Marlin, that separates I/O and network operations so that parallelism can be independently adjusted for each component. Marlin adopts online gradient descent algorithm to swiftly search the solution space and find the optimal level of parallelism for read, transfer, and write operations. Experimental results collected under various network settings show that Marlin can identify and use a minimum parallelism level for each component, reducing system overhead (e.g., low CPU usage and I/O contention) and improving fairness among competing transfers.
Degree
thesis:*- Level thesis:degree_level
- Doctorate Degree
- Department thesis:degree_department
- Computer Science
- Year dc:date.issued
- 2023
Author and committee
dc:creator, dc:contributor.*- Author dc:creator
-
- Arifuzzaman, Md
- Advisor dc:contributor.advisor
-
- Arslan, Engin
- Committee members dc:contributor.committeemember
-
- Harris, Frederick
- Sengupta, Shamik
- Papachristos, Christos
- Yim, Won
Subjects
dc:subject × 5Rights
dc:rights- Statement dc:rights
-
- Creative Commons Attribution 4.0 United States
- Author(s)
Identifiers
dc:identifier.*- Handle dc:identifier.uri
- http://hdl.handle.net/11714/10820