Back to search

Cornell University

BUILDING NETWORKED SYSTEMS FOR TERABIT ETHERNET

Abstract

dc:description.abstract

Over the last two decades, hardware in datacenters has shown diverging trends. On the one hand, access link bandwidth has increased rapidly, with datacenters now commonly supporting Terabit Ethernet (i.e., Ethernet with speeds above 100Gbps). Modern network hardware is capable of supporting microsecond-scale latency and multi-hundred-gigabit bandwidth. However, on the other hand, the slowdown of Moore’s Law and the end of Dennard scaling have resulted in total compute capacity (the number of CPU cores multiplied by per-core performance) remaining largely stagnant. As a result, network performance bottlenecks have shifted to the host network stacks, which are responsible for processing network packets to and from applications. The first contribution of this dissertation is to build an in-depth understanding the core challenges hindering existing host network stacks from fully leveraging modern network hardware. Our study reveals that the rapid increase in network link bandwidth has made data movement overheads (such as transferring data from NICs to CPUs) a bottleneck for scaling single-core performance. Typically, on the receiver side, after the NIC DMAs data to memory, the limited memory bandwidth leads to poor CPU efficiency, as CPUs must stall while waiting for data to be loaded from memory into CPU registers. We find out existing optimization techniques like DDIO, which enables NICs to directly read/write data from/to CPU caches, are unable to improve CPU efficiency. This is because the increase in bandwidth-delay products has outpaced the increase in cache sizes, leading to high cache miss rates and poor CPU efficiency. With today's network stacks, multiple cores are needed to fully exploit the capabilities of Terabit network hardware. However, our study shows that using multiple cores, while saturating the link bandwidth, leads to even worse CPU efficiency compared to the single-core case. This is because host resources like cache and access link bandwidth are contended among different cores/applications. The second contribution of this dissertation is to introduce NetChannel—a new network stack architecture that enables host network stacks to leverage network hardware without requiring application modifications. NetChannel disaggregates network stacks into multiple loosely-coupled layers, allowing each layer to scale and schedule across multiple cores independently. Using an end-to-end NetChannel realization within the Linux network stack, we demonstrate that NetChannel enables new operating points—(1) enabling a single application thread to saturate multi-hundred-gigabit access link bandwidth; (2) enabling near-linear scalability for small message processing with an increasing number of cores, independent of the number of application threads; and, (3) enabling isolation of latency-sensitive applications, allowing them to maintain μs-scale tail latency even when competing with throughput-bound applications operating at near-line rate. This thesis leaves open several interesting directions of future research: 1) improving CPU efficiency by reducing both data movement and CPU processing overheads; 2) extending NetChannel to function in more realistic scenarios, allowing us to fully realize its potential; and 3) extending the study to understand network stack overheads, not only in terms of CPU efficiency and throughput but also network latency, as achieving low latency is also crucial for applications.

Degree

thesis:*
Name thesis:degree_name
Ph. D., Computer Science
Level thesis:degree_level
Doctor of Philosophy
Discipline thesis:degree_discipline
Computer Science
Grantor
Cornell University
Year dc:date.issued
2024

Author and committee

dc:creator, dc:contributor.*
Author dc:creator
  • Cai, Qizhe
Chair dc:contributor.chair
  • Agarwal, Rachit
Committee members dc:contributor.committeemember
  • Shmoys, David
  • Sampson, Adrian

Subjects

dc:subject × 4

Rights

Language dc:language.iso
en

Identifiers

dc:identifier.*
Dc Identifier Bibid
16612018
Dc Identifier Other
ProQuest Submission ID: 14532
ProQuest Publication ID: 31487552

Chain of custody

source
Harvested from
Cornell University
Base URL
ecommons.cornell.edu/server/oai/request
Last updated
2026-11-19
Source record
OAI-PMH GetRecord
citation

Cai, Qizhe. BUILDING NETWORKED SYSTEMS FOR TERABIT ETHERNET. Doctor of Philosophy thesis, Cornell University, 2024. https://hdl.handle.net/1813/116404