Efficient Connection-Oriented Communication on High-Performance Networks (Thesis)

Abstract:

Network protocols have traditionally been implemented in the kernel in
order to provide protected communication between processes. As
gigabit network fabrics become available, the kernel protocol stack
becomes a communication performance bottleneck, affecting both latency
and bandwidth. This dissertation investigates how to use protected,
user-level communication mechanisms to support efficient,
connection-oriented communication.

Most user-level communication subsystems are designed for custom
communication models. To support fast user-level communication, new
network interfaces place incoming data directly in user memory, at an
address specified by the sender. This capability enables the
construction of efficient communication subsystems since the network
interface can deliver messages without any software intervention.
Such communication subsystems do not support connection-oriented
communication well for two reasons. First, direct data transfer does
not provide message detection as required by the connection-oriented
model, and second, sender-based communication does not provide the
anonymity of receivers required by the connection-oriented model.

This dissertation studies how to support efficient, user-level,
connection-oriented communication on high-performance networks.
First, the issue of message arrival detection for user-level
communication models is addressed. The traditional approach uses
interrupts to indicate the arrival of new messages, which defeats the
benefit of user-level communication since a protection crossing is
needed with every message. To minimize the interrupt overhead, a
competitive mechanism that combines polling and interrupts to
efficiently detect the arrival of new messages at user-level is
proposed. Second, the mismatch between connection-oriented APIs and
sender-based communication models is addressed. The traditional
approach requires receivers to copy messages to their ultimate
destinations, which are unknown by senders. To enable zero-copy,
connection-oriented protocols, a mechanism that allows the receiver to
dynamically change the destination of a message in a sender-based
model is proposed.

These ideas were integrated into a user-level communication model and
evaluated using a stream sockets implementation on two different
multiprocessor platforms. The results from experiments with
benchmarks and applications show that these techniques provide
significant performance improvements for user-level,
connection-oriented communication.