Limits to the Performance of Software Shared Memory: A Layered Approach

Abstract:

Much research has been done in fast communication on clusters and in
protocols for supporting software shared memory across them. However,
the end performance of applications that were written for the more
proven hardware-coherent shared memory is still not very good on these
systems. Three major layers of software (and hardware) stand between
the end user and parallel performance, each with its own
functionality and performance characteristics. They include the
communication layer, the software protocol layer that supports the
programming model, and the application. These layers provide a
useful framework to identify the key remaining limitations and bottlenecks
in software shared memory systems across clusters, as well as the key
areas where efforts might yield major performance improvements.
This paper performs such an integrated study, using this framework of
layers, for two types of software shared memory systems: page-based shared
virtual memory (SVM) and fine-grained software systems.

For the two system layers (communication and protocol), the focus of
this study is on the performance costs of basic operations in the
layers rather than on their functionalities (which bears
further research). This is possible because their functionalities are
now fairly mature. The less mature applications
layer is treated through application restructuring. We examine the
layers individually and in combination, understanding their
implications for the two types of protocols and exposing the
synergies among layers.

Our highest level conclusions are as follows. (i) For currently
achievable system parameters the fine-grained and SVM approaches are
competitive in performance, at least at the scale we examine. (ii) For
the fine-grained approach, the most important layer to improve is the
communication layer. (iii) For SVM, all three layers are important,
and no one or even two of them will do it all alone. (iv) There is
substantial synergy among layers, and the dependence on all three
rather than a single layer is in some ways a promising sign,
especially when that single layer is difficult to improve.