Much research has been done in fast communication on clusters and in protocols for supporting software shared memory across them. However, the end performance of applications that were written for the more proven hardware-coherent shared memory is still not very good on these systems. Three major layers of software (and hardware) stand between the end user and parallel performance, each with its own functionality and performance characteristics. They include the communication layer, the software protocol layer that supports the programming model, and the application. These layers provide a useful framework to identify the key remaining limitations and bottlenecks in software shared memory systems across clusters, as well as the key areas where efforts might yield major performance improvements. This paper performs such an integrated study, using this framework of layers, for two types of software shared memory systems: page-based shared virtual memory (SVM) and fine-grained software systems.

For the two system layers (communication and protocol), the focus of this study is on the performance costs of basic operations in the layers rather than on their functionalities (which bears further research). This is possible because their functionalities are now fairly mature. The less mature applications layer is treated through application restructuring. We examine the layers individually and in combination, understanding their implications for the two types of protocols and exposing the synergies among layers.

Our highest level conclusions are as follows. (i) For currently achievable system parameters the fine-grained and SVM approaches are competitive in performance, at least at the scale we examine. (ii) For the fine-grained approach, the most important layer to improve is the communication layer. (iii) For SVM, all three layers are important, and no one or even two of them will do it all alone. (iv) There is substantial synergy among layers, and the dependence on all three rather than a single layer is in some ways a promising sign, especially when that single layer is difficult to improve.