Other configuration considerations: Optimal MTU size
Based on past experience, it is known that larger MTU sizes typically provide better overall performance than smaller MTU sizes. Larger MTU sizes reduce the number of packets needed to transfer a large amount of data. Much of the overhead in sending data across a network appears to be generated from the setup and processing of each packet. In general, using fewer packets typically reduces the overhead and improves overall performance.
To illustrate this point further, refer to the following graph:
This graph clearly shows that HiperSockets performance steadily improves as the MTU size increases. The results using MTU 56K are the baseline, and the relative differences measured when using the other MTU size choices are shown. As stated earlier, larger MTU sizes tend to provide the greatest benefit when the datagram or payload is larger, which is seen with all the workload tests that use a 30k datagram.
Unlike standard network adapters, datagrams smaller than MTU size (as seen with HiperSockets using 8K MTU) deliver a little less throughput than the larger MTU configurations. The cause for this is not yet understood, but it may be related to the HiperSockets firmware implementation.
Based on the observed behavior and results, a HiperSockets configuration using a 56K MTU size was chosen for the comparison to the OSA configuration.
Similar observations are found when using the OSA configuration. Larger MTU sizes tend to deliver better performance on large datagram workloads, while imparting no significant impacts when using small datagrams. Typically a large MTU size on a physical network uses jumbo frames that are 9K bytes in size. However, in reality 9K uses three 4K pages of memory. To save one page of memory, the MTU size was lowered to 8K. For these workload tests, an MTU size of 8K vs 9K made no observable difference. Therefore 8K MTU size was used for the OSA configuration.
Even larger MTU sizes are possible, but they need to be supported by all hardware along the used network path. However, unless the network path is completely contained in a controlled environment, Path MTU Discovery (PMTUD)1 may auto-adjust the MTU size smaller.
However, for long-running or memory-constrained systems, memory fragmentation can become an issue when using larger MTU sizes.2