PCIe (Peripheral Component Interconnect Express), as it speaks for itself, is a standard for interconnecting peripheral components of a system like the GPU, sound card, network card or SSD. Here is a PCIe adapter containing 4 PCIe slots.
Having a GPU connected to the system, CUDA kernels and data are dispatched by CPU from the system RAM to the GPU VRAM through this interface and the results are transmitted back. Below is a Nvidia PCIe GPU.
Did you notice that? I said "PCIe GPU". This "PCIe" refers to the PCIe interface as shown in the images above. This interface has shown physical and performance limitations when it comes to GPU. For example, its vertical mounting hinders more efficient cooling solutions or GPU-GPU interconnections are hard to take effect using this interface. So shines the SXM GPU, exclusive to Nvidia and an alternative to the PCIe one. SXM GPU still employs PCIe as the standard for connecting to the CPU but in a whole new form factor. Two images below depict a SXM GPU from front and back respectively.
This GPU form factor mostly suits the servers like the one containing the assemply of 4 GPUs below not the desktop PCs. Shouldn't we get familiar with Nvidia NVLink, we'd not understand the true benefit of the SXM GPUs.
NVLink is a high-throughput interconnecting technology from Nvidia for directly connecting two GPUs. It provides a much faster & more configurable solution to GPU-GPU data transmission than the PCIe-switch-mediated one. Below is the image of a PCIe switch board and an example of using NVLink in three configurations.
Not all Nvidia GPUs support NVLink. Also, some GPUs have multiple NVLink links. For example Nvidia V100 GPU has 6 links. The links could be grouped together i.e., two GPUs connect to each other with more than one links, so as to yet increase the connection throughput. It's worth noting that to take the most of NVLink, Nvidia has also put forward NVSwitch, a technology facilitating all-to-all GPU communications.
A fact about NVLink is that a PCIe GPU could connect to at most one other GPU with NVLink but there's no such limitation to SXM GPUs. Below is the image of a NVLink slot on a PCIe GPU and two PCIe GPUs connected through NVLink (by NVLink Bridge product). As a result, among the three configurations in the above image, only the leftmost is feasible with PCIe GPUs.
