1======================================== 2Symmetric Communication Interface (SCIF) 3======================================== 4 5The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low 6level communications API across PCIe currently implemented for MIC. Currently 7SCIF provides inter-node communication within a single host platform, where a 8node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of 9communicating over the PCIe bus while providing an API that is symmetric 10across all the nodes in the PCIe network. An important design objective for SCIF 11is to deliver the maximum possible performance given the communication 12abilities of the hardware. SCIF has been used to implement an offload compiler 13runtime and OFED support for MPI implementations for MIC coprocessors. 14 15SCIF API Components 16=================== 17 18The SCIF API has the following parts: 19 201. Connection establishment using a client server model 212. Byte stream messaging intended for short messages 223. Node enumeration to determine online nodes 234. Poll semantics for detection of incoming connections and messages 245. Memory registration to pin down pages 256. Remote memory mapping for low latency CPU accesses via mmap 267. Remote DMA (RDMA) for high bandwidth DMA transfers 278. Fence APIs for RDMA synchronization 28 29SCIF exposes the notion of a connection which can be used by peer processes on 30nodes in a SCIF PCIe "network" to share memory "windows" and to communicate. A 31process in a SCIF node initiates a SCIF connection to a peer process on a 32different node via a SCIF "endpoint". SCIF endpoints support messaging APIs 33which are similar to connection oriented socket APIs. Connected SCIF endpoints 34can also register local memory which is followed by data transfer using either 35DMA, CPU copies or remote memory mapping via mmap. SCIF supports both user and 36kernel mode clients which are functionally equivalent. 37 38SCIF Performance for MIC 39======================== 40 41DMA bandwidth comparison between the TCP (over ethernet over PCIe) stack versus 42SCIF shows the performance advantages of SCIF for HPC applications and 43runtimes:: 44 45 Comparison of TCP and SCIF based BW 46 47 Throughput (GB/sec) 48 8 + PCIe Bandwidth ****** 49 + TCP ###### 50 7 + ************************************** SCIF %%%%%% 51 | %%%%%%%%%%%%%%%%%%% 52 6 + %%%% 53 | %% 54 | %%% 55 5 + %% 56 | %% 57 4 + %% 58 | %% 59 3 + %% 60 | % 61 2 + %% 62 | %% 63 | % 64 1 + 65 + ###################################### 66 0 +++---+++--+--+-+--+--+-++-+--+-++-+--+-++-+- 67 1 10 100 1000 10000 100000 68 Transfer Size (KBytes) 69 70SCIF allows memory sharing via mmap(..) between processes on different PCIe 71nodes and thus provides bare-metal PCIe latency. The round trip SCIF mmap 72latency from the host to an x100 MIC for an 8 byte message is 0.44 usecs. 73 74SCIF has a user space library which is a thin IOCTL wrapper providing a user 75space API similar to the kernel API in scif.h. The SCIF user space library 76is distributed @ https://software.intel.com/en-us/mic-developer 77 78Here is some pseudo code for an example of how two applications on two PCIe 79nodes would typically use the SCIF API:: 80 81 Process A (on node A) Process B (on node B) 82 83 /* get online node information */ 84 scif_get_node_ids(..) scif_get_node_ids(..) 85 scif_open(..) scif_open(..) 86 scif_bind(..) scif_bind(..) 87 scif_listen(..) 88 scif_accept(..) scif_connect(..) 89 /* SCIF connection established */ 90 91 /* Send and receive short messages */ 92 scif_send(..)/scif_recv(..) scif_send(..)/scif_recv(..) 93 94 /* Register memory */ 95 scif_register(..) scif_register(..) 96 97 /* RDMA */ 98 scif_readfrom(..)/scif_writeto(..) scif_readfrom(..)/scif_writeto(..) 99 100 /* Fence DMAs */ 101 scif_fence_signal(..) scif_fence_signal(..) 102 103 mmap(..) mmap(..) 104 105 /* Access remote registered memory */ 106 107 /* Close the endpoints */ 108 scif_close(..) scif_close(..) 109