summaryrefslogtreecommitdiff
path: root/Documentation/mic/scif_overview.txt
diff options
context:
space:
mode:
authorLinus Torvalds <torvalds@linux-foundation.org>2015-06-26 14:51:15 -0700
committerLinus Torvalds <torvalds@linux-foundation.org>2015-06-26 14:51:15 -0700
commitd87823813fe498fdd47894bd28e460a9dee8d771 (patch)
tree214eaf3babd0d61f08022fc1edd99a5128616548 /Documentation/mic/scif_overview.txt
parente382608254e06c8109f40044f5e693f2e04f3899 (diff)
parent3dc196eae1db548f05e53e5875ff87b8ff79f249 (diff)
Merge tag 'char-misc-4.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc
Pull char/misc driver updates from Greg KH: "Here's the big char/misc driver pull request for 4.2-rc1. Lots of mei, extcon, coresight, uio, mic, and other driver updates in here. Full details in the shortlog. All of these have been in linux-next for some time with no reported problems" * tag 'char-misc-4.2-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/char-misc: (176 commits) mei: me: wait for power gating exit confirmation mei: reset flow control on the last client disconnection MAINTAINERS: mei: add mei_cl_bus.h to maintained file list misc: sram: sort and clean up included headers misc: sram: move reserved block logic out of probe function misc: sram: add private struct device and virt_base members misc: sram: report correct SRAM pool size misc: sram: bump error message level on unclean driver unbinding misc: sram: fix device node reference leak on error misc: sram: fix enabled clock leak on error path misc: mic: Fix reported static checker warning misc: mic: Fix randconfig build error by including errno.h uio: pruss: Drop depends on ARCH_DAVINCI_DA850 from config uio: pruss: Add CONFIG_HAS_IOMEM dependence uio: pruss: Include <linux/sizes.h> extcon: Redefine the unique id of supported external connectors without 'enum extcon' type char:xilinx_hwicap:buffer_icap - change 1/0 to true/false for bool type variable in function buffer_icap_set_configuration(). Drivers: hv: vmbus: Allocate ring buffer memory in NUMA aware fashion parport: check exclusive access before register w1: use correct lock on error in w1_seq_show() ...
Diffstat (limited to 'Documentation/mic/scif_overview.txt')
-rw-r--r--Documentation/mic/scif_overview.txt98
1 files changed, 98 insertions, 0 deletions
diff --git a/Documentation/mic/scif_overview.txt b/Documentation/mic/scif_overview.txt
new file mode 100644
index 000000000000..0a280d986731
--- /dev/null
+++ b/Documentation/mic/scif_overview.txt
@@ -0,0 +1,98 @@
+The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low
+level communications API across PCIe currently implemented for MIC. Currently
+SCIF provides inter-node communication within a single host platform, where a
+node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of
+communicating over the PCIe bus while providing an API that is symmetric
+across all the nodes in the PCIe network. An important design objective for SCIF
+is to deliver the maximum possible performance given the communication
+abilities of the hardware. SCIF has been used to implement an offload compiler
+runtime and OFED support for MPI implementations for MIC coprocessors.
+
+==== SCIF API Components ====
+The SCIF API has the following parts:
+1. Connection establishment using a client server model
+2. Byte stream messaging intended for short messages
+3. Node enumeration to determine online nodes
+4. Poll semantics for detection of incoming connections and messages
+5. Memory registration to pin down pages
+6. Remote memory mapping for low latency CPU accesses via mmap
+7. Remote DMA (RDMA) for high bandwidth DMA transfers
+8. Fence APIs for RDMA synchronization
+
+SCIF exposes the notion of a connection which can be used by peer processes on
+nodes in a SCIF PCIe "network" to share memory "windows" and to communicate. A
+process in a SCIF node initiates a SCIF connection to a peer process on a
+different node via a SCIF "endpoint". SCIF endpoints support messaging APIs
+which are similar to connection oriented socket APIs. Connected SCIF endpoints
+can also register local memory which is followed by data transfer using either
+DMA, CPU copies or remote memory mapping via mmap. SCIF supports both user and
+kernel mode clients which are functionally equivalent.
+
+==== SCIF Performance for MIC ====
+DMA bandwidth comparison between the TCP (over ethernet over PCIe) stack versus
+SCIF shows the performance advantages of SCIF for HPC applications and runtimes.
+
+ Comparison of TCP and SCIF based BW
+
+ Throughput (GB/sec)
+ 8 + PCIe Bandwidth ******
+ + TCP ######
+ 7 + ************************************** SCIF %%%%%%
+ | %%%%%%%%%%%%%%%%%%%
+ 6 + %%%%
+ | %%
+ | %%%
+ 5 + %%
+ | %%
+ 4 + %%
+ | %%
+ 3 + %%
+ | %
+ 2 + %%
+ | %%
+ | %
+ 1 +
+ + ######################################
+ 0 +++---+++--+--+-+--+--+-++-+--+-++-+--+-++-+-
+ 1 10 100 1000 10000 100000
+ Transfer Size (KBytes)
+
+SCIF allows memory sharing via mmap(..) between processes on different PCIe
+nodes and thus provides bare-metal PCIe latency. The round trip SCIF mmap
+latency from the host to an x100 MIC for an 8 byte message is 0.44 usecs.
+
+SCIF has a user space library which is a thin IOCTL wrapper providing a user
+space API similar to the kernel API in scif.h. The SCIF user space library
+is distributed @ https://software.intel.com/en-us/mic-developer
+
+Here is some pseudo code for an example of how two applications on two PCIe
+nodes would typically use the SCIF API:
+
+Process A (on node A) Process B (on node B)
+
+/* get online node information */
+scif_get_node_ids(..) scif_get_node_ids(..)
+scif_open(..) scif_open(..)
+scif_bind(..) scif_bind(..)
+scif_listen(..)
+scif_accept(..) scif_connect(..)
+/* SCIF connection established */
+
+/* Send and receive short messages */
+scif_send(..)/scif_recv(..) scif_send(..)/scif_recv(..)
+
+/* Register memory */
+scif_register(..) scif_register(..)
+
+/* RDMA */
+scif_readfrom(..)/scif_writeto(..) scif_readfrom(..)/scif_writeto(..)
+
+/* Fence DMAs */
+scif_fence_signal(..) scif_fence_signal(..)
+
+mmap(..) mmap(..)
+
+/* Access remote registered memory */
+
+/* Close the endpoints */
+scif_close(..) scif_close(..)