1INFINIBAND MIDLAYER LOCKING 2 3 This guide is an attempt to make explicit the locking assumptions 4 made by the InfiniBand midlayer. It describes the requirements on 5 both low-level drivers that sit below the midlayer and upper level 6 protocols that use the midlayer. 7 8Sleeping and interrupt context 9 10 With the following exceptions, a low-level driver implementation of 11 all of the methods in struct ib_device may sleep. The exceptions 12 are any methods from the list: 13 14 create_ah 15 modify_ah 16 query_ah 17 destroy_ah 18 post_send 19 post_recv 20 poll_cq 21 req_notify_cq 22 map_phys_fmr 23 24 which may not sleep and must be callable from any context. 25 26 The corresponding functions exported to upper level protocol 27 consumers: 28 29 ib_create_ah 30 ib_modify_ah 31 ib_query_ah 32 ib_destroy_ah 33 ib_post_send 34 ib_post_recv 35 ib_req_notify_cq 36 ib_map_phys_fmr 37 38 are therefore safe to call from any context. 39 40 In addition, the function 41 42 ib_dispatch_event 43 44 used by low-level drivers to dispatch asynchronous events through 45 the midlayer is also safe to call from any context. 46 47Reentrancy 48 49 All of the methods in struct ib_device exported by a low-level 50 driver must be fully reentrant. The low-level driver is required to 51 perform all synchronization necessary to maintain consistency, even 52 if multiple function calls using the same object are run 53 simultaneously. 54 55 The IB midlayer does not perform any serialization of function calls. 56 57 Because low-level drivers are reentrant, upper level protocol 58 consumers are not required to perform any serialization. However, 59 some serialization may be required to get sensible results. For 60 example, a consumer may safely call ib_poll_cq() on multiple CPUs 61 simultaneously. However, the ordering of the work completion 62 information between different calls of ib_poll_cq() is not defined. 63 64Callbacks 65 66 A low-level driver must not perform a callback directly from the 67 same callchain as an ib_device method call. For example, it is not 68 allowed for a low-level driver to call a consumer's completion event 69 handler directly from its post_send method. Instead, the low-level 70 driver should defer this callback by, for example, scheduling a 71 tasklet to perform the callback. 72 73 The low-level driver is responsible for ensuring that multiple 74 completion event handlers for the same CQ are not called 75 simultaneously. The driver must guarantee that only one CQ event 76 handler for a given CQ is running at a time. In other words, the 77 following situation is not allowed: 78 79 CPU1 CPU2 80 81 low-level driver -> 82 consumer CQ event callback: 83 /* ... */ 84 ib_req_notify_cq(cq, ...); 85 low-level driver -> 86 /* ... */ consumer CQ event callback: 87 /* ... */ 88 return from CQ event handler 89 90 The context in which completion event and asynchronous event 91 callbacks run is not defined. Depending on the low-level driver, it 92 may be process context, softirq context, or interrupt context. 93 Upper level protocol consumers may not sleep in a callback. 94 95Hot-plug 96 97 A low-level driver announces that a device is ready for use by 98 consumers when it calls ib_register_device(), all initialization 99 must be complete before this call. The device must remain usable 100 until the driver's call to ib_unregister_device() has returned. 101 102 A low-level driver must call ib_register_device() and 103 ib_unregister_device() from process context. It must not hold any 104 semaphores that could cause deadlock if a consumer calls back into 105 the driver across these calls. 106 107 An upper level protocol consumer may begin using an IB device as 108 soon as the add method of its struct ib_client is called for that 109 device. A consumer must finish all cleanup and free all resources 110 relating to a device before returning from the remove method. 111 112 A consumer is permitted to sleep in its add and remove methods. 113