1USERSPACE MAD ACCESS 2 3Device files 4 5 Each port of each InfiniBand device has a "umad" device and an 6 "issm" device attached. For example, a two-port HCA will have two 7 umad devices and two issm devices, while a switch will have one 8 device of each type (for switch port 0). 9 10Creating MAD agents 11 12 A MAD agent can be created by filling in a struct ib_user_mad_reg_req 13 and then calling the IB_USER_MAD_REGISTER_AGENT ioctl on a file 14 descriptor for the appropriate device file. If the registration 15 request succeeds, a 32-bit id will be returned in the structure. 16 For example: 17 18 struct ib_user_mad_reg_req req = { /* ... */ }; 19 ret = ioctl(fd, IB_USER_MAD_REGISTER_AGENT, (char *) &req); 20 if (!ret) 21 my_agent = req.id; 22 else 23 perror("agent register"); 24 25 Agents can be unregistered with the IB_USER_MAD_UNREGISTER_AGENT 26 ioctl. Also, all agents registered through a file descriptor will 27 be unregistered when the descriptor is closed. 28 29Receiving MADs 30 31 MADs are received using read(). The receive side now supports 32 RMPP. The buffer passed to read() must be at least one 33 struct ib_user_mad + 256 bytes. For example: 34 35 If the buffer passed is not large enough to hold the received 36 MAD (RMPP), the errno is set to ENOSPC and the length of the 37 buffer needed is set in mad.length. 38 39 Example for normal MAD (non RMPP) reads: 40 struct ib_user_mad *mad; 41 mad = malloc(sizeof *mad + 256); 42 ret = read(fd, mad, sizeof *mad + 256); 43 if (ret != sizeof mad + 256) { 44 perror("read"); 45 free(mad); 46 } 47 48 Example for RMPP reads: 49 struct ib_user_mad *mad; 50 mad = malloc(sizeof *mad + 256); 51 ret = read(fd, mad, sizeof *mad + 256); 52 if (ret == -ENOSPC)) { 53 length = mad.length; 54 free(mad); 55 mad = malloc(sizeof *mad + length); 56 ret = read(fd, mad, sizeof *mad + length); 57 } 58 if (ret < 0) { 59 perror("read"); 60 free(mad); 61 } 62 63 In addition to the actual MAD contents, the other struct ib_user_mad 64 fields will be filled in with information on the received MAD. For 65 example, the remote LID will be in mad.lid. 66 67 If a send times out, a receive will be generated with mad.status set 68 to ETIMEDOUT. Otherwise when a MAD has been successfully received, 69 mad.status will be 0. 70 71 poll()/select() may be used to wait until a MAD can be read. 72 73Sending MADs 74 75 MADs are sent using write(). The agent ID for sending should be 76 filled into the id field of the MAD, the destination LID should be 77 filled into the lid field, and so on. The send side does support 78 RMPP so arbitrary length MAD can be sent. For example: 79 80 struct ib_user_mad *mad; 81 82 mad = malloc(sizeof *mad + mad_length); 83 84 /* fill in mad->data */ 85 86 mad->hdr.id = my_agent; /* req.id from agent registration */ 87 mad->hdr.lid = my_dest; /* in network byte order... */ 88 /* etc. */ 89 90 ret = write(fd, &mad, sizeof *mad + mad_length); 91 if (ret != sizeof *mad + mad_length) 92 perror("write"); 93 94Transaction IDs 95 96 Users of the umad devices can use the lower 32 bits of the 97 transaction ID field (that is, the least significant half of the 98 field in network byte order) in MADs being sent to match 99 request/response pairs. The upper 32 bits are reserved for use by 100 the kernel and will be overwritten before a MAD is sent. 101 102P_Key Index Handling 103 104 The old ib_umad interface did not allow setting the P_Key index for 105 MADs that are sent and did not provide a way for obtaining the P_Key 106 index of received MADs. A new layout for struct ib_user_mad_hdr 107 with a pkey_index member has been defined; however, to preserve 108 binary compatibility with older applications, this new layout will 109 not be used unless the IB_USER_MAD_ENABLE_PKEY ioctl is called 110 before a file descriptor is used for anything else. 111 112 In September 2008, the IB_USER_MAD_ABI_VERSION will be incremented 113 to 6, the new layout of struct ib_user_mad_hdr will be used by 114 default, and the IB_USER_MAD_ENABLE_PKEY ioctl will be removed. 115 116Setting IsSM Capability Bit 117 118 To set the IsSM capability bit for a port, simply open the 119 corresponding issm device file. If the IsSM bit is already set, 120 then the open call will block until the bit is cleared (or return 121 immediately with errno set to EAGAIN if the O_NONBLOCK flag is 122 passed to open()). The IsSM bit will be cleared when the issm file 123 is closed. No read, write or other operations can be performed on 124 the issm file. 125 126/dev files 127 128 To create the appropriate character device files automatically with 129 udev, a rule like 130 131 KERNEL="umad*", NAME="infiniband/%k" 132 KERNEL="issm*", NAME="infiniband/%k" 133 134 can be used. This will create device nodes named 135 136 /dev/infiniband/umad0 137 /dev/infiniband/issm0 138 139 for the first port, and so on. The InfiniBand device and port 140 associated with these devices can be determined from the files 141 142 /sys/class/infiniband_mad/umad0/ibdev 143 /sys/class/infiniband_mad/umad0/port 144 145 and 146 147 /sys/class/infiniband_mad/issm0/ibdev 148 /sys/class/infiniband_mad/issm0/port 149