1 21. Control Interfaces 3 4The interfaces for receiving network packages timestamps are: 5 6* SO_TIMESTAMP 7 Generates a timestamp for each incoming packet in (not necessarily 8 monotonic) system time. Reports the timestamp via recvmsg() in a 9 control message as struct timeval (usec resolution). 10 11* SO_TIMESTAMPNS 12 Same timestamping mechanism as SO_TIMESTAMP, but reports the 13 timestamp as struct timespec (nsec resolution). 14 15* IP_MULTICAST_LOOP + SO_TIMESTAMP[NS] 16 Only for multicast:approximate transmit timestamp obtained by 17 reading the looped packet receive timestamp. 18 19* SO_TIMESTAMPING 20 Generates timestamps on reception, transmission or both. Supports 21 multiple timestamp sources, including hardware. Supports generating 22 timestamps for stream sockets. 23 24 251.1 SO_TIMESTAMP: 26 27This socket option enables timestamping of datagrams on the reception 28path. Because the destination socket, if any, is not known early in 29the network stack, the feature has to be enabled for all packets. The 30same is true for all early receive timestamp options. 31 32For interface details, see `man 7 socket`. 33 34 351.2 SO_TIMESTAMPNS: 36 37This option is identical to SO_TIMESTAMP except for the returned data type. 38Its struct timespec allows for higher resolution (ns) timestamps than the 39timeval of SO_TIMESTAMP (ms). 40 41 421.3 SO_TIMESTAMPING: 43 44Supports multiple types of timestamp requests. As a result, this 45socket option takes a bitmap of flags, not a boolean. In 46 47 err = setsockopt(fd, SOL_SOCKET, SO_TIMESTAMPING, (void *) val, &val); 48 49val is an integer with any of the following bits set. Setting other 50bit returns EINVAL and does not change the current state. 51 52 531.3.1 Timestamp Generation 54 55Some bits are requests to the stack to try to generate timestamps. Any 56combination of them is valid. Changes to these bits apply to newly 57created packets, not to packets already in the stack. As a result, it 58is possible to selectively request timestamps for a subset of packets 59(e.g., for sampling) by embedding an send() call within two setsockopt 60calls, one to enable timestamp generation and one to disable it. 61Timestamps may also be generated for reasons other than being 62requested by a particular socket, such as when receive timestamping is 63enabled system wide, as explained earlier. 64 65SOF_TIMESTAMPING_RX_HARDWARE: 66 Request rx timestamps generated by the network adapter. 67 68SOF_TIMESTAMPING_RX_SOFTWARE: 69 Request rx timestamps when data enters the kernel. These timestamps 70 are generated just after a device driver hands a packet to the 71 kernel receive stack. 72 73SOF_TIMESTAMPING_TX_HARDWARE: 74 Request tx timestamps generated by the network adapter. 75 76SOF_TIMESTAMPING_TX_SOFTWARE: 77 Request tx timestamps when data leaves the kernel. These timestamps 78 are generated in the device driver as close as possible, but always 79 prior to, passing the packet to the network interface. Hence, they 80 require driver support and may not be available for all devices. 81 82SOF_TIMESTAMPING_TX_SCHED: 83 Request tx timestamps prior to entering the packet scheduler. Kernel 84 transmit latency is, if long, often dominated by queuing delay. The 85 difference between this timestamp and one taken at 86 SOF_TIMESTAMPING_TX_SOFTWARE will expose this latency independent 87 of protocol processing. The latency incurred in protocol 88 processing, if any, can be computed by subtracting a userspace 89 timestamp taken immediately before send() from this timestamp. On 90 machines with virtual devices where a transmitted packet travels 91 through multiple devices and, hence, multiple packet schedulers, 92 a timestamp is generated at each layer. This allows for fine 93 grained measurement of queuing delay. 94 95SOF_TIMESTAMPING_TX_ACK: 96 Request tx timestamps when all data in the send buffer has been 97 acknowledged. This only makes sense for reliable protocols. It is 98 currently only implemented for TCP. For that protocol, it may 99 over-report measurement, because the timestamp is generated when all 100 data up to and including the buffer at send() was acknowledged: the 101 cumulative acknowledgment. The mechanism ignores SACK and FACK. 102 103 1041.3.2 Timestamp Reporting 105 106The other three bits control which timestamps will be reported in a 107generated control message. Changes to the bits take immediate 108effect at the timestamp reporting locations in the stack. Timestamps 109are only reported for packets that also have the relevant timestamp 110generation request set. 111 112SOF_TIMESTAMPING_SOFTWARE: 113 Report any software timestamps when available. 114 115SOF_TIMESTAMPING_SYS_HARDWARE: 116 This option is deprecated and ignored. 117 118SOF_TIMESTAMPING_RAW_HARDWARE: 119 Report hardware timestamps as generated by 120 SOF_TIMESTAMPING_TX_HARDWARE when available. 121 122 1231.3.3 Timestamp Options 124 125The interface supports one option 126 127SOF_TIMESTAMPING_OPT_ID: 128 129 Generate a unique identifier along with each packet. A process can 130 have multiple concurrent timestamping requests outstanding. Packets 131 can be reordered in the transmit path, for instance in the packet 132 scheduler. In that case timestamps will be queued onto the error 133 queue out of order from the original send() calls. This option 134 embeds a counter that is incremented at send() time, to order 135 timestamps within a flow. 136 137 This option is implemented only for transmit timestamps. There, the 138 timestamp is always looped along with a struct sock_extended_err. 139 The option modifies field ee_data to pass an id that is unique 140 among all possibly concurrently outstanding timestamp requests for 141 that socket. In practice, it is a monotonically increasing u32 142 (that wraps). 143 144 In datagram sockets, the counter increments on each send call. In 145 stream sockets, it increments with every byte. 146 147 1481.4 Bytestream Timestamps 149 150The SO_TIMESTAMPING interface supports timestamping of bytes in a 151bytestream. Each request is interpreted as a request for when the 152entire contents of the buffer has passed a timestamping point. That 153is, for streams option SOF_TIMESTAMPING_TX_SOFTWARE will record 154when all bytes have reached the device driver, regardless of how 155many packets the data has been converted into. 156 157In general, bytestreams have no natural delimiters and therefore 158correlating a timestamp with data is non-trivial. A range of bytes 159may be split across segments, any segments may be merged (possibly 160coalescing sections of previously segmented buffers associated with 161independent send() calls). Segments can be reordered and the same 162byte range can coexist in multiple segments for protocols that 163implement retransmissions. 164 165It is essential that all timestamps implement the same semantics, 166regardless of these possible transformations, as otherwise they are 167incomparable. Handling "rare" corner cases differently from the 168simple case (a 1:1 mapping from buffer to skb) is insufficient 169because performance debugging often needs to focus on such outliers. 170 171In practice, timestamps can be correlated with segments of a 172bytestream consistently, if both semantics of the timestamp and the 173timing of measurement are chosen correctly. This challenge is no 174different from deciding on a strategy for IP fragmentation. There, the 175definition is that only the first fragment is timestamped. For 176bytestreams, we chose that a timestamp is generated only when all 177bytes have passed a point. SOF_TIMESTAMPING_TX_ACK as defined is easy to 178implement and reason about. An implementation that has to take into 179account SACK would be more complex due to possible transmission holes 180and out of order arrival. 181 182On the host, TCP can also break the simple 1:1 mapping from buffer to 183skbuff as a result of Nagle, cork, autocork, segmentation and GSO. The 184implementation ensures correctness in all cases by tracking the 185individual last byte passed to send(), even if it is no longer the 186last byte after an skbuff extend or merge operation. It stores the 187relevant sequence number in skb_shinfo(skb)->tskey. Because an skbuff 188has only one such field, only one timestamp can be generated. 189 190In rare cases, a timestamp request can be missed if two requests are 191collapsed onto the same skb. A process can detect this situation by 192enabling SOF_TIMESTAMPING_OPT_ID and comparing the byte offset at 193send time with the value returned for each timestamp. It can prevent 194the situation by always flushing the TCP stack in between requests, 195for instance by enabling TCP_NODELAY and disabling TCP_CORK and 196autocork. 197 198These precautions ensure that the timestamp is generated only when all 199bytes have passed a timestamp point, assuming that the network stack 200itself does not reorder the segments. The stack indeed tries to avoid 201reordering. The one exception is under administrator control: it is 202possible to construct a packet scheduler configuration that delays 203segments from the same stream differently. Such a setup would be 204unusual. 205 206 2072 Data Interfaces 208 209Timestamps are read using the ancillary data feature of recvmsg(). 210See `man 3 cmsg` for details of this interface. The socket manual 211page (`man 7 socket`) describes how timestamps generated with 212SO_TIMESTAMP and SO_TIMESTAMPNS records can be retrieved. 213 214 2152.1 SCM_TIMESTAMPING records 216 217These timestamps are returned in a control message with cmsg_level 218SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type 219 220struct scm_timestamping { 221 struct timespec ts[3]; 222}; 223 224The structure can return up to three timestamps. This is a legacy 225feature. Only one field is non-zero at any time. Most timestamps 226are passed in ts[0]. Hardware timestamps are passed in ts[2]. 227 228ts[1] used to hold hardware timestamps converted to system time. 229Instead, expose the hardware clock device on the NIC directly as 230a HW PTP clock source, to allow time conversion in userspace and 231optionally synchronize system time with a userspace PTP stack such 232as linuxptp. For the PTP clock API, see Documentation/ptp/ptp.txt. 233 2342.1.1 Transmit timestamps with MSG_ERRQUEUE 235 236For transmit timestamps the outgoing packet is looped back to the 237socket's error queue with the send timestamp(s) attached. A process 238receives the timestamps by calling recvmsg() with flag MSG_ERRQUEUE 239set and with a msg_control buffer sufficiently large to receive the 240relevant metadata structures. The recvmsg call returns the original 241outgoing data packet with two ancillary messages attached. 242 243A message of cm_level SOL_IP(V6) and cm_type IP(V6)_RECVERR 244embeds a struct sock_extended_err. This defines the error type. For 245timestamps, the ee_errno field is ENOMSG. The other ancillary message 246will have cm_level SOL_SOCKET and cm_type SCM_TIMESTAMPING. This 247embeds the struct scm_timestamping. 248 249 2502.1.1.2 Timestamp types 251 252The semantics of the three struct timespec are defined by field 253ee_info in the extended error structure. It contains a value of 254type SCM_TSTAMP_* to define the actual timestamp passed in 255scm_timestamping. 256 257The SCM_TSTAMP_* types are 1:1 matches to the SOF_TIMESTAMPING_* 258control fields discussed previously, with one exception. For legacy 259reasons, SCM_TSTAMP_SND is equal to zero and can be set for both 260SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_TX_SOFTWARE. It 261is the first if ts[2] is non-zero, the second otherwise, in which 262case the timestamp is stored in ts[0]. 263 264 2652.1.1.3 Fragmentation 266 267Fragmentation of outgoing datagrams is rare, but is possible, e.g., by 268explicitly disabling PMTU discovery. If an outgoing packet is fragmented, 269then only the first fragment is timestamped and returned to the sending 270socket. 271 272 2732.1.1.4 Packet Payload 274 275The calling application is often not interested in receiving the whole 276packet payload that it passed to the stack originally: the socket 277error queue mechanism is just a method to piggyback the timestamp on. 278In this case, the application can choose to read datagrams with a 279smaller buffer, possibly even of length 0. The payload is truncated 280accordingly. Until the process calls recvmsg() on the error queue, 281however, the full packet is queued, taking up budget from SO_RCVBUF. 282 283 2842.1.1.5 Blocking Read 285 286Reading from the error queue is always a non-blocking operation. To 287block waiting on a timestamp, use poll or select. poll() will return 288POLLERR in pollfd.revents if any data is ready on the error queue. 289There is no need to pass this flag in pollfd.events. This flag is 290ignored on request. See also `man 2 poll`. 291 292 2932.1.2 Receive timestamps 294 295On reception, there is no reason to read from the socket error queue. 296The SCM_TIMESTAMPING ancillary data is sent along with the packet data 297on a normal recvmsg(). Since this is not a socket error, it is not 298accompanied by a message SOL_IP(V6)/IP(V6)_RECVERROR. In this case, 299the meaning of the three fields in struct scm_timestamping is 300implicitly defined. ts[0] holds a software timestamp if set, ts[1] 301is again deprecated and ts[2] holds a hardware timestamp if set. 302 303 3043. Hardware Timestamping configuration: SIOCSHWTSTAMP and SIOCGHWTSTAMP 305 306Hardware time stamping must also be initialized for each device driver 307that is expected to do hardware time stamping. The parameter is defined in 308/include/linux/net_tstamp.h as: 309 310struct hwtstamp_config { 311 int flags; /* no flags defined right now, must be zero */ 312 int tx_type; /* HWTSTAMP_TX_* */ 313 int rx_filter; /* HWTSTAMP_FILTER_* */ 314}; 315 316Desired behavior is passed into the kernel and to a specific device by 317calling ioctl(SIOCSHWTSTAMP) with a pointer to a struct ifreq whose 318ifr_data points to a struct hwtstamp_config. The tx_type and 319rx_filter are hints to the driver what it is expected to do. If 320the requested fine-grained filtering for incoming packets is not 321supported, the driver may time stamp more than just the requested types 322of packets. 323 324A driver which supports hardware time stamping shall update the struct 325with the actual, possibly more permissive configuration. If the 326requested packets cannot be time stamped, then nothing should be 327changed and ERANGE shall be returned (in contrast to EINVAL, which 328indicates that SIOCSHWTSTAMP is not supported at all). 329 330Only a processes with admin rights may change the configuration. User 331space is responsible to ensure that multiple processes don't interfere 332with each other and that the settings are reset. 333 334Any process can read the actual configuration by passing this 335structure to ioctl(SIOCGHWTSTAMP) in the same way. However, this has 336not been implemented in all drivers. 337 338/* possible values for hwtstamp_config->tx_type */ 339enum { 340 /* 341 * no outgoing packet will need hardware time stamping; 342 * should a packet arrive which asks for it, no hardware 343 * time stamping will be done 344 */ 345 HWTSTAMP_TX_OFF, 346 347 /* 348 * enables hardware time stamping for outgoing packets; 349 * the sender of the packet decides which are to be 350 * time stamped by setting SOF_TIMESTAMPING_TX_SOFTWARE 351 * before sending the packet 352 */ 353 HWTSTAMP_TX_ON, 354}; 355 356/* possible values for hwtstamp_config->rx_filter */ 357enum { 358 /* time stamp no incoming packet at all */ 359 HWTSTAMP_FILTER_NONE, 360 361 /* time stamp any incoming packet */ 362 HWTSTAMP_FILTER_ALL, 363 364 /* return value: time stamp all packets requested plus some others */ 365 HWTSTAMP_FILTER_SOME, 366 367 /* PTP v1, UDP, any kind of event packet */ 368 HWTSTAMP_FILTER_PTP_V1_L4_EVENT, 369 370 /* for the complete list of values, please check 371 * the include file /include/linux/net_tstamp.h 372 */ 373}; 374 3753.1 Hardware Timestamping Implementation: Device Drivers 376 377A driver which supports hardware time stamping must support the 378SIOCSHWTSTAMP ioctl and update the supplied struct hwtstamp_config with 379the actual values as described in the section on SIOCSHWTSTAMP. It 380should also support SIOCGHWTSTAMP. 381 382Time stamps for received packets must be stored in the skb. To get a pointer 383to the shared time stamp structure of the skb call skb_hwtstamps(). Then 384set the time stamps in the structure: 385 386struct skb_shared_hwtstamps { 387 /* hardware time stamp transformed into duration 388 * since arbitrary point in time 389 */ 390 ktime_t hwtstamp; 391}; 392 393Time stamps for outgoing packets are to be generated as follows: 394- In hard_start_xmit(), check if (skb_shinfo(skb)->tx_flags & SKBTX_HW_TSTAMP) 395 is set no-zero. If yes, then the driver is expected to do hardware time 396 stamping. 397- If this is possible for the skb and requested, then declare 398 that the driver is doing the time stamping by setting the flag 399 SKBTX_IN_PROGRESS in skb_shinfo(skb)->tx_flags , e.g. with 400 401 skb_shinfo(skb)->tx_flags |= SKBTX_IN_PROGRESS; 402 403 You might want to keep a pointer to the associated skb for the next step 404 and not free the skb. A driver not supporting hardware time stamping doesn't 405 do that. A driver must never touch sk_buff::tstamp! It is used to store 406 software generated time stamps by the network subsystem. 407- Driver should call skb_tx_timestamp() as close to passing sk_buff to hardware 408 as possible. skb_tx_timestamp() provides a software time stamp if requested 409 and hardware timestamping is not possible (SKBTX_IN_PROGRESS not set). 410- As soon as the driver has sent the packet and/or obtained a 411 hardware time stamp for it, it passes the time stamp back by 412 calling skb_hwtstamp_tx() with the original skb, the raw 413 hardware time stamp. skb_hwtstamp_tx() clones the original skb and 414 adds the timestamps, therefore the original skb has to be freed now. 415 If obtaining the hardware time stamp somehow fails, then the driver 416 should not fall back to software time stamping. The rationale is that 417 this would occur at a later time in the processing pipeline than other 418 software time stamping and therefore could lead to unexpected deltas 419 between time stamps. 420