1// SPDX-License-Identifier: GPL-2.0 2 3======== 4ublk-nbd 5======== 6 7Motivation 8========== 9 10As one attempt of using io_uring to implement network storage based on ublk 11framework, make one basic nbd client with ublk/io_uring, which could be for 12replacing linux kernel nbd driver. 13 14Howto 15===== 16 17ublk add -t nbd [-q $NR_QUEUES ] [ -d $QUEUE_DEPTH ] [--host $HOST_IP_OR_NAME | --unix $UNIX_SOCKET_PATH] [--send_zc] 18 19Like ``nbd-client`` [#nbd_client]_, ublk-nbd supports both tcp and unix socket. 20 21``--host $HOST_IP_OR_NAME`` points to nbd server's IP address or domain name if 22tcp socket is used. 23 24``--unix $UNIX_SOCKET_PATH`` points to unix socket path if unix socket is used. 25 26The option of ``--send_zc`` enables ``io_uring send zero copy`` 27[#io_uring_send_zc]_, which is only used for handling ublk write IO. 28 29Design 30====== 31 32Handshake 33--------- 34 35Borrow code from ``nbd`` [#nbd]_ project. 36 37Transmission 38------------ 39 40Traditionally the transmission phase is implemented as kernel driver of 41``nbd driver`` [#nbd_driver]_. Now we have ublk framework, so it is 42possible to move it out of linux kernel. 43 44NBD protocol [#nbd_protocol]_ is simple, for each block IO request, 45nbd client sends 24byte request header, nbd server responds with one 4616 byte nbd reply. For READ request, the returned IO data follows the 47reply, and now ublk-nbd implements nbd simple reply only, and doesn't 48support structured reply which isn't implemented by ``nbd driver`` 49[#nbd_driver]_ too. For WRITE request, IO data needs to follow the 5024byte request header. 51 52For every IO request delivered from ublk driver, ublk-nbd target code 53handles this IO in one dedicated coroutine bound to IO tag, the 54IO handling includes: 55 56- sending nbd request 57- sending WRITE data 58- reading nbd reply 59- reading nbd READ data 60 61One extra dedicated coroutine is responsible for reading reply and 62data in case of READ request via io_uring & recv(nonblocking) hybrid 63approach. recv(nonblocking) is always tried first: 64 65- if the whole reply or data in case of READ is done by recv, wakeup 66 IO handling coroutine for completing this IO 67 68- if partial reply or data is read, keep to read via recv(nonblocking) 69 until the whole reply or data is read or the max tries are reached. 70 71- otherwise, io_uring is used for handling the remained reply/data 72 73If io_uring is used finally for reading reply or data, when the CQE is 74received, wakeup IO handling coroutine for completing the IO. 75 76Each IO's handling coroutine is responsible for sending nbd request and 77WRITE data in case of WRITE request via io_uring SQE, then wait for 78reply or data in case of READ request, which is notified from the recv 79coroutine. 80 81Even though everything is actually done asynchronously in single pthread 82for each nbd queue, programming with coroutine still looks like every 83step done step by step, so it becomes easier to write efficient async 84IO code with coroutine. Like other ublk targets, c++20 coroutine is used, 85which is stackless and efficient. 86 87Given stream socket is used by nbd, sending request header and data to 88socket has to be serialized, and io_uring's SQE chain is taken with 89help of IOSQE_IO_LINK. There are two chains, one is current chain, another 90chain is next chain. Before each socket send IO in current chain is sent 91to socket, new IO request is staggered into next chain. After the whole 92current chain is done, the next chain is started to be submitted. And 93the chain stuff is handled in ublk target callback of ->handle_io_background(). 94 95Test 96==== 97 98make test T=nbd 99 100 101TODO 102==== 103 104TLS support 105----------- 106 107Timeout handling 108---------------- 109 110More NBD features 111----------------- 112 113- structured replies 114 115References 116========== 117 118.. [#nbd] https://github.com/NetworkBlockDevice/nbd 119.. [#nbd_client] https://github.com/NetworkBlockDevice/nbd/blob/master/nbd-client.c 120.. [#nbd_driver] https://github.com/torvalds/linux/blob/master/drivers/block/nbd.c 121.. [#nbd_protocol] https://github.com/NetworkBlockDevice/nbd/blob/master/doc/proto.md 122.. [#io_uring_send_zc] https://lwn.net/Articles/879724/ 123