1 2============================ 3Userspace block driver(ublk) 4============================ 5 6Introduction 7============ 8 9This is the userspace daemon part(ublksrv) of the ublk framework, the other 10part is ``ublk driver`` [#userspace]_ which supports multiple queue. 11 12The two parts communicate by io_uring's IORING_OP_URING_CMD with one 13per-queue shared cmd buffer for storing io command, and the buffer is 14read only for ublksrv, each io command can be indexed by io request tag 15directly, and the command is written by ublk driver, and read by ublksrv 16after getting notification from ublk driver. 17 18For example, when one READ io request is submitted to ublk block driver, ublk 19driver stores the io command into cmd buffer first, then completes one 20IORING_OP_URING_CMD for notifying ublksrv, and the URING_CMD is issued to 21ublk driver beforehand by ublksrv for getting notification of any new io 22request, and each URING_CMD is associated with one io request by tag, 23so depth for URING_CMD is same with queue depth of ublk block device. 24 25After ublksrv gets the io command, it translates and handles the ublk io 26request, such as, for the ublk-loop target, ublksrv translates the request 27into same request on another file or disk, like the kernel loop block 28driver. In ublksrv's implementation, the io is still handled by io_uring, 29and share same ring with IORING_OP_URING_CMD command. When the target io 30request is done, the same IORING_OP_URING_CMD is issued to ublk driver for 31both committing io request result and getting future notification of new 32io request. 33 34So far, the ublk driver needs to copy io request pages into userspace buffer 35(pages) first for write before notifying the request to ublksrv, and copy 36userspace buffer(pages) to the io request pages after ublksrv handles 37READ. Also looks linux-mm can't support zero copy for this case yet. [#zero_copy]_ 38 39More ublk targets will be added with this framework in future even though only 40ublk-loop and ublk-null are implemented now. 41 42libublksrv is also generated, and it helps to integrate ublk into existed 43project. One example of demo_null is provided for how to make a ublk 44device over libublksrv. 45 46Quick start 47=========== 48 49how to build ublksrv: 50-------------------- 51 52.. code-block:: console 53 54 autoreconf -i 55 ./configure #pkg-config and libtool is usually needed 56 make 57 58note: './configure' requires liburing 2.2 package installed, if liburing 2.2 59isn't available in your distribution, please configure via the following 60command, or refer to ``build_with_liburing_src`` [#build_with_liburing_src]_ 61 62.. code-block:: console 63 64 PKG_CONFIG_PATH=${LIBURING_DIR} \ 65 ./configure \ 66 CFLAGS="-I${LIBURING_DIR}/src/include" \ 67 CXXFLAGS="-I${LIBURING_DIR}/src/include" \ 68 LDFLAGS="-L${LIBURING_DIR}/src" 69 70and LIBURING_DIR points to directory of liburing source code, and liburing 71needs to be built before running above commands. Also IORING_SETUP_SQE128 72has to be supported in the liburing source. 73 74c++20 is required for building ublk utility, but libublksrv and demo_null.c & 75demo_event.c can be built independently: 76 77- build libublksrv :: 78 79 make -C lib/ 80 81- build demo_null && demo_event :: 82 83 make -C lib/ 84 make demo_null demo_event 85 86help 87---- 88 89- ublk help 90 91add one ublk-null disk 92---------------------- 93 94- ublk add -t null 95 96 97add one ublk-loop disk 98---------------------- 99 100- ublk add -t loop -f /dev/vdb 101 102or 103 104- ublk add -t loop -f 1.img 105 106remove one ublk disk 107-------------------- 108 109- ublk del -n 0 #remove /dev/ublkb0 110 111- ublk del -a #remove all ublk devices 112 113list ublk devices 114--------------------- 115 116- ublk list 117 118- ublk list -v #with all device info dumped 119 120 121unprivileged mode 122================== 123 124Typical use case is container [#stefan_container]_ in which user 125can manage its own devices not exposed to other containers. 126 127At default, controlling ublk device needs privileged user, since 128/dev/ublk-control is permitted for administrator only, and this 129is called privileged mode. 130 131For unprivilege mode, /dev/ublk-control needs to be allowed for 132all users, so the following udev rule need to be added: 133 134KERNEL=="ublk-control", MODE="0666", OPTIONS+="static_node=ublk-control" 135 136Also when new ublk device is added, we need ublk to change device 137ownership to the device's real owner, so the following rules are 138needed: :: 139 140 KERNEL=="ublkc*",RUN+="ublk_chown.sh %k" 141 KERNEL=="ublkb*",RUN+="ublk_chown.sh %k" 142 143``ublk_chown.sh`` can be found under ``utils/`` too. 144 145``utils/ublk_dev.rules`` includes the above rules. 146 147With the above two administrator changes, unprivileged user can 148create/delete/list/use ublk device, also anyone which isn't permitted 149can't access and control this ublk devices(ublkc*/ublkb*) 150 151Unprivileged user can pass '--unprevileged' to 'ublk add' for creating 152unprivileged ublk device, then the created ublk device is only available 153for the owner and administrator. 154 155use unprivileged ublk in docker 156------------------------------- 157 158- install the following udev rules in host machine: :: 159 160 ACTION=="add",KERNEL=="ublk[bc]*",RUN+="/usr/local/sbin/ublk_chown_docker.sh %k 'add' '%M' '%m'" 161 ACTION=="remove",KERNEL=="ublk[bc]*",RUN+="/usr/local/sbin/ublk_chown_docker.sh %k 'remove' '%M' '%m'" 162 163``ublk_chown_docker.sh`` can be found under ``utils/``. 164 165- run one container and install ublk & its dependency packages 166 167.. code-block:: console 168 169 docker run \ 170 --name fedora \ 171 --hostname=ublk-docker.example.com \ 172 --device=/dev/ublk-control \ 173 --device-cgroup-rule='a *:* rmw' \ 174 --tmpfs /tmp \ 175 --tmpfs /run \ 176 --volume /sys/fs/cgroup:/sys/fs/cgroup:ro \ 177 -ti \ 178 fedora:38 179 180.. code-block:: console 181 182 #run the following commands inside the above container 183 dnf install -y git libtool automake autoconf g++ liburing-devel 184 git clone https://github.com/ming1/ubdsrv.git 185 cd ubdsrv 186 autoreconf -i&& ./configure&& make -j 4&& make install 187 188- add/delete ublk device inside container by unprivileged user 189 190.. code-block:: console 191 192 docker exec -u 1001:1001 -ti fedora /bin/bash 193 194.. code-block:: console 195 196 #run the following commands inside the above container 197 bash-5.2$ ublk add -t null --unprivileged 198 dev id 0: nr_hw_queues 1 queue_depth 128 block size 512 dev_capacity 524288000 199 max rq size 524288 daemon pid 178 flags 0x62 state LIVE 200 ublkc: 237:0 ublkb: 259:1 owner: 1001:1001 201 queue 0: tid 179 affinity(0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ) 202 target {"dev_size":268435456000,"name":"null","type":0} 203 204 bash-5.2$ ls -l /dev/ublk* 205 crw-rw-rw-. 1 root root 10, 123 May 1 04:35 /dev/ublk-control 206 brwx------. 1 1001 1001 259, 1 May 1 04:36 /dev/ublkb0 207 crwx------. 1 1001 1001 237, 0 May 1 04:36 /dev/ublkc0 208 209 bash-5.2$ ublk del -n 0 210 bash-5.2$ ls -l /dev/ublk* 211 crw-rw-rw-. 1 root root 10, 123 May 1 04:35 /dev/ublk-control 212 213- example of ublk in docker: ``tests/debug/ublk_docker`` 214 215test 216==== 217 218run all built tests 219------------------- 220 221make test T=all 222 223 224run test group 225-------------- 226 227make test T=null 228 229make test T=loop 230 231make test T=generic 232 233 234run single test 235--------------- 236 237make test T=generic/001 238 239make test T=null/001 240 241make test T=loop/001 242... 243 244run specified tests or test groups 245---------------------------------- 246 247make test T=generic:loop/001:null 248 249 250Debug 251===== 252 253ublksrv is running as one daemon process, so most of debug messages won't be 254shown in terminal. If any issue is observed, please collect log via command 255of "journalctl | grep ublksrvd" 256 257``./configure --enable-debug`` can build a debug version of ublk which 258dumps lots of runtime debug messages, and can't be used in production 259environment, should be for debug purpose only. For debug version of 260ublksrv, 'ublk add --debug_mask=0x{MASK}' can control which kind of 261debug log dumped, see ``UBLK_DBG_*`` defined in include/ublksrv_utils.h 262for each kind of debug log. 263 264libublksrv API doc 265================== 266 267API is documented in include/ublksrv.h, and doxygen doc can be generated 268by running 'make doxygen_doc', the generated html docs are in doc/html. 269 270Contributing 271============ 272 273Any kind of contribution is welcome! 274 275Development is done over github. 276 277 278Todo: 279==== 280 281libublk 282------ 283 284Move libublksrv out of ublksrv project, and make it as one standalone repo 285and name it as libublk. 286 287It is planned to do it when ublk driver UAPI changes(feature addition) is slow down. 288 289License 290======= 291 292nlohmann(include/nlohmann/json.hpp) is from [#nlohmann]_, which is covered 293by MIT license. 294 295The library functions (all code in lib/ directory and include/ublksrv.h) 296are covered by dual licensed LGPL and MIT, see COPYING.LGPL and LICENSE. 297 298All other source code are covered by dual licensed GPL and MIT, see 299COPYING and LICENSE. 300 301References 302========== 303 304.. [#ublk_driver] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/block/ublk_drv.c?h=v6.0 305.. [#zero_copy] https://lore.kernel.org/all/20220318095531.15479-1-xiaoguang.wang@linux.alibaba.com/ 306.. [#nlohmann] https://github.com/nlohmann/json 307.. [#build_with_liburing_src] https://github.com/ming1/ubdsrv/blob/master/build_with_liburing_src 308.. [#stefan_container] https://lore.kernel.org/linux-block/YoOr6jBfgVm8GvWg@stefanha-x1.localdomain/ 309