1erofs-utils 2=========== 3 4erofs-utils includes user-space tools for erofs filesystem. 5Currently mkfs.erofs and erofsfuse (experimental) are available. 6 7Dependencies & build 8-------------------- 9 10 lz4 1.8.0+ for lz4 enabled [2], lz4 1.9.3+ highly recommended [4][5]. 11 12 libfuse 2.6+ for erofsfuse enabled as a plus. 13 14How to build for lz4-1.9.0 or above 15~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 16 17To build, you can run the following commands in order: 18 19:: 20 21 $ ./autogen.sh 22 $ ./configure 23 $ make 24 25mkfs.erofs binary will be generated under mkfs folder. 26 27* For lz4 < 1.9.2, there are some stability issues about 28 LZ4_compress_destSize(). (lz4hc isn't impacted) [3]. 29 30** For lz4 = 1.9.2, there is a noticeable regression about 31 LZ4_decompress_safe_partial() [5], which impacts erofsfuse 32 functionality for legacy images (without 0PADDING). 33 34How to build for lz4-1.8.0~1.8.3 35~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 36 37For these old lz4 versions, lz4hc algorithm cannot be supported 38without lz4-static installed due to LZ4_compress_HC_destSize() 39unstable api usage, which means lz4 will only be available if 40lz4-static isn't found. 41 42On Fedora, lz4-static can be installed by using: 43 44 yum install lz4-static.x86_64 45 46However, it's still not recommended using those versions directly 47since there are serious bugs in these compressors, see [2] [3] [4] 48as well. 49 50mkfs.erofs 51---------- 52 53two main kinds of erofs images can be generated: (un)compressed. 54 55 - For uncompressed images, there will be none of compression 56 files in these images. However, it can decide whether the tail 57 block of a file should be inlined or not properly [1]. 58 59 - For compressed images, it will try to use lz4(hc) algorithm 60 first for each regular file and see if storage space can be 61 saved with compression. If not, fallback to an uncompressed 62 file. 63 64How to generate erofs images 65~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 66 67Currently lz4 and lz4hc are available for compression, e.g. 68 $ mkfs.erofs -zlz4hc foo.erofs.img foo/ 69 70Or leave all files uncompressed as an option: 71 $ mkfs.erofs foo.erofs.img foo/ 72 73How to generate legacy erofs images 74~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 75 76Decompression inplace and compacted indexes have been introduced in 77linux-5.3, which are not forward-compatible with older kernels. 78 79In order to generate _legacy_ erofs images for old kernels, 80consider adding "-E legacy-compress" to the command line, e.g. 81 82 $ mkfs.erofs -E legacy-compress -zlz4hc foo.erofs.img foo/ 83 84Obsoleted erofs.mkfs 85~~~~~~~~~~~~~~~~~~~~ 86 87There is an original erofs.mkfs version developed by Li Guifu, 88which was replaced by the new erofs-utils implementation. 89 90git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b obsoleted_mkfs 91 92PLEASE NOTE: This version is highly _NOT recommended_ now. 93 94erofsfuse (experimental, unstable) 95---------------------------------- 96 97erofsfuse is introduced to support erofs format for various platforms 98(including older linux kernels) and new on-disk features iteration. 99It can also be used as an unpacking tool for unprivileged users. 100 101It supports fixed-sized output decompression *without* any in-place 102I/O or in-place decompression optimization. Also like the other FUSE 103implementations, it suffers from most common performance issues (e.g. 104significant I/O overhead, double caching, etc.) 105 106Therefore, NEVER use it if performance is the top concern. 107 108Note that xattr & ACL aren't implemented yet due to the current Android 109use-case vs limited time. If you have some interest, contribution is, 110as always, welcome. 111 112How to build erofsfuse 113~~~~~~~~~~~~~~~~~~~~~~ 114 115It's disabled by default as an experimental feature for now, to enable 116and build it manually: 117 118 $ ./configure --enable-fuse 119 $ make 120 121erofsfuse binary will be generated under fuse folder. 122 123How to mount an erofs image with erofsfuse 124~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 125 126As the other FUSE implementations, it's quite simple to mount with 127erofsfuse, e.g.: 128 $ erofsfuse foo.erofs.img foo/ 129 130Alternatively, to make it run in foreground (with debugging level 3): 131 $ erofsfuse -f --dbglevel=3 foo.erofs.img foo/ 132 133To debug erofsfuse (also automatically run in foreground): 134 $ erofsfuse -d foo.erofs.img foo/ 135 136To unmount an erofsfuse mountpoint as a non-root user: 137 $ fusermount -u foo/ 138 139Contribution 140------------ 141 142erofs-utils is under GPLv2+ as a part of erofs project, 143feel free to send patches or feedback to us. 144 145To: 146 linux-erofs mailing list <linux-erofs@lists.ozlabs.org> 147 Li Guifu <bluce.liguifu@huawei.com> 148 Miao Xie <miaoxie@huawei.com> 149 Fang Wei <fangwei1@huawei.com> 150 151Cc: 152 Gao Xiang <xiang@kernel.org> 153 Chao Yu <yuchao0@huawei.com> 154 155Comments 156-------- 157 158[1] According to the erofs on-disk format, the tail block of files 159 could be inlined aggressively with its metadata in order to reduce 160 the I/O overhead and save the storage space (called tail-packing). 161 162[2] There was a bug until lz4-1.8.3, which can crash erofs-utils 163 randomly. Fortunately bugfix by our colleague Qiuyang Sun was 164 merged in lz4-1.9.0. 165 166 For more details, please refer to 167 https://github.com/lz4/lz4/commit/660d21272e4c8a0f49db5fc1e6853f08713dff82 168 169[3] There were many bugfixes merged into lz4-1.9.2 for 170 LZ4_compress_destSize(), and I once ran into some crashs due to 171 those issues. * Again lz4hc is not affected. * 172 173 [LZ4_compress_destSize] Allow 2 more bytes of match length 174 https://github.com/lz4/lz4/commit/690009e2c2f9e5dcb0d40e7c0c40610ce6006eda 175 176 [LZ4_compress_destSize] Fix rare data corruption bug 177 https://github.com/lz4/lz4/commit/6bc6f836a18d1f8fd05c8fc2b42f1d800bc25de1 178 179 [LZ4_compress_destSize] Fix overflow condition 180 https://github.com/lz4/lz4/commit/13a2d9e34ffc4170720ce417c73e396d0ac1471a 181 182 [LZ4_compress_destSize] Fix off-by-one error in fix 183 https://github.com/lz4/lz4/commit/7c32101c655d93b61fc212dcd512b87119dd7333 184 185 [LZ4_compress_destSize] Fix off-by-one error 186 https://github.com/lz4/lz4/commit/d7cad81093cd805110291f84d64d385557d0ffba 187 188 since upstream lz4 doesn't have stable branch for old versions, it's 189 preferred to use latest upstream lz4 library (although some regressions 190 could happen since new features are also introduced to latest upstream 191 version as well) or backport all stable bugfixes to old stable versions, 192 e.g. our unofficial lz4 fork: https://github.com/erofs/lz4 193 194[4] LZ4HC didn't compress long zeroed buffer properly with 195 LZ4_compress_HC_destSize() 196 https://github.com/lz4/lz4/issues/784 197 198 which has been resolved in 199 https://github.com/lz4/lz4/commit/e7fe105ac6ed02019d34731d2ba3aceb11b51bb1 200 201 and already included in lz4-1.9.3, see: 202 https://github.com/lz4/lz4/releases/tag/v1.9.3 203 204[5] LZ4_decompress_safe_partial is broken in 1.9.2 205 https://github.com/lz4/lz4/issues/783 206 207 which is also resolved in lz4-1.9.3. 208 209