1erofs-utils 2=========== 3 4erofs-utils includes user-space tools for EROFS filesystem. 5Currently mkfs.erofs, (experimental) erofsfuse, dump.erofs, fsck.erofs 6are available. 7 8Dependencies & build 9-------------------- 10 11 lz4 1.8.0+ for lz4 enabled [2], lz4 1.9.3+ highly recommended [4][5]. 12 XZ Utils 5.3.2alpha [6] or later versions for MicroLZMA enabled. 13 14 libfuse 2.6+ for erofsfuse enabled as a plus. 15 16How to build with lz4-1.9.0 or above 17~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 18 19To build, you can run the following commands in order: 20 21:: 22 23 $ ./autogen.sh 24 $ ./configure 25 $ make 26 27mkfs.erofs binary will be generated under mkfs folder. 28 29* For lz4 < 1.9.2, there are some stability issues about 30 LZ4_compress_destSize(). (lz4hc isn't impacted) [3]. 31 32** For lz4 = 1.9.2, there is a noticeable regression about 33 LZ4_decompress_safe_partial() [5], which impacts erofsfuse 34 functionality for legacy images (without 0PADDING). 35 36How to build with lz4-1.8.0~1.8.3 37~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 38 39For these old lz4 versions, lz4hc algorithm cannot be supported 40without lz4-static installed due to LZ4_compress_HC_destSize() 41unstable api usage, which means lz4 will only be available if 42lz4-static isn't found. 43 44On Fedora, lz4-static can be installed by using: 45 46 yum install lz4-static.x86_64 47 48However, it's still not recommended using those versions directly 49since there are serious bugs in these compressors, see [2] [3] [4] 50as well. 51 52How to build with liblzma 53~~~~~~~~~~~~~~~~~~~~~~~~~ 54 55In order to enable LZMA support, build with the following commands: 56 $ ./configure --enable-lzma 57 $ make 58 59Additionally, you could specify liblzma build paths with: 60 --with-liblzma-incdir and --with-liblzma-libdir 61 62mkfs.erofs 63---------- 64 65two main kinds of EROFS images can be generated: (un)compressed. 66 67 - For uncompressed images, there will be none of compression 68 files in these images. However, it can decide whether the tail 69 block of a file should be inlined or not properly [1]. 70 71 - For compressed images, it'll try to use specific algorithms 72 first for each regular file and see if storage space can be 73 saved with compression. If not, fallback to an uncompressed 74 file. 75 76How to generate EROFS images (lz4 for Linux 5.3+, lzma for Linux 5.16+) 77~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 78 79Currently lz4(hc) and lzma are available for compression, e.g. 80 $ mkfs.erofs -zlz4hc foo.erofs.img foo/ 81 82Or leave all files uncompressed as an option: 83 $ mkfs.erofs foo.erofs.img foo/ 84 85In addition, you could specify a higher compression level to get a 86(slightly) better compression ratio than the default level, e.g. 87 $ mkfs.erofs -zlz4hc,12 foo.erofs.img foo/ 88 89Note that all compressors are still single-threaded for now, thus it 90could take more time on the multiprocessor platform. Multi-threaded 91approach is already in our TODO list. 92 93How to generate EROFS big pcluster images (Linux 5.13+) 94~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 95 96In order to get much better compression ratios (thus better sequential 97read performance for common storage devices), big pluster feature has 98been introduced since linux-5.13, which is not forward-compatible with 99old kernels. 100 101In details, -C is used to specify the maximum size of each big pcluster 102in bytes, e.g. 103 $ mkfs.erofs -zlz4hc -C65536 foo.erofs.img foo/ 104 105So in that case, pcluster size can be 64KiB at most. 106 107Note that large pcluster size can cause bad random performance, so 108please evaluate carefully in advance. Or make your own per-(sub)file 109compression strategies according to file access patterns if needed. 110 111How to generate legacy EROFS images (Linux 4.19+) 112~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 113 114Decompression inplace and compacted indexes have been introduced in 115Linux upstream v5.3, which are not forward-compatible with older 116kernels. 117 118In order to generate _legacy_ EROFS images for old kernels, 119consider adding "-E legacy-compress" to the command line, e.g. 120 121 $ mkfs.erofs -E legacy-compress -zlz4hc foo.erofs.img foo/ 122 123For Linux kernel >= 5.3, legacy EROFS images are _NOT recommended_ 124due to runtime performance loss compared with non-legacy images. 125 126Obsoleted erofs.mkfs 127~~~~~~~~~~~~~~~~~~~~ 128 129There is an original erofs.mkfs version developed by Li Guifu, 130which was replaced by the new erofs-utils implementation. 131 132git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b obsoleted_mkfs 133 134PLEASE NOTE: This version is highly _NOT recommended_ now. 135 136erofsfuse (experimental) 137------------------------ 138 139erofsfuse is introduced to support EROFS format for various platforms 140(including older linux kernels) and new on-disk features iteration. 141It can also be used as an unpacking tool for unprivileged users. 142 143It supports fixed-sized output decompression *without* any in-place 144I/O or in-place decompression optimization. Also like the other FUSE 145implementations, it suffers from most common performance issues (e.g. 146significant I/O overhead, double caching, etc.) 147 148Therefore, NEVER use it if performance is the top concern. 149 150Note that xattr & ACL aren't implemented yet due to the current Android 151use-case vs limited time. If you have some interest, contribution is, 152as always, welcome. 153 154How to build erofsfuse 155~~~~~~~~~~~~~~~~~~~~~~ 156 157It's disabled by default as an experimental feature for now due to 158the extra libfuse dependency, to enable and build it manually: 159 160 $ ./configure --enable-fuse 161 $ make 162 163erofsfuse binary will be generated under fuse folder. 164 165How to mount an EROFS image with erofsfuse 166~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 167 168As the other FUSE implementations, it's quite simple to mount with 169erofsfuse, e.g.: 170 $ erofsfuse foo.erofs.img foo/ 171 172Alternatively, to make it run in foreground (with debugging level 3): 173 $ erofsfuse -f --dbglevel=3 foo.erofs.img foo/ 174 175To debug erofsfuse (also automatically run in foreground): 176 $ erofsfuse -d foo.erofs.img foo/ 177 178To unmount an erofsfuse mountpoint as a non-root user: 179 $ fusermount -u foo/ 180 181dump.erofs and fsck.erofs (experimental) 182~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 183 184dump.erofs and fsck.erofs are two new experimental tools to analyse 185and check EROFS file systems. 186 187They are still incomplete and actively under development by the 188community. But you could check them out if needed in advance. 189 190Report, feedback and/or contribution are welcomed. 191 192Contribution 193------------ 194 195erofs-utils is under GPLv2+ as a part of EROFS filesystem project, 196feel free to send patches or feedback to: 197 linux-erofs mailing list <linux-erofs@lists.ozlabs.org> 198 199Comments 200-------- 201 202[1] According to the EROFS on-disk format, the tail block of files 203 could be inlined aggressively with its metadata in order to reduce 204 the I/O overhead and save the storage space (called tail-packing). 205 206[2] There was a bug until lz4-1.8.3, which can crash erofs-utils 207 randomly. Fortunately bugfix by our colleague Qiuyang Sun was 208 merged in lz4-1.9.0. 209 210 For more details, please refer to 211 https://github.com/lz4/lz4/commit/660d21272e4c8a0f49db5fc1e6853f08713dff82 212 213[3] There were many bugfixes merged into lz4-1.9.2 for 214 LZ4_compress_destSize(), and I once ran into some crashs due to 215 those issues. * Again lz4hc is not affected. * 216 217 [LZ4_compress_destSize] Allow 2 more bytes of match length 218 https://github.com/lz4/lz4/commit/690009e2c2f9e5dcb0d40e7c0c40610ce6006eda 219 220 [LZ4_compress_destSize] Fix rare data corruption bug 221 https://github.com/lz4/lz4/commit/6bc6f836a18d1f8fd05c8fc2b42f1d800bc25de1 222 223 [LZ4_compress_destSize] Fix overflow condition 224 https://github.com/lz4/lz4/commit/13a2d9e34ffc4170720ce417c73e396d0ac1471a 225 226 [LZ4_compress_destSize] Fix off-by-one error in fix 227 https://github.com/lz4/lz4/commit/7c32101c655d93b61fc212dcd512b87119dd7333 228 229 [LZ4_compress_destSize] Fix off-by-one error 230 https://github.com/lz4/lz4/commit/d7cad81093cd805110291f84d64d385557d0ffba 231 232 since upstream lz4 doesn't have stable branch for old versions, it's 233 preferred to use latest upstream lz4 library (although some regressions 234 could happen since new features are also introduced to latest upstream 235 version as well) or backport all stable bugfixes to old stable versions, 236 e.g. our unofficial lz4 fork: https://github.com/erofs/lz4 237 238[4] LZ4HC didn't compress long zeroed buffer properly with 239 LZ4_compress_HC_destSize() 240 https://github.com/lz4/lz4/issues/784 241 242 which has been resolved in 243 https://github.com/lz4/lz4/commit/e7fe105ac6ed02019d34731d2ba3aceb11b51bb1 244 245 and already included in lz4-1.9.3, see: 246 https://github.com/lz4/lz4/releases/tag/v1.9.3 247 248[5] LZ4_decompress_safe_partial is broken in 1.9.2 249 https://github.com/lz4/lz4/issues/783 250 251 which is also resolved in lz4-1.9.3. 252 253[6] https://tukaani.org/xz/xz-5.3.2alpha.tar.xz 254