• Home
Name Date Size #Lines LOC

..--

LICENSES/03-May-2024-547456

dump/03-May-2024-656568

fsck/03-May-2024-864735

fuse/03-May-2024-321268

include/03-May-2024-2,7291,725

lib/03-May-2024-7,1895,704

man/03-May-2024-293277

mkfs/03-May-2024-765665

scripts/03-May-2024-3416

.gitignoreD03-May-2024289 3432

AUTHORSD03-May-2024274 109

Android.bpD03-May-20244.3 KiB192172

COPYINGD03-May-2024585 1611

ChangeLogD03-May-20243.7 KiB9170

LICENSED03-May-2024585 1611

METADATAD03-May-2024519 2018

MODULE_LICENSE_GPLD03-May-20240

Makefile.amD03-May-2024132 95

OWNERSD03-May-202460 43

READMED03-May-20249.6 KiB291197

VERSIOND03-May-202415 32

autogen.shD03-May-2024190 106

configure.acD03-May-202410.6 KiB372322

README

1erofs-utils
2===========
3
4userspace tools for EROFS filesystem, currently including:
5
6  mkfs.erofs    filesystem formatter
7  erofsfuse     FUSE daemon alternative
8  dump.erofs    filesystem analyzer
9  fsck.erofs    filesystem compatibility & consistency checker as well
10                as extractor
11
12Dependencies & build
13--------------------
14
15 lz4 1.8.0+ for lz4 enabled [2], lz4 1.9.3+ highly recommended [4][5].
16 XZ Utils 5.3.2alpha [6] or later versions for MicroLZMA enabled.
17
18 libfuse 2.6+ for erofsfuse enabled as a plus.
19
20How to build with lz4-1.9.0 or above
21~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
22
23To build, you can run the following commands in order:
24
25::
26
27	$ ./autogen.sh
28	$ ./configure
29	$ make
30
31mkfs.erofs binary will be generated under mkfs folder.
32
33* For lz4 < 1.9.2, there are some stability issues about
34  LZ4_compress_destSize(). (lz4hc isn't impacted) [3].
35
36** For lz4 = 1.9.2, there is a noticeable regression about
37   LZ4_decompress_safe_partial() [5], which impacts erofsfuse
38   functionality for legacy images (without 0PADDING).
39
40How to build with lz4-1.8.0~1.8.3
41~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
42
43For these old lz4 versions, lz4hc algorithm cannot be supported
44without lz4-static installed due to LZ4_compress_HC_destSize()
45unstable api usage, which means lz4 will only be available if
46lz4-static isn't found.
47
48On Fedora, lz4-static can be installed by using:
49
50	yum install lz4-static.x86_64
51
52However, it's still not recommended using those versions directly
53since there are serious bugs in these compressors, see [2] [3] [4]
54as well.
55
56How to build with liblzma
57~~~~~~~~~~~~~~~~~~~~~~~~~
58
59In order to enable LZMA support, build with the following commands:
60	$ ./configure --enable-lzma
61	$ make
62
63Additionally, you could specify liblzma build paths with:
64	--with-liblzma-incdir and --with-liblzma-libdir
65
66
67mkfs.erofs
68----------
69
70two main kinds of EROFS images can be generated: (un)compressed.
71
72 - For uncompressed images, there will be none of compression
73   files in these images. However, it can decide whether the tail
74   block of a file should be inlined or not properly [1].
75
76 - For compressed images, it'll try to use specific algorithms
77   first for each regular file and see if storage space can be
78   saved with compression. If not, fallback to an uncompressed
79   file.
80
81How to generate EROFS images (lz4 for Linux 5.3+, lzma for Linux 5.16+)
82~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
83
84Currently lz4(hc) and lzma are available for compression, e.g.
85 $ mkfs.erofs -zlz4hc foo.erofs.img foo/
86
87Or leave all files uncompressed as an option:
88 $ mkfs.erofs foo.erofs.img foo/
89
90In addition, you could specify a higher compression level to get a
91(slightly) better compression ratio than the default level, e.g.
92 $ mkfs.erofs -zlz4hc,12 foo.erofs.img foo/
93
94Note that all compressors are still single-threaded for now, thus it
95could take more time on the multiprocessor platform. Multi-threaded
96approach is already in our TODO list.
97
98How to generate EROFS big pcluster images (Linux 5.13+)
99~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
100
101In order to get much better compression ratios (thus better sequential
102read performance for common storage devices), big pluster feature has
103been introduced since linux-5.13, which is not forward-compatible with
104old kernels.
105
106In details, -C is used to specify the maximum size of each big pcluster
107in bytes, e.g.
108 $ mkfs.erofs -zlz4hc -C65536 foo.erofs.img foo/
109
110So in that case, pcluster size can be 64KiB at most.
111
112Note that large pcluster size can cause bad random performance, so
113please evaluate carefully in advance. Or make your own per-(sub)file
114compression strategies according to file access patterns if needed.
115
116How to generate legacy EROFS images (Linux 4.19+)
117~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
118
119Decompression inplace and compacted indexes have been introduced in
120Linux upstream v5.3, which are not forward-compatible with older
121kernels.
122
123In order to generate _legacy_ EROFS images for old kernels,
124consider adding "-E legacy-compress" to the command line, e.g.
125
126 $ mkfs.erofs -E legacy-compress -zlz4hc foo.erofs.img foo/
127
128For Linux kernel >= 5.3, legacy EROFS images are _NOT recommended_
129due to runtime performance loss compared with non-legacy images.
130
131Obsoleted erofs.mkfs
132~~~~~~~~~~~~~~~~~~~~
133
134There is an original erofs.mkfs version developed by Li Guifu,
135which was replaced by the new erofs-utils implementation.
136
137git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git -b obsoleted_mkfs
138
139PLEASE NOTE: This version is highly _NOT recommended_ now.
140
141
142erofsfuse
143---------
144
145erofsfuse is introduced to support EROFS format for various platforms
146(including older linux kernels) and new on-disk features iteration.
147It can also be used as an unpacking tool for unprivileged users.
148
149It supports fixed-sized output decompression *without* any in-place
150I/O or in-place decompression optimization. Also like the other FUSE
151implementations, it suffers from most common performance issues (e.g.
152significant I/O overhead, double caching, etc.)
153
154Therefore, NEVER use it if performance is the top concern.
155
156Note that extended attributes and ACLs aren't implemented yet due to
157the current Android use case vs limited time. If you are interested,
158contribution is, as always, welcome.
159
160How to build erofsfuse
161~~~~~~~~~~~~~~~~~~~~~~
162
163It's disabled by default as an experimental feature for now due to
164the extra libfuse dependency, to enable and build it manually:
165
166	$ ./configure --enable-fuse
167	$ make
168
169erofsfuse binary will be generated under fuse folder.
170
171How to mount an EROFS image with erofsfuse
172~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
173
174As the other FUSE implementations, it's quite simple to mount with
175erofsfuse, e.g.:
176 $ erofsfuse foo.erofs.img foo/
177
178Alternatively, to make it run in foreground (with debugging level 3):
179 $ erofsfuse -f --dbglevel=3 foo.erofs.img foo/
180
181To debug erofsfuse (also automatically run in foreground):
182 $ erofsfuse -d foo.erofs.img foo/
183
184To unmount an erofsfuse mountpoint as a non-root user:
185 $ fusermount -u foo/
186
187
188dump.erofs and fsck.erofs
189-------------------------
190
191dump.erofs and fsck.erofs are used to analyze, check, and extract
192EROFS filesystems. Note that extended attributes and ACLs are still
193unsupported when extracting images with fsck.erofs.
194
195Container images
196----------------
197
198EROFS filesystem is well-suitably used for container images with
199advanced features like chunk-based files, multi-devices (blobs)
200and new fscache backend for lazy pulling and cache management, etc.
201
202For example, CNCF Dragonfly Nydus image service [7] introduces an
203(EROFS-compatible) RAFS v6 image format to overcome flaws of the
204current OCIv1 tgz images so that:
205
206 - Images can be downloaded on demand in chunks aka lazy pulling with
207   new fscache backend (5.19+) or userspace block devices (5.16+);
208
209 - Finer chunk-based content-addressable data deduplication to minimize
210   storage, transmission and memory footprints;
211
212 - Merged filesystem tree to remove all metadata of intermediate layers
213   as an option;
214
215 - (e)stargz, zstd::chunked and other formats can be converted and run
216   on the fly;
217
218 - and more.
219
220Apart from Dragonfly Nydus, a native user daemon is planned to be added
221to erofs-utils to parse EROFS, (e)stargz and zstd::chunked images from
222network too as a real part of EROFS filesystem project.
223
224
225Contribution
226------------
227
228erofs-utils is a part of EROFS filesystem project, feel free to send
229patches or feedback to:
230  linux-erofs mailing list   <linux-erofs@lists.ozlabs.org>
231
232
233Comments
234--------
235
236[1] According to the EROFS on-disk format, the tail block of files
237    could be inlined aggressively with its metadata in order to reduce
238    the I/O overhead and save the storage space (called tail-packing).
239
240[2] There was a bug until lz4-1.8.3, which can crash erofs-utils
241    randomly. Fortunately bugfix by our colleague Qiuyang Sun was
242    merged in lz4-1.9.0.
243
244    For more details, please refer to
245    https://github.com/lz4/lz4/commit/660d21272e4c8a0f49db5fc1e6853f08713dff82
246
247[3] There were many bugfixes merged into lz4-1.9.2 for
248    LZ4_compress_destSize(), and I once ran into some crashs due to
249    those issues. * Again lz4hc is not affected. *
250
251    [LZ4_compress_destSize] Allow 2 more bytes of match length
252    https://github.com/lz4/lz4/commit/690009e2c2f9e5dcb0d40e7c0c40610ce6006eda
253
254    [LZ4_compress_destSize] Fix rare data corruption bug
255    https://github.com/lz4/lz4/commit/6bc6f836a18d1f8fd05c8fc2b42f1d800bc25de1
256
257    [LZ4_compress_destSize] Fix overflow condition
258    https://github.com/lz4/lz4/commit/13a2d9e34ffc4170720ce417c73e396d0ac1471a
259
260    [LZ4_compress_destSize] Fix off-by-one error in fix
261    https://github.com/lz4/lz4/commit/7c32101c655d93b61fc212dcd512b87119dd7333
262
263    [LZ4_compress_destSize] Fix off-by-one error
264    https://github.com/lz4/lz4/commit/d7cad81093cd805110291f84d64d385557d0ffba
265
266    since upstream lz4 doesn't have stable branch for old versions, it's
267    preferred to use latest upstream lz4 library (although some regressions
268    could happen since new features are also introduced to latest upstream
269    version as well) or backport all stable bugfixes to old stable versions,
270    e.g. our unofficial lz4 fork: https://github.com/erofs/lz4
271
272[4] LZ4HC didn't compress long zeroed buffer properly with
273    LZ4_compress_HC_destSize()
274    https://github.com/lz4/lz4/issues/784
275
276    which has been resolved in
277    https://github.com/lz4/lz4/commit/e7fe105ac6ed02019d34731d2ba3aceb11b51bb1
278
279    and already included in lz4-1.9.3, see:
280    https://github.com/lz4/lz4/releases/tag/v1.9.3
281
282[5] LZ4_decompress_safe_partial is broken in 1.9.2
283    https://github.com/lz4/lz4/issues/783
284
285    which is also resolved in lz4-1.9.3.
286
287[6] https://tukaani.org/xz/xz-5.3.2alpha.tar.xz
288
289[7] https://nydus.dev
290    https://github.com/dragonflyoss/image-service
291