README.md
1Zstandard library files
2================================
3
4The __lib__ directory is split into several sub-directories,
5in order to make it easier to select or exclude features.
6
7
8#### Building
9
10`Makefile` script is provided, supporting [Makefile conventions](https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html#Makefile-Conventions),
11including commands variables, staged install, directory variables and standard targets.
12- `make` : generates both static and dynamic libraries
13- `make install` : install libraries and headers in target system directories
14
15`libzstd` default scope is pretty large, including compression, decompression, dictionary builder,
16and support for decoding legacy formats >= v0.5.0.
17The scope can be reduced on demand (see paragraph _modular build_).
18
19
20#### Multithreading support
21
22When building with `make`, by default the dynamic library is multithreaded and static library is single-threaded (for compatibility reasons).
23
24Enabling multithreading requires 2 conditions :
25- set build macro `ZSTD_MULTITHREAD` (`-DZSTD_MULTITHREAD` for `gcc`)
26- for POSIX systems : compile with pthread (`-pthread` compilation flag for `gcc`)
27
28For convenience, we provide a build target to generate multi and single threaded libraries:
29- Force enable multithreading on both dynamic and static libraries by appending `-mt` to the target, e.g. `make lib-mt`.
30 Note that the `.pc` generated on calling `make lib-mt` will already include the require Libs and Cflags.
31- Force disable multithreading on both dynamic and static libraries by appending `-nomt` to the target, e.g. `make lib-nomt`.
32- By default, as mentioned before, dynamic library is multithreaded, and static library is single-threaded, e.g. `make lib`.
33
34When linking a POSIX program with a multithreaded version of `libzstd`,
35note that it's necessary to invoke the `-pthread` flag during link stage.
36
37The `.pc` generated from `make install` or `make install-pc` always assume a single-threaded static library
38is compiled. To correctly generate a `.pc` for the multi-threaded static library, set `MT=1` as ENV variable.
39
40Multithreading capabilities are exposed
41via the [advanced API defined in `lib/zstd.h`](https://github.com/facebook/zstd/blob/v1.4.3/lib/zstd.h#L351).
42
43
44#### API
45
46Zstandard's stable API is exposed within [lib/zstd.h](zstd.h).
47
48
49#### Advanced API
50
51Optional advanced features are exposed via :
52
53- `lib/zstd_errors.h` : translates `size_t` function results
54 into a `ZSTD_ErrorCode`, for accurate error handling.
55
56- `ZSTD_STATIC_LINKING_ONLY` : if this macro is defined _before_ including `zstd.h`,
57 it unlocks access to the experimental API,
58 exposed in the second part of `zstd.h`.
59 All definitions in the experimental APIs are unstable,
60 they may still change in the future, or even be removed.
61 As a consequence, experimental definitions shall ___never be used with dynamic library___ !
62 Only static linking is allowed.
63
64
65#### Modular build
66
67It's possible to compile only a limited set of features within `libzstd`.
68The file structure is designed to make this selection manually achievable for any build system :
69
70- Directory `lib/common` is always required, for all variants.
71
72- Compression source code lies in `lib/compress`
73
74- Decompression source code lies in `lib/decompress`
75
76- It's possible to include only `compress` or only `decompress`, they don't depend on each other.
77
78- `lib/dictBuilder` : makes it possible to generate dictionaries from a set of samples.
79 The API is exposed in `lib/dictBuilder/zdict.h`.
80 This module depends on both `lib/common` and `lib/compress` .
81
82- `lib/legacy` : makes it possible to decompress legacy zstd formats, starting from `v0.1.0`.
83 This module depends on `lib/common` and `lib/decompress`.
84 To enable this feature, define `ZSTD_LEGACY_SUPPORT` during compilation.
85 Specifying a number limits versions supported to that version onward.
86 For example, `ZSTD_LEGACY_SUPPORT=2` means : "support legacy formats >= v0.2.0".
87 Conversely, `ZSTD_LEGACY_SUPPORT=0` means "do __not__ support legacy formats".
88 By default, this build macro is set as `ZSTD_LEGACY_SUPPORT=5`.
89 Decoding supported legacy format is a transparent capability triggered within decompression functions.
90 It's also allowed to invoke legacy API directly, exposed in `lib/legacy/zstd_legacy.h`.
91 Each version does also provide its own set of advanced API.
92 For example, advanced API for version `v0.4` is exposed in `lib/legacy/zstd_v04.h` .
93
94- While invoking `make libzstd`, it's possible to define build macros
95 `ZSTD_LIB_COMPRESSION`, `ZSTD_LIB_DECOMPRESSION`, `ZSTD_LIB_DICTBUILDER`,
96 and `ZSTD_LIB_DEPRECATED` as `0` to forgo compilation of the
97 corresponding features. This will also disable compilation of all
98 dependencies (e.g. `ZSTD_LIB_COMPRESSION=0` will also disable
99 dictBuilder).
100
101- There are a number of options that can help minimize the binary size of
102 `libzstd`.
103
104 The first step is to select the components needed (using the above-described
105 `ZSTD_LIB_COMPRESSION` etc.).
106
107 The next step is to set `ZSTD_LIB_MINIFY` to `1` when invoking `make`. This
108 disables various optional components and changes the compilation flags to
109 prioritize space-saving.
110
111 Detailed options: Zstandard's code and build environment is set up by default
112 to optimize above all else for performance. In pursuit of this goal, Zstandard
113 makes significant trade-offs in code size. For example, Zstandard often has
114 more than one implementation of a particular component, with each
115 implementation optimized for different scenarios. For example, the Huffman
116 decoder has complementary implementations that decode the stream one symbol at
117 a time or two symbols at a time. Zstd normally includes both (and dispatches
118 between them at runtime), but by defining `HUF_FORCE_DECOMPRESS_X1` or
119 `HUF_FORCE_DECOMPRESS_X2`, you can force the use of one or the other, avoiding
120 compilation of the other. Similarly, `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT`
121 and `ZSTD_FORCE_DECOMPRESS_SEQUENCES_LONG` force the compilation and use of
122 only one or the other of two decompression implementations. The smallest
123 binary is achieved by using `HUF_FORCE_DECOMPRESS_X1` and
124 `ZSTD_FORCE_DECOMPRESS_SEQUENCES_SHORT` (implied by `ZSTD_LIB_MINIFY`).
125
126 On the compressor side, Zstd's compression levels map to several internal
127 strategies. In environments where the higher compression levels aren't used,
128 it is possible to exclude all but the fastest strategy with
129 `ZSTD_LIB_EXCLUDE_COMPRESSORS_DFAST_AND_UP=1`. (Note that this will change
130 the behavior of the default compression level.) Or if you want to retain the
131 default compressor as well, you can set
132 `ZSTD_LIB_EXCLUDE_COMPRESSORS_GREEDY_AND_UP=1`, at the cost of an additional
133 ~20KB or so.
134
135 For squeezing the last ounce of size out, you can also define
136 `ZSTD_NO_INLINE`, which disables inlining, and `ZSTD_STRIP_ERROR_STRINGS`,
137 which removes the error messages that are otherwise returned by
138 `ZSTD_getErrorName` (implied by `ZSTD_LIB_MINIFY`).
139
140 Finally, when integrating into your application, make sure you're doing link-
141 time optimization and unused symbol garbage collection (via some combination of,
142 e.g., `-flto`, `-ffat-lto-objects`, `-fuse-linker-plugin`,
143 `-ffunction-sections`, `-fdata-sections`, `-fmerge-all-constants`,
144 `-Wl,--gc-sections`, `-Wl,-z,norelro`, and an archiver that understands
145 the compiler's intermediate representation, e.g., `AR=gcc-ar`). Consult your
146 compiler's documentation.
147
148- While invoking `make libzstd`, the build macro `ZSTD_LEGACY_MULTITHREADED_API=1`
149 will expose the deprecated `ZSTDMT` API exposed by `zstdmt_compress.h` in
150 the shared library, which is now hidden by default.
151
152- The build macro `STATIC_BMI2` can be set to 1 to force usage of `bmi2` instructions.
153 It is generally not necessary to set this build macro,
154 because `STATIC_BMI2` will be automatically set to 1
155 on detecting the presence of the corresponding instruction set in the compilation target.
156 It's nonetheless available as an optional manual toggle for better control,
157 and can also be used to forcefully disable `bmi2` instructions by setting it to 0.
158
159- The build macro `DYNAMIC_BMI2` can be set to 1 or 0 in order to generate binaries
160 which can detect at runtime the presence of BMI2 instructions, and use them only if present.
161 These instructions contribute to better performance, notably on the decoder side.
162 By default, this feature is automatically enabled on detecting
163 the right instruction set (x64) and compiler (clang or gcc >= 5).
164 It's obviously disabled for different cpus,
165 or when BMI2 instruction set is _required_ by the compiler command line
166 (in this case, only the BMI2 code path is generated).
167 Setting this macro will either force to generate the BMI2 dispatcher (1)
168 or prevent it (0). It overrides automatic detection.
169
170- The build macro `ZSTD_NO_UNUSED_FUNCTIONS` can be defined to hide the definitions of functions
171 that zstd does not use. Not all unused functions are hidden, but they can be if needed.
172 Currently, this macro will hide function definitions in FSE and HUF that use an excessive
173 amount of stack space.
174
175- The build macro `ZSTD_NO_INTRINSICS` can be defined to disable all explicit intrinsics.
176 Compiler builtins are still used.
177
178- The build macro `ZSTD_DECODER_INTERNAL_BUFFER` can be set to control
179 the amount of extra memory used during decompression to store literals.
180 This defaults to 64kB. Reducing this value reduces the memory footprint of
181 `ZSTD_DCtx` decompression contexts,
182 but might also result in a small decompression speed cost.
183
184- The C compiler macros `ZSTDLIB_VISIBLE`, `ZSTDERRORLIB_VISIBLE` and `ZDICTLIB_VISIBLE`
185 can be overridden to control the visibility of zstd's API. Additionally,
186 `ZSTDLIB_STATIC_API` and `ZDICTLIB_STATIC_API` can be overridden to control the visibility
187 of zstd's static API. Specifically, it can be set to `ZSTDLIB_HIDDEN` to hide the symbols
188 from the shared library. These macros default to `ZSTDLIB_VISIBILITY`,
189 `ZSTDERRORLIB_VSIBILITY`, and `ZDICTLIB_VISIBILITY` if unset, for backwards compatibility
190 with the old macro names.
191
192- The C compiler macro `HUF_DISABLE_FAST_DECODE` disables the newer Huffman fast C
193 and assembly decoding loops. You may want to use this macro if these loops are
194 slower on your platform.
195
196#### Windows : using MinGW+MSYS to create DLL
197
198DLL can be created using MinGW+MSYS with the `make libzstd` command.
199This command creates `dll\libzstd.dll` and the import library `dll\libzstd.lib`.
200The import library is only required with Visual C++.
201The header file `zstd.h` and the dynamic library `dll\libzstd.dll` are required to
202compile a project using gcc/MinGW.
203The dynamic library has to be added to linking options.
204It means that if a project that uses ZSTD consists of a single `test-dll.c`
205file it should be linked with `dll\libzstd.dll`. For example:
206```
207 gcc $(CFLAGS) -Iinclude/ test-dll.c -o test-dll dll\libzstd.dll
208```
209The compiled executable will require ZSTD DLL which is available at `dll\libzstd.dll`.
210
211
212#### Advanced Build options
213
214The build system requires a hash function in order to
215separate object files created with different compilation flags.
216By default, it tries to use `md5sum` or equivalent.
217The hash function can be manually switched by setting the `HASH` variable.
218For example : `make HASH=xxhsum`
219The hash function needs to generate at least 64-bit using hexadecimal format.
220When no hash function is found,
221the Makefile just generates all object files into the same default directory,
222irrespective of compilation flags.
223This functionality only matters if `libzstd` is compiled multiple times
224with different build flags.
225
226The build directory, where object files are stored
227can also be manually controlled using variable `BUILD_DIR`,
228for example `make BUILD_DIR=objectDir/v1`.
229In which case, the hash function doesn't matter.
230
231
232#### Deprecated API
233
234Obsolete API on their way out are stored in directory `lib/deprecated`.
235At this stage, it contains older streaming prototypes, in `lib/deprecated/zbuff.h`.
236These prototypes will be removed in some future version.
237Consider migrating code towards supported streaming API exposed in `zstd.h`.
238
239
240#### Miscellaneous
241
242The other files are not source code. There are :
243
244 - `BUCK` : support for `buck` build system (https://buckbuild.com/)
245 - `Makefile` : `make` script to build and install zstd library (static and dynamic)
246 - `README.md` : this file
247 - `dll/` : resources directory for Windows compilation
248 - `libzstd.pc.in` : script for `pkg-config` (used in `make install`)
249