Lines Matching +full:- +full:- +full:with +full:- +full:zstd
1 zstd(1) -- zstd, zstdmt, unzstd, zstdcat - Compress or decompress .zst files
5 --------
7 `zstd` [<OPTIONS>] [-|<INPUT-FILE>] [-o <OUTPUT-FILE>]
9 `zstdmt` is equivalent to `zstd -T0`
11 `unzstd` is equivalent to `zstd -d`
13 `zstdcat` is equivalent to `zstd -dcf`
17 -----------
18 `zstd` is a fast lossless compression algorithm and data compression tool,
19 with command line syntax similar to `gzip`(1) and `xz`(1).
20 It is based on the **LZ77** family, with further FSE & huff0 entropy stages.
21 `zstd` offers highly configurable compression speed,
23 to strong modes with excellent compression ratios.
24 It also features a very fast decoder, with speeds > 500 MB/s per core,
27 `zstd` command line syntax is generally similar to gzip,
30 - Source files are preserved by default.
31 It's possible to remove them automatically by using the `--rm` command.
32 - When compressing a single file, `zstd` displays progress notifications
34 Use `-q` to turn them off.
35 - `zstd` displays a short help page when command line is an error.
36 Use `-q` to turn it off.
37 - `zstd` does not accept input from console,
39 - `zstd` does not store the input's filename or attributes, only its contents.
41 `zstd` processes each _file_ according to the selected operation mode.
42 If no _files_ are given or _file_ is `-`, `zstd` reads from standard input
44 `zstd` will refuse to write compressed data to standard output
46 Similarly, `zstd` will refuse to read compressed data from standard input
49 Unless `--stdout` or `-o` is specified, _files_ are written to a new file
57 ### Concatenation with .zst Files
58 It is possible to concatenate multiple `.zst` files. `zstd` will decompress
62 -------
82 * `-z`, `--compress`:
86 (for example, `unzstd` implies `--decompress`).
87 * `-d`, `--decompress`, `--uncompress`:
89 * `-t`, `--test`:
91 This option is equivalent to `--decompress --stdout > /dev/null`,
94 * `-b#`:
97 * `--train FILES`:
101 * `-l`, `--list`:
102 Display information related to a zstd compressed file, such as size, ratio, and checksum.
104 This command's output can be augmented with the `-v` modifier.
108 * `-#`:
109 selects `#` compression level \[1-19\] (default: 3).
114 * `--ultra`:
117 * `--fast[=#]`:
118 switch to ultra-fast compression levels.
123 Similarly, if a compression level is set after `--fast`, it overrides it.
124 * `-T#`, `--threads=#`:
128 which is either 64 in 32-bit mode, or 256 for 64-bit environments.
129 This modifier does nothing if `zstd` is compiled without multithread support.
130 * `--single-thread`:
132 As compression is serialized with I/O, this can be slightly slower.
133 Single-thread mode features significantly lower memory usage,
134 which can be useful for systems with limited amount of memory, such as 32-bit systems.
138 … Note 2: this mode is different from `-T1`, which spawns 1 compression thread in parallel with I/O.
139 Final compressed result is also slightly different from `-T1`.
140 * `--auto-threads={physical,logical} (default: physical)`:
141 When using a default amount of threads via `-T0`, choose the default based on the number
143 * `--adapt[=min=#,max=#]`:
144 `zstd` will dynamically adapt compression level to perceived I/O conditions.
145 Compression level adaptation can be observed live by using command `-v`.
147 The feature works when combined with multi-threading and `--long` mode.
148 It does not work with `--single-thread`.
152 _Note_: at the time of this writing, `--adapt` can remain stuck at low speed
153 when combined with multiple worker threads (>=2).
154 * `--long[=#]`:
155 enables long distance matching with `#` `windowLog`, if `#` is not
159 This setting is designed to improve the compression ratio for files with
162 Note: If `windowLog` is set to larger than 27, `--long=windowLog` or
163 `--memory=windowSize` needs to be passed to the decompressor.
164 * `--max`:
167 It's inappropriate for 32-bit mode and therefore disabled in this mode.
168 * `-D DICT`:
170 * `--patch-from FILE`:
171 Specify the file to be used as a reference point for zstd's diff engine.
172 This is effectively dictionary compression with some convenient parameter
175 Note: cannot use both this and `-D` together.
177 Note: `--long` mode will be automatically activated if _chainLog_ < _fileLog_
181 Note: up to level 15, you can use `--patch-from` in `--single-thread` mode
183 '--single-thread' above level 15 will lead to lower compression
187 of speed by specifying `--zstd=targetLength=` to be something large
188 (i.e. 4096), and by setting a large `--zstd=chainLog=`.
189 * `--rsyncable`:
190 `zstd` will periodically synchronize the compression state to make the
191 compressed file more rsync-friendly.
194 for example when combining `--rsyncable` with many parallel worker threads.
195 This feature does not work with `--single-thread`. You probably don't want
196 to use it with long range mode, since it will decrease the effectiveness of
198 * `-C`, `--[no-]check`:
200 * `--[no-]content-size`:
203 `--content-size` (meaning that the original size will be placed in the header).
204 * `--no-dictID`:
208 * `-M#`, `--memory=#`:
209 Set a memory usage limit. By default, `zstd` uses 128 MiB for decompression
214 This is also used during compression when using with `--patch-from=`. In this case,
218 overrides the default limit of 2 GiB. zstd will load training samples up to the memory limit
220 * `--stream-size=#`:
225 * `--size-hint=#`:
226 When handling input from a stream, `zstd` must guess how large the source size
232 * `--target-compressed-block-size=#`:
239 * `-f`, `--force`:
242 During decompression and when the output destination is stdout, pass-through
243 unrecognized formats as-is.
244 * `-c`, `--stdout`:
245 write to standard output (even if it is the console); keep original files (disable `--rm`).
246 * `-o FILE`:
248 Note that this operation is in conflict with `-c`.
250 * `--[no-]sparse`:
252 to make files with many zeroes smaller on disk.
258 * `--[no-]pass-through`
259 enable / disable passing through uncompressed files as-is. During
260 decompression when pass-through is enabled, unrecognized formats will be
261 copied as-is from the input to the output. By default, pass-through will
262 occur when the output destination is stdout and the force (`-f`) option is
264 * `--rm`:
267 If used in combination with `-o`,
268 …triggers a confirmation prompt (which can be silenced with `-f`), as this is a destructive operati…
269 * `-k`, `--keep`:
272 * `-r`:
278 * `--filelist FILE`
280 Format is compatible with `ls` output, with one file per line.
281 * `--output-dir-flat DIR`:
286 Collision resolution ensures first file with a given name will be present in `DIR`,
287 while in combination with `-f`, the last file will be present instead.
288 * `--output-dir-mirror DIR`:
289 similar to `--output-dir-flat`,
295 it will be stored into the "output-dir/var/tmp/abc".
297 name collision resolution will follow the same rules as `--output-dir-flat`.
298 * `--format=FORMAT`:
299 compress and decompress in other formats. If compiled with
300 support, zstd can compress to or decompress from other compression algorithm
301 formats. Possibly available options are `zstd`, `gzip`, `xz`, `lzma`, and `lz4`.
302 If no such format is provided, `zstd` is the default.
303 * `-h`/`-H`, `--help`:
305 * `-V`, `--version`:
307 note that, since it exits, flags specified after `-V` are effectively ignored.
308 Advanced: `-vV` also displays supported formats.
309 `-vvV` also displays POSIX support.
310 `-qV` will only display the version number, suitable for machine reading.
311 * `-v`, `--verbose`:
313 * `-q`, `--quiet`:
316 * `--no-progress`:
318 * `--show-default-cparams`:
321 * `--exclude-compressed`:
323 * `--`:
324 All arguments after `--` are treated as files
328 When invoked via a `gzip` symlink, `zstd` will support further
331 * `-n`, `--no-name`:
333 a file. This is the default behavior and hence a no-op.
334 * `--best`:
335 alias to the option `-9`.
345 If the value of `ZSTD_CLEVEL` is not a valid integer, it will be ignored with a warning message.
348 `ZSTD_NBTHREADS` can be used to set the number of threads `zstd` will attempt to use during compres…
349 If the value of `ZSTD_NBTHREADS` is not a valid unsigned integer, it will be ignored with a warning…
351 `zstd` must be compiled with multithread support for this variable to have any effect.
354 `-#` for compression level and `-T#` for number of compression threads.
358 ----------------------------
359 `zstd` provides 22 predefined regular compression levels plus the fast levels.
361 (one can observe the result of this translation with `--show-default-cparams`).
364 ### --zstd[=options]:
365 The _options_ are provided as a comma-separated list.
370 - `strategy`=_strat_, `strat`=_strat_:
378 - `windowLog`=_wlog_, `wlog`=_wlog_:
384 The minimum _wlog_ is 10 (1 KiB) and the maximum is 30 (1 GiB) on 32-bit
385 platforms and 31 (2 GiB) on 64-bit platforms.
387 Note: If `windowLog` is set to larger than 27, `--long=windowLog` or
388 `--memory=windowSize` needs to be passed to the decompressor.
390 - `hashLog`=_hlog_, `hlog`=_hlog_:
398 - `chainLog`=_clog_, `clog`=_clog_:
408 …m _clog_ is 6 (64 entries / 256 B) and the maximum is 29 (512M entries / 2 GiB) on 32-bit platforms
409 and 30 (1B entries / 4 GiB) on 64-bit platforms.
411 - `searchLog`=_slog_, `slog`=_slog_:
418 The minimum _slog_ is 1 and the maximum is 'windowLog' - 1.
420 - `minMatch`=_mml_, `mml`=_mml_:
428 - `targetLength`=_tlen_, `tlen`=_tlen_:
436 For `ZSTD_fast`, it triggers ultra-fast mode when > 0.
445 - `overlapLog`=_ovlog_, `ovlog`=_ovlog_:
455 Value 0 is special and means "default": _ovlog_ is automatically determined by `zstd`.
458 - `ldmHashRateLog`=_lhrlog_, `lhrlog`=_lhrlog_:
469 - `ldmHashLog`=_lhlog_, `lhlog`=_lhlog_:
477 The minimum _lhlog_ is 6 and the maximum is 30 (default: `windowLog - ldmHashRateLog`).
479 - `ldmMinMatch`=_lmml_, `lmml`=_lmml_:
488 - `ldmBucketSizeLog`=_lblog_, `lblog`=_lblog_:
504 `--zstd`=wlog=23,clog=23,hlog=22,slog=6,mml=3,tlen=48,strat=6
506 ### -B#:
508 This parameter is only available when multi-threading is enabled.
511 `-B#` makes it possible to manually select a custom size.
514 Different job sizes will lead to non-identical compressed frames.
518 ------------------
519 `zstd` offers _dictionary_ compression,
521 It's possible to train `zstd` with a set of samples,
524 using command `-D dictionaryFileName`.
527 * `--train FILEs`:
532 `--train` can be combined with `-r` to indicate a directory rather than listing all the files,
540 `--train` supports multithreading if `zstd` is compiled with threading support (default).
541 Additional advanced parameters can be specified with `--train-fastcover`.
542 The legacy dictionary builder can be accessed with `--train-legacy`.
543 The slower cover dictionary builder can be accessed with `--train-cover`.
544 Default `--train` is equivalent to `--train-fastcover=d=8,steps=4`.
546 * `-o FILE`:
548 * `--maxdict=#`:
553 * `-#`:
557 * `-B#`:
559 * `-M#`, `--memory=#`:
576 training of the same list of files with the same parameters
579 * `--dictID=#`:
582 By default, zstd will create a 4-bytes random number ID.
593 * `--train-cover[=k#,d=#,steps=#,split=#,shrink[=#]]`:
603 Selects segments of size _k_ with highest score to put in the dictionary.
607 algorithm will run faster with d <= _8_.
612 Supports multithreading if `zstd` is compiled with threading support.
619 `zstd --train-cover FILEs`
621 `zstd --train-cover=k=50,d=8 FILEs`
623 `zstd --train-cover=d=8,steps=500 FILEs`
625 `zstd --train-cover=k=50 FILEs`
627 `zstd --train-cover=k=50,split=60 FILEs`
629 `zstd --train-cover=shrink FILEs`
631 `zstd --train-cover=shrink=2 FILEs`
633 * `--train-fastcover[=k#,d=#,f=#,steps=#,split=#,accel=#]`:
634 Same as cover but with extra parameters _f_ and _accel_ and different default value of split
643 The subsegment is hashed to an index in the range [0,2^_f_ - 1].
649 `zstd --train-fastcover FILEs`
651 `zstd --train-fastcover=d=8,f=15,accel=2 FILEs`
653 * `--train-legacy[=selectivity=#]`:
654 Use legacy dictionary builder algorithm with the given dictionary
658 `--train-legacy=s=#` is also accepted.
662 `zstd --train-legacy FILEs`
664 `zstd --train-legacy=selectivity=8 FILEs`
668 ---------
669 The `zstd` CLI provides a benchmarking mode that can be used to easily find suitable compression pa…
670 `zstd -b [FILE(s)]` will benchmark `zstd` for both compression and decompression using default comp…
673 It's possible to pass multiple files to the benchmark, and even a directory with `-r DIRECTORY`.
678 * `-b#`:
680 * `-e#`:
681 benchmark file(s) using multiple compression levels, from `-b#` to `-e#` (inclusive)
682 * `-d`:
683 benchmark decompression speed only (requires providing a zstd-compressed content)
684 * `-i#`:
686 * `-B#`, `--block-size=#`:
688 * `-S`:
690 * `-D dictionary`
692 * `--priority=rt`:
693 set process priority to real-time (Windows)
695 …ompatible with other parameters, such as number of threads (`-T#`), advanced compression parameter…
697 **Output Format:** CompressionLevel#Filename: InputSize -> OutputSize (CompressionRatio), Compressi…
699 **Methodology:** For speed measurement, the entire input is compressed/decompressed in-memory to me…
703 --------
706 The <zstandard> format is specified in Y. Collet, "Zstandard Compression and the 'application/zstd'…
709 ----
710 Report bugs at: https://github.com/facebook/zstd/issues
713 ------