Lines Matching +full:gcc +full:- +full:7 +full:- +full:libzstd
5 * This source code is licensed under both the BSD-style license (found in the
8 * You may select, at your option, one of the above-listed licenses.
52 * typically with -Wno-deprecated-declarations for gcc or _CRT_SECURE_NO_WARNINGS in Visual.
77 real-time compression scenarios at zlib-level and better compression ratios.
78 The zstd compression library provides in-memory compression and decompression
82 which is currently 22. Levels >= 20, labeled `--ultra`, should be used with
88 - a single step (described as Simple API)
89 - a single step, reusing a context (described as Explicit context)
90 - unbounded multiple steps (described as Streaming compression)
94 - a single step (described as Simple dictionary API)
95 - a single step, reusing a dictionary (described as Bulk-processing
101 Advanced experimental APIs should never be used with a dynamically-linked
106 /*------ Version ------*/
136 /* All magic numbers are supposed read/written to/from files/memory using little-endian convention …
172 * @return : - decompressed size of `src` frame content, if known
173 * - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
174 …* - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too…
182 …* note 3 : decompressed size is always present when compression is completed using single-pass f…
184 * note 4 : decompressed size can be very large (64-bits value),
191 #define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1)
192 #define ZSTD_CONTENTSIZE_ERROR (0ULL - 2)
216 * maximum compressed size in worst case single-pass scenario.
217 * When invoking `ZSTD_compress()` or any other one-pass compression function,
232 …) ? 0 : (srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) /*…
233 …D_compressBound(size_t srcSize); /*!< maximum compressed size in worst case single-pass scenario */
254 * Note : re-using context is just a speed / resource optimization.
256 * Note 2 : In multi-threaded environments,
305 * __They do not apply to one-shot variants such as ZSTD_compressCCtx()__ .
321 ZSTD_btopt=7,
333 * See ZSTD_CCtx_refCDict() for more info (superseded-by-cdict). */
334 … ZSTD_c_compressionLevel=100, /* Set compression parameters according to pre-defined cLevel table.
347 ZSTD_c_windowLog=101, /* Maximum allowed back-reference distance, expressed as power of 2.
361 ZSTD_c_chainLog=103, /* Size of the multi-probe search table, as a power of 2.
399 … * This is helpful in low bandwidth streaming environments to improve end-to-end latency,
419 * default: windowlog - 7.
430 … * Must be clamped between 0 and (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN).
431 … * Default is MAX(0, (windowLog - ldmHashLog)), optimizing hash table usage.
441 …ZSTD_c_checksumFlag=201, /* A 32-bits checksum of content is written at end of frame (default:0) */
444 /* multi-threading parameters */
445 …/* These parameters are only active if multi-threading is enabled (compiled with build macro ZSTD_…
446 … * Otherwise, trying to set any other value than default (0) will be a no-op and return an error.
447 * In a situation where it's unknown if the linked library supports multi-threading or not,
457 … * Default value is `0`, aka "single-threaded mode" : no worker is spawned,
470 … * - 0 means "default" : value will be determined by the library, depending on strategy
471 * - 1 means "no overlap"
472 * - 9 means "full overlap", using a full window size.
474 …* 9: full window; 8: w/2; 7: w/4; 6: w/8; 5:w/16; 4: w/32; 3:w/64; 2:w/128; 1:no overlap; …
531 * - an error status field, which must be tested using ZSTD_isError()
532 * - lower and upper bounds, both inclusive
541 * Exception : when using multi-threading mode (nbWorkers >= 1),
574 * - The session : will stop compressing current frame, and make CCtx ready to start a new one.
580 * - The parameters : changes all parameters back to "default".
584 * - Both : similar to resetting the session, followed by resetting parameters.
593 …* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*…
594 * - The function is always blocking, returns when compression is completed.
622 …ameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
651 * - an error status field, which must be tested using ZSTD_isError()
652 * - both lower and upper bounds, inclusive
692 /*-***********************************************************************
693 * Streaming compression - HowTo
698 * It is recommended to reuse ZSTD_CStream since it will play nicer with system's memory, by re-usi…
728 * using ZSTD_compressStream2() with ZSTD_e_flush. `output->pos` will be updated.
729 * Note that, if `output->size` is too small, a single invocation with ZSTD_e_flush might not be en…
775 …* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*…
776 …* - Compression parameters cannot be changed once compression is started (save a list of exceptio…
777 * - output->pos must be <= dstCapacity, input->pos must be <= srcSize
778 …* - output->pos and input->pos will be updated. They are guaranteed to remain below their respect…
779 * - endOp must be a valid directive
780 …* - When nbWorkers==0 (default), function is blocking : it completes its job before returning to …
781 …* - When nbWorkers>=1, function is non-blocking : it copies a portion of input, distributes jobs …
784 …* - Exception : if the first call requests a ZSTD_e_end directive and provides enough dstCapacity…
785 * - @return provides a minimum amount of data remaining to be flushed from internal buffers
790 * - after a ZSTD_e_end directive, if internal buffer is not fully flushed (@return != 0),
794 * - note: if an operation ends with an error, it may leave @cctx in an undefined state.
796 * In order to be re-employed after an error, a state must be reset,
843 * the next read size (if non-zero and not an error). ZSTD_compressStream2()
844 * returns the minimum nb of bytes left to flush (if non-zero and not an error).
853 /*-***************************************************************************
854 * Streaming decompression - HowTo
902 * - `input.pos < input.size`, some input remaining and caller should provide remaining input
904 * - `output.pos < output.size`, decoder finished and flushed all remaining buffers.
905 * - `output.pos == output.size`, potentially uncflushed data present in the internal buffers,
915 * In order to re-use such a state, it must be first reset,
963 …Dict can be created once and shared by multiple threads concurrently, since its usage is read-only.
1017 * It can still be loaded, but as a content-only dictionary. */
1023 * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
1029 * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
1036 * - The frame does not require a dictionary to be decoded (most common case).
1037 …* - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a …
1038 * Note : this use case also happens when using a non-conformant dictionary.
1039 …* - `srcSize` is too small, and as a result, the frame header could not be decoded (only possible…
1040 * - This is not a Zstandard frame.
1053 * In contrast, Prefixes are single-use.
1061 * Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary,
1062 * meaning "return to no-dictionary mode".
1067 * It's also a CPU consuming operation, with non-negligible impact on latency.
1085 * The parameters ignored are labelled as "superseded-by-cdict" in the ZSTD_cParameter enum docs.
1086 * The ignored parameters will be used again if the CCtx is returned to no-dictionary mode.
1089 * Special : Referencing a NULL CDict means "return to no-dictionary mode".
1096 * Reference a prefix (single-usage dictionary) for next compressed frame.
1110 * It's a CPU consuming operation, with non-negligible impact on latency.
1122 * Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary,
1123 * meaning "return to no-dictionary mode".
1125 * which has a non-negligible impact on CPU usage and latency.
1148 * Special: referencing a NULL DDict means "return to no-dictionary mode".
1154 * Reference a prefix (single-usage dictionary) to decompress next frame.
1236 #define ZSTD_SEARCHLOG_MAX (ZSTD_WINDOWLOG_MAX-1)
1238 #define ZSTD_MINMATCH_MAX 7 /* only for ZSTD_fast, other strategies are limited to 6 */
1254 …* The limit does not apply for one-pass decoders (such as ZSTD_decompress()), since no additional …
1265 #define ZSTD_LDM_HASHRATELOG_MAX (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN)
1274 /* --- Advanced types --- */
1300 * rep == 1 --> offset == repeat_offset_1
1301 * rep == 2 --> offset == repeat_offset_2
1302 * rep == 3 --> offset == repeat_offset_3
1304 * rep == 1 --> offset == repeat_offset_2
1305 * rep == 2 --> offset == repeat_offset_3
1306 * rep == 3 --> offset == repeat_offset_1 - 1
1327 …int checksumFlag; /**< 1: generate a 32-bits checksum using XXH64 algorithm at end of frame, fo…
1344 …ZSTD_dlm_byRef = 1 /**< Reference dictionary content -- the dictionary buffer must outlive its …
1349 … ZSTD_f_zstd1_magicless = 1 /* Variant of zstd frame format, without initial 4-bytes magic number.
1374 * - The contents of the CDict can be copied into the working context. This
1377 * the compression faster per-byte of input. However, the initial copy of
1382 * - The CDict's tables can be used in-place. In this model, compression is
1384 * tables. However, this model incurs no start-up cost (as long as the
1388 * - The CDict's tables are not used at all, and instead we use the working
1420 ZSTD_ps_enable = 1, /* Force-enable the feature */
1432 * @return : - decompressed size of all data in all successive frames
1433 * - if the decompressed size cannot be determined: ZSTD_CONTENTSIZE_UNKNOWN
1434 * - if an error occurred: ZSTD_CONTENTSIZE_ERROR
1440 * note 3 : decompressed size can be very large (64-bits value),
1455 * @return : - upper-bound for the decompressed size of all data in all successive frames
1456 * - if an error occurred: ZSTD_CONTENTSIZE_ERROR
1459 …* note 2 : the upper-bound is exact when the decompressed size field is available in every ZSTD …
1461 …* note 3 : when the decompressed size field isn't available, the upper-bound for that frame is c…
1462 * upper-bound = # blocks * min(128 KB, Window_Size)
1497 * Zstd supports in-place decompression, where the input and output buffers overlap.
1506 * |---------------------------------------|-----------|----------|
1511 * NOTE: This applies only to single-pass decompression through ZSTD_decompress() or
1513 * NOTE: This function supports multi-frame input.
1526 * WARNING: This macro does not support multi-frame input, the input must be a single
1537 …((originalSize) == 0 ? 0 : 3 * (((originalSize) + (blockSize) - 1) / blockSize)) /* 3 bytes per bl…
1548 * @return : upper-bound for the number of sequences that can be generated
1551 * note : returns number of sequences - to get bytes, multiply by sizeof(ZSTD_Sequence).
1622 …* - ZSTD_c_minMatch MUST be set as less than or equal to the smallest match generated by the ma…
1623 …* - ZSTD_c_compressionLevel accordingly adjusts the strength of the entropy coder, as it would …
1624 …* - ZSTD_c_windowLog affects offset validation: this function will return an error at higher de…
1627 …* Note: Repcodes are, as of now, always re-calculated within this function, so ZSTD_Sequence::rep …
1641 * Skippable frames begin with a 4-byte magic number. There are 16 possible choices of magic number,
1647 …* with a 4-byte unsigned int, or if the parameter magicVariant is greater than 15 (and therefore i…
1658 * i.e. magicNumber - ZSTD_MAGIC_SKIPPABLE_START. This can be NULL if the caller is not interested
1686 * to compress data of any size using one-shot compression ZSTD_compressCCtx() or ZSTD_compress2()
1691 * Note that the size estimation is specific for one-shot compression,
1703 * Note : only single-threaded compression is supported.
1717 …ams() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compression …
1718 * Note : CStream size estimation is only correct for single-threaded compression.
1720 …* Note 2 : ZSTD_estimateCStreamSize* functions are not compatible with the Block-Level Sequence P…
1747 * Initialize an object using a pre-allocated fixed-size buffer.
1749 * Provided pointer *must be 8-bytes aligned*.
1764 * Limitation 2 : static cctx currently not compatible with multi-threading.
1863 …* same as ZSTD_getCParams(), but @return a full `ZSTD_parameters` object instead of sub-component…
1969 /* Force back-reference distances to remain < windowSize,
1986 * literals compression based on the compression parameters - specifically,
2027 * structure for the dictionary that is read-optimized.
2047 * In general, you should expect compression to be faster--sometimes very much
2048 * so--and CDict creation to be slightly slower. Eventually, we will probably
2075 * this flag is ALWAYS memory safe, and will never access out-of-bounds
2091 * calls. Specifically: (out.size - out.pos) will never grow. This gives the
2101 * Zstd will check that (out.size - out.pos) never grows and return an error
2148 * Set to ZSTD_ps_disable to never use row-based matchfinder.
2149 * Set to ZSTD_ps_enable to force usage of row-based matchfinder.
2152 * the row-based matchfinder based on support for SIMD instructions and the window log.
2164 * results than when the two buffers are non-contiguous. This flag forces zstd
2165 * to always load the prefix in non-contiguous mode, even if it happens to be
2180 * In some situations, zstd uses CDict tables in-place rather than copying them
2184 * when they are used in-place.
2194 * but in the future zstd may conditionally enable this feature via an auto-detection
2205 * is block-by-block: the internal sequence producer will only be called for blocks
2208 * normal (fully-internal) compression operation.
2210 * The user is strongly encouraged to read the full Block-Level Sequence Producer API
2228 * provided through the compressSequences() API or from an external block-level
2260 * - ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure
2261 * - ZSTD_CCtxParams_setParameter() : Push parameters one by one into
2265 * - ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to
2269 * - ZSTD_compressStream2() : Do compression using the CCtx.
2270 * - ZSTD_freeCCtxParams() : Free the memory, accept NULL pointer.
2273 * for static allocation of CCtx for single-threaded compression.
2375 …ameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
2410 * If you need to avoid the input buffer allocation use the buffer-less
2414 * this flag is ALWAYS memory safe, and will never access out-of-bounds
2490 * This instruction is mandatory to decode data without a fully-formed header,
2613 * This is typically useful to skip dictionary loading stage, since it will reuse it in-place.
2639 * Note : (ingested - consumed) is amount of input data buffered internally, not yet compressed.
2695 /* ********************* BLOCK-LEVEL SEQUENCE PRODUCER API *********************
2698 * The Block-Level Sequence Producer API allows users to provide their own custom
2699 * sequence producer which libzstd invokes to process each block. The produced list
2700 * of sequences (literals and matches) is then post-processed by libzstd to produce
2703 * This block-level offload API is a more granular complement of the existing
2704 * frame-level offload API compressSequences() (introduced in v1.5.1). It offers
2705 * an easier migration story for applications already integrated with libzstd: the
2711 * acceleration not available within libzstd itself.
2714 * Block-Level Sequence Producer API.
2719 * arguments to the user-provided function:
2721 * - sequenceProducerState: a pointer to a user-managed state for the sequence
2724 * - outSeqs, outSeqsCapacity: an output buffer for the sequence producer.
2728 * - src, srcSize: an input buffer for the sequence producer to parse.
2731 * - dict, dictSize: a history buffer, which may be empty, which the sequence
2736 * - compressionLevel: a signed integer representing the zstd compression level
2742 * - windowSize: a size_t representing the maximum allowed offset for external
2747 * The user-provided function shall return a size_t representing the number of
2749 * code if it is greater than outSeqsCapacity. The return value must be non-zero
2750 * if srcSize is non-zero. The ZSTD_SEQUENCE_PRODUCER_ERROR macro is provided
2754 * If the user-provided function does not return an error code, the sequences
2758 * - The sum of matchLengths and literalLengths must equal srcSize.
2759 * - All sequences in the parse, except for the final sequence, must have
2762 * - All offsets must respect the windowSize parameter as specified in
2764 * - If the final sequence has matchLength == 0, it must also have offset == 0.
2770 * If the user-provided function returns an error, zstd will either fall back
2792 * COMPRESSION WILL FAIL if it is enabled and the user tries to compress with a block-level
2794 * - Note that ZSTD_c_enableLongDistanceMatching is auto-enabled by default in some
2798 * - As of this writing, ZSTD_c_enableLongDistanceMatching is disabled by default
2800 * check the docs on ZSTD_c_enableLongDistanceMatching whenever the Block-Level Sequence
2805 * - Dictionaries are not currently supported. Compression will *not* fail if the user
2807 * - Stream history is not currently supported. All advanced compression APIs, including
2811 * Third, multi-threading within a single compression is not currently supported. In other words,
2813 * Multi-threading across compressions is fine: simply create one CCtx per thread.
2815 * Long-term, we plan to overcome all three limitations. There is no technical blocker to
2819 #define ZSTD_SEQUENCE_PRODUCER_ERROR ((size_t)(-1))
2831 * Instruct zstd to use a block-level external sequence producer function.
2874 * Buffer-less and synchronous inner streaming functions (DEPRECATED)
2888 Buffer-less streaming compression (synchronous mode)
2899 - ZSTD_compressContinue() has no internal buffer. It uses externally provided buffers only.
2900 - Interface is synchronous : input is consumed entirely and produces 1+ compressed blocks.
2901 …- Caller must ensure there is enough space in `dst` to store compressed data under worst case scen…
2904 …- ZSTD_compressContinue() presumes prior input ***is still accessible and unmodified*** (up to max…
2906 …- ZSTD_compressContinue() detects that prior input has been overwritten when `src` buffer overlaps.
2916 /*===== Buffer-less streaming compression functions =====*/
2917 ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.")
2919 ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.")
2921 ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.")
2928 ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.")
2930 ZSTD_DEPRECATED("The buffer-less API is deprecated in favor of the normal streaming API. See docs.")
2941 Buffer-less streaming decompression (synchronous mode)
2956 such as the dictionary ID, content size, or maximum back-reference distance (`windowSize`).
2966 …or that previous contiguous segment is large enough to properly handle maximum back-reference dist…
2971 …which can return an error code if required value is too large for current system (in 32-bits mode).
2981 as long as the encoder and decoder progress in "lock-step",
3003 Skippable frames allow integration of user-defined data into a flow of concatenated frames.
3006 a) Skippable frame ID - 4 Bytes, Little endian format, any value from 0x184D2A50 to 0x184D2A5F
3007 b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits
3008 c) Frame Content - any content (User Data) of length equal to Frame Size
3009 For skippable frames ZSTD_getFrameHeader() returns zfhPtr->frameType==ZSTD_skippableFrame.
3013 /*===== Buffer-less streaming decompression functions =====*/
3041 - ZSTD_c_format = ZSTD_f_zstd1_magicless
3042 - ZSTD_c_contentSizeFlag = 0
3043 - ZSTD_c_checksumFlag = 0
3044 - ZSTD_c_dictIDFlag = 0
3051 …Frame metadata cost is typically ~12 bytes, which can be non-negligible for very small blocks (< 1…
3055 - Compressing and decompressing require a context structure
3057 - It is necessary to init context before starting
3060 - Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB
3064 …- When a block is considered not compressible enough, ZSTD_compressBlock() result will be 0 (zero)…
3083 …Size); /**< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression.…