Lines Matching +full:zstd +full:- +full:src
5 * This source code is licensed under both the BSD-style license (found in the
8 * You may select, at your option, one of the above-listed licenses.
44 zstd, short for Zstandard, is a fast lossless compression algorithm, targeting
45 real-time compression scenarios at zlib-level and better compression ratios.
46 The zstd compression library provides in-memory compression and decompression
50 which is currently 22. Levels >= 20, labeled `--ultra`, should be used with
56 - a single step (described as Simple API)
57 - a single step, reusing a context (described as Explicit context)
58 - unbounded multiple steps (described as Streaming compression)
62 - a single step (described as Simple dictionary API)
63 - a single step, reusing a dictionary (described as Bulk-processing
67 `#define ZSTD_STATIC_LINKING_ONLY` before including zstd.h.
69 Advanced experimental APIs should never be used with a dynamically-linked
74 /*------ Version ------*/
104 /* All magic numbers are supposed read/written to/from files/memory using little-endian convention …
118 * Compresses `src` content as a single zstd compressed frame into already allocated `dst`.
123 const void* src, size_t srcSize,
133 const void* src, size_t compressedSize);
136 * `src` should point to the start of a ZSTD encoded frame.
139 * @return : - decompressed size of `src` frame content, if known
140 * - ZSTD_CONTENTSIZE_UNKNOWN if the size cannot be determined
141 …* - ZSTD_CONTENTSIZE_ERROR if an error occurred (e.g. invalid magic number, srcSize too…
149 …* note 3 : decompressed size is always present when compression is completed using single-pass f…
151 * note 4 : decompressed size can be very large (64-bits value),
158 #define ZSTD_CONTENTSIZE_UNKNOWN (0ULL - 1)
159 #define ZSTD_CONTENTSIZE_ERROR (0ULL - 2)
160 ZSTDLIB_API unsigned long long ZSTD_getFrameContentSize(const void *src, size_t srcSize);
167 * @return : decompressed size of `src` frame content _if known and not empty_, 0 otherwise. */
168 ZSTDLIB_API unsigned long long ZSTD_getDecompressedSize(const void* src, size_t srcSize);
171 * `src` should point to the start of a ZSTD frame or skippable frame.
173 * @return : the compressed size of the first frame starting at `src`,
176 ZSTDLIB_API size_t ZSTD_findFrameCompressedSize(const void* src, size_t srcSize);
180 …ize) ((srcSize) + ((srcSize)>>8) + (((srcSize) < (128<<10)) ? (((128<<10) - (srcSize)) >> 11) /*…
181 …D_compressBound(size_t srcSize); /*!< maximum compressed size in worst case single-pass scenario */
195 * and re-use it for each successive compression operation.
197 * Note : re-using context is just a speed / resource optimization.
199 * Note 2 : In multi-threaded environments,
216 const void* src, size_t srcSize,
222 * and re-use it for each successive compression operation.
236 const void* src, size_t srcSize);
248 * __They do not apply to "simple" one-shot variants such as ZSTD_compressCCtx()__ .
276 * See ZSTD_CCtx_refCDict() for more info (superseded-by-cdict). */
277 … ZSTD_c_compressionLevel=100, /* Set compression parameters according to pre-defined cLevel table.
290 ZSTD_c_windowLog=101, /* Maximum allowed back-reference distance, expressed as power of 2.
304 ZSTD_c_chainLog=103, /* Size of the multi-probe search table, as a power of 2.
349 * default: windowlog - 7.
360 … * Must be clamped between 0 and (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN).
361 … * Default is MAX(0, (windowLog - ldmHashLog)), optimizing hash table usage.
371 …ZSTD_c_checksumFlag=201, /* A 32-bits checksum of content is written at end of frame (default:0) */
374 /* multi-threading parameters */
375 …/* These parameters are only active if multi-threading is enabled (compiled with build macro ZSTD_…
376 … * Otherwise, trying to set any other value than default (0) will be a no-op and return an error.
377 * In a situation where it's unknown if the linked library supports multi-threading or not,
387 … * Default value is `0`, aka "single-threaded mode" : no worker is spawned,
400 … * - 0 means "default" : value will be determined by the library, depending on strategy
401 * - 1 means "no overlap"
402 * - 9 means "full overlap", using a full window size.
455 * - an error status field, which must be tested using ZSTD_isError()
456 * - lower and upper bounds, both inclusive
465 * Exception : when using multi-threading mode (nbWorkers >= 1),
498 * - The session : will stop compressing current frame, and make CCtx ready to start a new one.
504 * - The parameters : changes all parameters back to "default".
508 * - Both : similar to resetting the session, followed by resetting parameters.
516 …* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*…
517 * - The function is always blocking, returns when compression is completed.
524 const void* src, size_t srcSize);
544 …ameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
569 * - an error status field, which must be tested using ZSTD_isError()
570 * - both lower and upper bounds, inclusive
597 const void* src; /**< start of input buffer */ member
610 /*-***********************************************************************
611 * Streaming compression - HowTo
616 * It is recommended to re-use ZSTD_CStream since it will play nicer with system's memory, by re-us…
623 * it will re-use the same sticky parameters as previous compression session.
646 * using ZSTD_compressStream2() with ZSTD_e_flush. `output->pos` will be updated.
647 * Note that, if `output->size` is too small, a single invocation with ZSTD_e_flush might not be en…
693 …* - Compression parameters are pushed into CCtx before starting compression, using ZSTD_CCtx_set*…
694 …* - Compression parameters cannot be changed once compression is started (save a list of exceptio…
695 * - output->pos must be <= dstCapacity, input->pos must be <= srcSize
696 …* - output->pos and input->pos will be updated. They are guaranteed to remain below their respect…
697 * - endOp must be a valid directive
698 …* - When nbWorkers==0 (default), function is blocking : it completes its job before returning to …
699 …* - When nbWorkers>=1, function is non-blocking : it copies a portion of input, distributes jobs …
702 …* - Exception : if the first call requests a ZSTD_e_end directive and provides enough dstCapacity…
703 * - @return provides a minimum amount of data remaining to be flushed from internal buffers
708 * - after a ZSTD_e_end directive, if internal buffer is not fully flushed (@return != 0),
755 * the next read size (if non-zero and not an error). ZSTD_compressStream2()
756 * returns the minimum nb of bytes left to flush (if non-zero and not an error).
765 /*-***************************************************************************
766 * Streaming decompression - HowTo
770 * ZSTD_DStream objects can be re-used multiple times.
825 const void* src, size_t srcSize,
837 const void* src, size_t srcSize,
851 …Dict can be created once and shared by multiple threads concurrently, since its usage is read-only.
873 const void* src, size_t srcSize,
894 const void* src, size_t srcSize,
905 * It can still be loaded, but as a content-only dictionary. */
911 * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
917 * Non-conformant dictionaries can still be loaded, but as content-only dictionaries. */
921 * Provides the dictID required to decompressed the frame stored within `src`.
924 * - The frame does not require a dictionary to be decoded (most common case).
925 …* - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a …
926 * Note : this use case also happens when using a non-conformant dictionary.
927 …* - `srcSize` is too small, and as a result, the frame header could not be decoded (only possible…
928 * - This is not a Zstandard frame.
930 ZSTDLIB_API unsigned ZSTD_getDictID_fromFrame(const void* src, size_t srcSize);
939 * ZSTD_reset_session_and_parameters. Prefixes are single-use.
947 * Special: Loading a NULL (or 0-size) dictionary invalidates previous dictionary,
948 * meaning "return to no-dictionary mode".
950 * To return to "no-dictionary" situation, load a NULL dictionary (or reset parameters).
952 * It's also a CPU consuming operation, with non-negligible impact on latency.
966 * The parameters ignored are labelled as "superseded-by-cdict" in the ZSTD_cParameter enum docs.
967 * The ignored parameters will be used again if the CCtx is returned to no-dictionary mode.
970 * Special : Referencing a NULL CDict means "return to no-dictionary mode".
977 * Reference a prefix (single-usage dictionary) for next compressed frame.
986 * Note 2 : If the intention is to diff some large src data blob with some prior version of itself,
990 * It's a CPU consuming operation, with non-negligible impact on latency.
1002 * Special : Adding a NULL (or 0-size) dictionary invalidates any previous dictionary,
1003 * meaning "return to no-dictionary mode".
1005 * which has a non-negligible impact on CPU usage and latency.
1027 * Special: referencing a NULL DDict means "return to no-dictionary mode".
1033 * Reference a prefix (single-usage dictionary) to decompress next frame.
1091 * typically with -Wno-deprecated-declarations for gcc or _CRT_SECURE_NO_WARNINGS in Visual.
1137 #define ZSTD_SEARCHLOG_MAX (ZSTD_WINDOWLOG_MAX-1)
1154 …* The limit does not apply for one-pass decoders (such as ZSTD_decompress()), since no additional …
1165 #define ZSTD_LDM_HASHRATELOG_MAX (ZSTD_WINDOWLOG_MAX - ZSTD_HASHLOG_MIN)
1174 /* --- Advanced types --- */
1200 * rep == 1 --> offset == repeat_offset_1
1201 * rep == 2 --> offset == repeat_offset_2
1202 * rep == 3 --> offset == repeat_offset_3
1204 * rep == 1 --> offset == repeat_offset_2
1205 * rep == 2 --> offset == repeat_offset_3
1206 * rep == 3 --> offset == repeat_offset_1 - 1
1227 …int checksumFlag; /**< 1: generate a 32-bits checksum using XXH64 algorithm at end of frame, fo…
1244 …ZSTD_dlm_byRef = 1 /**< Reference dictionary content -- the dictionary buffer must outlive its …
1248 …ZSTD_f_zstd1 = 0, /* zstd frame format, specified in zstd_compression_format.md (default…
1249 … ZSTD_f_zstd1_magicless = 1 /* Variant of zstd frame format, without initial 4-bytes magic number.
1272 * Zstd currently supports the use of a CDict in three ways:
1274 * - The contents of the CDict can be copied into the working context. This
1277 * the compression faster per-byte of input. However, the initial copy of
1282 * - The CDict's tables can be used in-place. In this model, compression is
1284 * tables. However, this model incurs no start-up cost (as long as the
1288 * - The CDict's tables are not used at all, and instead we use the working
1294 * Zstd has a simple internal heuristic that selects which strategy to use
1296 * Zstd is making poor choices, it is possible to override that choice with
1315 …/* Note: This enum controls features which are conditionally beneficial. Zstd typically will make …
1320 ZSTD_ps_enable = 1, /* Force-enable the feature */
1329 * `src` should point to the start of a series of ZSTD encoded and/or skippable frames
1331 * (i.e. there should be a frame boundary at `src + srcSize`)
1332 * @return : - decompressed size of all data in all successive frames
1333 * - if the decompressed size cannot be determined: ZSTD_CONTENTSIZE_UNKNOWN
1334 * - if an error occurred: ZSTD_CONTENTSIZE_ERROR
1340 * note 3 : decompressed size can be very large (64-bits value),
1349 ZSTDLIB_STATIC_API unsigned long long ZSTD_findDecompressedSize(const void* src, size_t srcSize);
1352 * `src` should point to the start of a series of ZSTD encoded and/or skippable frames
1354 * (i.e. there should be a frame boundary at `src + srcSize`)
1355 * @return : - upper-bound for the decompressed size of all data in all successive frames
1356 * - if an error occurred: ZSTD_CONTENTSIZE_ERROR
1358 * note 1 : an error can occur if `src` contains an invalid or incorrectly formatted frame.
1359 …* note 2 : the upper-bound is exact when the decompressed size field is available in every ZSTD …
1361 …* note 3 : when the decompressed size field isn't available, the upper-bound for that frame is c…
1362 * upper-bound = # blocks * min(128 KB, Window_Size)
1364 ZSTDLIB_STATIC_API unsigned long long ZSTD_decompressBound(const void* src, size_t srcSize);
1370 ZSTDLIB_STATIC_API size_t ZSTD_frameHeaderSize(const void* src, size_t srcSize);
1394 size_t outSeqsSize, const void* src, size_t srcSize);
1427 …* - ZSTD_c_minMatch MUST be set as less than or equal to the smallest match generated by the ma…
1428 …* - ZSTD_c_compressionLevel accordingly adjusts the strength of the entropy coder, as it would …
1429 …* - ZSTD_c_windowLog affects offset validation: this function will return an error at higher de…
1432 …* Note: Repcodes are, as of now, always re-calculated within this function, so ZSTD_Sequence::rep …
1435 * @return : final compressed size or a ZSTD error.
1439 const void* src, size_t srcSize);
1443 * Generates a zstd skippable frame containing data given by src, and writes it to dst buffer.
1445 …* Skippable frames begin with a a 4-byte magic number. There are 16 possible choices of magic numb…
1451 …* with a 4-byte unsigned int, or if the parameter magicVariant is greater than 15 (and therefore i…
1453 * @return : number of bytes written or a ZSTD error.
1456 const void* src, size_t srcSize, unsigned magicVariant);
1459 * Retrieves a zstd skippable frame containing data given by src, and writes it to dst buffer.
1462 * i.e. magicNumber - ZSTD_MAGIC_SKIPPABLE_START. This can be NULL if the caller is not interested
1467 * @return : number of bytes written or a ZSTD error.
1470 const void* src, size_t srcSize);
1491 * Therefore, the estimation is only guaranteed for single-shot compressions, not streaming.
1503 * Note 2 : only single-threaded compression is supported.
1513 * It will also consider src size to be arbitrarily "large", which is worst case.
1516 …ams() can be used in tandem with ZSTD_CCtxParams_setParameter(). Only single-threaded compression …
1517 * Note : CStream size estimation is only correct for single-threaded compression.
1528 ZSTDLIB_STATIC_API size_t ZSTD_estimateDStreamSize_fromFrame(const void* src, size_t srcSize);
1531 …* ZSTD_estimateCDictSize() will bet that src size is relatively "small", and content is copied, l…
1540 * Initialize an object using a pre-allocated fixed-size buffer.
1542 * Provided pointer *must be 8-bytes aligned*.
1548 * Note : zstd will never resize nor malloc() when using a static buffer.
1550 * zstd will just error out (typically ZSTD_error_memory_allocation).
1557 * Limitation 2 : static cctx currently not compatible with multi-threading.
1656 …* same as ZSTD_getCParams(), but @return a full `ZSTD_parameters` object instead of sub-component…
1680 const void* src, size_t srcSize,
1691 const void* src, size_t srcSize,
1739 /* Force back-reference distances to remain < windowSize,
1756 * literals compression based on the compression parameters - specifically,
1802 * structure for the dictionary that is read-optimized.
1822 * In general, you should expect compression to be faster--sometimes very much
1823 * so--and CDict creation to be slightly slower. Eventually, we will probably
1833 * between calls, except for the modifications that zstd makes to pos (the
1836 * mode that makes sense is ZSTD_e_end, so zstd will error if ZSTD_e_end
1837 * is not used. The data in the ZSTD_inBuffer in the range [src, src + pos)
1840 * When this flag is enabled zstd won't allocate an input window buffer,
1850 * this flag is ALWAYS memory safe, and will never access out-of-bounds
1855 * is because zstd needs to reference data in the ZSTD_inBuffer to find
1856 * matches. Normally zstd maintains its own window buffer for this purpose,
1857 * but passing this flag tells zstd to use the user provided buffer.
1866 * calls. Specifically: (out.size - out.pos) will never grow. This gives the
1872 * When this flag is enabled zstd won't allocate an output buffer, because
1876 * Zstd will check that (out.size - out.pos) never grows and return an error
1899 * Without validation, providing a sequence that does not conform to the zstd spec will cause
1923 * Set to ZSTD_ps_disable to never use row-based matchfinder.
1924 * Set to ZSTD_ps_enable to force usage of row-based matchfinder.
1927 * the row-based matchfinder based on support for SIMD instructions and the window log.
1935 * Zstd produces different results for prefix compression when the prefix is
1937 * This is because zstd detects that the two buffers are contiguous and it can
1939 * results than when the two buffers are non-contiguous. This flag forces zstd
1940 * to always load the prefix in non-contiguous mode, even if it happens to be
1962 * - ZSTD_createCCtxParams() : Create a ZSTD_CCtx_params structure
1963 * - ZSTD_CCtxParams_setParameter() : Push parameters one by one into
1967 * - ZSTD_CCtx_setParametersUsingCCtxParams() : Apply parameters to
1971 * - ZSTD_compressStream2() : Do compression using the CCtx.
1972 * - ZSTD_freeCCtxParams() : Free the memory, accept NULL pointer.
1975 * for static allocation of CCtx for single-threaded compression.
2033 const void* src, size_t srcSize, size_t* srcPos,
2077 …ameter is only useful in streaming mode, since no internal buffer is allocated in single-pass mode.
2100 * between calls, except for the modifications that zstd makes to pos (the
2108 * When this flags is enabled zstd won't allocate an output buffer, because
2112 * If you need to avoid the input buffer allocation use the buffer-less
2116 * this flag is ALWAYS memory safe, and will never access out-of-bounds
2121 * is because zstd needs to reference data in the ZSTD_outBuffer to regenerate
2122 * matches. Normally zstd maintains its own buffer for this purpose, but passing
2123 * this flag tells zstd to use the user provided buffer.
2165 * This instruction is mandatory to decode data without a fully-formed header,
2180 const void* src, size_t srcSize, size_t* srcPos);
2204 ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions")
2217 * Note: dict is loaded with ZSTD_dct_auto (treated as a full zstd dictionary if
2221 ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions")
2229 * // Pseudocode: Set each zstd parameter and leave the rest as-is.
2241 ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions")
2255 ZSTD_DEPRECATED("use ZSTD_CCtx_reset and ZSTD_CCtx_refCDict, see zstd.h for detailed instructions")
2261 * // Pseudocode: Set each zstd frame parameter and leave the rest as-is.
2273 ZSTD_DEPRECATED("use ZSTD_CCtx_reset and ZSTD_CCtx_refCDict, see zstd.h for detailed instructions")
2288 * This is typically useful to skip dictionary loading stage, since it will re-use it in-place.
2297 ZSTD_DEPRECATED("use ZSTD_CCtx_reset, see zstd.h for detailed instructions")
2313 * Note : (ingested - consumed) is amount of input data buffered internally, not yet compressed.
2363 * re-use decompression parameters from previous init; saves dictionary loading
2370 * Buffer-less and synchronous inner streaming functions
2378 Buffer-less streaming compression (synchronous mode)
2382 ZSTD_CCtx object can be re-used multiple times within successive compression operations.
2390 - ZSTD_compressContinue() has no internal buffer. It uses externally provided buffers only.
2391 - Interface is synchronous : input is consumed entirely and produces 1+ compressed blocks.
2392 …- Caller must ensure there is enough space in `dst` to store compressed data under worst case scen…
2395 …- ZSTD_compressContinue() presumes prior input ***is still accessible and unmodified*** (up to max…
2397 …- ZSTD_compressContinue() detects that prior input has been overwritten when `src` buffer overlaps.
2404 `ZSTD_CCtx` object can be re-used (ZSTD_compressBegin()) to compress again.
2407 /*===== Buffer-less streaming compression functions =====*/
2413 …_compressContinue(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
2414 … ZSTD_compressEnd(ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
2422 Buffer-less streaming decompression (synchronous mode)
2426 A ZSTD_DCtx object can be re-used multiple times.
2437 such as the dictionary ID, content size, or maximum back-reference distance (`windowSize`).
2447 …or that previous contiguous segment is large enough to properly handle maximum back-reference dist…
2452 …which can @return an error code if required value is too large for current system (in 32-bits mode…
2462 as long as the encoder and decoder progress in "lock-step",
2484 Skippable frames allow integration of user-defined data into a flow of concatenated frames.
2487 a) Skippable frame ID - 4 Bytes, Little endian format, any value from 0x184D2A50 to 0x184D2A5F
2488 b) Frame Size - 4 Bytes, Little endian format, unsigned 32-bits
2489 c) Frame Content - any content (User Data) of length equal to Frame Size
2490 For skippable frames ZSTD_getFrameHeader() returns zfhPtr->frameType==ZSTD_skippableFrame.
2494 /*===== Buffer-less streaming decompression functions =====*/
2511 ZSTDLIB_STATIC_API size_t ZSTD_getFrameHeader(ZSTD_frameHeader* zfhPtr, const void* src, size_t src…
2515 ZSTDLIB_STATIC_API size_t ZSTD_getFrameHeader_advanced(ZSTD_frameHeader* zfhPtr, const void* src, s…
2523 …ecompressContinue(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
2538 Block functions produce and decode raw zstd blocks, without frame metadata.
2539 …Frame metadata cost is typically ~12 bytes, which can be non-negligible for very small blocks (< 1…
2543 - Compressing and decompressing require a context structure
2545 - It is necessary to init context before starting
2549 - Block size is limited, it must be <= ZSTD_getBlockSize() <= ZSTD_BLOCKSIZE_MAX == 128 KB
2553 …- When a block is considered not compressible enough, ZSTD_compressBlock() result will be 0 (zero)…
2564 /*===== Raw zstd block functions =====*/
2566 …D_compressBlock (ZSTD_CCtx* cctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
2567 …D_decompressBlock(ZSTD_DCtx* dctx, void* dst, size_t dstCapacity, const void* src, size_t srcSize);
2568 …Size); /**< insert uncompressed block into `dctx` history. Useful for multi-blocks decompression.…