• Home
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1# LZ4 Streaming API Basics
2by *Takayuki Matsuoka*
3## LZ4 API sets
4
5LZ4 has the following API sets :
6
7 - "Auto Framing" API (lz4frame.h) :
8   This is most recommended API for usual application.
9   It guarantees interoperability with other LZ4 framing format compliant tools/libraries
10   such as LZ4 command line utility, node-lz4, etc.
11 - "Block" API : This is recommended for simple purpose.
12   It compress single raw memory block to LZ4 memory block and vice versa.
13 - "Streaming" API : This is designed for complex thing.
14   For example, compress huge stream data in restricted memory environment.
15
16Basically, you should use "Auto Framing" API.
17But if you want to write advanced application, it's time to use Block or Streaming APIs.
18
19
20## What is difference between Block and Streaming API ?
21
22Block API (de)compresses single contiguous memory block.
23In other words, LZ4 library find redundancy from single contiguous memory block.
24Streaming API does same thing but (de)compress multiple adjacent contiguous memory block.
25So LZ4 library could find more redundancy than Block API.
26
27The following figure shows difference between API and block sizes.
28In these figures, original data is splitted to 4KiBytes contiguous chunks.
29
30```
31Original Data
32    +---------------+---------------+----+----+----+
33    | 4KiB Chunk A  | 4KiB Chunk B  | C  | D  |... |
34    +---------------+---------------+----+----+----+
35
36Example (1) : Block API, 4KiB Block
37    +---------------+---------------+----+----+----+
38    | 4KiB Chunk A  | 4KiB Chunk B  | C  | D  |... |
39    +---------------+---------------+----+----+----+
40    | Block #1      | Block #2      | #3 | #4 |... |
41    +---------------+---------------+----+----+----+
42
43                    (No Dependency)
44
45
46Example (2) : Block API, 8KiB Block
47    +---------------+---------------+----+----+----+
48    | 4KiB Chunk A  | 4KiB Chunk B  | C  | D  |... |
49    +---------------+---------------+----+----+----+
50    |            Block #1           |Block #2 |... |
51    +--------------------+----------+-------+-+----+
52          ^              |             ^    |
53          |              |             |    |
54          +--------------+             +----+
55          Internal Dependency          Internal Dependency
56
57
58Example (3) : Streaming API, 4KiB Block
59    +---------------+---------------+-----+----+----+
60    | 4KiB Chunk A  | 4KiB Chunk B  | C   | D  |... |
61    +---------------+---------------+-----+----+----+
62    | Block #1      | Block #2      | #3  | #4 |... |
63    +---------------+----+----------+-+---+-+--+----+
64          ^              |   ^        | ^   |
65          |              |   |        | |   |
66          +--------------+   +--------+ +---+
67          Dependency         Dependency Dependency
68```
69
70 - In example (1), there is no dependency.
71   All blocks are compressed independently.
72 - In example (2), naturally 8KiBytes block has internal dependency.
73   But still block #1 and #2 are compressed independently.
74 - In example (3), block #2 has dependency to #1,
75   also #3 has dependency to #2 and #1, #4 has #3, #2 and #1, and so on.
76
77Here, we can observe difference between example (2) and (3).
78In (2), there's no dependency between chunk B and C, but (3) has dependency between B and C.
79This dependency improves compression ratio.
80
81
82## Restriction of Streaming API
83
84For the efficiency, Streaming API doesn't keep mirror copy of dependent (de)compressed memory.
85This means users should keep these dependent (de)compressed memory explicitly.
86Usually, "Dependent memory" is previous adjacent contiguous memory up to 64KiBytes.
87LZ4 will not access further memories.
88