• Home
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1# LZ4 Streaming API Example : Double Buffer
2by *Takayuki Matsuoka*
3
4`blockStreaming_doubleBuffer.c` is LZ4 Straming API example which implements double buffer (de)compression.
5
6Please note :
7
8 - Firstly, read "LZ4 Streaming API Basics".
9 - This is relatively advanced application example.
10 - Output file is not compatible with lz4frame and platform dependent.
11
12
13## What's the point of this example ?
14
15 - Handle huge file in small amount of memory
16 - Always better compression ratio than Block API
17 - Uniform block size
18
19
20## How the compression works
21
22First of all, allocate "Double Buffer" for input and LZ4 compressed data buffer for output.
23Double buffer has two pages, "first" page (Page#1) and "second" page (Page#2).
24
25```
26        Double Buffer
27
28      Page#1    Page#2
29    +---------+---------+
30    | Block#1 |         |
31    +----+----+---------+
32         |
33         v
34      {Out#1}
35
36
37      Prefix Dependency
38         +---------+
39         |         |
40         v         |
41    +---------+----+----+
42    | Block#1 | Block#2 |
43    +---------+----+----+
44                   |
45                   v
46                {Out#2}
47
48
49   External Dictionary Mode
50         +---------+
51         |         |
52         |         v
53    +----+----+---------+
54    | Block#3 | Block#2 |
55    +----+----+---------+
56         |
57         v
58      {Out#3}
59
60
61      Prefix Dependency
62         +---------+
63         |         |
64         v         |
65    +---------+----+----+
66    | Block#3 | Block#4 |
67    +---------+----+----+
68                   |
69                   v
70                {Out#4}
71```
72
73Next, read first block to double buffer's first page. And compress it by `LZ4_compress_continue()`.
74For the first time, LZ4 doesn't know any previous dependencies,
75so it just compress the line without dependencies and generates compressed block {Out#1} to LZ4 compressed data buffer.
76After that, write {Out#1} to the file.
77
78Next, read second block to double buffer's second page. And compress it.
79In this time, LZ4 can use dependency to Block#1 to improve compression ratio.
80This dependency is called "Prefix mode".
81
82Next, read third block to double buffer's *first* page. And compress it.
83Also this time, LZ4 can use dependency to Block#2.
84This dependency is called "External Dictonaly mode".
85
86Continue these procedure to the end of the file.
87
88
89## How the decompression works
90
91Decompression will do reverse order.
92
93 - Read first compressed block.
94 - Decompress it to the first page and write that page to the file.
95 - Read second compressed block.
96 - Decompress it to the second page and write that page to the file.
97 - Read third compressed block.
98 - Decompress it to the *first* page and write that page to the file.
99
100Continue these procedure to the end of the compressed file.
101