1<!-- 2Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al. 3 4SPDX-License-Identifier: curl 5--> 6 7# bufq 8 9This is an internal module for managing I/O buffers. A `bufq` can be written 10to and read from. It manages read and write positions and has a maximum size. 11 12## read/write 13 14Its basic read/write functions have a similar signature and return code 15handling as many internal curl read and write ones. 16 17 18``` 19ssize_t Curl_bufq_write(struct bufq *q, const unsigned char *buf, size_t len, CURLcode *err); 20 21- returns the length written into `q` or -1 on error. 22- writing to a full `q` returns -1 and set *err to CURLE_AGAIN 23 24ssize_t Curl_bufq_read(struct bufq *q, unsigned char *buf, size_t len, CURLcode *err); 25 26- returns the length read from `q` or -1 on error. 27- reading from an empty `q` returns -1 and set *err to CURLE_AGAIN 28 29``` 30 31To pass data into a `bufq` without an extra copy, read callbacks can be used. 32 33``` 34typedef ssize_t Curl_bufq_reader(void *reader_ctx, unsigned char *buf, size_t len, 35 CURLcode *err); 36 37ssize_t Curl_bufq_slurp(struct bufq *q, Curl_bufq_reader *reader, void *reader_ctx, 38 CURLcode *err); 39``` 40 41`Curl_bufq_slurp()` invokes the given `reader` callback, passing it its own 42internal buffer memory to write to. It may invoke the `reader` several times, 43as long as it has space and while the `reader` always returns the length that 44was requested. There are variations of `slurp` that call the `reader` at most 45once or only read in a maximum amount of bytes. 46 47The analog mechanism for write out buffer data is: 48 49``` 50typedef ssize_t Curl_bufq_writer(void *writer_ctx, const unsigned char *buf, size_t len, 51 CURLcode *err); 52 53ssize_t Curl_bufq_pass(struct bufq *q, Curl_bufq_writer *writer, void *writer_ctx, 54 CURLcode *err); 55``` 56 57`Curl_bufq_pass()` invokes the `writer`, passing its internal memory and 58remove the amount that `writer` reports. 59 60## peek and skip 61 62It is possible to get access to the memory of data stored in a `bufq` with: 63 64``` 65bool Curl_bufq_peek(const struct bufq *q, const unsigned char **pbuf, size_t *plen); 66``` 67 68On returning TRUE, `pbuf` points to internal memory with `plen` bytes that one 69may read. This is only valid until another operation on `bufq` is performed. 70 71Instead of reading `bufq` data, one may simply skip it: 72 73``` 74void Curl_bufq_skip(struct bufq *q, size_t amount); 75``` 76 77This removes `amount` number of bytes from the `bufq`. 78 79## unwrite 80 81It is possible to undo writes by calling: 82 83``` 84CURLcode Curl_bufq_unwrite(struct bufq *q, size_t len); 85``` 86 87This removes `len` bytes from the end of the bufq again. When removing more 88bytes than are present, CURLE_AGAIN is returned and bufq is cleared. 89 90## lifetime 91 92`bufq` is initialized and freed similar to the `dynbuf` module. Code using 93`bufq` holds a `struct bufq` somewhere. Before it uses it, it invokes: 94 95``` 96void Curl_bufq_init(struct bufq *q, size_t chunk_size, size_t max_chunks); 97``` 98 99The `bufq` is told how many "chunks" of data it shall hold at maximum and how 100large those "chunks" should be. There are some variants of this, allowing for 101more options. How "chunks" are handled in a `bufq` is presented in the section 102about memory management. 103 104The user of the `bufq` has the responsibility to call: 105 106``` 107void Curl_bufq_free(struct bufq *q); 108``` 109to free all resources held by `q`. It is possible to reset a `bufq` to empty via: 110 111``` 112void Curl_bufq_reset(struct bufq *q); 113``` 114 115## memory management 116 117Internally, a `bufq` uses allocation of fixed size, e.g. the "chunk_size", up 118to a maximum number, e.g. "max_chunks". These chunks are allocated on demand, 119therefore writing to a `bufq` may return `CURLE_OUT_OF_MEMORY`. Once the max 120number of chunks are used, the `bufq` reports that it is "full". 121 122Each chunks has a `read` and `write` index. A `bufq` keeps its chunks in a 123list. Reading happens always at the head chunk, writing always goes to the 124tail chunk. When the head chunk becomes empty, it is removed. When the tail 125chunk becomes full, another chunk is added to the end of the list, becoming 126the new tail. 127 128Chunks that are no longer used are returned to a `spare` list by default. If 129the `bufq` is created with option `BUFQ_OPT_NO_SPARES` those chunks are freed 130right away. 131 132If a `bufq` is created with a `bufc_pool`, the no longer used chunks are 133returned to the pool. Also `bufq` asks the pool for a chunk when it needs one. 134More in section "pools". 135 136## empty, full and overflow 137 138One can ask about the state of a `bufq` with methods such as 139`Curl_bufq_is_empty(q)`, `Curl_bufq_is_full(q)`, etc. The amount of data held 140by a `bufq` is the sum of the data in all its chunks. This is what is reported 141by `Curl_bufq_len(q)`. 142 143Note that a `bufq` length and it being "full" are only loosely related. A 144simple example: 145 146* create a `bufq` with chunk_size=1000 and max_chunks=4. 147* write 4000 bytes to it, it reports "full" 148* read 1 bytes from it, it still reports "full" 149* read 999 more bytes from it, and it is no longer "full" 150 151The reason for this is that full really means: *bufq uses max_chunks and the 152last one cannot be written to*. 153 154When you read 1 byte from the head chunk in the example above, the head still 155hold 999 unread bytes. Only when those are also read, can the head chunk be 156removed and a new tail be added. 157 158There is another variation to this. If you initialized a `bufq` with option 159`BUFQ_OPT_SOFT_LIMIT`, it allows writes **beyond** the `max_chunks`. It 160reports **full**, but one can **still** write. This option is necessary, if 161partial writes need to be avoided. It means that you need other checks to keep 162the `bufq` from growing ever larger and larger. 163 164 165## pools 166 167A `struct bufc_pool` may be used to create chunks for a `bufq` and keep spare 168ones around. It is initialized and used via: 169 170``` 171void Curl_bufcp_init(struct bufc_pool *pool, size_t chunk_size, size_t spare_max); 172 173void Curl_bufq_initp(struct bufq *q, struct bufc_pool *pool, size_t max_chunks, int opts); 174``` 175 176The pool gets the size and the mount of spares to keep. The `bufq` gets the 177pool and the `max_chunks`. It no longer needs to know the chunk sizes, as 178those are managed by the pool. 179 180A pool can be shared between many `bufq`s, as long as all of them operate in 181the same thread. In curl that would be true for all transfers using the same 182multi handle. The advantages of a pool are: 183 184* when all `bufq`s are empty, only memory for `max_spare` chunks in the pool 185 is used. Empty `bufq`s holds no memory. 186* the latest spare chunk is the first to be handed out again, no matter which 187 `bufq` needs it. This keeps the footprint of "recently used" memory smaller. 188