1zram: Compressed RAM based block devices 2---------------------------------------- 3 4* Introduction 5 6The zram module creates RAM based block devices named /dev/zram<id> 7(<id> = 0, 1, ...). Pages written to these disks are compressed and stored 8in memory itself. These disks allow very fast I/O and compression provides 9good amounts of memory savings. Some of the usecases include /tmp storage, 10use as swap disks, various caches under /var and maybe many more :) 11 12Statistics for individual zram devices are exported through sysfs nodes at 13/sys/block/zram<id>/ 14 15* Usage 16 17There are several ways to configure and manage zram device(-s): 18a) using zram and zram_control sysfs attributes 19b) using zramctl utility, provided by util-linux (util-linux@vger.kernel.org). 20 21In this document we will describe only 'manual' zram configuration steps, 22IOW, zram and zram_control sysfs attributes. 23 24In order to get a better idea about zramctl please consult util-linux 25documentation, zramctl man-page or `zramctl --help'. Please be informed 26that zram maintainers do not develop/maintain util-linux or zramctl, should 27you have any questions please contact util-linux@vger.kernel.org 28 29Following shows a typical sequence of steps for using zram. 30 31WARNING 32======= 33For the sake of simplicity we skip error checking parts in most of the 34examples below. However, it is your sole responsibility to handle errors. 35 36zram sysfs attributes always return negative values in case of errors. 37The list of possible return codes: 38-EBUSY -- an attempt to modify an attribute that cannot be changed once 39the device has been initialised. Please reset device first; 40-ENOMEM -- zram was not able to allocate enough memory to fulfil your 41needs; 42-EINVAL -- invalid input has been provided. 43 44If you use 'echo', the returned value that is changed by 'echo' utility, 45and, in general case, something like: 46 47 echo 3 > /sys/block/zram0/max_comp_streams 48 if [ $? -ne 0 ]; 49 handle_error 50 fi 51 52should suffice. 53 541) Load Module: 55 modprobe zram num_devices=4 56 This creates 4 devices: /dev/zram{0,1,2,3} 57 58num_devices parameter is optional and tells zram how many devices should be 59pre-created. Default: 1. 60 612) Set max number of compression streams 62Regardless the value passed to this attribute, ZRAM will always 63allocate multiple compression streams - one per online CPUs - thus 64allowing several concurrent compression operations. The number of 65allocated compression streams goes down when some of the CPUs 66become offline. There is no single-compression-stream mode anymore, 67unless you are running a UP system or has only 1 CPU online. 68 69To find out how many streams are currently available: 70 cat /sys/block/zram0/max_comp_streams 71 723) Select compression algorithm 73Using comp_algorithm device attribute one can see available and 74currently selected (shown in square brackets) compression algorithms, 75change selected compression algorithm (once the device is initialised 76there is no way to change compression algorithm). 77 78Examples: 79 #show supported compression algorithms 80 cat /sys/block/zram0/comp_algorithm 81 lzo [lz4] 82 83 #select lzo compression algorithm 84 echo lzo > /sys/block/zram0/comp_algorithm 85 86For the time being, the `comp_algorithm' content does not necessarily 87show every compression algorithm supported by the kernel. We keep this 88list primarily to simplify device configuration and one can configure 89a new device with a compression algorithm that is not listed in 90`comp_algorithm'. The thing is that, internally, ZRAM uses Crypto API 91and, if some of the algorithms were built as modules, it's impossible 92to list all of them using, for instance, /proc/crypto or any other 93method. This, however, has an advantage of permitting the usage of 94custom crypto compression modules (implementing S/W or H/W compression). 95 964) Set Disksize 97Set disk size by writing the value to sysfs node 'disksize'. 98The value can be either in bytes or you can use mem suffixes. 99Examples: 100 # Initialize /dev/zram0 with 50MB disksize 101 echo $((50*1024*1024)) > /sys/block/zram0/disksize 102 103 # Using mem suffixes 104 echo 256K > /sys/block/zram0/disksize 105 echo 512M > /sys/block/zram0/disksize 106 echo 1G > /sys/block/zram0/disksize 107 108Note: 109There is little point creating a zram of greater than twice the size of memory 110since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of the 111size of the disk when not in use so a huge zram is wasteful. 112 1135) Set memory limit: Optional 114Set memory limit by writing the value to sysfs node 'mem_limit'. 115The value can be either in bytes or you can use mem suffixes. 116In addition, you could change the value in runtime. 117Examples: 118 # limit /dev/zram0 with 50MB memory 119 echo $((50*1024*1024)) > /sys/block/zram0/mem_limit 120 121 # Using mem suffixes 122 echo 256K > /sys/block/zram0/mem_limit 123 echo 512M > /sys/block/zram0/mem_limit 124 echo 1G > /sys/block/zram0/mem_limit 125 126 # To disable memory limit 127 echo 0 > /sys/block/zram0/mem_limit 128 1296) Activate: 130 mkswap /dev/zram0 131 swapon /dev/zram0 132 133 mkfs.ext4 /dev/zram1 134 mount /dev/zram1 /tmp 135 1367) Add/remove zram devices 137 138zram provides a control interface, which enables dynamic (on-demand) device 139addition and removal. 140 141In order to add a new /dev/zramX device, perform read operation on hot_add 142attribute. This will return either new device's device id (meaning that you 143can use /dev/zram<id>) or error code. 144 145Example: 146 cat /sys/class/zram-control/hot_add 147 1 148 149To remove the existing /dev/zramX device (where X is a device id) 150execute 151 echo X > /sys/class/zram-control/hot_remove 152 1538) Stats: 154Per-device statistics are exported as various nodes under /sys/block/zram<id>/ 155 156A brief description of exported device attributes. For more details please 157read Documentation/ABI/testing/sysfs-block-zram. 158 159Name access description 160---- ------ ----------- 161disksize RW show and set the device's disk size 162initstate RO shows the initialization state of the device 163reset WO trigger device reset 164mem_used_max WO reset the `mem_used_max' counter (see later) 165mem_limit WO specifies the maximum amount of memory ZRAM can use 166 to store the compressed data 167writeback_limit WO specifies the maximum amount of write IO zram can 168 write out to backing device as 4KB unit 169writeback_limit_enable RW show and set writeback_limit feature 170max_comp_streams RW the number of possible concurrent compress operations 171comp_algorithm RW show and change the compression algorithm 172compact WO trigger memory compaction 173debug_stat RO this file is used for zram debugging purposes 174backing_dev RW set up backend storage for zram to write out 175idle WO mark allocated slot as idle 176 177 178User space is advised to use the following files to read the device statistics. 179 180File /sys/block/zram<id>/stat 181 182Represents block layer statistics. Read Documentation/block/stat.txt for 183details. 184 185File /sys/block/zram<id>/io_stat 186 187The stat file represents device's I/O statistics not accounted by block 188layer and, thus, not available in zram<id>/stat file. It consists of a 189single line of text and contains the following stats separated by 190whitespace: 191 failed_reads the number of failed reads 192 failed_writes the number of failed writes 193 invalid_io the number of non-page-size-aligned I/O requests 194 notify_free Depending on device usage scenario it may account 195 a) the number of pages freed because of swap slot free 196 notifications or b) the number of pages freed because of 197 REQ_DISCARD requests sent by bio. The former ones are 198 sent to a swap block device when a swap slot is freed, 199 which implies that this disk is being used as a swap disk. 200 The latter ones are sent by filesystem mounted with 201 discard option, whenever some data blocks are getting 202 discarded. 203 204File /sys/block/zram<id>/mm_stat 205 206The stat file represents device's mm statistics. It consists of a single 207line of text and contains the following stats separated by whitespace: 208 orig_data_size uncompressed size of data stored in this disk. 209 This excludes same-element-filled pages (same_pages) since 210 no memory is allocated for them. 211 Unit: bytes 212 compr_data_size compressed size of data stored in this disk 213 mem_used_total the amount of memory allocated for this disk. This 214 includes allocator fragmentation and metadata overhead, 215 allocated for this disk. So, allocator space efficiency 216 can be calculated using compr_data_size and this statistic. 217 Unit: bytes 218 mem_limit the maximum amount of memory ZRAM can use to store 219 the compressed data 220 mem_used_max the maximum amount of memory zram have consumed to 221 store the data 222 same_pages the number of same element filled pages written to this disk. 223 No memory is allocated for such pages. 224 pages_compacted the number of pages freed during compaction 225 huge_pages the number of incompressible pages 226 227File /sys/block/zram<id>/bd_stat 228 229The stat file represents device's backing device statistics. It consists of 230a single line of text and contains the following stats separated by whitespace: 231 bd_count size of data written in backing device. 232 Unit: 4K bytes 233 bd_reads the number of reads from backing device 234 Unit: 4K bytes 235 bd_writes the number of writes to backing device 236 Unit: 4K bytes 237 2389) Deactivate: 239 swapoff /dev/zram0 240 umount /dev/zram1 241 24210) Reset: 243 Write any positive value to 'reset' sysfs node 244 echo 1 > /sys/block/zram0/reset 245 echo 1 > /sys/block/zram1/reset 246 247 This frees all the memory allocated for the given device and 248 resets the disksize to zero. You must set the disksize again 249 before reusing the device. 250 251* Optional Feature 252 253= writeback 254 255With CONFIG_ZRAM_WRITEBACK, zram can write idle/incompressible page 256to backing storage rather than keeping it in memory. 257To use the feature, admin should set up backing device via 258 259 "echo /dev/sda5 > /sys/block/zramX/backing_dev" 260 261before disksize setting. It supports only partition at this moment. 262If admin want to use incompressible page writeback, they could do via 263 264 "echo huge > /sys/block/zramX/write" 265 266To use idle page writeback, first, user need to declare zram pages 267as idle. 268 269 "echo all > /sys/block/zramX/idle" 270 271From now on, any pages on zram are idle pages. The idle mark 272will be removed until someone request access of the block. 273IOW, unless there is access request, those pages are still idle pages. 274 275Admin can request writeback of those idle pages at right timing via 276 277 "echo idle > /sys/block/zramX/writeback" 278 279With the command, zram writeback idle pages from memory to the storage. 280 281If there are lots of write IO with flash device, potentially, it has 282flash wearout problem so that admin needs to design write limitation 283to guarantee storage health for entire product life. 284 285To overcome the concern, zram supports "writeback_limit" feature. 286The "writeback_limit_enable"'s default value is 0 so that it doesn't limit 287any writeback. IOW, if admin want to apply writeback budget, he should 288enable writeback_limit_enable via 289 290 $ echo 1 > /sys/block/zramX/writeback_limit_enable 291 292Once writeback_limit_enable is set, zram doesn't allow any writeback 293until admin set the budget via /sys/block/zramX/writeback_limit. 294 295(If admin doesn't enable writeback_limit_enable, writeback_limit's value 296assigned via /sys/block/zramX/writeback_limit is meaninless.) 297 298If admin want to limit writeback as per-day 400M, he could do it 299like below. 300 301 $ MB_SHIFT=20 302 $ 4K_SHIFT=12 303 $ echo $((400<<MB_SHIFT>>4K_SHIFT)) > \ 304 /sys/block/zram0/writeback_limit. 305 $ echo 1 > /sys/block/zram0/writeback_limit_enable 306 307If admin want to allow further write again once the bugdet is exausted, 308he could do it like below 309 310 $ echo $((400<<MB_SHIFT>>4K_SHIFT)) > \ 311 /sys/block/zram0/writeback_limit 312 313If admin want to see remaining writeback budget since he set, 314 315 $ cat /sys/block/zramX/writeback_limit 316 317If admin want to disable writeback limit, he could do 318 319 $ echo 0 > /sys/block/zramX/writeback_limit_enable 320 321The writeback_limit count will reset whenever you reset zram(e.g., 322system reboot, echo 1 > /sys/block/zramX/reset) so keeping how many of 323writeback happened until you reset the zram to allocate extra writeback 324budget in next setting is user's job. 325 326If admin want to measure writeback count in a certain period, he could 327know it via /sys/block/zram0/bd_stat's 3rd column. 328 329= memory tracking 330 331With CONFIG_ZRAM_MEMORY_TRACKING, user can know information of the 332zram block. It could be useful to catch cold or incompressible 333pages of the process with*pagemap. 334If you enable the feature, you could see block state via 335/sys/kernel/debug/zram/zram0/block_state". The output is as follows, 336 337 300 75.033841 .wh. 338 301 63.806904 s... 339 302 63.806919 ..hi 340 341First column is zram's block index. 342Second column is access time since the system was booted 343Third column is state of the block. 344(s: same page 345w: written page to backing store 346h: huge page 347i: idle page) 348 349First line of above example says 300th block is accessed at 75.033841sec 350and the block's state is huge so it is written back to the backing 351storage. It's a debugging feature so anyone shouldn't rely on it to work 352properly. 353 354Nitin Gupta 355ngupta@vflare.org 356