1Introduction 2============ 3 4dm-cache is a device mapper target written by Joe Thornber, Heinz 5Mauelshagen, and Mike Snitzer. 6 7It aims to improve performance of a block device (eg, a spindle) by 8dynamically migrating some of its data to a faster, smaller device 9(eg, an SSD). 10 11This device-mapper solution allows us to insert this caching at 12different levels of the dm stack, for instance above the data device for 13a thin-provisioning pool. Caching solutions that are integrated more 14closely with the virtual memory system should give better performance. 15 16The target reuses the metadata library used in the thin-provisioning 17library. 18 19The decision as to what data to migrate and when is left to a plug-in 20policy module. Several of these have been written as we experiment, 21and we hope other people will contribute others for specific io 22scenarios (eg. a vm image server). 23 24Glossary 25======== 26 27 Migration - Movement of the primary copy of a logical block from one 28 device to the other. 29 Promotion - Migration from slow device to fast device. 30 Demotion - Migration from fast device to slow device. 31 32The origin device always contains a copy of the logical block, which 33may be out of date or kept in sync with the copy on the cache device 34(depending on policy). 35 36Design 37====== 38 39Sub-devices 40----------- 41 42The target is constructed by passing three devices to it (along with 43other parameters detailed later): 44 451. An origin device - the big, slow one. 46 472. A cache device - the small, fast one. 48 493. A small metadata device - records which blocks are in the cache, 50 which are dirty, and extra hints for use by the policy object. 51 This information could be put on the cache device, but having it 52 separate allows the volume manager to configure it differently, 53 e.g. as a mirror for extra robustness. 54 55Fixed block size 56---------------- 57 58The origin is divided up into blocks of a fixed size. This block size 59is configurable when you first create the cache. Typically we've been 60using block sizes of 256k - 1024k. 61 62Having a fixed block size simplifies the target a lot. But it is 63something of a compromise. For instance, a small part of a block may be 64getting hit a lot, yet the whole block will be promoted to the cache. 65So large block sizes are bad because they waste cache space. And small 66block sizes are bad because they increase the amount of metadata (both 67in core and on disk). 68 69Writeback/writethrough 70---------------------- 71 72The cache has two modes, writeback and writethrough. 73 74If writeback, the default, is selected then a write to a block that is 75cached will go only to the cache and the block will be marked dirty in 76the metadata. 77 78If writethrough is selected then a write to a cached block will not 79complete until it has hit both the origin and cache devices. Clean 80blocks should remain clean. 81 82A simple cleaner policy is provided, which will clean (write back) all 83dirty blocks in a cache. Useful for decommissioning a cache. 84 85Migration throttling 86-------------------- 87 88Migrating data between the origin and cache device uses bandwidth. 89The user can set a throttle to prevent more than a certain amount of 90migration occuring at any one time. Currently we're not taking any 91account of normal io traffic going to the devices. More work needs 92doing here to avoid migrating during those peak io moments. 93 94For the time being, a message "migration_threshold <#sectors>" 95can be used to set the maximum number of sectors being migrated, 96the default being 204800 sectors (or 100MB). 97 98Updating on-disk metadata 99------------------------- 100 101On-disk metadata is committed every time a REQ_SYNC or REQ_FUA bio is 102written. If no such requests are made then commits will occur every 103second. This means the cache behaves like a physical disk that has a 104write cache (the same is true of the thin-provisioning target). If 105power is lost you may lose some recent writes. The metadata should 106always be consistent in spite of any crash. 107 108The 'dirty' state for a cache block changes far too frequently for us 109to keep updating it on the fly. So we treat it as a hint. In normal 110operation it will be written when the dm device is suspended. If the 111system crashes all cache blocks will be assumed dirty when restarted. 112 113Per-block policy hints 114---------------------- 115 116Policy plug-ins can store a chunk of data per cache block. It's up to 117the policy how big this chunk is, but it should be kept small. Like the 118dirty flags this data is lost if there's a crash so a safe fallback 119value should always be possible. 120 121For instance, the 'mq' policy, which is currently the default policy, 122uses this facility to store the hit count of the cache blocks. If 123there's a crash this information will be lost, which means the cache 124may be less efficient until those hit counts are regenerated. 125 126Policy hints affect performance, not correctness. 127 128Policy messaging 129---------------- 130 131Policies will have different tunables, specific to each one, so we 132need a generic way of getting and setting these. Device-mapper 133messages are used. Refer to cache-policies.txt. 134 135Discard bitset resolution 136------------------------- 137 138We can avoid copying data during migration if we know the block has 139been discarded. A prime example of this is when mkfs discards the 140whole block device. We store a bitset tracking the discard state of 141blocks. However, we allow this bitset to have a different block size 142from the cache blocks. This is because we need to track the discard 143state for all of the origin device (compare with the dirty bitset 144which is just for the smaller cache device). 145 146Target interface 147================ 148 149Constructor 150----------- 151 152 cache <metadata dev> <cache dev> <origin dev> <block size> 153 <#feature args> [<feature arg>]* 154 <policy> <#policy args> [policy args]* 155 156 metadata dev : fast device holding the persistent metadata 157 cache dev : fast device holding cached data blocks 158 origin dev : slow device holding original data blocks 159 block size : cache unit size in sectors 160 161 #feature args : number of feature arguments passed 162 feature args : writethrough. (The default is writeback.) 163 164 policy : the replacement policy to use 165 #policy args : an even number of arguments corresponding to 166 key/value pairs passed to the policy 167 policy args : key/value pairs passed to the policy 168 E.g. 'sequential_threshold 1024' 169 See cache-policies.txt for details. 170 171Optional feature arguments are: 172 writethrough : write through caching that prohibits cache block 173 content from being different from origin block content. 174 Without this argument, the default behaviour is to write 175 back cache block contents later for performance reasons, 176 so they may differ from the corresponding origin blocks. 177 178A policy called 'default' is always registered. This is an alias for 179the policy we currently think is giving best all round performance. 180 181As the default policy could vary between kernels, if you are relying on 182the characteristics of a specific policy, always request it by name. 183 184Status 185------ 186 187<#used metadata blocks>/<#total metadata blocks> <#read hits> <#read misses> 188<#write hits> <#write misses> <#demotions> <#promotions> <#blocks in cache> 189<#dirty> <#features> <features>* <#core args> <core args>* <#policy args> 190<policy args>* 191 192#used metadata blocks : Number of metadata blocks used 193#total metadata blocks : Total number of metadata blocks 194#read hits : Number of times a READ bio has been mapped 195 to the cache 196#read misses : Number of times a READ bio has been mapped 197 to the origin 198#write hits : Number of times a WRITE bio has been mapped 199 to the cache 200#write misses : Number of times a WRITE bio has been 201 mapped to the origin 202#demotions : Number of times a block has been removed 203 from the cache 204#promotions : Number of times a block has been moved to 205 the cache 206#blocks in cache : Number of blocks resident in the cache 207#dirty : Number of blocks in the cache that differ 208 from the origin 209#feature args : Number of feature args to follow 210feature args : 'writethrough' (optional) 211#core args : Number of core arguments (must be even) 212core args : Key/value pairs for tuning the core 213 e.g. migration_threshold 214#policy args : Number of policy arguments to follow (must be even) 215policy args : Key/value pairs 216 e.g. 'sequential_threshold 1024 217 218Messages 219-------- 220 221Policies will have different tunables, specific to each one, so we 222need a generic way of getting and setting these. Device-mapper 223messages are used. (A sysfs interface would also be possible.) 224 225The message format is: 226 227 <key> <value> 228 229E.g. 230 dmsetup message my_cache 0 sequential_threshold 1024 231 232Examples 233======== 234 235The test suite can be found here: 236 237https://github.com/jthornber/thinp-test-suite 238 239dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \ 240 /dev/mapper/ssd /dev/mapper/origin 512 1 writeback default 0' 241dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \ 242 /dev/mapper/ssd /dev/mapper/origin 1024 1 writeback \ 243 mq 4 sequential_threshold 1024 random_threshold 8' 244