1<!DOCTYPE html> 2<html> 3<head> 4<link rel="stylesheet" type="text/css" href="doc.css" /> 5<title>Leveldb file layout and compactions</title> 6</head> 7 8<body> 9 10<h1>Files</h1> 11 12The implementation of leveldb is similar in spirit to the 13representation of a single 14<a href="http://research.google.com/archive/bigtable.html"> 15Bigtable tablet (section 5.3)</a>. 16However the organization of the files that make up the representation 17is somewhat different and is explained below. 18 19<p> 20Each database is represented by a set of files stored in a directory. 21There are several different types of files as documented below: 22<p> 23<h2>Log files</h2> 24<p> 25A log file (*.log) stores a sequence of recent updates. Each update 26is appended to the current log file. When the log file reaches a 27pre-determined size (approximately 4MB by default), it is converted 28to a sorted table (see below) and a new log file is created for future 29updates. 30<p> 31A copy of the current log file is kept in an in-memory structure (the 32<code>memtable</code>). This copy is consulted on every read so that read 33operations reflect all logged updates. 34<p> 35<h2>Sorted tables</h2> 36<p> 37A sorted table (*.sst) stores a sequence of entries sorted by key. 38Each entry is either a value for the key, or a deletion marker for the 39key. (Deletion markers are kept around to hide obsolete values 40present in older sorted tables). 41<p> 42The set of sorted tables are organized into a sequence of levels. The 43sorted table generated from a log file is placed in a special <code>young</code> 44level (also called level-0). When the number of young files exceeds a 45certain threshold (currently four), all of the young files are merged 46together with all of the overlapping level-1 files to produce a 47sequence of new level-1 files (we create a new level-1 file for every 482MB of data.) 49<p> 50Files in the young level may contain overlapping keys. However files 51in other levels have distinct non-overlapping key ranges. Consider 52level number L where L >= 1. When the combined size of files in 53level-L exceeds (10^L) MB (i.e., 10MB for level-1, 100MB for level-2, 54...), one file in level-L, and all of the overlapping files in 55level-(L+1) are merged to form a set of new files for level-(L+1). 56These merges have the effect of gradually migrating new updates from 57the young level to the largest level using only bulk reads and writes 58(i.e., minimizing expensive seeks). 59 60<h2>Manifest</h2> 61<p> 62A MANIFEST file lists the set of sorted tables that make up each 63level, the corresponding key ranges, and other important metadata. 64A new MANIFEST file (with a new number embedded in the file name) 65is created whenever the database is reopened. The MANIFEST file is 66formatted as a log, and changes made to the serving state (as files 67are added or removed) are appended to this log. 68<p> 69<h2>Current</h2> 70<p> 71CURRENT is a simple text file that contains the name of the latest 72MANIFEST file. 73<p> 74<h2>Info logs</h2> 75<p> 76Informational messages are printed to files named LOG and LOG.old. 77<p> 78<h2>Others</h2> 79<p> 80Other files used for miscellaneous purposes may also be present 81(LOCK, *.dbtmp). 82 83<h1>Level 0</h1> 84When the log file grows above a certain size (1MB by default): 85<ul> 86<li>Create a brand new memtable and log file and direct future updates here 87<li>In the background: 88<ul> 89<li>Write the contents of the previous memtable to an sstable 90<li>Discard the memtable 91<li>Delete the old log file and the old memtable 92<li>Add the new sstable to the young (level-0) level. 93</ul> 94</ul> 95 96<h1>Compactions</h1> 97 98<p> 99When the size of level L exceeds its limit, we compact it in a 100background thread. The compaction picks a file from level L and all 101overlapping files from the next level L+1. Note that if a level-L 102file overlaps only part of a level-(L+1) file, the entire file at 103level-(L+1) is used as an input to the compaction and will be 104discarded after the compaction. Aside: because level-0 is special 105(files in it may overlap each other), we treat compactions from 106level-0 to level-1 specially: a level-0 compaction may pick more than 107one level-0 file in case some of these files overlap each other. 108 109<p> 110A compaction merges the contents of the picked files to produce a 111sequence of level-(L+1) files. We switch to producing a new 112level-(L+1) file after the current output file has reached the target 113file size (2MB). We also switch to a new output file when the key 114range of the current output file has grown enough to overlap more then 115ten level-(L+2) files. This last rule ensures that a later compaction 116of a level-(L+1) file will not pick up too much data from level-(L+2). 117 118<p> 119The old files are discarded and the new files are added to the serving 120state. 121 122<p> 123Compactions for a particular level rotate through the key space. In 124more detail, for each level L, we remember the ending key of the last 125compaction at level L. The next compaction for level L will pick the 126first file that starts after this key (wrapping around to the 127beginning of the key space if there is no such file). 128 129<p> 130Compactions drop overwritten values. They also drop deletion markers 131if there are no higher numbered levels that contain a file whose range 132overlaps the current key. 133 134<h2>Timing</h2> 135 136Level-0 compactions will read up to four 1MB files from level-0, and 137at worst all the level-1 files (10MB). I.e., we will read 14MB and 138write 14MB. 139 140<p> 141Other than the special level-0 compactions, we will pick one 2MB file 142from level L. In the worst case, this will overlap ~ 12 files from 143level L+1 (10 because level-(L+1) is ten times the size of level-L, 144and another two at the boundaries since the file ranges at level-L 145will usually not be aligned with the file ranges at level-L+1). The 146compaction will therefore read 26MB and write 26MB. Assuming a disk 147IO rate of 100MB/s (ballpark range for modern drives), the worst 148compaction cost will be approximately 0.5 second. 149 150<p> 151If we throttle the background writing to something small, say 10% of 152the full 100MB/s speed, a compaction may take up to 5 seconds. If the 153user is writing at 10MB/s, we might build up lots of level-0 files 154(~50 to hold the 5*10MB). This may signficantly increase the cost of 155reads due to the overhead of merging more files together on every 156read. 157 158<p> 159Solution 1: To reduce this problem, we might want to increase the log 160switching threshold when the number of level-0 files is large. Though 161the downside is that the larger this threshold, the more memory we will 162need to hold the corresponding memtable. 163 164<p> 165Solution 2: We might want to decrease write rate artificially when the 166number of level-0 files goes up. 167 168<p> 169Solution 3: We work on reducing the cost of very wide merges. 170Perhaps most of the level-0 files will have their blocks sitting 171uncompressed in the cache and we will only need to worry about the 172O(N) complexity in the merging iterator. 173 174<h2>Number of files</h2> 175 176Instead of always making 2MB files, we could make larger files for 177larger levels to reduce the total file count, though at the expense of 178more bursty compactions. Alternatively, we could shard the set of 179files into multiple directories. 180 181<p> 182An experiment on an <code>ext3</code> filesystem on Feb 04, 2011 shows 183the following timings to do 100K file opens in directories with 184varying number of files: 185<table class="datatable"> 186<tr><th>Files in directory</th><th>Microseconds to open a file</th></tr> 187<tr><td>1000</td><td>9</td> 188<tr><td>10000</td><td>10</td> 189<tr><td>100000</td><td>16</td> 190</table> 191So maybe even the sharding is not necessary on modern filesystems? 192 193<h1>Recovery</h1> 194 195<ul> 196<li> Read CURRENT to find name of the latest committed MANIFEST 197<li> Read the named MANIFEST file 198<li> Clean up stale files 199<li> We could open all sstables here, but it is probably better to be lazy... 200<li> Convert log chunk to a new level-0 sstable 201<li> Start directing new writes to a new log file with recovered sequence# 202</ul> 203 204<h1>Garbage collection of files</h1> 205 206<code>DeleteObsoleteFiles()</code> is called at the end of every 207compaction and at the end of recovery. It finds the names of all 208files in the database. It deletes all log files that are not the 209current log file. It deletes all table files that are not referenced 210from some level and are not the output of an active compaction. 211 212</body> 213</html> 214