SQUASHFS CHANGE LOG 4.3 12 MAY 2014 New compressor options, new Mksquashfs/Unsquashfs functionality, duplicate checking optimisations, stability improvements (option/file parsing, buffer/memory overflow checks, filesystem hardening on corrupted filesystems), CVE fixes. Too many changes to do the traditional custom changelog. But, this is now unnecessary, so instead list most significant 15% of commits from git changelog in chronological order. - unsquashfs: add checks for corrupted data in opendir functions - unsquashfs: completely empty filesystems incorrectly generate an error - unsquashfs: fix open file limit - mksquashfs: Use linked list to store directory entries rather - mksquashfs: Remove qsort and add a bottom up linked list merge sort - mksquashfs: optimise lookup_inode2() for dirs - pseudo: fix handling of modify pseudo files - pseudo: fix handling of directory pseudo files - xattr: Fix ERROR() so that it is synchronised with the progress bar - mksquashfs/sort: Fix INFO() so that it is synced with the progress bar - mksquashfs: Add -progress to force progress bar when using -info - error.h: consolidate the various error macros into one header file - mksquashfs: fix stack overflow in write_fragment_table() - mksquashfs: move list allocation from off the stack - unsquashfs: fix oversight in directory permission setting - mksquashfs: dynamically allocate recovery_file - mksquashfs: dynamically allocate buffer in subpathname() - mksquashfs: dynamically allocate buffer in pathname() - unsquashfs: fix CVE-2012-4024 - unsquashfs: fix CVE-2012-4025 - mksquashfs: fix potential stack overflow in get_component() - mksquashfs: add parse_number() helper for numeric command line options - mksquasfs: check return value of fstat() in reader_read_file() - mksquashfs: dynamically allocate filename in old_add_exclude() - unsquashfs: dynamically allocate pathname in dir_scan() - unsquashfs: dynamically allocate pathname in pre_scan() - sort: dynamically allocate filename in add_sort_list() - mksquashfs: fix dir_scan() exit if lstat of source directory fails - pseudo: fix memory leak in read_pseudo_def() if exec_file() fails - pseudo: dynamically allocate path in dump_pseudo() - mksquashfs: dynamically allocate path in display_path2() - mksquashfs: dynamically allocate b_buffer in getbase() - pseudo: fix potential stack overflow in get_component() - pseudo: avoid buffer overflow in read_pseudo_def() using sscanf() - pseudo: dynamically allocate filename in exec_file() - pseudo: avoid buffer overflow in read_sort_file() using fscanf() - sort: tighten up sort file parsing - unsquashfs: fix name under-allocation in process_extract_files() - unsquashfs: avoid buffer overflow in print_filename() using sprintf() - Fix some limits in the file parsing routines - pseudo: Rewrite pseudo file processing - read_fs: fix small memory leaks in read_filesystem() - mksquashfs: fix fclose leak in reader_read_file() on I/O error - mksquashfs: fix frag struct leak in write_file_{process|blocks|frag} - unsquashfs_xattr: fix memory leak in write_xattr() - read_xattrs: fix xattr free in get_xattr() in error path - unsquashfs: add -user-xattrs option to only extract user.xxx xattrs - unsquashfs: add code to only print "not superuser" error message once - unsquashfs: check for integer overflow in user input - mksquashfs: check for integer overflow in user input - mksquashfs: fix "new" variable leak in dir_scan1() - read_fs: prevent buffer {over|under}flow in read_block() with corrupted filesystems - read_fs: check metadata blocks are expected size in scan_inode_table() - read_fs: check the root inode block is found in scan_inode_table() - read_fs: Further harden scan_inode_table() against corrupted filesystems - unsquashfs: prevent buffer {over|under}flow in read_block() with corrupted filesystems - read_xattrs: harden xattr data reading against corrupted filesystems - unsquash-[23]: harden frag table reading against corrupted filesystems - unsquash-4.c: harden uid/gid & frag table reading against corruption - unsquashfs: harden inode/directory table reading against corruption - mksquashfs: improve out of space in output filesystem handling - mksquashfs: flag lseek error in writer as probable out of space - mksquashfs: flag lseek error in write_destination as probable out of space - mksquashfs: print file being squashed when ^\ (SIGQUIT) typed - mksquashfs: make EXIT_MKSQUASHFS() etc restore via new restore thread - mksquashfs: fix recursive restore failure check - info: dump queue and cache status if ^\ hit twice within one second - mksquashfs: fix rare race condition in "locked fragment" queueing - lz4: add experimental support for lz4 compression - lz4: add support for lz4 "high compression" - lzo_wrapper: new implementation with compression options - gzip_wrapper: add compression options - mksquashfs: redo -comp parsing - mksquashfs: display compressor options when -X option isn't recognised - mksquashfs: add -Xhelp option - mksquashfs/unsquashfs: fix mtime signedness - Mksquashfs: optimise duplicate checking when appending - Mksquashfs: introduce additional per CPU fragment process threads - Mksquashfs: significantly optimise fragment duplicate checking - read_fs: scan_inode_table(), fix memory leak on filesystem corruption - pseudo: add_pseudo(), fix use of freed variable - mksquashfs/unsquashfs: exclude/extract/pseudo files, fix handling of leaf name - mksquashfs: rewrite default queue size so it's based on physical mem - mksquashfs: add a new -mem option - mksquashfs: fix limit on the number of dynamic pseudo files - mksquashfs: make -mem take a normal byte value, optionally with a K, M or G 4.2 28 FEB 2011 XZ compression, and compression options support 1. Filesystem improvements: 1.1 Added XZ compression 1.2 Added compression options support 2. Miscellaneous improvements/bug fixes 1.1 Add missing NO_XATTR filesystem flag to indicate no-xattrs option was specified and no xattrs should be stored when appending. 1.2 Add suppport in Unquashfs -stat option for displaying NO_XATTR flag. 1.3 Remove checkdata entry from Unsquashfs -stat option if a 4.0 filesystem - checkdata is no longer supported. 1.4 Fix appending bug when appending to an empty filesystem - this would be incorrectly treated as an error. 1.5 Use glibc sys/xattr.h include rather than using attr/xattr.h which isn't present by default on some distributions. 1.6 Unsquashfs, fix block calculation error with regular files when file size is between 2^32-block_size+1 and 2^32-1. 1.7 Unsquashfs, fix sparse file writing when holes are larger than 2^31-1. 1.8 Add external CFLAGS and LDFLAGS support to Makefile, and allow build options to be specified on command line. Also don't over-write passed in CFLAGS definition. 4.1 19 SEPT 2010 Major filesystem and tools improvements 1. Filesystem improvements: 1.1 Extended attribute support 1.2 New compression framework 1.3 Support for LZO compression 1.4 Support for LZMA compression (not yet in mainline) 2. Mksquashfs improvements: 1.1 Enhanced pseudo file support 1.2 New options for choosing compression algorithm used 1.3 New options for controlling extended attributes 1.4 Fix misalignment issues with memcpy etc. seen on ARM 1.5 Fix floating point error in progress_bar when max == 0 1.6 Removed use of get_nproc() call unavailable in ulibc 1.7 Reorganised help text 3. Unsquashfs improvements: 1.1 New options for controlling extended attributes 1.2 Fix misalignment issues with memcpy etc. seen on ARM 1.3 Fix floating point error in progress_bar when max == 0 1.4 Removed use of get_nproc() call unavailable in ulibc 4.0 5 APR 2009 Major filesystems improvements 1. Kernel code improvements: 1.1 Fixed little endian layout adopted. All swapping macros removed, and in-line swapping added for big-endian architectures. 1.2 Kernel code substantially improved and restructured. 1.3 Kernel code split into separate files along functional lines. 1.4 Vmalloc usage removed, and code changed to use separately allocated 4K buffers 2. Unsquashfs improvements: 2.1 Support for 4.0 filesystems added. 2.2 Swapping macros rewritten. 2.3 Unsquashfs code restructured and split into separate files. 3. Mksquashfs improvements: 3.1 Swapping macros rewritten. Fixed little-endian layout allows code to be optimised and only added at compile time for big endian systems. 3.2 Support for pseudo files added. 3.4 26 AUG 2008 Performance improvements to Unsquashfs, Mksquashfs and the kernel code. Plus many small bug fixes. 1. Kernel code improvements: 1.1 Internal Squashfs kernel metadata and fragment cache implementations have been merged and optimised. Spinlocks are now used, locks are held for smaller periods and wakeups have been minimised. Small race condition fixed where if two or more processes tried to read the same cache block simultaneously they would both read and decompress it. 10-20%+ speed improvement has been seen on tests. 1.2 NFS export code rewritten following VFS changes in linux-2.6.24. 1.3 New patches for linux-2.6.25, linux-2.6.26, and linux-2.6.27. Fixed patch for linux-2.6.24. 1.4 Fixed small buffer_head leak in squashfs_read_data when handling badly corrupted filesystems. 1.5 Fixed bug in get_dir_index_using_offset. 2. Unsquashfs improvements: 2.1 Unsquashfs has been parallelised. Filesystem reading, writing and decompression is now multi-threaded. Up to 40% speed improvement seen on tests. 2.2 Unsquashfs now has a progress bar. Use -no-progress to disable it. 2.3 Fixed small bug where unistd.h wasn't being included on some distributions, leading to lseek being used rather than lseek64 - which meant on these distributions Unsquashfs couldn't unsquash filesystems larger than 4GB. 3. Mksquashfs improvements: 3.1 Removed some small remaining parallelisation bottlenecks. Depending on source filesystem, up to 10%+ speed improvement. 3.2 Progress bar improved, and moved to separate thread. 3.3 Sparse file handling bug in Mksquashfs 3.3 fixed. 3.4 Two rare appending restore bugs fixed (when ^C hit twice). 3.3 1 NOV 2007 Increase in block size, sparse file support, Mksquashfs and Unsquashfs extended to use pattern matching in exclude/extract files, plus many more improvements and bug fixes. 1. Filesystem improvements: 1.1. Maximum block size has been increased to 1Mbyte, and the default block size has been increased to 128 Kbytes. This improves compression. 1.2. Sparse files are now supported. Sparse files are files which have large areas of unallocated data commonly called holes. These files are now detected by Squashfs and stored more efficiently. This improves compression and read performance for sparse files. 2. Mksquashfs improvements: 2.1. Exclude files have been extended to use wildcard pattern matching and regular expressions. Support has also been added for non-anchored excludes, which means it is now possible to specify excludes which match anywhere in the filesystem (i.e. leaf files), rather than always having to specify exclude files starting from the root directory (anchored excludes). 2.2. Recovery files are now created when appending to existing Squashfs filesystems. This allows the original filesystem to be recovered if Mksquashfs aborts unexpectedly (i.e. power failure). 3. Unsquashfs improvements: 3.1. Multiple extract files can now be specified on the command line, and the files/directories to be extracted can now also be given in a file. 3.2. Extract files have been extended to use wildcard pattern matching and regular expressions. 3.3. Filename printing has been enhanced and Unquashfs can now display filenames with file attributes ('ls -l' style output). 3.4. A -stat option has been added which displays the filesystem superblock information. 3.5. Unsquashfs now supports 1.x filesystems. 4. Miscellaneous improvements/bug fixes: 4.1. Squashfs kernel code improved to use SetPageError in squashfs_readpage() if I/O error occurs. 4.2. Fixed Squashfs kernel code bug preventing file seeking beyond 2GB. 4.3. Mksquashfs now detects file size changes between first phase directory scan and second phase filesystem create. It also deals better with file I/O errors. 3.2-r2 15 JAN 2007 Kernel patch update and progress bar bug fix 1. Kernel patches 2.6.19/2.6.20 have been updated to use const structures and mutexes rather than older semaphores. 2. Minor SMP bug fixes. 3. Progress bar broken on x86-64. Fixed. 3.2 2 JAN 2007 NFS support, improvements to the Squashfs-tools, major bug fixes, lots of small improvements/bug fixes, and new kernel patches. Improvements: 1. Squashfs filesystems can now be exported via NFS. 2. Unsquashfs now supports 2.x filesystems. 3. Mksquashfs now displays a progress bar. 4. Squashfs kernel code has been hardened against accidently or maliciously corrupted Squashfs filesystems. Bug fixes: 5. Race condition occurring on S390 in readpage() fixed. 6. Odd behaviour of MIPS memcpy in read_data() routine worked-around. 7. Missing cache_flush in Squashfs symlink_readpage() added. 3.1-r2 30 AUG 2006 Mksquashfs -sort bug fix A code optimisation after testing unfortunately broke sorting in Mksquashfs. This has been fixed. 3.1 19 AUG 2006 This release has some major improvements to the squashfs-tools, a couple of major bug fixes, lots of small improvements/bug fixes, and new kernel patches. 1. Mksquashfs has been rewritten to be multi-threaded. It has the following improvements 1.1. Parallel compression. By default as many compression and fragment compression threads are created as there are available processors. This significantly speeds up performance on SMP systems. 1.2. File input and filesystem output is peformed in parallel on separate threads to maximise I/O performance. Even on single processor systems this speeds up performance by at least 10%. 1.3. Appending has been significantly improved, and files within the filesystem being appended to are no longer scanned and checksummed. This significantly improves append time for large filesystems. 1.4. File duplicate checking has been optimised, and split into two separate phases. Only files which are considered possible duplicates after the first phase are checksummed and cached in memory. 1.5 The use of swap memory was found to significantly impact performance. The amount of memory used to cache files is now a command line option, by default this is 512 Mbytes. 2. Unsquashfs has the following improvements 2.1 Unsquashfs now allows you to specify the filename or the directory within the Squashfs filesystem that is to be extracted, rather than always extracting the entire filesystem. 2.2 A new -force option has been added which forces Unsquashfs to output to the destination directory even if files and directories already exist in the destination directory. This allows you to update an already existing directory tree, or to Unsquashfs to a partially filled directory tree. Without the -force option Unsquashfs will refuse to output. 3. The following major bug fixes have been made 3.1 A fragment table rounding bug has been fixed in Mksquashfs. Previously if the number of fragments in the filesystem were a multiple of 512, Mksquashfs would generate an incorrect filesystem. 3.2 A rare SMP bug which occurred when simultaneously acccessing multiply mounted Squashfs filesystems has been fixed. 4. Miscellaneous improvements/bug fixes 4.1 Kernel code stack usage has been reduced. This is to ensure Squashfs works with 4K stacks. 4.2 Readdir (Squashfs kernel code) has been fixed to always return 0, rather than the number of directories read. Squashfs should now interact better with NFS. 4.3 Lseek bug in Mksquashfs when appending to larger than 4GB filesystems fixed. 4.4 Squashfs 2.x initrds can now been mounted. 4.5 Unsquashfs exit status fixed. 4.6 New patches for linux-2.6.18 and linux-2.4.33. 3.0 15 MAR 2006 Major filesystem improvements 1. Filesystems are no longer limited to 4 GB. In theory 2^64 or 4 exabytes is now supported. 2. Files are no longer limited to 4 GB. In theory the maximum file size is 4 exabytes. 3. Metadata (inode table and directory tables) are no longer restricted to 16 Mbytes. 4. Hardlinks are now suppported. 5. Nlink counts are now supported. 6. Readdir now returns '.' and '..' entries. 7. Special support for files larger than 256 MB has been added to the Squashfs kernel code for faster read access. 8. Inode numbers are now stored within the inode rather than being computed from inode location on disk (this is not so much an improvement, but a change forced by the previously listed improvements). 2.2-r2 8 SEPT 2005 Second release of 2.2, this release fixes a couple of small bugs, a couple of small documentation mistakes, and adds a patch for kernel 2.6.13. 1. Mksquashfs now deletes the output filesystem image file if an error occurs whilst generating the filesystem. Previously on error the image file was left empty or partially written. 2. Updated mksquashfs so that it doesn't allow you to generate filesystems with block sizes smaller than 4K. Squashfs hasn't supported block sizes less than 4K since 2.0-alpha. 3. Mksquashfs now ignores missing files/directories in sort files. This was the original behaviour before 2.2. 4. Fixed small mistake in fs/Kconfig where the version was still listed as 2.0. 5. Updated ACKNOWLEDGEMENTS file. 2.2 3 JUL 2005 This release has some small improvements, bug fixes and patches for new kernels. 1. Sort routine re-worked and debugged from release 2.1. It now allows you to give Mkisofs style sort files and checks for filenames that don't match anything. Sort priority has also been changed to conform to Mkisofs usage, highest priority files are now placed at the start of the filesystem (this means they will be on the inside of a CD or DVD). 2. New Configure options for embedded systems (memory constrained systems). See INSTALL file for further details. 3. Directory index bug fixed where chars were treated as signed on some architectures. A file would not be found in the rare case that the filename started with a chracter greater than 127. 4. Bug introduced into the read_data() routine when sped up to use data block queueing fixed. If the second or later block resulted in an I/O error this was not checked. 5. Append bug introduced in 2.1 fixed. The code to compute the new compressed and uncompressed directory parts after appending was wrong. 6. Metadata block length read routine altered to not perform a misaligned short read. This was to fix reading on an ARM7 running uCLinux without a misaligned read interrupt handler. 7. Checkdata bug introduced in 2.1 fixed. 2.1-r2 15 DEC 2004 Code changed so it can be compiled with gcc 2.x 1. In some of the code added for release 2.1 I unknowingly used some gcc extensions only supported by 3.x compilers. I have received a couple of reports that the 2.1 release doesn't build on 2.x and so people are clearly still using gcc 2.x. The code has been rewritten to remove these extensions. 2.1 10 DEC 2004 Significantly improved directory handling plus numerous other smaller improvements 1. Fast indexed directories implemented. These speed up directory operations (ls, file lookup etc.) significantly for directories larger than 8 KB. 2. All directories are now sorted in alphabetical order. This again speeds up directory operations, and in some cases it also results in a small compression improvement (greater data similarity between files with alphabetically similar names). 3. Maximum directory size increased from 512 KB to 128 MB. 4. Duplicate fragment checking and appending optimised in mksquashfs, depending on filesystem, this is now up to 25% faster. 5. Mksquashfs help information reformatted and reorganised. 6. The Squashfs version and release date is now printed at kernel boot-time or module insertion. This addition will hopefully help to reduce the growing problem where the Squashfs version supported by a kernel is unknown and the kernel source is unavailable. 7. New PERFORMANCE.README file. 8. New -2.0 mksquashfs option. 9. CHANGES file reorganised. 10. README file reorganised, clarified and updated to include the 2.0 mksquashfs options. 11. New patch for Linux 2.6.9. 12. New patch for Linux 2.4.28. 2.0r2 29 AUG 2004 Workaround for kernel bug in kernels 2.6.8 and newer added 1. New patch for kernel 2.6.8.1. This includes a workaround for a kernel bug introduced in 2.6.7bk14, which is present in all later versions of the kernel. If you're using a 2.6.8 kernel or later then you must use this 2.6.8.1 patch. If you've experienced hangs or oopses using Squashfs with a 2.6.8 or later kernel then you've hit this bug, and this patch will fix it. It is worth mentioning that this kernel bug potentially affects other filesystems. If you receive odd results with other filesystems you may be experiencing this bug with that filesystem. I submitted a patch but this has not yet gone into the kernel, hopefully the bug will be fixed in later kernels. 2.0 13 JULY 2004 A couple of new options, and some bug fixes 1. New mksquashfs -all-root, -root-owned, -force-uid, and -force-gid options. These allow the uids/gids of files in the generated filesystem to be specified, overriding the uids/gids in the source filesystem. 2. Initrds are now supported for kernels 2.6.x. 3. amd64 bug fixes. If you use an amd64, please read the README-AMD64 file. 4. Check-data and gid bug fixes. With 2.0-alpha when mounting 1.x filesystems in certain cases file gids were corrupted. 5. New patch for Linux 2.6.7. 2.0-ALPHA 21 MAY 2004 Filesystem changes and compression improvements 1. Squashfs 2.0 has added the concept of fragment blocks. Files smaller than the file block size and optionally the remainder of files that do not fit fully into a block (i.e. the last 32K in a 96K file) are packed into shared fragments and compressed together. This achieves on average 5 - 20% better compression than Squashfs 1.x. 2. The maximum block size has been increased to 64K (in the ALPHA version of Squashfs 2.0). 3. The maximum number of UIDs has been increased to 256 (from 48 in 1.x). 4. The maximum number of GIDs has been increased to 256 (from 15 in 1.x). 5. Removal of sleep_on() function call in 2.6.x patch, to allow Squashfs to work on the Fedora rc2 kernel. 6. Numerous small bug fixes have been made. 1.3r3 18 JAN 2004 Third release of 1.3, this adds a new mksquashfs option, some bug fixes, and extra patches for new kernels 1. New mksquashfs -ef exclude option. This option reads the exclude dirs/files from an exclude file, one exclude dir/file per line. This avoids the command line size limit when using the -e exclude option, 2. When appending to existing filesystems, if mksquashfs experiences a fatal error (e.g. out of space when adding to the destination), the original filesystem is restored, 3. Mksquashfs now builds standalone, without the kernel needing to be patched. 4. Bug fix in the kernel squashfs filesystem, where the pages being filled were not kmapped. This seems to only have caused problems on an Apple G5, 5. New patch for Linux 2.4.24, 6. New patch for Linux 2.6.1, this replaces the patch for 2.6.0-test7. 1.3r2 14 OCT 2003 Second release of 1.3, bug fixes and extra patches for new kernels 1. Bug fix in routine that adds files to the filesystem being generated in mksquashfs. This bug was introduced in 1.3 (not enough testing...) when I rewrote it to handle files larger than available memory. This bug caused a SEGV, so if you've ever got that, it is now fixed, 2. Long running bug where ls -s and du reported wrong block size fixed. I'm pretty sure this used to work many kernel versions ago (2.4.7) but it broke somewhere along the line since then, 3. New patch for Linux 2.4.22, 4. New patch for 2.6.0-test7, this replaces the patch for 2.6.0-test1. 1.3 29 JUL 2003 FIFO/Socket support added plus optimisations and improvements 1. FIFOs and Socket inodes are now supported, 2. Mksquashfs can now compress files larger than available memory, 3. File duplicate check routine optimised, 4. Exit codes fixed in Mksquashfs, 5. Patch for Linux 2.4.21, 6. Patch for Linux 2.6.0-test1. Hopefully, this will work for the next few releases of 2.6.0-testx, otherwise, I'll be releasing a lot of updates to the 2.6.0 patch... 1.2 13 MAR 2003 Append feature and new mksquashfs options added Mksquashfs can now add to existing squashfs filesystems. Three extra options "-noappend", "-keep-as-directory", and "root-becomes" have been added. The append option with file duplicate detection, means squashfs can be used as a simple versioning archiving filesystem. A squashfs filesystem can be created with for example the linux-2.4.19 source. Appending the linux-2.4.20 source will create a filesystem with the two source trees, but only the changed files will take extra room, the unchanged files will be detected as duplicates. See the README file for usage changes. 1.1b 16 JAN 2003 Bug fix release Fixed readpage deadlock bug. This was a rare deadlock bug that happened when pushing pages into the page cache when using greater than 4K blocks. I never got this bug when I tested the filesystem, but two people emailed me on the same day about the problem! I fixed it by using a page cache function that wasn't there when I originally did the work, which was nice :-) 1.1 8 JAN 2003 Added features 1. Kernel squashfs can now mount different byte order filesystems. 2. Additional features added to mksquashfs. Mksquashfs now supports exclude files and multiple source files/directories can be specified. A nopad option has also been added, which informs mksquashfs not to pad filesystems to a multiple of 4K. See README for mksquashfs usage changes. 3. Greater than 2GB filesystems bug fix. Filesystems greater than 2GB can now be created. 1.0c 14 NOV 2002 Bug fix release Fixed bugs with initrds and device nodes 1.0 23 OCT 2002 Initial release