littlefs

4d068a1 Update README example code in accordance to the block_cycles change
ba088aa lfs_dir_*: Cast error return codes to int.
955b296 lfs_file_rewind: Cast error return codes to int.
241dbc6 lfs_stat: Cast error return codes to int.
8cca58f lfs_file_truncate: ensure lfs_file_seek return code is lsf_soff_t and cast error returns
97f86af lfs_remove: Cast tag/error return codes to int.
d40302c lfs_rename: Cast error return codes to int.
0b5a78e Adjust lfs_dir_find return code to ensure 32 bit value.
27b6cc8 Fixed off-by-one null terminator in tests

965d29b build: Fix warnings about shift count width difference for 16 bit compiler
f7fd7d9 Corrections for typos and grammar
d5aba27 Fix for issue #260
0c77123 lfs: Validate lfs-cfg sizes before performing arithmetic logics with them

151104c Changed CI to create release note for patches
303ffb2 Fix build with -Wundef
5bf71fa lfs: do not reposition seek pointer on truncate
55fb141 lfs: initialize file offs field
dc031ce lfs: use meaningful names for magic block identifiers
f85ff1d lfs: correct alignment restriction on lookahead buffer

Note!

The behavior of the config option block_cycles has changed in this release. Before, block_cycles = 0 would disable wear-leveling. This was a mistake as it made it easy to accidentally disable wear-leveling.

This has been changed so that block_cycles = -1 disables wear-leveling. block_cycles = 0 is now illegal and will trigger an assert.

lfs.c:3381: lfs_init: Assertion `lfs->cfg->block_cycles != 0' failed.

If you get this assert, you will need to change your block_cycles configuration. Suggested values are in the range 100-1000, with large values having better performance at the cost of less consistent wear distribution. -1 will keep the previous behavior by disabling wear-leveling.

// Number of erase cycles before littlefs evicts metadata logs and moves 
// the metadata to another block. Suggested values are in the
// range 100-1000, with large values having better performance at the cost
// of less consistent wear distribution.
//
// Set to -1 to disable block-level wear-leveling.
int32_t block_cycles;

More info here: https://github.com/ARMmbed/littlefs/pull/233

What's new?

Debugging improvements!

• Thanks to @Ar2rL, we now have asserts against operations on closed files

• RDONLY/WRONLY mistakes are now asserts instead of errors

• Added LFS_TRACE statements that can be turned on by defining LFS_YES_TRACE at build time:

   $ make CFLAGS+=-DLFS_YES_TRACE=1 test_attrs
   ./tests/test_attrs.sh
   === Attr tests ===
   lfs_trace:66: lfs_emubd_create(0x40cee0 {.context=0x610ae0, .read=0x40c0a7, .prog=0x40c34c, .erase=0x40c6e7, .sync=0x40c9b7, .read_size=16, .prog_size=16, .block_size=512, .block_count=1024
   }, "blocks")
   lfs_trace:139: lfs_emubd_create -> 0
   lfs_trace:3461: lfs_format(0x610140, 0x40cee0 {.context=0x610ae0, .read=0x40c0a7, .prog=0x40c34c, .erase=0x40c6e7, .sync=0x40c9b7, .read_size=16, .prog_size=16, .block_size=512, .block_count=1024, .block_cycles=1024, .cache_size=64, .lookahead_size=16, .read_buffer=(nil), .prog_buffer=(nil), .lookahead_buffer=(nil), .name_max=0, .file_max=0, .attr_max=0})
   lfs_trace:155: lfs_emubd_read(0x40cee0, 0x1, 0, 0x1553130, 16)
   lfs_trace:201: lfs_emubd_read -> 0
   lfs_trace:279: lfs_emubd_erase(0x40cee0, 0x0)
   lfs_trace:321: lfs_emubd_erase -> 0
   lfs_trace:155: lfs_emubd_read(0x40cee0, 0x0, 64, 0x1553130, 16)
   lfs_trace:201: lfs_emubd_read -> 0
   lfs_trace:208: lfs_emubd_prog(0x40cee0, 0x0, 0, 0x1553180, 64)
   lfs_trace:274: lfs_emubd_prog -> 0
   lfs_trace:326: lfs_emubd_sync(0x40cee0)
   lfs_trace:402: lfs_emubd_sync -> 0
   lfs_trace:155: lfs_emubd_read(0x40cee0, 0x0, 0, 0x1553130, 48)
   lfs_trace:201: lfs_emubd_read -> 0
   ~ snip ~
  

• Added asserts help stop positive error codes from block devices from causing the filesystem to break, this has been a reoccurring issue https://github.com/ARMmbed/littlefs/issues/215, https://github.com/ARMmbed/littlefs/issues/169

• Several tweaks to debug output and test framework

• Significantly improved handling of devices with large prog sizes (prog_size >1KiB)

• Changed block_cycles = 0 to assert, replaced with block_cycles = -1 (see above)

Changes

db05468 Bump version to v2.1
7872918 Fixed issue where lfs_migrate would relocate root and corrupt superblock
e249854 Removed dependency on uninitialized value in lfs_file_t struct
0d4c0b1 Fixed issue where inline files were not cleaned up
4850e01 Changed rdonly/wronly mistakes to assert
4ec4425 Fixed overlapping memcpy in emubd
3806d88 Fixed seek-related typos in lfs.h
de59726 Fixed license header in lfs.c
e8c023a Changed FUSE branch to v2 (previously v2-alpha)
38a2a8d Minor improvement to documentation over block_cycles
51fabc6 Switched to using hex for blocks and ids in debug output
1983837 Fixed issue where sed buffering (QUIET=1) caused Travis timeout
312326c Added a better solution for large prog sizes
ef1c926 Increased testing to include geometries that can't be fully tested
72e3bb4 Refactored a handful of things in tests
649640c Fixed workaround for erase sizes >1024 B
eb013e6 lfs: correct documentation on lookahead-related values
7e1bad3 Set LFS_F_OPENED flag at places required by lfs internal logic.
72a3758 Use LFS_F_OPENED flag to protect against use of not opened or closed file.
df2e676 Add necessary flag to mark file as being opened.
53a6e04 Changed block_cycles disable from 0 to -1
1aaf1cb Minor improvements to testing framework
52a90b8 Added asserts on positive return values from block device functions
e279c8f Tweaked debug output
6a1ee91 Added trace statements through LFS_YES_TRACE
2e92f7a actually removed <dirent.h>
2588948 removed <dirent.h> preventing compile on some archs
abd90cb Fixed 32-bit/64-bit Ubuntu multilib issue in Travis
b73ac59 Fixed issues with reading and caching inline files
614f7b1 Fixed accidental truncate after seek on inline files
a9a61a3 Added redundant compaction to lfs_format/lfs_migrate
36973d8 Fixed missing cache flush in lfs_migrate
3fb242f Mark all Python 2 scripts as Python 2
ef77195 Fixed limit of inline files based on LFS_ATTR_MAX
12e464e Fixed issue with writes following a truncate
9899c7f Fixed read cache amount based on hint and offset
2533a0f Make lfs1_crc static so it doesn't conflict with prefixed LFS1 code
2a7f0ed Fix compilation with -Wundef
f35fb8c Fixed migration test condition for prefix branches
fdd239f Don't bypass cache in lfs_cache_prog() and lfs_cache_read()
780ef2f Fixed buffer overflow due to mistaking prog_size for cache_size
25a843a Fixed .travis.yml to use explicit branch names for migration testing
905727b Fix: length more than LFS_FILE_MAX should return error
26d2560 Fixed documentation for return lfs_dir_read return value.

What's new?

Big things

1. Metadata logging

   Metadata pairs now act as small two block logs. This allows incremental updates without erasing metadata blocks, which is a significant performance improvement. Additionally, this enables dynamically sized entries, which is necessary for the two following features.

2. Custom attributes

   It's now possible to get and set custom attributes on files, directories, and the even the filesystem itself. Custom attributes use a byte as an identifier and can be up to 1022 bytes large. This is useful for things such as timestamps, permissions, hashes, or other attributes.

3. Inline files

   Files can now be inlined directly in the directory block. This allows multiple small files to be coalesced into a single block instead of wasting an entire block for each file. This opens up the possibility for littlefs to be used on NAND flash and MCU internal flash, both which often have block sizes >16KiB. Note that inline files are limited by the cache size.

4. Independent cache sizes

   littlefs now has a separate cache_size config option. This config option determines the size of the read cache, prog cache, and file caches independently of the storage's physical read/prog size. This allows more granular reads and writes without artificially limiting the cache, improving performance and log utilization.

5. XOR global state, no more move problem

   XOR global state is a mechanism that allows littlefs to store a small amount of global state distributed across the filesystem. This global state is now used internal to fix moves in O(1), a significant improvement over the previous O(n) search. Additionally XOR global state may be useful for future improvements.

6. True dynamic wear-leveling

   littlefs now proactively levels wear over dynamic blocks instead of reacting only to bad blocks. This offers better protection against wear that causes stored data to deteriorate.

   Dynamic wear-leveling is provided by evicting blocks after a bounded number of writes determined by the block_cycles config option. A combination of random offset and incremental allocation of blocks provide a uniform distribution of wear over the storage.

7. Expanding superblocks

   The superblock no longer has a flat cost of 2 blocks. Instead, the superblock doubles as the root directory until block_cycles are passed. After every block_cycles number of erases, the superblock is expanded to include another metadata pair if space is available. This allows littlefs to use as few superblocks as is needed to avoid an early death.

   At the smallest, littlefs can now fit in only 2 blocks.

Small things

1. Configurable name max, file max, and attr max

   These limits can now be set during format and should be respected on other devices. This means that the RAM cost of stat can be reduced without losing portability.

2. Added lfs_fs_size

   A function to get a count of the used blocks on the filesystem. Likely easier to use than lfs_fs_traverse.

3. Better update tracking

   Metadata logging, with resizable entries, means that there's a lot of state changes flying around. This required a review and ultimately a rewrite of the way littlefs manages open files and dirs. This should be much more robust moving forward and also has a small performance improvement.

Important changes

1. Must set cache_size. If read_buffer or prog_buffer are provided manually, these must be cache_size.
2. Highly suggested to set block_cycles in the range of 100-1000. This enables wear leveling.
3. Change lookahead -> lookahead_size. The lookahead buffer must now be aligned to a 64-bit boundary.
4. Drop file_buffer in favor of per-file buffers.
5. Renamed lfs_traverse -> lfs_fs_traverse. Note the addition of lfs_fs_size can replace the most common use of lfs_traverse.
6. Change lfs_crc to not use a pointer if it was overridden.

Migration

Following the discussion on #127, releases now have three special branch/tags that are generated as a part of CI:

• A patch-specific tag (v1.7.2), immutable
• A major version branch (v1), automatically updated every patch release
• A major version prefix branch (v1-prefix) aromatically generated every patch release. Note that this is generated as a merge commit, making it easy to update code that needs to avoid name conflicts between multiple versions of littlefs.

To help adoption with the breaking disk changes, this version includes a migration function designed to help migrate v1 filesystems to v2.

Note that this migration is best-effort, if it fails, an error is returned and the v1 filesystem is left unmodified.

Enabling the migration requires two steps:

1. Defining LFS_MIGRATE
2. Call lfs_migrate (only available with the above macro)

More info about this here.

Documentation

Documentation has been brought up to date with the changes. More info on how littlefs v2 works can be found in the updated DESIGN.md and SPEC.md.

Performance comparison of v1 and v2

Calculated by simulating v1 and v2 with various file counts and file sizes. Results are divided by the count*size of the files to get an easier-to-compare multiplicative cost. Note that these results are still on the smaller scale for storage. Smaller numbers are better.

As expected, the performance does not change much for reading (and even gets worse on NAND). However, the prog/erase performance is much better. This is desired as erasing has a much higher runtime penalty than reading,

There's also a moderate improvement to storage consumption thanks to inline files, though note that the storage consumption converges as the file size increases.

Note!

This is the alpha release of littlefs v2. There are still a handful of tasks that need to be completed before completing the work around v2, most notably 1. implement a migration function from v1 and 2. update documentation. But at the moment littlefs v2 is code complete and passing all the tests I've been able to throw at it.

If you any issues with v2-alpha, please raise them on GitHub as that will help make the v2 release that much more stable.

More info about v2.0 progress can be found here: https://github.com/ARMmbed/littlefs/pull/85

What's new?

Big things

• Metadata logging - this is a constant-time improvement to write efficiency and wear distribution
• Custom attributes - a much requested feature for attaching small attributes to files (such as timestamps)
• Inline files - significantly improved handling of small files which can now have multiple in a single block
• XOR global state - moves and deletes should now be much faster
• Independent cache sizes - better handling of the RAM you give littlefs
• Expanding superblocks - no longer a flat cost of 2 blocks for the superblock
• True dynamic wear-leveling - no longer relies on reporting of block errors

Small things

• Configurable name/file/attr maximums - reduces RAM consumption and is still portable
• Added lfs_fs_size - no need for the extra code with lfs_fs_traverse
• Dropped global file buffer - now buffers are provided on a per-file basis
• Better update tracking - minor miscellaneous improvements

Detailed info in the PR: https://github.com/ARMmbed/littlefs/pull/85

Changes

Right now the best source of info regarding the low-level changes can be found in the commit history, which contains a detailed log of all of the changes and reasoning behind different decisions:

https://github.com/ARMmbed/littlefs/commits/v2-alpha

Note!

littlefs v2.0 is almost here! This will be the last minor release for littlefs v1. I'll still be making patch releases as necessary, but any new features will be moved to littlefs v2.

More info about v2.0 progress can be found here: https://github.com/ARMmbed/littlefs/pull/85

What's new?

• Added 2GiB file size limit and proper reporting of EFBIG errors.

Changes

ec4d8b6 Changed release script to generate drafts
c7894a6 Added a handful of links to related projects
1950758 Added 2GiB file size limit and EFBIG reporting
97d8d5e Fixed issue where a rename causes a split and pushes dir out of sync
0bb1f7a Modified release script to create notes only on minor releases
28d2d96 Fix -Wsign-compare error
cb62bf2 Fixed release script issue with fetching recent tags
646b1b5 Added -Wjump-misses-init and fixed uninitialized warnings
e5a6938 Fixed possible infinite loop in deorphan step
6ad544f If stats file doesn't exist lfs_emubd_create will fail.
3419284 Fixed issue with corruption due to different cache sizes

What's new?

• Improved release process. No more mutable versions. See https://github.com/ARMmbed/littlefs/pull/81 for more info.

Changes

510cd13 Bumped minor version to v1.6
f5e0539 Fixed issue with release script non-standard version tags
0664480 Moved SPDX and license info into README
0234c77 Simplified release process based on feedback

What's new?

• Added possibility to open multiple files with LFS_NO_MALLOC enabled
• Added file config structure and lfs_file_opencfg
• Added support for corrupt as initial state of blocks

Changes

0422c55 Fix memory leaks in lfs_mount and lfs_format
961fab7 Added file config structure and lfs_file_opencfg
041e90a Added handling for corrupt as initial state of blocks

What's new?

• Relicensed to BSD-3-Clause

Changes

577d777 Add C++ guards to public headers
7e67f93 Use PRIu32 and PRIx32 format specifiers to fix warnings
5a17fa4 Fixed script issue with bash expansion inside makefile parameter
eed1eec Fixed information leaks through reused caches
4a86370 Added quality of life improvements for main.c/test.c issues
51346b8 Fixed shadowed variable warnings
6beff50 Changed license to BSD-3-Clause

What's new?

• Support for big-endian machines
• Software-only implementations of intrinsics (LFS_NO_INTRINSICS)
• Wrappers for malloc/free/assert
• Conditional debug and assert statements

Changes

c5e2b33 Added error when opening multiple files with a statically allocated buffer
015b86b Fixed issue with trailing dots in file paths
9637b96 Fixed lookahead overflow and removed unbounded lookahead pointers
89a7630 Fixed issue with lookahead trusting old lookahead blocks
43eac30 Renamed test_parallel tests to test_interespersed
dbc3cb1 Fixed Travis rate-limit issue with Github requests
93ece2e Removed outdated note about moves and powerloss
d9c076d Removed the uninitialized read for invalid superblocks
f72f6d6 Removed out of date note about endianness
5c4ee21 Added a note about the callback functions
1552246 Fixed Travis issue with deploy stage in PRs
9ee112a Fixed issue updating dir struct when extended dir chain
d9c3637 Fixed handling of root as target for create operations
1476181 Added LFS_CONFIG for user provided configuration of the utils
b2124a5 Fixed multiple deploy steps in Travis
67daf9e Added cross-compile targets for testing
a3fd2d4 Added more configurable utils
a0a55fb Added conversion to/from little-endian on disk
4f08424 Added software implementations of bitwise instructions

What's new?

• Added lfs_file_truncate
• Versioning!

Changes

2f8ae34 Added a git ignore file with .o .d blocks dir and lfs bin
e611cf5 Fix incorrect lookahead population before ack
a25743a Fixed some minor error code differences
6716b55 Fixed error check when truncating files to larger size
dc513b1 Silenced more of aldot's warnings
aa50e03 Commentary typo fix
6d55755 tests: Silence warnings in template
029361e Silence shadow warnings
fd04ed4 Added autogenerated release notes from commits
3101bc9 Do not print command invocation if QUIET
436707c doc: Editorial tweaks
3457252 doc: Spelling fixes
6d8e0e2 Moved -Werror flag to CI only
88f678f Fixed self-assign warning in tests
3ef4847 Added remove step in tests to force rebuild
5a38d00 Added deploy step in Travis to push new version as tags
035552a Add version info for software library and on-disk structures
997c2e5 Fixed incorrect reliance on errno in emubd
d88f0ac Added lfs_file_truncate

What's new?

• Atomic moves
• Significant amount of bug fixes
• A lot of tweaks based on initial feedback

Changes

2ad435e Added files test to littlefs-fuse tests in Travis
1fb6a19 Reduced ctz traverse runtime by 2x
db88727 Added error code LFS_ERR_NOTEMPTY
c2fab8f Added asserts on geometry and updated config documentation
472ccc4 Fixed file truncation without writes
aea3d3d Fixed positive seek bounds checking
be22d34 Updated links to Mbed OS
425aa3c Fixed issue with immediate exhaustion and small unaligned storage
5ee20e8 Fixed pipefail issue that was preventing CI from reporting errors
bf78b09 Added directory list for synchronizing in flight directories
e169d06 Removed vestigial function declaration
996cd8a Revisited documentation
78c79ec Added QUIET flag to tests so CI is readable
f9f4f5c Fixed standard name mismatch LFS_ERR_EXISTS -> LFS_ERR_EXIST
843e3c6 Added sticky-bit for preventing file syncs after write errors
2612e1b Modified lfs_ctz_extend to be a little bit safer
6664723 Fixed issue with committing directories to bad-blocks that are stuck
3f31c8c Fixed corner case with immediate exhaustion and lookahead==block_count
f4aeb83 Fixed issue with aggressively rounding down lookahead configuration
db51a39 Removed stray newline in LFS_ERROR for version
2ab150c Removed toolchain specific warnings
0825d34 Adopted alternative implementation for lfs_ctz_index
46e22b2 Adopted lfs_ctz_index implementation using popcount
4fdca15 Slight name change with ctz skip-list functions
454b588 Updated SPEC.md and DESIGN.md based on recent changes
f3578e3 Removed clamping to block size in ctz linked-list
83d4c61 Updated copyright
539409e Refactored deduplicate/deorphan step to single deorphan step
2936514 Added atomic move using dirty tag in entry type
ac9766e Added self-hosting fuzz test using littlefs-fuse
9db1a86 Added specification document
9843402 Fixed incorrect return value from lfs_file_seek
273cb7c Fixed problem with lookaheads larger than block device
d9367e0 Fixed collection of multiblock directories
a83b2fe Added checks for out-of-bound seeks
a8fa5e6 Fixed some corner cases with paths
26dd49a Fixed issue with negative modulo with unaligned lookaheads
0982020 Fixed issue with cold-write after seek to block boundary

Initial release

littlefs is a little fail-safe filesystem designed for embedded systems.

More information

• README.md - Landing page and introduction of littlefs
• DESIGN.md - High-level design overview of littlefs