irmin

CHANGES:

Added

• irmin-pack
  • Add forbid_empty_dir_persistence in store configuration. (#1789,
    @ngoguey42)
  • Add Store.Snapshot to expose the inodes for tezos snapshots (#1757,
    @icristescu).

Changed

• irmin
  • Add error types in the API or proof verifiers. (#1791, @icristescu)
  • Reduced the memory footprint of Tree.fold ~uniq:`True by a factor of 2.
    (#1770, @CraigFe)

CHANGES:

Fixed

• irmin-pack

  • Drop unnecessary runtime dependency on ppx_irmin. (#1782, @hhugo)
  • Split the unix part of irmin-pack into irmin-pack.unix (#1783, @hhugo)

• irmin-unix

  • Fix conflicting command line arguments for push, pull, fetch and
    clone (#1776, @zshipko)
  • Fix issues with Sync functions by provided a better default Mimic.ctx. A
    side-effect of this update is that the remote function now returns an Lwt
    promise. (#1778, @zshipko)

Added

• libirmin
  • Create libirmin package providing a C interface to the irmin API
    (#1713, @zshipko)

Changed

• irmin-bench
  • Make trace replay API public and simpler (#1781, @Ngoguey42)

CHANGES:

Fixed

• irmin

  • Fix the implementation of comparison on Irmin.Tree objects to use the
    comparison defined on hashes. The previous implementation was unstable.
    (#1519, @CraigFe)
  • Default implementation for contents, nodes and commits can no longer trigger
    pre_hash collisions. This changes their hash. (#1715, @Ngoguey42,
    @icristescu)

• irmin-pack

  • Improve the performance of Index encode and decode operations by
    eliminating intermediate allocations. (#1577, @CraigFe)

• irmin-unix

  • Fix terms that can be manipulated at runtime by delaying computation
    (#1645, @zshipko)

Added

• irmin

  • Add Read_only.S and Read_only.Maker module types (#1343, @samoht)
  • Append-only and content-addressable backend implementations have to
    provide close and batch functions (#1345, @samoht)
  • Atomic-write backend implementations have to provide a close function
    (#1345, @samoht)
  • Add a function Store.Tree.singleton for building trees with a single
    contents binding. (#1567, @CraigFe)
  • Add with_handler and head to Store.Backend.Node and
    Store.Backend.Node_portableto work with recursive node structures from
    irmin core. (#1712, #1746 @Ngoguey42). Forward
    port of #1692 and #1670.
  • Add proof, to_proof and of_proof to Store.Backend.Node_portable
    (#1716, @Ngoguey42). Forward port from #1583.
  • Add hash_exn to Store.Backend.Node.Val and Store.Backend.Node_portable
    (#TODO, @Ngoguey42) Forward ported from #1741.
  • Add a Store.Tree.kinded_hash function. (#1767, @Ngoguey) Forward ported
    from #1625.
  • Add Contents.String_v2, Node.Generic_key.Make_v2 and
    Commit.Generic_key.Make_v2 for backward compatibility with older stores.
    (#1715, @icristescu)

• irmin-bench

  • Many improvements to the actions trace replay:
    • Support for the layered store (#1293, @Ngoguey42)
    • Fix replay for the first ~650k commits (was ~13k) (#1314, @Ngoguey42)
    • Can change inode configuration prior to replay (#1326, @Ngoguey42)
    • Check hash of commits (#1328, @icristescu)
    • Fix the path flattening technique (#1357, @Ngoguey42)
    • Introduce a new actions trace that can support replaying up to ~1300k
      commits. (#1358, @Ngoguey42)
    • Improve the stats collection and stats report (#1367, #1384, #1403,
      #1404, #1416, #1429, #1438, #1501, #1616, @Ngoguey42, @maiste)
    • Enable replay in CI (#1430, @Ngoguey42)
    • Enable replay in CB (#1441, @Ngoguey42)

• irmin-mem

  • Add Irmin_mem.Content_addressable (#1369, @samoht)

• irmin-pack

  • Add a stat-store command to irmin-fsck to output stats on the tree
    under a specified commit (#1391, @icristescu, @Ngoguey42, @CraigFe).
  • Add new counters in Stats (#1570, @Ngoguey42).
  • Add an option to configure the index function and pick the relevant bits
    in a cryptographic hash by default (#1677 #1699, @samoht)
  • Verify inode depth invariants (#1711, @Ngoguey42). Forward port of #1665.

• irmin-unix

  • Update irmin CLI to raise an exception when an invalid/non-existent
    config file is specified (#1413, @zshipko)
  • Add --commit flag to CLI to load a store from a specific commit hash
    (#1721, @zshipko)

• irmin-tezos

  • Add a new package to mirror Tezos tezos-context.encoding library.
    That'll simplify building benchmarks and custom tools (#1579, @samoht)

Changed

• irmin

  • Irmin.Sync is now a namespace: use Irmin.Sync.Make(S) instead of
    Irmin.Sync(S) (#1338, @samoht)

  • Irmin.Private is now Irmin.Backend (#1530, @CraigFe)

  • Store.master is now Store.main. The existing Store.master function is
    deprecated and will be removed in a future release. (#1564, @CraigFe)

  • Store.Private is now Store.Backend (#1530, @CraigFe)

  • Store.Private.Sync is now Store.Backend.Remote (#1338, @samoht)

  • Irmin.Branch.S.master is now Irmin.Branch.S.main (#1564, @CraigFe)

  • Irmin.Private.{Commit,Node} are now Irmin.{Node,Commit}. (#1471,
    @CraigFe)

  • All module types are now using snake-case and are not capitalized anymore.
    (#1341, @samoht)

  • Move signatures for backend stores into their own modules. All the
    X_STORE sigs have moved to X.S:

    • APPEND_ONLY_STORE is now Append_only.S
    • CONTENT_ADDRESSABLE_STORE is now Content_addressable.S
    • ATOMIC_WRITE_STORE is now Irmin.Atomic_write.S
      And all the X_STORE_MAKER have moved to X.Maker:
    • APPEND_ONLY_STORE_MAKER is now Append_only.Maker
    • CONTENT_ADDRESSABLE_STORE_MAKER is now Content_addressable.Maker
    • ATOMIC_WRITE_STORE_MAKER is now Atomic_write.Maker
      This gives some space to move convenient functors closer to where they
      belong:
    • Content_addressable is now Content_addressable.Make
    • New Content_adddressable.Check_closed and Atomic_write.Check_closed
      (#1342, @samoht)

  • Rename Irmin.Make into Irmin.Maker ; stage its result to return
    Make functor once provided with a content-addressable and an
    atomic-writes stores (#1369, @samoht)

  • Rename Irmin.Make_ext into Irmin.Maker_ext ; stage its result to
    return Make functor once provided with a content-addressable and an
    atomic-writes stores, as well as node and commit makers (#1369, @samoht)

  • Require at least lwt.5.3.0 to use Lwt.Syntax in the codebase
    (#1401, @samoht)

  • Info implementations are not part of store: use S.Info.v
    instead of Irmin.Info.v (#1400, @samoht)

  • Rename Commit.V1 to Commit.V1.Make. This functor now takes separate
    hash and key implementations as arguments. (#1431 #1634, @CraigFe
    @icristescu)

  • Introduce a Schema module to hold all the types that users can
    define in an Irmin store. Use this as a parameter to every Maker
    functor. This is a large change which touches all the backends.
    (#1470, @samoht, @CraigFe)

  • Add Irmin.Backend.Conf.Schema for grouping configuration keys. Now
    Irmin.Backend.Conf.key takes an additional ~spec parameter.
    (#1492, @zshipko)

  • Tree.empty and Node.empty now both take a unit argument. (#1566 #1629,
    @CraigFe)

  • Rename key type to path and Key module to Path when it is in a path
    context in Tree and Store. (#1569, @maiste)

  • Move Node.default metadata default values into a Node.Metadata.default
    to give room for other metadata values (#1611, @samoht)

  • Add support for non-content-addressed ("generic key") backend stores. This
    allows Irmin to work with backends in which not all values are addressed by
    their hash. In particular, this includes:

    • New functions: Store.{Commit,Contents,Tree}.of_key.
    • Adds Irmin.{Node,Commit}.Generic_key modules.
    • Adds new types that must be provided by backends: Node.Portable and
      Commit.Portable.
    • Adds a new type of backend store: Irmin.Indexable.S.
      (#1510 #1647, @CraigFe)

  • Cache hits in several Tree functions are more frequent than before.
    (#1724, @Ngoguey42, @CraigFe)

  • Add a new Pruned_hash tag to the error case of several Store.Tree
    functions (#1744 @Ngoguey42). Forward ported from #1583.

• irmin-containers

  • Removed Irmin_containers.Store_maker; this is now equivalent to
    Irmin.Content_addressable.S (#1369, @samoht)
  • Renamed Irmin_containers.CAS_maker to
    Irmin_containers.Content_addressable (#1369, @samoht)

• irmin-fs

  • Renamed Irmin_fs.Make into Irmin_fs.Maker (#1369, @samoht)
  • Renamed Irmin_fs.Make_ext into Irmin_fs.Maker_ext (#1369, @samoht)

• irmin-git

  • All of the configuration keys have moved into their own namespace:
    • Irmin_git.root is now Irmin_git.Conf.root
    • Irmin_git.head is now Irmin_git.Conf.head
    • Irmin_git.bare is now Irmin_git.Conf.bare
    • Irmin_git.level is now Irmin_git.Conf.level
    • Irmin_git.buffers is now Irmin_git.Conf.buffers
    • Irmin_git.dot_git is now Irmin_git.Conf.dot_git
      (#1347, @samoht)
  • Renamed Irmin_git.Make into Irmin_git.Maker (#1369, @samoht)
  • Require at least git.3.7.0 in the codebase (#1632, @dinosaure)

• irmin-graphql:

  • Changed the name of the default branch node from master to main in the
    GraphQL API. (#1564, @CraigFe)
  • Updated to be compatible with generic keys.
    • The Key type is now called Path to match the new name in irmin
    • All key fields and parameters have been renamed to path
      (#1618, @zshipko)

• irmin-mirage

  • Renamed Irmin_mirage_git.Make into Irmin_mirage_git.Maker
    (#1369, @samoht)

• irmin-pack

  • Changed the implementation of backend store keys to use direct pointers to
    store contents (by offset in the pack file) when possible, rather than
    querying the index on each lookup. (#1659, @CraigFe @ngoguey42 @icristescu)
  • The Irmin_pack.Maker module type now no longer takes a Conf argument.
    (#1641, @CraigFe)
  • The backend configuration type Conf.S requires a new parameter
    contents_length_header that (optionally) further specifies the encoding
    format used for commits in order to improve performance. (#1644, @CraigFe)
  • Upgraded on-disk format of pack files to support more efficient lookups and
    reduce indexing overhead. This change is fully backwards-compatible with
    existing stores using irmin-pack.2.x versions, but not
    forwards compatible. (#1649 #1655, @CraigFe @Ngoguey42)
  • Added support for user-specified indexing strategies. The default strategy
    is to index all objects appended to the pack file (as before), but users may
    now choose to index fewer objects in order to improve the write performance
    of the store, at the cost of introducing potential duplicate values to the
    pack file. (#1664, #1761, @CraigFe, @maiste)

• irmin-unix

  • Clean up command line interface. Allow config file to be specified when
    using Irmin_unix.Resolver.load_config and make command line options
    take precedence over config options.
    (#1464, #1543, #1607 @zshipko)
  • Irmin_unix.Resolver.destruct has been removed (and partially replaced by
    Resolver.spec). (#1603, @CraigFe)
  • Update irmin CLI to support empty path in list subcommand.
    (#1575, @maiste)
  • Add new commands to CLI: branches for listing available branches and
    log which is similar to git log (#1609, #1727, @zshipko)
  • Update irmin watch to take parameters to specify a command that should
    be executed when there are new changes (#1608, @zshipko)

Removed

• irmin-pack

  • Removed the irmin-pack.layered library. Support for the layered store
    will be restored on a future release of irmin-pack. (#1651, @CraigFe)
  • Removed support for the clear operation in irmin-pack. This operation
    is incompatible with performance optimisations made in this release.
    (#1655, @CraigFe)

• irmin-layers

  • This experimental package has been removed.

CHANGES:

Fixed

• irmin

  • Fixed a bug causing stream proof extender nodes to have their segments be
    returned in reverse order (i.e. bottom to top, rather then top-down).
    (#1742, @CraigFe)

  • Fixed a bug that allowed the creation of overly-large stable inodes via
    stream proofs. (#1741, @Ngoguey42)

Added

• irmin
  • Add Store.Private.Node.Val.hash_exn (#1741, @Ngoguey42)

CHANGES:

Fixed

• irmin
  • Fix bug introduced in #1683 which causes Tree.seq and Tree.list to
    produce pruned children (#1720, @Ngoguey42)

CHANGES:

Fixed

• irmin
  • Better support for s390x to workaround
    https://github.com/ocaml/ocaml/issues/10857 (#1694, @icristescu)

CHANGES:

Fixed

• irmin

  • Conversion between proofs and trees are now done in CPS (#1624, @samoht)
  • Better support for s390x to workaround https://github.com/ocaml/ocaml/issues/10857
    (#1694, @icristescu)

• irmin-pack

  • Fix proofs for large inodes by tracking side-effects reads inside the
    inode implementation (#1670, @samoht, @Ngoguey42)
  • Flush branch store without calling Repo.close (#1707, @zshipko)

Added

• irmin

  • Add Tree.produce_proof and Tree.verify_proof to produce and verify
    proofs from complex computations. produce_proof and verify_proof
    takes a callback over tree and instead of a static list of operations
    -- this now means that the full Tree API can now be used in proofs,
    including sub-tree operations, folds and paginated lists
    (#1625, #1663, #1683, @samoht, @Ngoguey42)
  • Add Tree.produce_stream and Tree.verify_stream to produce and
    verify stream proofs (#1684, #1692, #1691, @samoht, @Ngoguey42, @icristescu)

• irmin-pack

  • Verify inode depth invariants (#1665, @samoht)

• irmin-unix

  • Add tezos store type for irmin command-line (#1678, @zshipko)

Changed

• irmin

  • Remove Tree.Proof.of_keys. Use Tree.produce_proof instead
    (#1625, @samoht)
  • Tree.empty now takes a unit argument. (#1566, @CraigFe)
  • Tree.length now takes a tree as argument (#1676, @samoht)
  • Tree.Proof.t now uses a more precise datatype to encode value
    invariants (#1688, @samoht)

• irmin-pack

  • irmin-pack: add an option to configure the index function and pick
    the relevant bits in cryptographic a hash by default (#1677, @samoht)

• irmin-git

  • Require at least git.3.7.0 in the codebase (#1637, @dinosaure)

CHANGES:

Fixed

• irmin-pack
  • Improved the performance of Index encode and decode operations by
    eliminating intermediate allocations (up to 5% fewer minor words
    allocated) (#1577, @CraigFe)
  • Reduce the number of backend nodes built during export
    (up to 20% fewer minor words allocated) (#1553, @Ngoguey42)

Added

• irmin
  • Add Merkle Proofs and expose function to convert a proof to and from a tree.
    Once converted, normal tree operations can be performed on the proof, as
    long at it access values contained in the proof.
    (#1583, @samoht, @Ngoguey42, @icristescu)

Changed

• irmin-pack
  • Limit inode depth (#1596, #samoht)
  • Adapt to index 1.5.0 (#1593, @icristescu)

CHANGES:

Fixed

• irmin
  • Tree operations now raise a Dangling_hash exception when called with a
    path that contains dangling hashes in the underlying store, rather than
    interpreting such paths as ending with empty nodes (#1477, @CraigFe)
  • Fix the pre-hashing function for big-endian architectures. (#1505,
    @Ngoguey42, @dinosaure)
  • Fix a bug in Tree.export where nodes could be exported before
    some of their contents, resulting in indirect hashes in irmin-pack
    (#1508, @Ngoguey42)

Added

• irmin
  • Node.seq and Node.of_seq are added to avoid allocating intermediate
    lists when it is not necessary (#1508, @samoht)
  • New optional cache parameter to Tree.hash, Tree.Contents.hash,
    Tree.list, Node.list, Node.seq and Node.find to control the storing
    of lazily loaded data (#1526, @Ngoguey42)
  • Add Node.clear to clear internal caches (#1526, @Ngoguey42)
  • Added a tree argument to Tree.fold to manipulate the subtrees (#1527,
    @icristescu, @Ngoguey42)
  • Add a function Store.Tree.pruned for building purely in-memory tree
    objects with known hashes. (#1537, @CraigFe)
  • Added a order argument to specify the order of traversal in Tree.fold
    (#1548, @icristescu, @CraigFe)

Changed

• irmin

  • Node.v is renamed to Node.of_list (#1508, @samoht)
  • Rewrite Tree.export in order to minimise the memory footprint.
    (#1508, @Ngoguey42)
  • Remove the ~force:`And_clear case parameter from Tree.fold,
    ~force:`True ~cache:false is the new equivalent. (#1526, @Ngoguey42)
  • `Tree.fold ~force:`True and `Tree.fold ~force:`False don't
    cache the lazily loaded data any more. Pass ~cache:true to enable it
    again. (#1526, @Ngoguey42)
  • Do not allocate large lists in Irmin.Tree.clear (#1515, @samoht)

• irmin-git

  • Upgrade irmin-git to git.3.5.0. (#1495, @dinosaure)

CHANGES:

Added

• irmin-pack
  • Added integrity-check-index command in irmin-fsck. (#1480, #1487
    @icristescu, @samoht)

Changed

• irmin-pack
  • reconstruct_index is now traverse_pack_file, it allows for both index
    reconstruction and index checking (#1478, @Ngoguey42)

CHANGES:

Fixed

• irmin-pack
  • Fix termination condition of reconstruct index (#1468, @Ngoguey42)

CHANGES:

Fixed

• irmin

  • Added Store.Tree.length. (#1316, @Ngoguey42)
  • Fixed fold for non-persisted, cleared trees (#1442, @samoht, @Ngoguey42)

• irmin-layers

  • Do not fail on double-close errors for private nodes (#1421, @samoht)

• irmin-pack

  • Do not clear and bump the generation for empty files (#1420, @samoht)

Added

• irmin-pack

  • Added Irmin_pack.Version.{V1,V2} modules for convenience. (#1457,
    @CraigFe)
  • Added a irmin-pack.mem package (#1436, @icristescu, @craigfe)

• irmin-graphql

  • Added last_modified field to GraphQL interface (#1393, @kluvin)

Changed

• irmin-layers

  • Remove copy_in_upper from the repo configuration. The default is now to
    copy. (#1322, @Ngoguey42)
  • Simplify the API of freeze. It is now possible to specify two distinct
    commit closures for the copy to lower and the copy to next upper.
    (#1322, @Ngoguey42)
  • Renamed Irmin_layered_pack.Make and Irmin_layers.MakeintoIrmin_layered_pack.MakerandIrmin_layers.Maker` (#1369, @samoht)
  • Renamed Irmin_layered_pack.Make_ext and and Irmin_layers.Make_extinto intoIrmin_layered_pack.Maker_extandIrmin_layers.Maker_ext`
    (#1369, @samoht)
  • Renamed Irmin_layered_pack.Config into Irmin_layered_pack.Conf
    (#1370, @samoht)
  • Readonly instances can check for an ongoing freeze (#1382, @icristescu,
    @Ngoguey42)

• irmin-pack

  • It is no longer possible to modify an inode that doesn't point to the root
    of a directory. (#1292, @Ngoguey42)
  • When configuring a store, is it no longer possible to set entries to a
    value larger than stable_hash. (#1292, @Ngoguey42)
  • Added number of objects to the output of stat-pack command in
    irmin-fsck. (#1311, @icristescu)
  • Renamed the Version module type into Version.S and io_version into
    version. The Pack.File and Atomic_write functors now take
    Version as their first parameter (#1352, @samoht)
  • Renamed Irmin_pack.Make into Irmin_pack.V1 (#1369, @samoht)
  • Renamed Irmin_pack.Config into Irmin_pack.Conf (#1370, @samoht)
  • Renamed Irmin_pack.Pack into Irmin_pack.Content_addressable and
    Irmin_pack.Pack.File into Irmin_pack.Content_addressable.Maker
    (#1377, @samoht)
  • Moved Irmin_pack.Store.Atomic_write into its own module (#1378, @samoht)
  • Checks.Reconstruct_index.run now takes an optional index_log_size
    parameter for customising the interval between merges during
    reconstruction. (#1459, @CraigFe)

CHANGES:

This release contains 2.6.0 plus the changes described in 2.5.4.

CHANGES:

Fixed

• irmin-pack

  • Revert a patch introduced in 2.3.0 which was calling Index.try_merge.
    This function was supposed to hint index to schedule merges after
    every commit. However, Index.try_merge is buggy and stacks merges
    which causes the node to block and wait for any existing merge to
    complete. We will revisit that feature in future once we fix
    Index.try_merge (#1409, @CraigFe)

• irmin

  • Fix peformance issue in Tree.update_tree and Tree.add_tree for
    large directories (#1315, @Ngoguey42)

Added

• irmin-pack
  • Expose internal inode trees (#1273, @mattiasdrp, @samoht)

CHANGES:

Fixed

• irmin

  • Fix stack overflow exception when working with wide trees (#1313, @zshipko)

  • Tree.of_concrete now prunes empty subdirectories, and raises
    Invalid_argument if the input contains duplicate bindings. (#1385,
    @CraigFe)

• irmin-chunk

  • Use the pre_hash function to compute entry keys instead of
    their raw binary representation (#1308, @samoht)

Changed

• irmin-git
  • Upgrade irmin-git with git.3.4.0. (#1392, @dinosaure)

CHANGES:

Fixed

• irmin
  • Fixed a bug causing equality functions derived from Store.tree_t to return
    false-negatives. (#1371, @CraigFe)

Added

• irmin
  • Add Store.Tree.is_empty. (#1373, @CraigFe)

CHANGES:

Fixed

• irmin

  • The Tree.update_tree and Tree.add_tree functions now interpret adding
    an empty subtree as a remove operation, rather than adding an empty
    directory. (#1335, @craigfe)

• irmin-pack

  • Fix a performance regression where all caches where always cleaned by
    Store.sync when using the V1 format (#1360, @samoht)

CHANGES:

• irmin-git
  • Use the last version of git 3.3.0. It fixes a bug about trailing LF on
    message. For Irmin users, it should not change anything (#1301, @dinosaure,
    @CraigFe)

CHANGES:

Changed

• irmin

  • Store.Tree.remove is now much faster when operating on large directories.
    The commits following removals are also much faster. (#1289, @Ngoguey42)

  • Changed Store.Tree.{of_hash, shallow} to take kinded hashes, allowing the
    creation of unforced contents values. (#1285, @CraigFe)

  • Changed Tree.destruct to return lazy contents values, which may be forced
    with Tree.Contents.force. (#1285, @CraigFe)

• irmin-bench

  • New features in benchmarks for tree operations (#1269, @Ngoguey42)

CHANGES:

Fixed

• irmin-pack

  • Fix a bug in inode where the remove function could cause hashing
    instabilities. No user-facing change since this function is not being used
    yet. (#1247, @Ngoguey42, @icristescu)

• irmin

  • Ensure that Tree.add_tree t k v complexity does not depend on v size.
    (#1267, @samoht @Ngoguey42 and @CraigFe)

Added

• irmin

  • Added a Perms module containing helper types for using phantom-typed
    capabilities as used by the store backends. (#1262, @CraigFe)

  • Added an Exported_for_stores module containing miscellaneous helper types
    for building backends. (#1262, @CraigFe)

  • Added new operations Tree.update and Tree.update_tree for efficient
    read-and-set on trees. (#1274, @CraigFe)

• irmin-pack:

  • Added integrity-check-inodes command to irmin-fsck for checking the
    integrity of inodes. (#1253, @icristescu, @Ngoguey42)

• irmin-bench

  • Added benchmarks for tree operations. (#1237, @icristescu, @Ngoguey42,
    @Craigfe)

Changed

• The irmin-mem package is now included with the irmin package under the
  library name irmin.mem. It keeps the same top-level module name of
  Irmin_mem. (#1276, @CraigFe)

Removed

• Irmin_mem no longer provides the layered in-memory store Make_layered.
  This can be constructed manually via Irmin_layers.Make. (#1276, @CraigFe)