nessie

Nessie: Transactional Catalog for Data Lakes with Git-like semantics

APACHE-2.0 License

Downloads
3.9K
Stars
824
Committers
57

Bot releases are hidden (Show)

nessie - Nessie 0.90.4

Published by snazy 4 months ago

Nessie 0.90.4 release

Try it

The attached nessie-quarkus-0.90.4-runner.jar is a standalone uber-jar file that runs on Java 17 or newer and it is also available via Maven Central. Download and run it (requires Java 17):

wget https://github.com/projectnessie/nessie/releases/download/nessie-0.90.4/nessie-quarkus-0.90.4-runner.jar
java -jar nessie-quarkus-0.90.4-runner.jar

Nessie GC tool is attached as nessie-gc-0.90.4.jar, which is a standalone uber-jar file that runs on Java 11 or newer. Shell completion can be generated from the tool, check its help command. Nessie GC tool is also available as a Docker image: docker run --rm ghcr.io/projectnessie/nessie-gc:0.90.4 --help.

Nessie Server Admin tool is attached as nessie-server-admin-tool-0.90.4-runner.jar, which is a standalone uber-jar file that runs on Java 17 or newer. Shell completion can be generated from the tool, check its help command. Nessie Server Admin tool is also available as a Docker image: docker run --rm ghcr.io/projectnessie/nessie-server-admin:0.90.4 --help.

The attached nessie-helm-0.90.4.tgz is a packaged Helm chart, which can be downloaded and installed via Helm. There is also the Nessie Helm chart repo, which can be added and used to install the Nessie Helm chart.

Changelog

New Features

  • Support for token exchange in the Nessie client has been completely redesigned. The new API and
    configuration options are described in the [Nessie authentication settings]. If this feature is
    enabled, each time a new access token is obtained, the client will exchange it for another one by
    performing a token exchange with the authorization server. We hope that this new feature will
    unlock many advanced use cases for Nessie users, such as impersonation and delegation. Please note
    that token exchange is considered in beta state and both the API and configuration options are
    subject to change at any time; we appreciate early feedback, comments and suggestions.

Full Changelog (minus renovate commits):

  • Skip 0.90.3
  • Fix compilation failure (#8817)
  • Revert commits for 0.90.3 release
  • JDBC: add proper support for H2 (#8751)
  • Fix cache-deserialization for UpdateableObj (#8815)
  • OAuth2Client: refactor scopes (#8814)
  • Token exchange: documentation enhancements (#8813)
  • Token exchange: more flexible subject / actor configuration (#8812)
  • OAuth2Client: integration tests for token exchange (#8810)
  • Fix docker-compose files after Keycloak upgrade to 25.0 (#8804)
  • HTTP client: introduce specialized HttpClientResponseException (#8806)
  • OAuth2Client: internal code cleanup (#8807)
  • OAuthClient: configurable subject and actor token types (#8805)
  • OAuth2Client: introduce proper support for token exchange (#8803)
  • OAuth2Client: don't refresh tokens using token exchange (#8790)
  • Disable a test on macOS (#8799)
  • Helm chart: minor improvement to JDBC section in docs (#8717)
  • Nessie w/ Iceberg REST / post-release site changes (#8510)
  • Ninja: fix release-create workflow issues in "create github release" job
nessie - Nessie 0.90.2

Published by snazy 4 months ago

Nessie 0.90.2 release

Try it

The attached nessie-quarkus-0.90.2-runner.jar is a standalone uber-jar file that runs on Java 17 or newer and it is also available via Maven Central. Download and run it (requires Java 17):

wget https://github.com/projectnessie/nessie/releases/download/nessie-0.90.2/nessie-quarkus-0.90.2-runner.jar
java -jar nessie-quarkus-0.90.2-runner.jar

Nessie GC tool is attached as nessie-gc-0.90.2.jar, which is a standalone uber-jar file that runs on Java 11 or newer. Shell completion can be generated from the tool, check its help command. Nessie GC tool is also available as a Docker image: docker run --rm ghcr.io/projectnessie/nessie-gc:0.90.2 --help.

Nessie Server Admin tool is attached as nessie-server-admin-tool-0.90.2-runner.jar, which is a standalone uber-jar file that runs on Java 17 or newer. Shell completion can be generated from the tool, check its help command. Nessie Server Admin tool is also available as a Docker image: docker run --rm ghcr.io/projectnessie/nessie-server-admin:0.90.2 --help.

The attached nessie-helm-0.90.2.tgz is a packaged Helm chart, which can be downloaded and installed via Helm. There is also the Nessie Helm chart repo, which can be added and used to install the Nessie Helm chart.

Changelog

Highlights

  • Nessie has got support for Iceberg REST.
  • MySQL users can now configure the JDBC connection using the quarkus.datasource.mysql.*
    properties. Also, JDBC URLs can now use the mysql prefix, e.g.
    jdbc:mysql://example.com:3306/my_db.

Upgrade notes

  • Helm chart: the cassandra section now has support for pulling credentials from a secret. The
    old cassandra.auth.username and cassandra.auth.password properties still work, but are now
    deprecated and will be removed in a future release. Use the cassandra.secret property instead.

New Features

  • Support for Iceberg REST is in "beta" state. We appreciate early feedback, comments and suggestions.
    Take a look at the Guides and Docs
    on our web site projectnessie.org for more information.
  • CEL access check scripts now receive the variable roles that can be used to check whether the current
    principal has a role assigned using a CEL expression like 'rolename' in roles.

Deprecations

  • Support for Java 8 is officially deprecated and users are encouraged to upgrade all clients to
    at least Java 11, better Java 17 or 21, if possible. Current Spark versions 3.3, 3.4 and 3.5
    work with Java 11 and 17. Support for Java 8 will eventually be removed.
  • For JDBC version stores, the following settings, which never worked as expected, are now
    deprecated and will be removed in a future release. The catalog and the schema must always be
    specified explicitly in the JDBC URL.
    • nessie.version.store.persist.jdbc.catalog
    • nessie.version.store.persist.jdbc.schema

Fixes

  • A bug in the API compatibility filter has been discovered and fixed: when OAuth2 authentication is
    being used, the filter causes the OAuth2 client to close prematurely, thus triggering unauthorized
    errors. A workaround is to simply disable the filter (set nessie.enable-api-compatibility-check
    to false), but this is no longer necessary.

Full Changelog (minus renovate commits):

  • Manually bump to 0.90.1-SNAPSHOT for next release
  • Revert "[release] release nessie-0.90.1"
  • Another release-publish bug
  • Manually bump to 0.90.1
  • Revert "[release] release nessie-0.90.0"
  • Revert 0.90.0 commits
  • Ninja: fix
  • Revert 0.90.0 release commits
  • Ninja: Fix Swaggerhub
  • Prepare bump to v0.90.0 (#8792)
  • Persist: introduce type-safe bulk-fetch functions (#8763)
  • Persist-cache: add negative obj cache (#8762)
  • Remove NessieDockerTestResourceLifecycleManager (#8788)
  • Catalog tests: disable token refresh (#8786)
  • OAuth2Client: fix race condition when scheduling refreshes (#8787)
  • OAuth2 client: add support for RFC 8414 metadata discovery (#8789)
  • Support distributed invalidation for Persist cache (#8463)
  • API compatibility filter: don't share authentication with main client (#8769)
  • [Catalog] Restrict S3 signing to URIs within table base prefix (#8654)
  • Persist: add fetchObjsIfExist() (#8761)
  • Refactor Backend.setupSchema to return schema info (#8753)
  • Add workaround for nip.io for macOS, optimize for Linux (#8759)
  • Cleanup nessie-quarkus build file (#8757)
  • [Catalog] Recognize s3a and s3n, behave like s3 (#8755)
  • Expose 'roles' in CEL access check scripts (#8752)
  • Site: Add RSS+JSON feeds (#8754)
  • Docs-gen: more documented types, don't expose "Java types" (#8705)
  • Allow secrets in a Keystore (#8710)
  • [Catalog] Secrets suppliers/managers API (#8706)
  • JDBC: remove options 'catalog' and 'schema' and revisit schema creation (#8723)
  • Helm chart: add support for OIDC client secret (#8750)
  • Site: Clarify "Authorization" and remove confusing paragraph (#8749)
  • Helm chart: fix a few property names (#8743)
  • Add explicit authorization-type = CEL (#8746)
  • Add isAnonymous() to AccessContext (#8745)
  • Add a JSON view to SmileSerialization (#8744)
  • Helm chart: reload pods when configmap changes (#8725)
  • Replace :latest with versions in docker-compose files (#8724)
  • Nit: fix docker-compose paths (#8722)
  • Avoid invalid test names due to non-printable characters (#8721)
  • Helm chart / Cassandra: add support for pulling credentials from secrets (#8716)
  • Helm chart: fix wrong env var + nits (#8718)
  • Helm chart: mention removal of legacy store types in 0.75.0 (#8719)
  • Fix API examples for merge conflict responses (#8712)
  • Prevent Quarkus warning about unindexed dependency (#8711)
  • Helm chart: create ConfigMap for application.properties file (#8709)
  • [Catalog] S3CredentialsResolver: remove one second from session end (#8703)
  • Update URLs in ConfigChecks (#8704)
  • Do not pin digests for Dockerfiles and docker-compose files (#8689)
  • Strip leading / (only for S3) (#8681)
  • [Catalog] Implement paging for Iceberg REST (#8633)
  • Adjust all logback*.xml files to new syntax (#8678)
  • Testing: less console log during BigTable tests (#8677)
  • Helm chart: add support for JDBC catalog and schema (#8672)
  • GC: fix incorrect paths when listing prefixes with ADLS (#8661)
  • [Catalog] return metadata-location in object store (at least for now) (#8669)
  • [Catalog] Handle CompletionStage from snapshot store (#8667)
  • [Catalog] Remove unused parameters/code (#8666)
  • Deprecate support for Java 8 (#8650)
  • [Catalog] Simplify tokens-uri resolution (#8649)
  • [Catalog] update CHANGELOG (#8651)
  • README: remove link to demos repository (#8652)
  • Un-ignore SetLocation directives (#8646)
  • [Catalog] remove Google projectId for S3 (#8635)
  • Ninja: fix renovate config
  • Renovate: more automatic merges (#8640)
  • Ninja: rename snapshot-publish workflow
  • [FEATURE] Nessie integration of Iceberg REST (#7043)
  • Authelia tests: remove nip.io (#8632)
  • Split Release workflow job into multiple, retryable jobs (#8596)
  • Custom keycloak container: add function to create a bearer token (#8622)
  • OAuth2 client: add integration tests with Authelia (#8618)
  • Docker compose: add example with Authelia (#8619)
  • Fix jaegertracing image spec (#8615)
  • Fix downloads on site + publish CLI uber jar (#8595)
  • Fix incremental generation of site/docs (#8591)
  • Separate MariaDB and MySQL datasources (#8585)
  • Clean up failed 0.83.0 + 0.83.1 releases (#8590)
  • Snapshot-Publishing: job splitting for snapshot releases (#8589)
  • Renovate: there's more than just dockerfile (#8586)
  • Remove deprecated HttpClientBuilder (#7803)
nessie - Nessie 0.7.0

Published by snazy over 3 years ago

Highlights

Try it

The attached executable file nessie-quarkus-0.7.0-runner is a x86-64 native binary image and only works on compatible Linux environments.

The attached nessie-quarkus-0.7.0-runner.jar is a standalone uber-jar file that runs on Java 11 or newer and it is also available via Maven Central. Download and run it (requires Java 11):

wget https://github.com/projectnessie/nessie/releases/download/nessie-0.7.0/nessie-quarkus-0.7.0-runner.jar
java -jar nessie-quarkus-0.7.0-runner.jar

Full Changelog (minus dependabot commits):

  • Prepare 0.7.0 release (#1427)
  • Build the Docker image with the Quarkus native image (#1431)
  • Throw exception when conflicting keys are modified during merge without a common ancestor (#1416)
  • ./mvnw spotless:apply
  • ./gradlew spotlessApply
  • Use Spotless & Google Java Style
  • Mention Deltalake demo and the nessie-demos repo (#1408)
  • Add missing @Nonnull annotations to overriding methods (#1400)
  • fix mypy typing checks (#1399)
  • Add server-side filtering for /api/v1/tree/{ref}/entries by namespace (#1390)
  • Distinguish between author & committer in the Python CLI (#1385)
  • Allow setting author when committing via Python CLI (#1384)
  • Update iceberg.md (#1379)
  • Add server-side filters to /api/v1/trees/tree/{ref}/log (#1325)
  • Make ContentsKey immutable (#1362)
  • fix naming in nessie client merge operation (#1357)
  • Add OWASP Dependency Check to CI (#1257)
  • Revert "Bump pyspark from 3.0.2 to 3.1.2 in /python (#1349)" (#1353)
  • main+pr builds: maven/gradle build order fix, test-logs-artifact fix (#1345)
  • Update Maven plugin version (#1343)
  • loosen pins for client install on python cli (#1332)
  • Revert "Bump pyspark from 3.0.2 to 3.1.2 in /python (#1329)" (#1331)
  • Update ErrorProne to 2.7.1 (#1321)
  • Fix finding the previous tag + generation of release-notes (#1313)
  • Mention 0.6.1 release (#1314)
  • Verify that the HISTORY.rst and releases.md files contain information about the release (#1315)
nessie - Nessie 0.6.1

Published by snazy over 3 years ago

Highlights

Try it

The attached executable file nessie-quarkus-0.6.1-runner is a native binary image and only works on compatible Linux environments.

The attached uber-jar nessie-quarkus-0.6.1-runner.jar is a standalone uber-jar file that runs on Java 11 or newer and it is also available via Maven Central. Download and run it (requires Java 11):

wget https://github.com/projectnessie/nessie/releases/download/nessie-0.6.1/nessie-quarkus-0.6.1-runner.jar
java -jar nessie-quarkus-0.6.1-runner.jar

Full Changelog (minus dependabot commits):

  • release-workflow: Handle unfinished checks (.conclusion == null) (#1311)
  • Attach quarkus-server-uber-runner-jar to github releases (#1276)
  • Handle NessieConflictException in Delta clients + handling of multiple-branches (#1249)
  • Bump Node from v12.16.2 to v16.2.0 (#1293)
  • Remove Nullable on Contents#getId() (#1256)
  • Bump the version in ui/package*.json in the release-create workflow (#1289)
  • Introduce AssertJ for better test code readability (#1271)
  • Some more fixes to site's spark.md (#1296)
  • Clean up + adjust tools/spark.md (#1275)
  • Fix typo in dependabot.yml (#1274)
  • [site] Add links to Nessie demos (#1246)
  • Add NPM ecosystem to dependabot configuration (#1273)
  • Describe how to configure the Checkstyle formatter rules for Intellij IDEA (#1272)
  • Auto-release on Sonatype (#1247)
  • Re-enable status-checks for release-create workflow (#1255)
  • Step name cleanup (#1248)
  • Add a BOM artifact for Nessie (#1234)
  • Enable Java11 for Quarkus modules (#1245)
  • Gradle plugin: proper fix to run multiple gradle test tasks + support "reducing the log level" (#1243)
  • Mention new Jupyter NB Demos (#1244)
  • Fix input-parameter for release-publish.yml
  • Fix quotes for release-publish.yml step-conditions
nessie - Nessie 0.6.0

Published by snazy over 3 years ago

Highlights

Full Changelog (minus dependabot commits):

  • Finally fix the git-push against protected-branch in release-create.yml workflow
  • Rename .github/README.md so GH uses the top-level README.md
  • Another try to push back to "main" (Sorry for the commits)
  • Another try to fix GH auth for push-to-main
  • Add Robert+Eduard to python/AUTHORS.rst
  • Add Eduard
  • Fix use of access-token in release-create.yml + temporarily disable commit-status-check
  • Use a PAT secret for git push in release-create.yml (#1232)
  • Exclude Release workflow runs from Release workflow commit status check (#1231)
  • Prepare Nessie 0.6.0 release (#1230)
  • Exclude commits from dependabot-preview and dependabot in release notes (#1229)
  • Change scope of nessie-deltalake-core to provided (#1223)
  • Allow multiple Test tasks per Gradle project (#1221)
  • Make pynessie depend on python-dateutil>=2.8.0 for Flink compatibility (#1220)
  • Fix github-workflows-readme (#1212)
  • Check for nessie.url if nessie.uri is not present (#1211)
  • Add release-workflow (#1070)
  • Improve/Fix README for Python Demo Steps (#1187)
  • Improve CONTRIBUTING.md (#1163)
  • Update .gitignore for IntelliJ files (#1164)
  • Bump Maven to 3.8.1 and update wrapper (#1154)
  • Add spec doc to clarify external caching of nessie objects. (#1049)
  • Update spec.md (#1146)
  • fix jacoco for test classes (#1139)
  • Iceberg GC actions and a process to execute GC algorithm (#1032)
  • add unit tests for SerDe of commit messages and contents (#1113)
  • Add some "get involved" on the web-site start page (#1112)
  • Add Iceberg spec to docs (#1107)
  • Add Robert (#1106)
  • fix noise from jacoco (#1105)
  • Add key to gc data set. (#1031)
  • Let VersionStore.create()+commit() return the current hash (#1089)
  • Add profile "docker" to :nessie-quarkus (#1097)
  • Also cache the Gradle wrapper in GH workflows (#1096)
  • Rename the GH site-workflow from Main CI, which is used by the main-branch-CI (#1095)
  • add an id to all contents types (#1023)
  • pin test nessie instance to the one released with iceberg (#1092)
  • Apply the workaround to fix the reverse-DNS issue on GitHub Ubuntu runners (#1091)
  • Update readme to record new vcr cassettes for Python tests (#1090)
  • Fix flaky Spark tests (#1071)
nessie - Nessie 0.5.1

Published by rymurr over 3 years ago

Highlights

Full Changelog (minus dependabot commits):

  • [release] prepare release nessie-0.5.1
  • Prepare for Nessie 0.5.1 (#1065)
  • Add tests for the Gradle Plugin (#1069)
  • Update UI openapi definition (#1068)
  • Improve dependency ordering in the Gradle plugin (#1064)
  • [release] prepare for next development iteration
nessie - Nessie 0.5.0

Published by rymurr over 3 years ago

Highlights

Full Changelog (minus dependabot commits):

  • [release] prepare release nessie-0.5.0
  • Update docs for Nessie 0.5.0 (#1058)
  • Change log-level in CI to WARN (#1056)
  • Fix formatting in OpenAPI example objects (#1048)
  • Backwards compatible payload parsing in Dynamo. (#1029)
  • Correct OpenAPI and add examples (#1043)
  • Support lazy start of Nessie-Quarkus-App via Gradle plugin (#1042)
  • Switch Clock to use Javas clock (#1030)
  • Add TracingStore + TracingVersionStore facades (#986)
  • Add metrics to version-store (#985)
  • set read timeout on client (#1016)
  • Update nessie-iceberg-demo.ipynb, fix catalog creation (#1017)
  • Fix handling of unsaved commits in getUpdateState() (#915)
  • Add/update unit tests to verify that tiered-version-store commit-retries work (#1015)
  • Simplify fetching of iceberg dependency (#1014)
  • Add filtering by entity type in getEntities (#837)
  • fix build - missed rebase (#1009)
  • add WithPayload class to VersionStore interface (#866)
  • Richer commit metadata object (#949)
  • Handle RuntimeException + DynamoDbException in DynamoStore (#938)
  • Fix handling of server-side IllegalArgumentException (#1003)
  • Enable Quarkus Jacoco extension (#994)
  • remove stack traces, reduce noise in pytest related diffs (#972)
  • Use Iceberg's CatalogUtil function in demo (#973)
  • Paging for get-commit-log (#935)
  • Fix parameters in python demo (#965)
  • Fix issue when loading the Gradle plugin into IntelliJ ("missing JAVA_HOME") (#960)
  • Remove AddClause from UpdateExpression as not used. (#950)
  • Revert "Don't let mvn verify fail for deltalake-core, add verify to gh-actions (#941)" (#948)
  • Fix for Quarkus deprecated native-image goal (#943)
  • Don't let mvn verify fail for deltalake-core, add verify to gh-actions (#941)
  • bad merge left these hanging files (#937)
  • log inconsistencies during commit() (#936)
  • GC support for iceberg (#803)
  • [release] prepare for next development iteration
nessie - Nessie 0.4.0

Published by rymurr over 3 years ago

Highlights

Full Changelog (minus dependabot commits):

  • [release] prepare release nessie-0.4.0 (#924)
  • Configurable log-level for tests (#910)
  • Change DynamoDB default billing mode (#916)
  • Update expression visitor (#894)
  • Allow a little bit of SubstrateVM inspection (#914)
  • Tweak logging and don't log HTTP accesses to the console (#913)
  • Close Streams to allow proper tracing+metrics (#909)
  • Bit more verbose error messages (#911)
  • Start version-store when Quarkus app starts (#912)
  • Refactor Quarkus version store configuration (#893)
  • JAX-RS server cleanup (#888)
  • New module for TableCommitMetaStoreWorker (#895)
  • Better modularity in GC (#843)
  • Create SECURITY.md (#876)
  • Create CODE_OF_CONDUCT.md (#875)
  • Create CONTRIBUTING.md (#870)
  • Delete SECURITY.md
  • Create SECURITY.md
  • Add tracing to our http-client, make NessieClient an interface (#835)
  • Use Quarkus BOM version of MongoDB (#850)
  • remove unused transient dependency (#848)
  • Remove unused "backends-type" stuff from ApplicationConfig (#833)
  • Fix config application.properties to match name in ApplicationConfig.java (#832)
  • Report proper exception message, telling the user what actually happened (#826)
  • Handle errors during (Version)Store initialization in Quarkus (#825)
  • Fix concurrent commits against same branch fail (#820)
  • changes to gc algo in prep for iceberg (#811)
  • Respect DynamoDB limit of max 25 requests per batch-write (#798)
  • add system props config to app runner (#802)
  • Rename com.dremio.nessie package to org.projectnessie (#810)
  • Add additional base store tests to AbstractTestStore. (#796)
  • Add ConditionExpression and UpdateExpression visitors (#741)
  • Only log stacktrace for "final" updatedBranch.getUpdateState(store).ensureAvailable with DEBUG logging (#797)
  • Fix property name for table-prefix in application.properties (#795)
  • Use initialState in InternalBranch.collapseIntentionLog (#794)
  • Update to Checkstyle 8.40 (#787)
  • Add metrics and tracing to dynamo store (#755)
  • for / with typo (#773)
  • Fix passing file.encoding system property + update maven-failsafe-plugin as well (#767)
  • API call-order constraints for Ref-tag/branch + BranchCommit-(un)saved (#700)
  • re-enable tests (#758)
  • Fix perftest (#754)
  • Make build+test use en/US (#751)
  • Add test ensuring that assets of dangling values within slop time are considered referenced (#740)
  • Update nessie-iceberg-demo.ipynb (#735)
  • Add "About Nessie" for humans (#599)
  • clarify language for Nessie/Iceberg jar (#730)
  • fix references to iceberg (#719)
  • remove iceberg client and update docs to reflect release 0.11.0 from iceberg project (#713)
  • Implement Garbage Collection Algorithm (#684)
  • Introduce DT Field and refactor Asset Key Interface (#701)
  • Upper-case + ImmutableMap (#699)
  • Move tiered modules to a tiered subdirectory. (#690)
  • Remove Consumer suffix from external ValueType value classes (and ren… (#689)
  • Consumer-pattern for stores (#615)
  • Fix site/README.md to mention venv, add venv to top-level .gitignore (#659)
  • Exclude pentaho-aggdesigner-algorithm (#671)
  • Mongo load support (#545)
  • [release] prepare for next development iteration
nessie - Nessie 0.3.0

Published by rymurr almost 4 years ago

Updates and improvements:

  • 118 commits since 0.2.1
  • Replace jax-rs client with one based on HttpURLConnection
  • Update Quarkus to 1.10.5
  • Improvements to Server including better UI routing, validation checks on inputs etc
  • Various improvements to python client and cli. Including python3.9 support

Maven Central: https://search.maven.org/search?q=g:org.projectnessie
Docker Hub: https://hub.docker.com/r/projectnessie/nessie
PyPI: https://pypi.org/project/pynessie/