opendal

Apache OpenDAL: access data freely.

APACHE-2.0 License

Downloads
16.2K
Stars
3.1K
Committers
226
View Code on GitHub Visit Website
Ecosystems: Apache Spark, Cordova, Redis, Rust

Bot releases are visible (Hide)

opendal - v0.27.1

Published by Xuanwo over 1 year ago

What's Changed

New Contributors

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.27.0...v0.27.1

opendal - v0.27.0

Published by Xuanwo over 1 year ago

Upgrade to v0.27

In v0.27, we refactored our list related logic and added scan support. So make Pager and BlockingPager associated types in Accessor too!

pub trait Accessor: Send + Sync + Debug + Unpin + 'static {
    type Reader: output::Read;
    type BlockingReader: output::BlockingRead;
+    type Pager: output::Page;
+    type BlockingPager: output::BlockingPage;
}

User defined layers

Due to this change, all layers implementation should be changed. If there is not changed over pager, they can by changed like the following:

impl<A: Accessor> LayeredAccessor for MyAccessor<A> {
    type Inner = A;
    type Reader = MyReader<A::Reader>;
    type BlockingReader = MyReader<A::BlockingReader>;
+    type Pager = A::Pager;
+    type BlockingPager = A::BlockingPager;

+    async fn list(&self, path: &str, args: OpList) -> Result<(RpList, Self::Pager)> {
+        self.inner.list(path, args).await
+    }

+    async fn scan(&self, path: &str, args: OpScan) -> Result<(RpScan, Self::Pager)> {
+        self.inner.scan(path, args).await
+    }

+    fn blocking_list(&self, path: &str, args: OpList) -> Result<(RpList, Self::BlockingPager)> {
+        self.inner.blocking_list(path, args)
+    }

+    fn blocking_scan(&self, path: &str, args: OpScan) -> Result<(RpScan, Self::BlockingPager)> {
+        self.inner.blocking_scan(path, args)
+    }
}

Usage of ops

To reduce the understanding overhead, we move all OpXxx into opendal::ops now. User may need to change:

- use opendal::OpWrite;
+ use opendal::ops::OpWrite;

Usage of RetryLayer

backon is the implementation detail of our RetryLayer, so we hide it from our public API. Users of RetryLayer need to change the code like:

- RetryLayer::new(backon::ExponentialBackoff::default())
+ RetryLayer::new()

What's Changed

New Contributors

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.26.2...v0.27.0

opendal - v0.26.2

Published by Xuanwo over 1 year ago

What's Changed

New Contributors

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.26.1...v0.26.2

opendal - v0.26.1

Published by Xuanwo over 1 year ago

v0.26.1 - 2023-02-05

Changed

  • refactor: Remove not used layer subdir (#1280)

Docs

  • docs: Add v0.26 upgrade guide (#1276)
  • docs: Add feature sets in services (#1277)
  • docs: Migrate all docs in rustdoc instead (#1281)
  • docs: Fix index page not redirected (#1282)
opendal - v0.26.0

Published by Xuanwo over 1 year ago

Upgrade to v0.26

In v0.26 we have replaced all internal dynamic dispatch usage with static dispatch. With this change, we can ensure that all operations performed inside OpenDAL are zero cost.

Due to this change, we have to refactor the logic of Operator's init logic. In v0.26, we added opendal::Builder trait and opendal::OperatorBuilder. For the first glance, the only change to existing code will be like:

- let op = Operator::new(builder.build()?);
+ let op = Operator::new(builder.build()?).finish();

By adding a finish() call, we will erase all generic types so that Operator can still be easily to used everywhere as before.

Accessor

In v0.26, Accessor has been changed into trait with associated types.

All services need to decalare the types returned as Reader or BlockingReader:

pub trait Accessor: Send + Sync + Debug + Unpin + 'static {
    type Reader: output::Read;
    type BlockingReader: output::BlockingRead;
}

If your service doesn't support read or blocking_read, we can use () to represent an dummy reader:

impl Accessor for MyDummyAccessor {
    type Reader = ();
    type BlockingReader = ();
}

Layer

As described before, OpenDAL prefer to use static dispatch. Layers are required to implement the new Layer and LayeredAccessor trait:

pub trait Layer<A: Accessor> {
    type LayeredAccessor: Accessor;

    fn layer(&self, inner: A) -> Self::LayeredAccessor;
}

#[async_trait]
pub trait LayeredAccessor: Send + Sync + Debug + Unpin + 'static {
    type Inner: Accessor;
    type Reader: output::Read;
    type BlockingReader: output::BlockingRead;
}

LayeredAccessor is a wrapper of Accessor with the typed Innder. All methods that not implemented will be forward to inner instead.

Builder

Since v0.26, we implement opendal::Builder for all services, and services' mod will not be exported.

- use opendal::services::s3::Builder;
+ use opendal::services::S3;

Conclusion

Sorry again for the big changes in this release. It's a big step for OpenDAL to work in more critical systems.

What's Changed

New Contributors

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.25.2...v0.26.0

opendal - v0.25.2

Published by Xuanwo over 1 year ago

What's Changed

New Contributors

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.25.1...v0.25.2

opendal - v0.25.1

Published by Xuanwo over 1 year ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.25.0...v0.25.1

opendal - v0.25.0

Published by Xuanwo almost 2 years ago

What's Changed

New Contributors

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.24.6...v0.25.0

opendal - v0.24.6

Published by Xuanwo almost 2 years ago

What's Changed

New Contributors

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.24.5...v0.24.6

opendal - v0.24.5

Published by Xuanwo almost 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.24.4...v0.24.5

opendal - v0.24.4

Published by Xuanwo almost 2 years ago

What's Changed

New Contributors

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.24.3...v0.24.4

opendal - v0.24.3

Published by Xuanwo almost 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.24.2...v0.24.3

opendal - v0.24.2

Published by Xuanwo almost 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.24.1...v0.24.2

opendal - v0.24.1

Published by Xuanwo almost 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.24.0...v0.24.1

opendal - v0.24.0

Published by Xuanwo almost 2 years ago

Upgrade to v0.24

In v0.24, we made a big refactor on our internal IO-related traits. In this version, we split our IO traits into input and output versions:

Take Reader as an example:

input::Reader is the user input reader, which only requires futures::AsyncRead + Send.

output::Reader is the reader returned by OpenDAL, which implements futures::AsyncRead, futures::AsyncSeek, and futures::Stream<Item=io::Result<Bytes>>. Besides, output::Reader also implements Send + Sync, which makes it useful for users.

Due to this change, all code that depends on BytesReader should be refactored.

  • BytesReader => input::Reader
  • OutputBytesReader => output::Reader

Thanks to the change of IO trait split, we make ObjectReader implements all needed traits:

  • futures::AsyncRead
  • futures::AsyncSeek
  • futures::Stream<Item=io::Result<Bytes>>

Thus, we removed the seekable_reader API. They can be replaced by range_reader:

  • o.seekable_reader => o.range_reader

Most changes only happen inside. Users not using opendal::raw::* will not be affected.

Sorry for the inconvenience. I think those changes are required and make OpenDAL better! Welcome any comments at Discussion.

What's Changed

New Contributors

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.23.0...v0.24.0

opendal - v0.23.0

Published by Xuanwo almost 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.6...v0.23.0

opendal - v0.22.6

Published by Xuanwo almost 2 years ago

What's Changed

New Contributors

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.5...v0.22.6

opendal - v0.22.5

Published by Xuanwo almost 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.4...v0.22.5

opendal - v0.22.4

Published by Xuanwo almost 2 years ago

What's Changed

New Contributors

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.3...v0.22.4

opendal - v0.22.3

Published by Xuanwo almost 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.22.2...v0.22.3

Package Rankings
Top 3.17% on Crates.io
Top 6.51% on Proxy.golang.org
Top 2.98% on Pypi.org
Top 8.47% on Npmjs.org
Badges
Extracted from project README
Rust Core Image Docs Release Docs Dev Docs Dev Docs Dev Go Binding Image Docs Release Java Binding Image Docs Release Docs Dev Node.js Binding Image Docs Dev Python Binding Image Docs Dev oay image oli image ofs image dav-server image Docs Release Docs Dev object_store image Docs Release Docs Dev fuse3 image Docs Release Docs Dev virtiofs image Docs Release Docs Dev unftp-sbe image Docs Release Docs Dev parquet image Docs Release Docs Dev
Related Projects
arrow-rs

Official Rust implementation of Apache Arrow

17 Apr 2021 2,226
Understanding_in_Rust

【最安全的编程语言】Rust工程师枕边资料，大牛文章，开源框架，官方文档，视频，推荐书籍，学习干货，大牛语录

04 Sep 2021 1,038
opendal-go-services

Apache OpenDAL Go Binding Services Releases

13 Jul 2024 6