opendal

Apache OpenDAL: access data freely.

APACHE-2.0 License

Downloads
16.2K
Stars
3.1K
Committers
226
View Code on GitHub Visit Website
Ecosystems: Apache Spark, Cordova, Redis, Rust

Bot releases are visible (Hide)

opendal - v0.8.0

Published by Xuanwo over 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.7.3...v0.8.0

opendal - v0.7.3

Published by Xuanwo over 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.7.2...v0.7.3

opendal - v0.7.2

Published by Xuanwo over 2 years ago

What's Changed

New Contributors

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.7.1...v0.7.2

opendal - v0.7.1

Published by Xuanwo over 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.7.0...v0.7.1

opendal - v0.7.0

Published by Xuanwo over 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.6.3...v0.7.0

opendal - v0.6.3

Published by Xuanwo over 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.6.2...v0.6.3

opendal - v0.6.2

Published by Xuanwo over 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.6.1...v0.6.2

opendal - v0.6.1

Published by Xuanwo over 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.6.0...v0.6.1

opendal - v0.6.0

Published by Xuanwo over 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.5.2...v0.6.0

opendal - v0.5.2

Published by Xuanwo over 2 years ago

What's Changed

New Contributors

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.5.1...v0.5.2

opendal - v0.5.1

Published by Xuanwo over 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.5.0...v0.5.1

opendal - v0.5.0

Published by Xuanwo over 2 years ago

This release introduces the following new features:

Object::create API

Now, we can create an empty file or directory without a trick!

Read Object::create to know more!

Create empty file

let o = op.object("path/to/file");
let _ = o.create().await?;

Create dir

let o = op.object("path/to/dir/");
let _ = o.create().await?;

Native decompress read support

Now, we can read a compressed file natively!

Enable compress features:

opendal = {version="0.5.0", feautres=["compress"]}

Read with decompress_read() or decompress_reader()!

let o = op.object("path/to/file.gz");
let bs = o.decompress_read().await?;

Azblob support

With the help from @D2Lark, we have official support for azblob now!

Refer to azblob for more information.

// Create azblob backend builder.
let mut builder: Builder = azblob::Backend::build();
// Set the root for azblob, all operations will happen under this root.
//
// NOTE: the root must be absolute path.
builder.root("/path/to/dir");
// Set the container name, this is required.
builder.container("test");
// Set the endpoint, this is required.
//
// For examples:
// - "http://127.0.0.1:10000/devstoreaccount1"
// - "https://accountname.blob.core.windows.net"
builder.endpoint("http://127.0.0.1:10000/devstoreaccount1");
// Set the account_name and account_key.
//
// OpenDAL will try load credential from the env.
// If credential not set and no valid credential in env, OpenDAL will
// send request without signing like anonymous user.
builder.account_name("devstoreaccount1");
builder.account_key("Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==");
// Build the `Accessor`.
let accessor: Arc<dyn Accessor> = builder.finish().await?;

// `Accessor` provides the low level APIs, we will use `Operator` normally.
let op: Operator = Operator::new(accessor);

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.4.2...v0.5.0

opendal - v0.4.2

Published by Xuanwo over 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.4.1...v0.4.2

opendal - v0.4.1

Published by Xuanwo over 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.4.0...v0.4.1

opendal - v0.4.0

Published by Xuanwo over 2 years ago

What's Changed

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.3.0...v0.4.0

Refer to Upgrade From v0.3 to v0.4 section for more upgrade details.

From v0.3 to v0.4

OpenDAL introduces many breaking changes in v0.4.

Object::reader() is not AsyncSeek anymore

Since v0.4, Object::reader() will return impl BytesRead instead of Reader that implements AsyncRead and AsyncSeek. Users who want AsyncSeek please wrapped with opendal::io_util::seekable_read:

use opendal::io_util::seekable_read;

let o = op.object("test");
let mut r = seekable_read(&o, 10..);
r.seek(SeekFrom::Current(10)).await?;
let mut bs = vec![0;10];
r.read(&mut bs).await?;

Use RangeBounds instead

Since v0.4, the following APIs will be removed.

  • Object::limited_reader(size: u64)
  • Object::offset_reader(offset: u64)
  • Object::range_reader(offset: u64, size: u64)

Instead, OpenDAL is providing a more general range_reader powered by RangeBounds:

pub async fn range_reader(&self, range: impl RangeBounds<u64>) -> Result<impl BytesRead>

Users can use their familiar rust range syntax:

let r = o.range_reader(1024..2048).await?;

Return io::Result instead

Since v0.4, all functions in OpenDAL will return std::io::Result instead.

Please check via std::io::ErrorKind directly:

use std::io::ErrorKind;

if let Err(e) = op.object("test_file").metadata().await {
    if e.kind() == ErrorKind::NotFound {
        println!("object not exist")
    }
}

Removing Credential

Since v0.4, Credential has been removed, please use the API provided by Builder directly.

builder.access_key_id("access_key_id");
builder.secret_access_key("secret_access_key");

Write returns BytesWriter instead

Since v0.4, Accessor::write will return a BytesWriter instead accepting a BoxedAsyncReader.

Along with this change, the old Writer has been replaced by a new set of write functions:

pub async fn write(&self, bs: impl AsRef<[u8]>) -> Result<()> {}
pub async fn writer(&self, size: u64) -> Result<impl BytesWrite> {}

Users can write into an object more easily:

let _ = op.object("path/to/file").write("Hello, World!").await?;

io_util replaces readers

Since v0.4, mod io_util will replace readers. In io_utils, OpenDAL provides helpful functions like:

  • into_reader: Convert BytesStream into BytesRead
  • into_sink: Convert BytesWrite into BytesSink
  • into_stream: Convert BytesRead into BytesStream
  • into_writer: Convert BytesSink into BytesWrite
  • observe_read: Add callback for BytesReader
  • observe_write: Add callback for BytesWrite

New type alias

For better naming, types that OpenDAL returns have been renamed:

  • AsyncRead + Unpin + Send => BytesRead
  • BoxedAsyncReader => BytesReader
  • AsyncWrite + Unpin + Send => BytesWrite
  • BoxedAsyncWriter => BytesWriter
  • ObjectStream => ObjectStreamer
opendal - v0.3.0

Published by Xuanwo over 2 years ago

What's Changed

New Contributors

Full Changelog: https://github.com/datafuselabs/opendal/compare/v0.2.5...v0.3.0

opendal -

Published by Xuanwo over 2 years ago

v0.2.5 - 2022-03-22

Added

  • feat: Adopt quick_xml to parse xml (#164)
  • test: Add behavior test for not exist object (#166)
  • feat: Allow user input region (#168)

Changed

  • feat: Improve error handle for s3 service (#169)
  • feat: Read error response for better debugging (#170)
  • examples: Improve examples for s3 (#171)
opendal -

Published by Xuanwo over 2 years ago

v0.2.4 - 2022-03-18

Added

  • feat: Add content_md5 and last_modified in metadata (#158)

Changed

  • refactor: Say goodbye to aws-s3-sdk (#152)
opendal -

Published by Xuanwo over 2 years ago

v0.2.3 - 2022-03-14

Added

  • feat: Export BoxedObjectStream so that users can implement Layer (#147)
opendal -

Published by Xuanwo over 2 years ago

v0.2.2 - 2022-03-14

Fixed

  • services/fs: Refactor via tokio::fs (#142)
  • fix: Stat root should return a dir object (#143)
Package Rankings
Top 3.17% on Crates.io
Top 6.51% on Proxy.golang.org
Top 2.98% on Pypi.org
Top 8.47% on Npmjs.org
Badges
Extracted from project README
Rust Core Image Docs Release Docs Dev Docs Dev Docs Dev Go Binding Image Docs Release Java Binding Image Docs Release Docs Dev Node.js Binding Image Docs Dev Python Binding Image Docs Dev oay image oli image ofs image dav-server image Docs Release Docs Dev object_store image Docs Release Docs Dev fuse3 image Docs Release Docs Dev virtiofs image Docs Release Docs Dev unftp-sbe image Docs Release Docs Dev parquet image Docs Release Docs Dev
Related Projects
arrow-rs

Official Rust implementation of Apache Arrow

17 Apr 2021 2,226
opendal-go-services

Apache OpenDAL Go Binding Services Releases

13 Jul 2024 6
Understanding_in_Rust

【最安全的编程语言】Rust工程师枕边资料，大牛文章，开源框架，官方文档，视频，推荐书籍，学习干货，大牛语录

04 Sep 2021 1,038