opendal

Apache OpenDAL: access data freely.

APACHE-2.0 License

Downloads
16.2K
Stars
3.1K
Committers
226
View Code on GitHub Visit Website
Ecosystems: Apache Spark, Cordova, Redis, Rust

Bot releases are hidden (Show)

opendal - v0.49.0 Latest Release

Published by Xuanwo 2 months ago

Release List

Name Next
core 0.49.0
integrations/dav-server 0.0.7
integrations/fuse3 0.0.4
integrations/object_store 0.46.0
integrations/parquet 0.1.0
integrations/unftp-sbe 0.0.4
bin/oay 0.41.8
bin/ofs 0.0.9
bin/oli 0.41.8
bindings/c 0.44.10
bindings/cpp 0.45.8
bindings/dotnet 0.1.6
bindings/go 0.1.0
bindings/haskell 0.44.8
bindings/java 0.47.0
bindings/lua 0.1.6
bindings/nodejs 0.47.2
bindings/php 0.1.6
bindings/python (*) 0.45.8
bindings/ruby 0.1.6

OpenDAL Core Upgrade to v0.49

Public API

Configurator now returns associated builder instead

Configurator used to return impl Builder, but now it returns associated builder type directly. This will allow users to use the builder in a more flexible way.

impl Configurator for MemoryConfig {
-    fn into_builder(self) -> impl Builder {
+    type Builder = MemoryBuilder;
+    fn into_builder(self) -> Self::Builder {
        MemoryBuilder { config: self }
    }
}

LoggingLayer now accepts LoggingInterceptor

LoggingLayer now accepts LoggingInterceptor trait instead of configuration. This change will allow users to customize the logging behavior more flexibly.

pub trait LoggingInterceptor: Debug + Clone + Send + Sync + Unpin + 'static {
    fn log(
        &self,
        info: &AccessorInfo,
        operation: Operation,
        context: &[(&str, &str)],
        message: &str,
        err: Option<&Error>,
    );
}

Users can now implement the log in the way they want.

OpenDAL Java Binding Upgrade to v0.47

Breaking change

artifactId of the opendal-java has changed from to opendal to align with the convention of entire OpenDAL project.

<dependencies>
<dependency>
  <groupId>org.apache.opendal</groupId>
-  <artifactId>opendal-java</artifactId>
+  <artifactId>opendal</artifactId>
  <version>${opendal.version}</version>
</dependency>
<dependency>
  <groupId>org.apache.opendal</groupId>
-  <artifactId>opendal-java</artifactId>
+  <artifactId>opendal</artifactId>
  <version>${opendal.version}</version>
  <classifier>${os.detected.classifier}</classifier>
</dependency>
</dependencies>

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/opendal/compare/v0.48.0...v0.49.0

opendal - v0.48.0

Published by Xuanwo 3 months ago

Release List

Name Version
core 0.48.0
integrations/cloudfilter 0.0.0
integrations/dav-server 0.0.6
integrations/fuse3 0.0.3
integrations/object_store 0.45.0
integrations/unftp-sbe 0.0.3
bin/oay 0.41.7
bin/ofs 0.0.8
bin/oli 0.41.7
bindings/c 0.44.9
bindings/cpp 0.45.7
bindings/dotnet 0.1.5
bindings/haskell 0.44.7
bindings/java 0.46.4
bindings/lua 0.1.5
bindings/nodejs 0.47.1
bindings/php 0.1.5
bindings/python 0.45.7
bindings/ruby 0.1.5

OpenDAL Core Upgrade to v0.48

Public API

Typo in customized_credential_load

Since v0.48, the customed_credential_load function has been renamed to customized_credential_load to fix the typo of customized.

- builder.customed_credential_load(v);
+ builder.customized_credential_load(v);

S3 service rename security_token to session_token

In 2014 Amazon switched from AWS_SECURITY_TOKEN to AWS_SESSION_TOKEN. To be consistent with the naming of AWS STS, we have renamed the security_token field to session_token in the S3 service.

- builder.security_token(v);
+ builder.session_token(v);

Operator from_iter and via_iter replaces from_map and via_map

Since v0.48, Operator's new APIs from_iter and via_iter methods have deprecated the from_map and via_map methods.

- Operator::from_map::<Fs>(map)?.finish();
+ Operator::from_iter::<Fs>(map)?.finish();

New API from_iter and via_iter should cover all use cases of from_map and via_map.

Service builder now takes ownership

Since v0.48, all service builder now takes ownership self instead of &mut self. This change will allow users to configure the service in a more flexible way.

- let mut builder = S3::default();
- builder.bucket("test");
- builder.root("/path/to/root");
+ let builder = S3::default().bucket("test").root("/path/to/root");
  let op = Operator::new(builder)?.finish();

Raw API

oio::Write::write will write the whole buffer

Starting from version 0.48, oio::Write::write now writes the entire buffer. This update aligns the API more closely with oio::Read::read and simplifies the implementation of concurrent writing.

  trait Write {
-     fn write(&mut self, bs: Buffer) -> impl Future<Output = Result<usize>>;
+     fn write(&mut self, bs: Buffer) -> impl Future<Output = Result<()>>;
  }

write will now return Result<()> instead of Result<usize>. The number of bytes written can be obtained from the buffer's length.

Access::metadata() will return Arc<AccessInfo>

Starting from version 0.48, Access::metadata() will return Arc<AccessInfo> instead of AccessInfo. This change is intended to improve performance and reduce memory usage.

  trait Access {
-     fn metadata(&self) -> AccessInfo;
+     fn metadata(&self) -> Arc<AccessInfo>;
  }

MinitraceLayer renamed to FastraceLayer

The MinitraceLayer has been renamed to FastraceLayer to respond to the transition from minitrace to fastrace.

- use opendal::layers::MinitraceLayer;
+ use opendal::layers::FastraceLayer;

Use Configurator to replace Builder::from_config

Since v0.48, the Builder::from_config and Builder::from_map method has been replaced by the Configurator trait. The Configurator trait provides a more flexible and extensible way to configure OpenDAL.

Service implementers should update their code to use the Configurator trait instead:

impl Configurator for MemoryConfig {
    fn into_builder(self) -> impl Builder {
        MemoryBuilder { config: self }
    }
}

impl Builder for MemoryBuilder {
    const SCHEME: Scheme = Scheme::Memory;
    type Config = MemoryConfig;

    fn build(self) -> Result<impl Access> {
        ...
    }
}

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/opendal/compare/v0.47.3...v0.48.0

opendal - v0.47.3

Published by Xuanwo 3 months ago

Release List

Name Version Next
core 0.47.2 0.47.3
integrations/dav-server 0.0.4 0.0.5
integrations/fuse3 0.0.1 0.0.2
integrations/object_store 0.44.2 0.44.3
integrations/unftp-sbe 0.0.1 0.0.2
bin/oay 0.41.5 0.41.6
bin/ofs 0.0.6 0.0.7
bin/oli 0.41.5 0.41.6
bindings/c 0.44.7 0.44.8
bindings/cpp 0.45.5 0.45.6
bindings/dotnet 0.1.3 0.1.4
bindings/haskell 0.44.5 0.44.6
bindings/java 0.46.2 0.46.3
bindings/lua 0.1.3 0.1.4
bindings/nodejs 0.47.0 0.47.0
bindings/php 0.1.3 0.1.4
bindings/python 0.45.5 0.45.6
bindings/ruby 0.1.3 0.1.4

What's Changed

Changed

Fixed

Docs

Chore

Full Changelog: https://github.com/apache/opendal/compare/v0.47.2...v0.47.3

opendal - v0.47.2

Published by Xuanwo 4 months ago

Relelase List

Name Version Next
core 0.47.1 0.47.2
integrations/cloudfilter 0.0.0 0.0.0
integrations/dav-server 0.0.3 0.0.4
integrations/fuse3 0.0.0 0.0.1
integrations/object_store 0.44.1 0.44.2
integrations/unftp-sbe 0.0.0 0.0.1
integrations/virtiofs 0.0.0 0.0.0
bin/oay 0.41.4 0.41.5
bin/ofs 0.0.5 0.0.6
bin/oli 0.41.4 0.41.5
bindings/c 0.44.6 0.44.7
bindings/cpp 0.45.4 0.45.5
bindings/dotnet 0.1.2 0.1.3
bindings/go 0.0.0 0.0.0
bindings/haskell 0.44.4 0.44.5
bindings/java 0.46.1 0.46.2
bindings/lua 0.1.2 0.1.3
bindings/nodejs 0.46.2 0.47.0
bindings/ocaml 0.0.0 0.0.0
bindings/php 0.1.2 0.1.3
bindings/python 0.45.4 0.45.5
bindings/ruby 0.1.2 0.1.3
bindings/swift 0.0.0 0.0.0
bindings/zig 0.0.0 0.0.0
  • bindings/nodejs 0.47.0 release failed, we will get it fixed in next release.

Breaking changes

Binding Node.js Public API

Now, the append operation has been removed. You can use below code instead.

op.write("path/to/file", Buffer.from("hello world"), { append: true });

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/opendal/compare/v0.47.1...v0.47.2

opendal - v0.47.1

Published by Xuanwo 4 months ago

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/opendal/compare/v0.47.0...v0.47.1

opendal - v0.47.0

Published by tisonkun 4 months ago

Core - Upgrade to v0.47

Public API

Reader into_xxx APIs

Since v0.47, Reader's into_xxx APIs requires async and returns Result instead.

- let r = op.reader("test.txt").await?.into_futures_async_read(1024..2048);
+ let r = op.reader("test.txt").await?.into_futures_async_read(1024..2048).await?;

Affected API includes:

  • Reader::into_futures_async_read
  • Reader::into_bytes_stream
  • BlockingReader::into_std_read
  • BlockingReader::into_bytes_iterator

Raw API

Bring Streaming Read Back

As explained in core: Bring Streaming Read Back, we do need read streaming back for better performance and low memory usage.

So our oio::Read changed back to streaming read instead:

trait Read {
-  async fn read(&self, offset: u64, size: usize) -> Result<Buffer>;
+  async fn read(&mut self) -> Result<Buffer>;
}

All services and layers should be updated to meet this change.

Java Binding - Upgrade to v0.46

Breaking change

PR-4641 renames async Operator to AsyncOperator and BlockingOperator to Operator.

New features

PR-4626 implements OperatorInputStream and OperatorOutputStream which implements Java's core IO abstractions InputStream and OutputStream. Users can now read/write bytes streamlined without loading/preparing the bytes fully in memory.

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/opendal/compare/v0.46.0...v0.47.0

opendal - v0.46.0

Published by tisonkun 6 months ago

Upgrade to v0.46

Public API

MSRV Changed to 1.75

Since 0.46, OpenDAL requires Rust 1.75.0 or later to use features like RPITIT and AFIT.

Services Feature Flag

Starting with version 0.46, OpenDAL only includes the memory service by default to prevent compiling unnecessary service code. To use other services, please activate their respective feature flags.

Additionally, we have removed all reqwest-related feature flags:

  • Users must now directly use reqwest's feature flags for options like rustls, native-tls, etc.
  • The rustls feature is no longer enabled by default; it must be activated manually.
  • OpenDAL no longer offers the trust-dns option; users should configure the client builder directly.

Range Based Read

Since v0.46, OpenDAL transformed it's Read IO trait to range based instead of stateful poll based IO. This change will make the IO more efficient, easier for concurrency and ready for completion based IO.

opendal::Reader now have APIs like:

let r = op.reader("test.txt").await?;
let buf = r.read(1024..2048).await?;

Buffer Based IO

Since version 0.46, OpenDAL features a native Buffer struct that supports both contiguous and non-contiguous buffers. This update enhances IO efficiency by minimizing unnecessary byte copying and enabling vectored IO.

OpenDAL's Reader will return Buffer and Writer will accept Buffer as input. Users who have implemented their own IO traits should update their code to use the new Buffer struct.

let r = op.reader("test.txt").await?;
// read returns `Buffer`
let buf: Buffer = r.read(1024..2048).await?;

let w = op.writer("test.txt").await?;

// Buffer can be created from continues bytes.
w.write("hello, world").await?;
// Buffer can also be created from non-continues bytes.
w.write(vec![Bytes::from("hello,"), Bytes::from("world!")]).await?;

// Make sure file has been written completely.
w.close().await?;

To enhance usability, we've integrated bridges into bytes::Buf and bytes::BufMut, allowing users to directly interact with the bytes API.

let r = op.reader("test.txt").await?;
let mut bs = vec![];
// read_into accepts bytes::BufMut
let buf: Buffer = r.read_into(&mut bs, 1024..2048).await?;

let w = op.writer("test.txt").await?;

// write_from accepts bytes::Buf
w.write_from("hello, world".as_bytes()).await?;

// Make sure file has been written completely.
w.close().await?;

Bridge API

OpenDAL's Reader and Writer previously implemented APIs such as AsyncRead and AsyncWrite directly. This design was not user-friendly, as it could lead to unexpected costs that users were unaware of in advance.

Since v0.46, OpenDAL provides bridge APIs for Reader and Writer instead.

let r = op.reader("test.txt").await?;

// Convert into futures AsyncRead + AsyncSeek.
let reader = r.into_futures_async_read(1024..2048);
// Convert into futures bytes stream.
let stream = r.into_bytes_stream(1024..2048);

let w = op.writer("test.txt").await?;

// Convert into futures AsyncWrite
let writer = w.into_futures_async_write();
// Convert into futures bytes sink;
let sink = w.into_bytes_sink();

Raw API

Async in IO trait

Since version 0.46, OpenDAL has adopted Rust's native async_in_trait for our core IO traits, including oio::Read, oio::Write, and oio::List.

This update eliminates the need for manually written, poll-based state machines and simplifies the codebase. Consequently, OpenDAL now requires Rust version 1.75.0 or later.

Users who have implemented their own IO traits should update their code to use the new async trait syntax.

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/opendal/compare/v0.45.1...v0.46.0

opendal - v0.45.1

Published by Xuanwo 8 months ago

Pacakages

Name Version
core 0.45.1
bin/oay 0.41.1
bin/oli 0.41.1
bin/ofs 0.0.2
bindings/python 0.45.1
bindings/nodejs 0.45.1
bindings/c 0.44.3
bindings/zig 0.0.0
bindings/dotnet 0.1.1
bindings/haskell 0.44.3
bindings/java 0.45.1
bindings/lua 0.1.1
bindings/ruby 0.1.1
bindings/swift 0.0.0
bindings/ocaml 0.0.0
bindings/php 0.1.1
bindings/cpp 0.44.3
bindings/go 0.0.0
integrations/object_store 0.43.0
integrations/dav-server 0.0.1

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/opendal/compare/v0.45.0...v0.45.1

opendal - v0.45.0

Published by morristai 9 months ago

Upgrade to v0.45

Core

Public API

BlockingLayer is not enabled by default

To further enhance the optionality of tokio, we have introduced a new feature called layers-blocking. The default usage of the blocking layer has been disabled. To utilize the BlockingLayer, please enable the layers-blocking feature.

TimeoutLayer deprecated with_speed

The with_speed API has been deprecated. Please use with_io_timeout instead.

Raw API

No raw API changes.

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/opendal/compare/v0.44.2...v0.45.0

opendal - v0.44.2

Published by Zheaoli 9 months ago

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/opendal/compare/v0.44.1...v0.44.2

opendal - v0.44.1

Published by Xuanwo 10 months ago

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.44.0...v0.44.1

opendal - v0.44.0

Published by dqhl76 10 months ago

Upgrade Note

Rust core

Public API

Moka Service Configuration

  • The thread_pool_enabled option has been removed.

List Prefix Supported

After RFC: List Prefix landed, we have changed the behavior of list a path without /. OpenDAL used to return NotADirectory error, but now we will return the list of entries that start with given prefix instead.

Nodejs binding

Public API

Now, the list operation returns Array<Entry> instead of a lister.
Also, we removed scan, you can use list('some/path', {recursive: true})/listSync('some/path', {recursive: true}) instead of scan('some/path')/scanSync('some/path').

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.43.0...v0.44.0

opendal - v0.43.0

Published by G-XD 11 months ago

Upgrade Note

Rust Core

Public API

List Recursive

After RFC-3526: List Recursive landed, we have changed the list API to accept recursive instead of delimiter:

Users will need to change the following usage:

  • op.list_with(path).delimiter("") -> op.list_with(path).recursive(true)
  • op.list_with(path).delimiter("/") -> op.list_with(path).recursive(false)

delimiter other than "" and "/" is not supported anymore.

Stat a dir path

After RFC: List Prefix landed, we have changed the behavior of stat a dir path:

Here are the behavior list:

Case Path Result
stat existing dir abc/ Metadata with dir mode
stat existing file abc/def_file Metadata with file mode
stat dir without / abc/def_dir Error NotFound or metadata with dir mode
stat file with / abc/def_file/ Error NotFound
stat not existing path xyz Error NotFound

Services like s3, azblob can handle stat("abc/") correctly by check if there are objects with prefix abc/.

Raw API

Lister Align

We changed our internal lister implementation to align with the list public API for better performance and readability.

  • trait Page => List
  • struct Pager => Lister
  • trait BlockingPage => BlockingList
  • struct BlockingPager => BlockingLister

Every call to next will return an entry instead a page of entries. Also, we changed our async list api into poll based instead of async_trait.

Java binding

Breaking change

Because of a TLS lib issue, we temporarily disable the services-ftp feature.

Node.js binding

Breaking change

Because of a TLS lib issue, we temporarily disable the services-ftp feature.

Python binding

Breaking change

Because of a TLS lib issue, we temporarily disable the services-ftp feature.

C binding

There are no API changes.

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.42.0...v0.43.0

opendal - v0.42.0

Published by silver-ymz 11 months ago

Upgrade Note

Rust Core

MSRV Changed

OpenDAL bumps it's MSRV to 1.67.0.

S3 Service Configuration

  • The enable_exact_buf_write option has been deprecated and is superseded by BufferedWriter, introduced in version 0.40.

Oss Service Configuration

  • The write_min_size option has been deprecated and replaced by BufferedWriter, also introduced in version 0.40.
  • A new setting, allow_anonymous, has been added. Since v0.41, OSS will now return an error if credential loading fails. Enabling allow_anonymous to fallback to request without credentials.

Ghac Service Configuration

  • The enable_create_simulation option has been removed. We add this option to allow ghac simulate create empty file, but it's could result in unexpected behavior when users create a file with content length 1. So we remove it.

Wasabi Service Removed

wasabi service native support has been removed. Users who want to access wasabi can use our s3 service instead.

Java binding

There are no API changes.

Node.js binding

There are no API changes.

Python binding

Breaking change for layers

Operator and BlockingOperator won't accept layers anymore. Instead, we provide a layer API:

op = opendal.Operator("memory").layer(opendal.layers.RetryLayer())

We removed not used layers ConcurrentLimitLayer and ImmutableIndexLayer along with this change.

File and AsyncFile

OpenDAL removes Reader and AsyncReader classes, instead, we provide file-like object File and AsyncFile.

Open a file for reading in blocking way:

with op.open(filename, "rb") as r:
    content = r.read()

Open a file for reading in async way:

async with await op.open(filename, "rb") as r:
    content = await r.read()

Breaking change for Errors

We remove the old error classes and provide a couple of Exception based class for the error handling.

  1. opendal.Error is based class for all the exceptions now.
  2. opendal.exceptions.Unexpected is added.
  3. opendal.exceptions.Unsupported is added.
  4. opendal.exceptions.ConfigInvalid is added.
  5. opendal.exceptions.NotFound is added.
  6. opendal.exceptions.PermissionDenied is added.
  7. opendal.exceptions.IsADirectory is added.
  8. opendal.exceptions.NotADirectory is added.
  9. opendal.exceptions.AlreadyExists is added.
  10. opendal.exceptions.IsSameFile is added.
  11. opendal.exceptions.ConditionNotMatch is added.
  12. opendal.exceptions.ContentTruncated is added.
  13. opendal.exceptions.ContentIncomplete is added.
  14. opendal.exceptions.InvalidInput is added.

C binding

The naming convention for C binding has been altered.

Struct Naming

Renaming certain struct names for consistency.

  • opendal_operator_ptr => opendal_operator
  • opendal_blocking_lister => opendal_lister
  • opendal_list_entry => opendal_entry

API Naming

We've eliminated the blocking_ prefix from our API because C binding doesn't currently support async. In the future, we plan to introduce APIs such as opendal_operator_async_write.

  • opendal_operator_blocking_write => opendal_operator_write
  • opendal_operator_blocking_read => opendal_operator_read

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.41.0...v0.42.0

opendal - v0.41.0

Published by suyanhanx about 1 year ago

Upgrade Note

Rust Core

There are no public API and raw API changes.

Java binding

Breaking change for constructing operators

PR-3166 changes the API for constructing operators:

Previous:

new BlockingOperator(scheme, config);
new Operator(scheme, config);

Current:

BlockingOperator.of(scheme, config);
Operator.of(scheme, config);

Now, there is no public constructor for operators, but only factory methods. In this way, the APIs are free to do arbitrary verifications and preparations before constructing operators.

Node.js binding

There are no API changes.

Python binding

There are no API changes.

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.40.0...v0.41.0

opendal - v0.40.0

Published by Xuanwo about 1 year ago

Checkout our OwO #1 to know more about this release!

Upgrade Note

Public API

RFC-2578 Merge Append Into Write

RFC-2578 merges append into write and removes append API.

  • For writing a file at once, please use op.write() for convenience.
  • For appending a file, please use op.write_with().append(true) instead of op.append().

The same rule applies to writer() and writer_with().

RFC-2774 Lister API

RFC-2774 proposes a new lister API to replace current list and scan. And we add a new API list to return entries directly.

  • For listing a directory at once, please use list() for convenience.
  • For listing a directory recursively, please use list_with().delimiter("") or lister_with().delimiter("") instead of scan().
  • For listing in streaming, please use lister() or lister_with() instead.

RFC-2779 List With Metakey

RFC-2779 proposes a new op.list_with().metakey() API to allow list with metakey and removes op.metadata(&entry) API.

Please use op.list_with().metakey() instead of op.metadata(&entry), for example:

// Before
let entries: Vec<Entry> = op.list("dir/").await?;
for entry in entris {
  let meta = op.metadata(&entry, Metakey::ContentLength | Metakey::ContentType).await?;
  println!("{} {}", entry.name(), entry.metadata().content_length());
}

// After
let entries: Vec<Entry> = op
  .list_with("dir/")
  .metakey(Metakey::ContentLength | Metakey::ContentType).await?;
for entry in entris {
  println!("{} {}", entry.name(), entry.metadata().content_length());
}

RFC-2852: Native Capability

RFC-2852 proposes new native_capability and full_capability API to allow users to check if the underlying service supports a capability natively.

  • native_capability returns true if the capability is supported natively.
  • full_capability returns true if the capability is supported, maybe via a layer.

Most of time, you can use full_capability to replace capability call. But if to check if the capability is supported natively for better performance design, please use native_capability instead.

Buffered Writer

OpenDAL v0.40 added buffered writer support!

Users don't need to specify the content_length() for writer anymore!

- let mut w = op.writer_with("path/to/file").content_length(1024).await?;
+ let mut w = op.writer_with("path/to/file").await?;

Users can specify the buffer() to control the size we call underlying storage:

let mut w = op.writer_with("path/to/file").buffer(8 * 1024 * 1024).await?;

If buffer is not specified, we will call underlying storage everytime we call write. Otherwise, we will make sure to call underlying storage when buffer is full or close is called.

Raw API

RFC-3017 Remove Write Copy From

RFC-3017 removes copy_from API from the oio::Write trait. Users who implements services and layers by hand should remove this API.

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.39.0...v0.40.0

opendal - v0.39.0

Published by oowl about 1 year ago

Upgrade to v0.39

Public API

Service S3 Role Arn Behavior

In PR #2687, OpenDAL changed the behavior when role_arn has been specified.

OpenDAL used to override role_arn simply. But since this version, OpenDAL will make sure to use assume_role with specified role_arn and external_id (if supplied).

RetryLayer supports RetryInterceptor

In PR #2666, RetryLayer supports RetryInterceptor. To implement this change, RetryLayer changed it's in-memory layout by adding a new generic parameter I to RetryLayer<I>.

Users who stores RetryLayer in struct or enum will need to change the type if they don't want to use default behavior.

Raw API

In PR #2698, OpenDAL re-org the internal structure of opendal::raw::oio and changed some APIs name.

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.38.1...v0.39.0

opendal - v0.38.1

Published by Xuanwo over 1 year ago

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.38.0...v0.38.1

opendal - v0.38.0

Published by PsiACE over 1 year ago

Upgrade to v0.38

There are no public API changes.

Raw API

OpenDAL add the Write::sink API to enable streaming writing. This is a breaking change for users who depend on the raw API.

For a quick fix, users who have implemented opendal::raw::oio::Write can return an Unsupported error for Write::sink().

More detailes could be found at RFC: Writer sink API.

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.37.0...v0.38.0

opendal - v0.37.0

Published by suyanhanx over 1 year ago

Upgrade to v0.37

In v0.37.0, OpenDAL bump the version of reqsign to v0.13.0.

There are no public API and raw API changes.

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.36.0...v0.37.0

Package Rankings
Top 3.17% on Crates.io
Top 6.51% on Proxy.golang.org
Top 2.98% on Pypi.org
Top 8.47% on Npmjs.org
Badges
Extracted from project README
Rust Core Image Docs Release Docs Dev Docs Dev Docs Dev Go Binding Image Docs Release Java Binding Image Docs Release Docs Dev Node.js Binding Image Docs Dev Python Binding Image Docs Dev oay image oli image ofs image dav-server image Docs Release Docs Dev object_store image Docs Release Docs Dev fuse3 image Docs Release Docs Dev virtiofs image Docs Release Docs Dev unftp-sbe image Docs Release Docs Dev parquet image Docs Release Docs Dev
Related Projects
Understanding_in_Rust

【最安全的编程语言】Rust工程师枕边资料，大牛文章，开源框架，官方文档，视频，推荐书籍，学习干货，大牛语录

04 Sep 2021 1,038
arrow-rs

Official Rust implementation of Apache Arrow

17 Apr 2021 2,226
opendal-go-services

Apache OpenDAL Go Binding Services Releases

13 Jul 2024 6