Apache OpenDAL: access data freely.
APACHE-2.0 License
Bot releases are hidden (Show)
Name | Next |
---|---|
core | 0.49.0 |
integrations/dav-server | 0.0.7 |
integrations/fuse3 | 0.0.4 |
integrations/object_store | 0.46.0 |
integrations/parquet | 0.1.0 |
integrations/unftp-sbe | 0.0.4 |
bin/oay | 0.41.8 |
bin/ofs | 0.0.9 |
bin/oli | 0.41.8 |
bindings/c | 0.44.10 |
bindings/cpp | 0.45.8 |
bindings/dotnet | 0.1.6 |
bindings/go | 0.1.0 |
bindings/haskell | 0.44.8 |
bindings/java | 0.47.0 |
bindings/lua | 0.1.6 |
bindings/nodejs | 0.47.2 |
bindings/php | 0.1.6 |
bindings/python (*) | 0.45.8 |
bindings/ruby | 0.1.6 |
0.45.8
, but due to https://github.com/apache/opendal/issues/5000, we were unable to. We will release it next time.
Configurator
now returns associated builder insteadConfigurator
used to return impl Builder
, but now it returns associated builder type directly. This will allow users to use the builder in a more flexible way.
impl Configurator for MemoryConfig {
- fn into_builder(self) -> impl Builder {
+ type Builder = MemoryBuilder;
+ fn into_builder(self) -> Self::Builder {
MemoryBuilder { config: self }
}
}
LoggingLayer
now accepts LoggingInterceptor
LoggingLayer
now accepts LoggingInterceptor
trait instead of configuration. This change will allow users to customize the logging behavior more flexibly.
pub trait LoggingInterceptor: Debug + Clone + Send + Sync + Unpin + 'static {
fn log(
&self,
info: &AccessorInfo,
operation: Operation,
context: &[(&str, &str)],
message: &str,
err: Option<&Error>,
);
}
Users can now implement the log in the way they want.
artifactId of the opendal-java
has changed from to opendal
to align with the convention of entire OpenDAL project.
<dependencies>
<dependency>
<groupId>org.apache.opendal</groupId>
- <artifactId>opendal-java</artifactId>
+ <artifactId>opendal</artifactId>
<version>${opendal.version}</version>
</dependency>
<dependency>
<groupId>org.apache.opendal</groupId>
- <artifactId>opendal-java</artifactId>
+ <artifactId>opendal</artifactId>
<version>${opendal.version}</version>
<classifier>${os.detected.classifier}</classifier>
</dependency>
</dependencies>
put_multipart
in object_store
by @Rachelint in https://github.com/apache/opendal/pull/4793
AsyncWriter
for parquet integrations by @WenyXu in https://github.com/apache/opendal/pull/4958
AsyncReader
for parquet integrations by @WenyXu in https://github.com/apache/opendal/pull/4972
allow_anonymous
by @jdockerty in https://github.com/apache/opendal/pull/4979
Full Changelog: https://github.com/apache/opendal/compare/v0.48.0...v0.49.0
Published by Xuanwo 3 months ago
Name | Version |
---|---|
core | 0.48.0 |
integrations/cloudfilter | 0.0.0 |
integrations/dav-server | 0.0.6 |
integrations/fuse3 | 0.0.3 |
integrations/object_store | 0.45.0 |
integrations/unftp-sbe | 0.0.3 |
bin/oay | 0.41.7 |
bin/ofs | 0.0.8 |
bin/oli | 0.41.7 |
bindings/c | 0.44.9 |
bindings/cpp | 0.45.7 |
bindings/dotnet | 0.1.5 |
bindings/haskell | 0.44.7 |
bindings/java | 0.46.4 |
bindings/lua | 0.1.5 |
bindings/nodejs | 0.47.1 |
bindings/php | 0.1.5 |
bindings/python | 0.45.7 |
bindings/ruby | 0.1.5 |
customized_credential_load
Since v0.48, the customed_credential_load
function has been renamed to customized_credential_load
to fix the typo of customized
.
- builder.customed_credential_load(v);
+ builder.customized_credential_load(v);
security_token
to session_token
In 2014 Amazon switched from AWS_SECURITY_TOKEN
to AWS_SESSION_TOKEN
. To be consistent with the naming of AWS STS, we have renamed the security_token
field to session_token
in the S3 service.
- builder.security_token(v);
+ builder.session_token(v);
from_iter
and via_iter
replaces from_map
and via_map
Since v0.48, Operator's new APIs from_iter
and via_iter
methods have deprecated the from_map
and via_map
methods.
- Operator::from_map::<Fs>(map)?.finish();
+ Operator::from_iter::<Fs>(map)?.finish();
New API from_iter
and via_iter
should cover all use cases of from_map
and via_map
.
Since v0.48, all service builder now takes ownership self
instead of &mut self
. This change will allow users to configure the service in a more flexible way.
- let mut builder = S3::default();
- builder.bucket("test");
- builder.root("/path/to/root");
+ let builder = S3::default().bucket("test").root("/path/to/root");
let op = Operator::new(builder)?.finish();
oio::Write::write
will write the whole bufferStarting from version 0.48, oio::Write::write
now writes the entire buffer. This update aligns the API more closely with oio::Read::read
and simplifies the implementation of concurrent writing.
trait Write {
- fn write(&mut self, bs: Buffer) -> impl Future<Output = Result<usize>>;
+ fn write(&mut self, bs: Buffer) -> impl Future<Output = Result<()>>;
}
write
will now return Result<()>
instead of Result<usize>
. The number of bytes written can be obtained from the buffer's length.
Access::metadata()
will return Arc<AccessInfo>
Starting from version 0.48, Access::metadata()
will return Arc<AccessInfo>
instead of AccessInfo
. This change is intended to improve performance and reduce memory usage.
trait Access {
- fn metadata(&self) -> AccessInfo;
+ fn metadata(&self) -> Arc<AccessInfo>;
}
MinitraceLayer
renamed to FastraceLayer
The MinitraceLayer
has been renamed to FastraceLayer
to respond to the transition from minitrace
to fastrace
.
- use opendal::layers::MinitraceLayer;
+ use opendal::layers::FastraceLayer;
Configurator
to replace Builder::from_config
Since v0.48, the Builder::from_config
and Builder::from_map
method has been replaced by the Configurator
trait. The Configurator
trait provides a more flexible and extensible way to configure OpenDAL.
Service implementers should update their code to use the Configurator
trait instead:
impl Configurator for MemoryConfig {
fn into_builder(self) -> impl Builder {
MemoryBuilder { config: self }
}
}
impl Builder for MemoryBuilder {
const SCHEME: Scheme = Scheme::Memory;
type Config = MemoryConfig;
fn build(self) -> Result<impl Access> {
...
}
}
Buffer::to_bytes()
by cloning contiguous bytes by @LDeakin in https://github.com/apache/opendal/pull/4858
root
on using PrometheusClientLayer by @flaneur2020 in https://github.com/apache/opendal/pull/4907
Content-Type
based on the extension in the path. by @czy-29 in https://github.com/apache/opendal/pull/4912
Arc<AccessInfo>
for metadata by @Lzzzzzt in https://github.com/apache/opendal/pull/4883
MimeGuessLayer
. by @czy-29 in https://github.com/apache/opendal/pull/4916
Full Changelog: https://github.com/apache/opendal/compare/v0.47.3...v0.48.0
Published by Xuanwo 3 months ago
Name | Version | Next |
---|---|---|
core | 0.47.2 | 0.47.3 |
integrations/dav-server | 0.0.4 | 0.0.5 |
integrations/fuse3 | 0.0.1 | 0.0.2 |
integrations/object_store | 0.44.2 | 0.44.3 |
integrations/unftp-sbe | 0.0.1 | 0.0.2 |
bin/oay | 0.41.5 | 0.41.6 |
bin/ofs | 0.0.6 | 0.0.7 |
bin/oli | 0.41.5 | 0.41.6 |
bindings/c | 0.44.7 | 0.44.8 |
bindings/cpp | 0.45.5 | 0.45.6 |
bindings/dotnet | 0.1.3 | 0.1.4 |
bindings/haskell | 0.44.5 | 0.44.6 |
bindings/java | 0.46.2 | 0.46.3 |
bindings/lua | 0.1.3 | 0.1.4 |
bindings/nodejs | 0.47.0 | 0.47.0 |
bindings/php | 0.1.3 | 0.1.4 |
bindings/python | 0.45.5 | 0.45.6 |
bindings/ruby | 0.1.3 | 0.1.4 |
oio::Write
. by @yuchanns in https://github.com/apache/opendal/pull/4821
Full Changelog: https://github.com/apache/opendal/compare/v0.47.2...v0.47.3
Published by Xuanwo 4 months ago
Name | Version | Next |
---|---|---|
core | 0.47.1 | 0.47.2 |
integrations/cloudfilter | 0.0.0 | 0.0.0 |
integrations/dav-server | 0.0.3 | 0.0.4 |
integrations/fuse3 | 0.0.0 | 0.0.1 |
integrations/object_store | 0.44.1 | 0.44.2 |
integrations/unftp-sbe | 0.0.0 | 0.0.1 |
integrations/virtiofs | 0.0.0 | 0.0.0 |
bin/oay | 0.41.4 | 0.41.5 |
bin/ofs | 0.0.5 | 0.0.6 |
bin/oli | 0.41.4 | 0.41.5 |
bindings/c | 0.44.6 | 0.44.7 |
bindings/cpp | 0.45.4 | 0.45.5 |
bindings/dotnet | 0.1.2 | 0.1.3 |
bindings/go | 0.0.0 | 0.0.0 |
bindings/haskell | 0.44.4 | 0.44.5 |
bindings/java | 0.46.1 | 0.46.2 |
bindings/lua | 0.1.2 | 0.1.3 |
bindings/nodejs | 0.46.2 | 0.47.0 |
bindings/ocaml | 0.0.0 | 0.0.0 |
bindings/php | 0.1.2 | 0.1.3 |
bindings/python | 0.45.4 | 0.45.5 |
bindings/ruby | 0.1.2 | 0.1.3 |
bindings/swift | 0.0.0 | 0.0.0 |
bindings/zig | 0.0.0 | 0.0.0 |
Now, the append
operation has been removed. You can use below code instead.
op.write("path/to/file", Buffer.from("hello world"), { append: true });
Access
impl by @George-Miao in https://github.com/apache/opendal/pull/4693
OpendalStorage
by @George-Miao in https://github.com/apache/opendal/pull/4765
WriteOptions
for write methods by @bxb100 in https://github.com/apache/opendal/pull/4785
OpendalMetadata
by @George-Miao in https://github.com/apache/opendal/pull/4819
MadsimLayer
by @zzzk1 in https://github.com/apache/opendal/pull/4788
Full Changelog: https://github.com/apache/opendal/compare/v0.47.1...v0.47.2
Published by Xuanwo 4 months ago
size
for RangeWrite
. by @reswqa in https://github.com/apache/opendal/pull/4755
/
by @yuchanns in https://github.com/apache/opendal/pull/4754
Full Changelog: https://github.com/apache/opendal/compare/v0.47.0...v0.47.1
Published by tisonkun 4 months ago
into_xxx
APIsSince v0.47, Reader
's into_xxx
APIs requires async
and returns Result
instead.
- let r = op.reader("test.txt").await?.into_futures_async_read(1024..2048);
+ let r = op.reader("test.txt").await?.into_futures_async_read(1024..2048).await?;
Affected API includes:
Reader::into_futures_async_read
Reader::into_bytes_stream
BlockingReader::into_std_read
BlockingReader::into_bytes_iterator
As explained in core: Bring Streaming Read Back, we do need read streaming back for better performance and low memory usage.
So our oio::Read
changed back to streaming read instead:
trait Read {
- async fn read(&self, offset: u64, size: usize) -> Result<Buffer>;
+ async fn read(&mut self) -> Result<Buffer>;
}
All services and layers should be updated to meet this change.
PR-4641 renames async Operator
to AsyncOperator
and BlockingOperator
to Operator
.
PR-4626 implements OperatorInputStream
and OperatorOutputStream
which implements Java's core IO abstractions InputStream
and OutputStream
. Users can now read/write bytes streamlined without loading/preparing the bytes fully in memory.
AccessorDyn
by @morristai in https://github.com/apache/opendal/pull/4574
Full Changelog: https://github.com/apache/opendal/compare/v0.46.0...v0.47.0
Published by tisonkun 6 months ago
Since 0.46, OpenDAL requires Rust 1.75.0 or later to use features like RPITIT
and AFIT
.
Starting with version 0.46, OpenDAL only includes the memory service by default to prevent compiling unnecessary service code. To use other services, please activate their respective feature flags.
Additionally, we have removed all reqwest
-related feature flags:
reqwest
's feature flags for options like rustls
, native-tls
, etc.rustls
feature is no longer enabled by default; it must be activated manually.trust-dns
option; users should configure the client builder directly.Since v0.46, OpenDAL transformed it's Read IO trait to range based instead of stateful poll based IO. This change will make the IO more efficient, easier for concurrency and ready for completion based IO.
opendal::Reader
now have APIs like:
let r = op.reader("test.txt").await?;
let buf = r.read(1024..2048).await?;
Since version 0.46, OpenDAL features a native Buffer
struct that supports both contiguous and non-contiguous buffers. This update enhances IO efficiency by minimizing unnecessary byte copying and enabling vectored IO.
OpenDAL's Reader
will return Buffer
and Writer
will accept Buffer
as input. Users who have implemented their own IO traits should update their code to use the new Buffer
struct.
let r = op.reader("test.txt").await?;
// read returns `Buffer`
let buf: Buffer = r.read(1024..2048).await?;
let w = op.writer("test.txt").await?;
// Buffer can be created from continues bytes.
w.write("hello, world").await?;
// Buffer can also be created from non-continues bytes.
w.write(vec![Bytes::from("hello,"), Bytes::from("world!")]).await?;
// Make sure file has been written completely.
w.close().await?;
To enhance usability, we've integrated bridges into bytes::Buf
and bytes::BufMut
, allowing users to directly interact with the bytes API.
let r = op.reader("test.txt").await?;
let mut bs = vec![];
// read_into accepts bytes::BufMut
let buf: Buffer = r.read_into(&mut bs, 1024..2048).await?;
let w = op.writer("test.txt").await?;
// write_from accepts bytes::Buf
w.write_from("hello, world".as_bytes()).await?;
// Make sure file has been written completely.
w.close().await?;
OpenDAL's Reader
and Writer
previously implemented APIs such as AsyncRead
and AsyncWrite
directly. This design was not user-friendly, as it could lead to unexpected costs that users were unaware of in advance.
Since v0.46, OpenDAL provides bridge APIs for Reader
and Writer
instead.
let r = op.reader("test.txt").await?;
// Convert into futures AsyncRead + AsyncSeek.
let reader = r.into_futures_async_read(1024..2048);
// Convert into futures bytes stream.
let stream = r.into_bytes_stream(1024..2048);
let w = op.writer("test.txt").await?;
// Convert into futures AsyncWrite
let writer = w.into_futures_async_write();
// Convert into futures bytes sink;
let sink = w.into_bytes_sink();
Since version 0.46, OpenDAL has adopted Rust's native async_in_trait
for our core IO traits, including oio::Read
, oio::Write
, and oio::List
.
This update eliminates the need for manually written, poll-based state machines and simplifies the codebase. Consequently, OpenDAL now requires Rust version 1.75.0 or later.
Users who have implemented their own IO traits should update their code to use the new async trait syntax.
Arc::from_iter
by @Xuanwo in https://github.com/apache/opendal/pull/4440
HttpClient::with()
constructor by @waynexia in https://github.com/apache/opendal/pull/4447
Buffer
as cache in RangeWrite
by @reswqa in https://github.com/apache/opendal/pull/4476
Buffer
as cache in OneshotWrite
by @reswqa in https://github.com/apache/opendal/pull/4477
Buffer
by @George-Miao in https://github.com/apache/opendal/pull/4530
IoBuf
for Buffer
by @George-Miao in https://github.com/apache/opendal/pull/4532
oio::Read
to async in trait by @Xuanwo in https://github.com/apache/opendal/pull/4336
oio::List
to async fn in trait by @Xuanwo in https://github.com/apache/opendal/pull/4352
oio::Write
to async in trait by @Xuanwo in https://github.com/apache/opendal/pull/4358
File::read
to PyResult<&PyBytes>
by @reswqa in https://github.com/apache/opendal/pull/4360
oio::WriteBuf
and oio::ChunkedBytes
after refactor by @Xuanwo in https://github.com/apache/opendal/pull/4361
ofs
API public by @ho-229 in https://github.com/apache/opendal/pull/4387
Accessor
to async fn in trait by @George-Miao in https://github.com/apache/opendal/pull/4562
Accessor
by @George-Miao in https://github.com/apache/opendal/pull/4564
size
argument for python File::read
by @reswqa in https://github.com/apache/opendal/pull/4359
File::write
should return the written bytes by @reswqa in https://github.com/apache/opendal/pull/4367
WebdavFile
continuous reading and writing by @G-XD in https://github.com/apache/opendal/pull/4374
service-*
from default features by @xxchan in https://github.com/apache/opendal/pull/4311
yaml
file type by @jbampton in https://github.com/apache/opendal/pull/4339
rdf
file type as indent_size = 2
by @jbampton in https://github.com/apache/opendal/pull/4341
indent_style = tab
for Go by @jbampton in https://github.com/apache/opendal/pull/4342
Full Changelog: https://github.com/apache/opendal/compare/v0.45.1...v0.46.0
Published by Xuanwo 8 months ago
Name | Version |
---|---|
core | 0.45.1 |
bin/oay | 0.41.1 |
bin/oli | 0.41.1 |
bin/ofs | 0.0.2 |
bindings/python | 0.45.1 |
bindings/nodejs | 0.45.1 |
bindings/c | 0.44.3 |
bindings/zig | 0.0.0 |
bindings/dotnet | 0.1.1 |
bindings/haskell | 0.44.3 |
bindings/java | 0.45.1 |
bindings/lua | 0.1.1 |
bindings/ruby | 0.1.1 |
bindings/swift | 0.0.0 |
bindings/ocaml | 0.0.0 |
bindings/php | 0.1.1 |
bindings/cpp | 0.44.3 |
bindings/go | 0.0.0 |
integrations/object_store | 0.43.0 |
integrations/dav-server | 0.0.1 |
=
by @Xuanwo in https://github.com/apache/opendal/pull/4105
core/src/services/http/docs.md
by @jbampton in https://github.com/apache/opendal/pull/4226
website/README.md
by @jbampton in https://github.com/apache/opendal/pull/4228
github
to GitHub
by @jbampton in https://github.com/apache/opendal/pull/4232
bindings/c/CONTRIBUTING.md
by @jbampton in https://github.com/apache/opendal/pull/4235
CONTRIBUTING
by @jbampton in https://github.com/apache/opendal/pull/4237
bindings/README.md
by @jbampton in https://github.com/apache/opendal/pull/4238
examples/rust
by @jbampton in https://github.com/apache/opendal/pull/4241
Who is using OpenDAL
list by @jbampton in https://github.com/apache/opendal/pull/4263
Full Changelog: https://github.com/apache/opendal/compare/v0.45.0...v0.45.1
Published by morristai 9 months ago
To further enhance the optionality of tokio
, we have introduced a new feature called layers-blocking
. The default usage of the blocking layer has been disabled. To utilize the BlockingLayer
, please enable the layers-blocking
feature.
with_speed
The with_speed
API has been deprecated. Please use with_io_timeout
instead.
No raw API changes.
IntoFuture
for operator futures to remove an alloc by @Xuanwo in https://github.com/apache/opendal/pull/4098
Full Changelog: https://github.com/apache/opendal/compare/v0.44.2...v0.45.0
Published by Zheaoli 9 months ago
MultipartUploadWriter
by @WenyXu in https://github.com/apache/opendal/pull/3915
RangeWriter
by @WenyXu in https://github.com/apache/opendal/pull/3923
concurrent
and buffer
parameters into FuzzInput by @WenyXu in https://github.com/apache/opendal/pull/3921
next_offset
by @WenyXu in https://github.com/apache/opendal/pull/3927
Full Changelog: https://github.com/apache/opendal/compare/v0.44.1...v0.44.2
Published by Xuanwo 10 months ago
CompleteReader
by @WenyXu in https://github.com/apache/incubator-opendal/pull/3861
bindings_python.yml
by @messense in https://github.com/apache/incubator-opendal/pull/3837
BufferReader
by @WenyXu in https://github.com/apache/incubator-opendal/pull/3864
cur
of BufferReader
by @WenyXu in https://github.com/apache/incubator-opendal/pull/3865
Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.44.0...v0.44.1
Published by dqhl76 10 months ago
thread_pool_enabled
option has been removed.After RFC: List Prefix landed, we have changed the behavior of list
a path without /
. OpenDAL used to return NotADirectory
error, but now we will return the list of entries that start with given prefix instead.
Now, the list
operation returns Array<Entry>
instead of a lister.
Also, we removed scan
, you can use list('some/path', {recursive: true})
/listSync('some/path', {recursive: true})
instead of scan('some/path')
/scanSync('some/path')
.
actual
& expected
to pass ci by @Young-Flash in https://github.com/apache/incubator-opendal/pull/3733
get_file_id_by_path
by @G-XD in https://github.com/apache/incubator-opendal/pull/3801
Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.43.0...v0.44.0
Published by G-XD 11 months ago
After RFC-3526: List Recursive landed, we have changed the list
API to accept recursive
instead of delimiter
:
Users will need to change the following usage:
op.list_with(path).delimiter("")
-> op.list_with(path).recursive(true)
op.list_with(path).delimiter("/")
-> op.list_with(path).recursive(false)
delimiter
other than ""
and "/"
is not supported anymore.
After RFC: List Prefix landed, we have changed the behavior of stat
a dir path:
Here are the behavior list:
Case | Path | Result |
---|---|---|
stat existing dir | abc/ |
Metadata with dir mode |
stat existing file | abc/def_file |
Metadata with file mode |
stat dir without /
|
abc/def_dir |
Error NotFound or metadata with dir mode |
stat file with /
|
abc/def_file/ |
Error NotFound
|
stat not existing path | xyz |
Error NotFound
|
Services like s3, azblob can handle stat("abc/")
correctly by check if there are objects with prefix abc/
.
We changed our internal lister
implementation to align with the list
public API for better performance and readability.
Page
=> List
Pager
=> Lister
BlockingPage
=> BlockingList
BlockingPager
=> BlockingLister
Every call to next
will return an entry instead a page of entries. Also, we changed our async list api into poll based instead of async_trait
.
Because of a TLS lib issue, we temporarily disable the services-ftp
feature.
Because of a TLS lib issue, we temporarily disable the services-ftp
feature.
Because of a TLS lib issue, we temporarily disable the services-ftp
feature.
There are no API changes.
ftp
for python and java binding by @ZutJoe in https://github.com/apache/incubator-opendal/pull/3659
Page
to List
by @Xuanwo in https://github.com/apache/incubator-opendal/pull/3589
raw::tests
been excluded unexpectedly by @Xuanwo in https://github.com/apache/incubator-opendal/pull/3623
stat
and stat_with
by @Xuanwo in https://github.com/apache/incubator-opendal/pull/3657
Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.42.0...v0.43.0
Published by silver-ymz 11 months ago
OpenDAL bumps it's MSRV to 1.67.0.
enable_exact_buf_write
option has been deprecated and is superseded by BufferedWriter
, introduced in version 0.40.write_min_size
option has been deprecated and replaced by BufferedWriter
, also introduced in version 0.40.allow_anonymous
, has been added. Since v0.41, OSS will now return an error if credential loading fails. Enabling allow_anonymous
to fallback to request without credentials.enable_create_simulation
option has been removed. We add this option to allow ghac simulate create empty file, but it's could result in unexpected behavior when users create a file with content length 1
. So we remove it.wasabi
service native support has been removed. Users who want to access wasabi can use our s3
service instead.
There are no API changes.
There are no API changes.
Operator and BlockingOperator won't accept layers
anymore. Instead, we provide a layer
API:
op = opendal.Operator("memory").layer(opendal.layers.RetryLayer())
We removed not used layers ConcurrentLimitLayer
and ImmutableIndexLayer
along with this change.
OpenDAL removes Reader
and AsyncReader
classes, instead, we provide file-like object File
and AsyncFile
.
Open a file for reading in blocking way:
with op.open(filename, "rb") as r:
content = r.read()
Open a file for reading in async way:
async with await op.open(filename, "rb") as r:
content = await r.read()
We remove the old error classes and provide a couple of Exception based class for the error handling.
opendal.Error
is based class for all the exceptions now.opendal.exceptions.Unexpected
is added.opendal.exceptions.Unsupported
is added.opendal.exceptions.ConfigInvalid
is added.opendal.exceptions.NotFound
is added.opendal.exceptions.PermissionDenied
is added.opendal.exceptions.IsADirectory
is added.opendal.exceptions.NotADirectory
is added.opendal.exceptions.AlreadyExists
is added.opendal.exceptions.IsSameFile
is added.opendal.exceptions.ConditionNotMatch
is added.opendal.exceptions.ContentTruncated
is added.opendal.exceptions.ContentIncomplete
is added.opendal.exceptions.InvalidInput
is added.The naming convention for C binding has been altered.
Renaming certain struct names for consistency.
opendal_operator_ptr
=> opendal_operator
opendal_blocking_lister
=> opendal_lister
opendal_list_entry
=> opendal_entry
We've eliminated the blocking_
prefix from our API because C binding doesn't currently support async. In the future, we plan to introduce APIs such as opendal_operator_async_write
.
opendal_operator_blocking_write
=> opendal_operator_write
opendal_operator_blocking_read
=> opendal_operator_read
rename
support by @G-XD in https://github.com/apache/incubator-opendal/pull/3238
Metadata
a POJO by @G-XD in https://github.com/apache/incubator-opendal/pull/3277
memoryview
instead of bytes
to avoid copy by @messense in https://github.com/apache/incubator-opendal/pull/3310
Metadata::metakey()
api by @G-XD in https://github.com/apache/incubator-opendal/pull/3373
list_with
in Operator::list
by @G-XD in https://github.com/apache/incubator-opendal/pull/3305
OPENDAL_DISABLE_RANDOM_ROOT
by @suyanhanx in https://github.com/apache/incubator-opendal/pull/3456
connection_string
config description in doc by @xring in https://github.com/apache/incubator-opendal/pull/3388
range_reader
change in upgrade doc by @wcy-fdu in https://github.com/apache/incubator-opendal/pull/3401
testWriteFileWithNonAsciiName
behavior test by @G-XD in https://github.com/apache/incubator-opendal/pull/3424
linux-aarch_64
by @amunra in https://github.com/apache/incubator-opendal/pull/3527
Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.41.0...v0.42.0
Published by suyanhanx about 1 year ago
There are no public API and raw API changes.
PR-3166 changes the API for constructing operators:
Previous:
new BlockingOperator(scheme, config);
new Operator(scheme, config);
Current:
BlockingOperator.of(scheme, config);
Operator.of(scheme, config);
Now, there is no public constructor for operators, but only factory methods. In this way, the APIs are free to do arbitrary verifications and preparations before constructing operators.
There are no API changes.
There are no API changes.
prometheus-client
crate with PrometheusClientLayer by @flaneur2020 in https://github.com/apache/incubator-opendal/pull/3134
BlockingLayer
for non-blocking services that don't support blocking by @messense in https://github.com/apache/incubator-opendal/pull/3198
copy
test by @G-XD in https://github.com/apache/incubator-opendal/pull/3207
WebdavFs
to dav-server-fs-opendal by @Young-Flash in https://github.com/apache/incubator-opendal/pull/3119
Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.40.0...v0.41.0
Published by Xuanwo about 1 year ago
Checkout our OwO #1 to know more about this release!
RFC-2578 merges append
into write
and removes append
API.
op.write()
for convenience.op.write_with().append(true)
instead of op.append()
.The same rule applies to writer()
and writer_with()
.
RFC-2774 proposes a new lister
API to replace current list
and scan
. And we add a new API list
to return entries directly.
list()
for convenience.list_with().delimiter("")
or lister_with().delimiter("")
instead of scan()
.lister()
or lister_with()
instead.RFC-2779 proposes a new op.list_with().metakey()
API to allow list with metakey and removes op.metadata(&entry)
API.
Please use op.list_with().metakey()
instead of op.metadata(&entry)
, for example:
// Before
let entries: Vec<Entry> = op.list("dir/").await?;
for entry in entris {
let meta = op.metadata(&entry, Metakey::ContentLength | Metakey::ContentType).await?;
println!("{} {}", entry.name(), entry.metadata().content_length());
}
// After
let entries: Vec<Entry> = op
.list_with("dir/")
.metakey(Metakey::ContentLength | Metakey::ContentType).await?;
for entry in entris {
println!("{} {}", entry.name(), entry.metadata().content_length());
}
RFC-2852 proposes new native_capability
and full_capability
API to allow users to check if the underlying service supports a capability natively.
native_capability
returns true
if the capability is supported natively.full_capability
returns true
if the capability is supported, maybe via a layer.Most of time, you can use full_capability
to replace capability
call. But if to check if the capability is supported natively for better performance design, please use native_capability
instead.
OpenDAL v0.40 added buffered writer support!
Users don't need to specify the content_length()
for writer anymore!
- let mut w = op.writer_with("path/to/file").content_length(1024).await?;
+ let mut w = op.writer_with("path/to/file").await?;
Users can specify the buffer()
to control the size we call underlying storage:
let mut w = op.writer_with("path/to/file").buffer(8 * 1024 * 1024).await?;
If buffer is not specified, we will call underlying storage everytime we call write
. Otherwise, we will make sure to call underlying storage when buffer is full or close
is called.
RFC-3017 removes copy_from
API from the oio::Write
trait. Users who implements services and layers by hand should remove this API.
read_dir
by @Young-Flash in https://github.com/apache/incubator-opendal/pull/2736
OpMonad
to support custom IO monad by @silver-ymz in https://github.com/apache/incubator-opendal/pull/2789
WebdavMetaData
by @Young-Flash in https://github.com/apache/incubator-opendal/pull/2857
_pin
with !Unpin
as argument by @morristai in https://github.com/apache/incubator-opendal/pull/2886
content_length
to writer by @Xuanwo in https://github.com/apache/incubator-opendal/pull/3044
parse_batch_delete_response
with Multipart::parse
by @G-XD in https://github.com/apache/incubator-opendal/pull/3071
webdav_put
signatures by using OpWrite
. by @laipz8200 in https://github.com/apache/incubator-opendal/pull/3076
webdav_get
. by @laipz8200 in https://github.com/apache/incubator-opendal/pull/3081
oss_put_object
signatures by using OpWrite by @sysu-yunz in https://github.com/apache/incubator-opendal/pull/3080
http
methods signature by using OpRead/OpStat by @miroim in https://github.com/apache/incubator-opendal/pull/3083
gcs
methods signature by using OpXxxx by @wavty in https://github.com/apache/incubator-opendal/pull/3087
fixtures
from core/src/services/{service}
to top-level fixtures/{service}
by @G-XD in https://github.com/apache/incubator-opendal/pull/3088
webhdfs
methods signature by using OpXxxx
by @cxorm in https://github.com/apache/incubator-opendal/pull/3109
us-east-1
for non-aws s3 by @G-XD in https://github.com/apache/incubator-opendal/pull/2812
flush
before sync_all
by @WenyXu in https://github.com/apache/incubator-opendal/pull/3053
Github
to GitHub
by @jbampton in https://github.com/apache/incubator-opendal/pull/3012
MacOS
to macOS
by @jbampton in https://github.com/apache/incubator-opendal/pull/3013
Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.39.0...v0.40.0
Published by oowl about 1 year ago
In PR #2687, OpenDAL changed the behavior when role_arn
has been specified.
OpenDAL used to override role_arn simply. But since this version, OpenDAL will make sure to use assume_role with specified role_arn
and external_id
(if supplied).
In PR #2666, RetryLayer
supports RetryInterceptor
. To implement this change, RetryLayer
changed it's in-memory layout by adding a new generic parameter I
to RetryLayer<I>
.
Users who stores RetryLayer
in struct or enum will need to change the type if they don't want to use default behavior.
In PR #2698, OpenDAL re-org the internal structure of opendal::raw::oio
and changed some APIs name.
stat
by @suyanhanx in https://github.com/apache/incubator-opendal/pull/2649
HADOOP_CONF_DIR
setting in guidance document by @wcy-fdu in https://github.com/apache/incubator-opendal/pull/2713
Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.38.1...v0.39.0
Published by Xuanwo over 1 year ago
oay.toml
by @messense in https://github.com/apache/incubator-opendal/pull/2615
delete_with
on blocking operator by @suyanhanx in https://github.com/apache/incubator-opendal/pull/2633
Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.38.0...v0.38.1
Published by PsiACE over 1 year ago
There are no public API changes.
OpenDAL add the Write::sink
API to enable streaming writing. This is a breaking change for users who depend on the raw API.
For a quick fix, users who have implemented opendal::raw::oio::Write
can return an Unsupported
error for Write::sink()
.
More detailes could be found at RFC: Writer sink
API.
is_exist
create_dir
copy
rename
delete
by @silver-ymz in https://github.com/apache/incubator-opendal/pull/2475
Monad
wrapper by @silver-ymz in https://github.com/apache/incubator-opendal/pull/2482
std
feature for futures to allow futures::AsyncRead
by @Xuanwo in https://github.com/apache/incubator-opendal/pull/2450
vercel artifacts
for FromStr
by @cijiugechu in https://github.com/apache/incubator-opendal/pull/2519
create
with new
by @NiwakaDev in https://github.com/apache/incubator-opendal/pull/2427
Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.37.0...v0.38.0
Published by suyanhanx over 1 year ago
In v0.37.0, OpenDAL bump the version of reqsign
to v0.13.0.
There are no public API and raw API changes.
Full Changelog: https://github.com/apache/incubator-opendal/compare/v0.36.0...v0.37.0