06 May 01:26

tisonkun

84586e5

v0.46.0 Latest

Latest

Upgrade to v0.46

Public API

MSRV Changed to 1.75

Since 0.46, OpenDAL requires Rust 1.75.0 or later to use features like RPITIT and AFIT.

Services Feature Flag

Starting with version 0.46, OpenDAL only includes the memory service by default to prevent compiling unnecessary service code. To use other services, please activate their respective feature flags.

Additionally, we have removed all reqwest-related feature flags:

Users must now directly use reqwest's feature flags for options like rustls, native-tls, etc.
The rustls feature is no longer enabled by default; it must be activated manually.
OpenDAL no longer offers the trust-dns option; users should configure the client builder directly.

Range Based Read

Since v0.46, OpenDAL transformed it's Read IO trait to range based instead of stateful poll based IO. This change will make the IO more efficient, easier for concurrency and ready for completion based IO.

opendal::Reader now have APIs like:

let r = op.reader("test.txt").await?;
let buf = r.read(1024..2048).await?;

Buffer Based IO

Since version 0.46, OpenDAL features a native Buffer struct that supports both contiguous and non-contiguous buffers. This update enhances IO efficiency by minimizing unnecessary byte copying and enabling vectored IO.

OpenDAL's Reader will return Buffer and Writer will accept Buffer as input. Users who have implemented their own IO traits should update their code to use the new Buffer struct.

let r = op.reader("test.txt").await?;
// read returns `Buffer`
let buf: Buffer = r.read(1024..2048).await?;

let w = op.writer("test.txt").await?;

// Buffer can be created from continues bytes.
w.write("hello, world").await?;
// Buffer can also be created from non-continues bytes.
w.write(vec![Bytes::from("hello,"), Bytes::from("world!")]).await?;

// Make sure file has been written completely.
w.close().await?;

To enhance usability, we've integrated bridges into bytes::Buf and bytes::BufMut, allowing users to directly interact with the bytes API.

let r = op.reader("test.txt").await?;
let mut bs = vec![];
// read_into accepts bytes::BufMut
let buf: Buffer = r.read_into(&mut bs, 1024..2048).await?;

let w = op.writer("test.txt").await?;

// write_from accepts bytes::Buf
w.write_from("hello, world".as_bytes()).await?;

// Make sure file has been written completely.
w.close().await?;

Bridge API

OpenDAL's Reader and Writer previously implemented APIs such as AsyncRead and AsyncWrite directly. This design was not user-friendly, as it could lead to unexpected costs that users were unaware of in advance.

Since v0.46, OpenDAL provides bridge APIs for Reader and Writer instead.

let r = op.reader("test.txt").await?;

// Convert into futures AsyncRead + AsyncSeek.
let reader = r.into_futures_async_read(1024..2048);
// Convert into futures bytes stream.
let stream = r.into_bytes_stream(1024..2048);

let w = op.writer("test.txt").await?;

// Convert into futures AsyncWrite
let writer = w.into_futures_async_write();
// Convert into futures bytes sink;
let sink = w.into_bytes_sink();

Raw API

Async in IO trait

Since version 0.46, OpenDAL has adopted Rust's native async_in_trait for our core IO traits, including oio::Read, oio::Write, and oio::List.

This update eliminates the need for manually written, poll-based state machines and simplifies the codebase. Consequently, OpenDAL now requires Rust version 1.75.0 or later.

Users who have implemented their own IO traits should update their code to use the new async trait syntax.

What's Changed

Added

feat(services/github): add github contents support by @hoslo in #4281
feat: Allow selecting webpki for reqwest by @arlyon in #4303
feat(services/swift): add support for storage_url configuration in swift service by @zjregee in #4302
feat(services/swift): add ceph test setup for swift by @zjregee in #4307
docs(website): add local content search based on lunr plugin by @m1911star in #4348
feat(services/sled): add SledConfig by @yufan022 in #4351
feat : Implementing config for part of services by @AnuRage-git in #4344
feat(bindings/java): explicit async runtime by @tisonkun in #4376
feat(services/surrealdb): support surrealdb service by @yufan022 in #4375
feat(bindings/java): avoid double dispose NativeObject by @tisonkun in #4377
feat : Implement config for services/foundationdb by @AnuRage-git in #4355
feat: add ofs ctrl-c exit unmount hook by @oowl in #4393
feat: Implement RFC-4382 Range Based Read by @Xuanwo in #4381
feat(ofs): rename2 lseek copy_file_range getattr API support by @oowl in #4395
feat(services/github): make access_token optional by @hoslo in #4404
feat(core/oio): Add readable buf by @Xuanwo in #4407
feat(ofs): add freebsd OS support by @oowl in #4403
feat(core/raw/oio): Add Writable Buf by @Xuanwo in #4410
feat(bin/ofs): Add behavior test for ofs by @ho-229 in #4373
feat(core/raw/buf): Reduce one allocation by Arc::from_iter by @Xuanwo in #4440
feat: ?Send async trait for HttpBackend when the target is wasm32 by @waynexia in #4444
feat: add HttpClient::with() constructor by @waynexia in #4447
feat: Move Buffer as public API by @Xuanwo in #4450
feat: Optimize buffer implementation and add stream support by @Xuanwo in #4458
feat(core): Implement FromIterator for Buffer by @Xuanwo in #4459
feat(services/ftp): Support multiple write by @xxxuuu in #4425
feat(raw/oio): block write change to buffer by @hoslo in #4466
feat(core): Implement read and read_into for Reader by @Xuanwo in #4467
feat(core): Implement into_stream for Reader by @Xuanwo in #4473
feat(core): Tune buffer operations based on benchmark results by @Xuanwo in #4468
feat(raw/oio): Use Buffer as cache in RangeWrite by @reswqa in #4476
feat(raw/oio): Use Buffer as cache in OneshotWrite by @reswqa in #4477
feat: add list/copy/rename for dropbox by @zjregee in #4424
feat(core): Implement write and write_from for Writer by @zjregee in #4482
feat(core): Add auto ranged read and concurrent read support by @Xuanwo in #4486
feat(core): Implement fetch for Reader by @Xuanwo in #4488
feat(core): Allow concurrent reading on bytes stream by @Xuanwo in #4499
feat: provide send-wrapper to contidionally implement Send for operators by @waynexia in #4443
feat(bin/ofs): privileged mount by @ho-229 in #4507
feat(services/compfs): init compfs by @George-Miao in #4519
feat(raw/oio): Add PooledBuf to allow reuse buffer by @Xuanwo in #4522
feat(services/fs): Add PooledBuf in fs to allow reusing memory by @Xuanwo in #4525
feat(core): add access methods for Buffer by @George-Miao in #4530
feat(core): implement IoBuf for Buffer by @George-Miao in #4532
feat(services/compfs): compio runtime and compfs structure by @George-Miao in #4534
feat(core): change Result to default error by @George-Miao in #4535
feat(services/github): support list recursive by @hoslo in #4423
feat: Add crc32c checksums to S3 Service by @JWackerbauer in #4533
feat: Add into_bytes_sink for Writer by @Xuanwo in #4541

Changed

refactor(core/raw): Migrate oio::Read to async in trait by @Xuanwo in #4336
refactor(core/raw): Align oio::BlockingRead API with oio::Read by @Xuanwo in #4349
refactor(core/oio): Migrate oio::List to async fn in trait by @Xuanwo in #4352
refactor(core/raw): Migrate oio::Write from WriteBuf to Bytes by @Xuanwo in https://github...

Contributors

jbampton, mobiusklein, and 26 other contributors

Assets 2

02 Mar 13:38

Xuanwo

v0.45.1

5079125

v0.45.1

Pacakages

Name	Version
core	0.45.1
bin/oay	0.41.1
bin/oli	0.41.1
bin/ofs	0.0.2
bindings/python	0.45.1
bindings/nodejs	0.45.1
bindings/c	0.44.3
bindings/zig	0.0.0
bindings/dotnet	0.1.1
bindings/haskell	0.44.3
bindings/java	0.45.1
bindings/lua	0.1.1
bindings/ruby	0.1.1
bindings/swift	0.0.0
bindings/ocaml	0.0.0
bindings/php	0.1.1
bindings/cpp	0.44.3
bindings/go	0.0.0
integrations/object_store	0.43.0
integrations/dav-server	0.0.1

What's Changed

Added

feat(services/vercel_blob): support vercel blob by @hoslo in #4103
feat(bindings/python): add ruff as linter by @asukaminato0721 in #4135
feat(services/hdfs-native): Add capabilities for hdfs-native service by @jihuayu in #4174
feat(services/sqlite): Add list capability supported for sqlite by @jihuayu in #4180
feat(services/azblob): support multi write for azblob by @wcy-fdu in #4181
feat(release): Implement releasing OpenDAL components seperately by @Xuanwo in #4196
feat: object store adapter based on v0.9 by @waynexia in #4233
feat(bin/ofs): implement fuse for linux by @ho-229 in #4179
feat(services/memcached): change to binary protocal by @hoslo in #4252
feat(services/memcached): setup auth test for memcached by @hoslo in #4259
feat(services/yandex_disk): setup test for yandex disk by @hoslo in #4264
feat: add ci support for ceph_rados by @ZhengLin-Li in #4191
feat: Implement Config for part of services by @Xuanwo in #4277
feat: add jfrog test setup for webdav by @zjregee in #4265

Changed

refactor(bindings/python): simplify async writer aexit by @suyanhanx in #4128
refactor(service/d1): Add D1Config by @jihuayu in #4129
refactor: Rewrite webdav to improve code quality by @Xuanwo in #4280

Fixed

fix: Azdls returns 403 while continuation contains = by @Xuanwo in #4105
fix(bindings/python): missed to call close for the file internally by @zzl221000 in #4122
fix(bindings/python): sync writer exit close raise error by @suyanhanx in #4127
fix(services/chainsafe): fix 423 http status by @hoslo in #4148
fix(services/webdav): Add possibility to answer without response if file isn't exist by @AJIOB in #4170
fix(services/webdav): Recreate root directory if need by @AJIOB in #4173
fix(services/webdav): remove base_dir component by @hoslo in #4231
fix(core): Poll TimeoutLayer::sleep once to make sure timer registered by @Xuanwo in #4230
fix(services/webdav): Fix endpoint suffix not handled by @Xuanwo in #4257
fix(services/webdav): Fix possible error with value loosing from config by @AJIOB in #4262

Docs

docs: add request for add secrets of services by @suyanhanx in #4104
docs(website): announce release v0.45.0 to news by @morristai in #4152
docs(services/gdrive): Update Google Drive capabilities list docs by @jihuayu in #4158
docs: Fix docs build by @Xuanwo in #4162
docs: add docs for Ceph Rados Gateway S3 by @ZhengLin-Li in #4190
docs: Fix typo in core/src/services/http/docs.md by @jbampton in #4226
docs: Fix spelling in Rust files by @jbampton in #4227
docs: fix typo in website/README.md by @jbampton in #4228
docs: fix spelling by @jbampton in #4229
docs: fix spelling; change github to GitHub by @jbampton in #4232
docs: fix typo by @jbampton in #4234
docs: fix typo in bindings/c/CONTRIBUTING.md by @jbampton in #4235
docs: fix spelling in code comments by @jbampton in #4236
docs: fix spelling in CONTRIBUTING by @jbampton in #4237
docs: fix Markdown link in bindings/README.md by @jbampton in #4238
docs: fix links and spelling in Markdown by @jbampton in #4239
docs: fix grammar and spelling in Markdown in examples/rust by @jbampton in #4241
docs: remove unneeded duplicate words from Rust files by @jbampton in #4243
docs: fix grammar and spelling in Markdown by @jbampton in #4245
docs: Add architectural.png to website by @Xuanwo in #4261
docs: Re-org project README by @Xuanwo in #4260
docs: order the README Who is using OpenDAL list by @jbampton in #4263

CI

ci: Use old version of seafile mc instead by @Xuanwo in #4107
ci: Refactor workflows layout by @Xuanwo in #4139
ci: Migrate hdfs default setup by @Xuanwo in #4140
ci: Refactor check.sh into check.py to get ready for multi components release by @Xuanwo in #4159
ci: Add test case for hdfs over gcs bucket by @ArmandoZ in #4145
ci: Add hdfs test case for s3 by @ArmandoZ in #4184
ci: Add hdfs test case for azurite by @ArmandoZ in #4185
ci: Add support for releasing all rust packages by @Xuanwo in #4200
ci: Fix dependabot not update by @Xuanwo in #4202
ci: reduce the open pull request limits to 1 by @jbampton in #4225
ci: Remove version suffix from package versions by @Xuanwo in #4254
ci: Fix fuzz test for minio s3 name change by @Xuanwo in #4266
ci: Mark python 3.13 is not supported by @Xuanwo in #4269
ci: Disable yandex disk test for too slow by @Xuanwo in #4274
ci: Split python CI into release and checks by @Xuanwo in #4275
ci(release): Make sure LICENSE and NOTICE files are included by @Xuanwo in #4283
ci(release): Refactor and merge the check.py into verify.py by @Xuanwo in #4284

Chore

chore(deps): bump actions/cache from 3 to 4 by @dependabot in #4118
chore(deps): bump toml from 0.8.8 to 0.8.9 by @dependabot in #4109
chore(deps): bump dav-server from 0.5.7 to 0.5.8 by @dependabot in #4111
chore(deps): bump assert_cmd from 2.0.12 to 2.0.13 by @dependabot in #4112
chore(deps): bump actions/setup-dotnet from 3 to 4 by @dependabot in #4115
chore(deps): bump mongodb from 2.7.1 to 2.8.0 by @dependabot in #4110
chore(deps): bump quick-xml from 0.30.0 to 0.31.0 by @dependabot in #4113
chore: Make every components seperate, remove workspace by @Xuanwo in #4134
chore: Fix build of core by @Xuanwo in #4137
chore: Fix workflow links for opendal by @Xuanwo in #4147
chore(website): Bump download link for 0.45.0 release by @morristai in #4149
chore: Fix name DtraceLayerWrapper by @jayvdb in #4165
chore: Align core version by @Xuanwo in #4197
chore: update benchmark doc by @wcy-fdu in #4201
chore(deps): bump clap from 4.4.18 to 4.5.1 in /bin/oli by @dependabot in #4221
chore(deps): bump serde from 1.0.196 to 1.0.197 in /bin/oay by @Depe...

Contributors

jayvdb, jbampton, and 15 other contributors

Assets 2

0 Join discussion

05 Feb 15:55

morristai

v0.45.0

7190c07

v0.45.0

Upgrade to v0.45

Core

Public API

BlockingLayer is not enabled by default

To further enhance the optionality of tokio, we have introduced a new feature called layers-blocking. The default usage of the blocking layer has been disabled. To utilize the BlockingLayer, please enable the layers-blocking feature.

TimeoutLayer deprecated `with_speed`

The with_speed API has been deprecated. Please use with_io_timeout instead.

Raw API

No raw API changes.

What's Changed

Added

feat(ofs): introduce ofs execute bin by @oowl in #4033
feat: add a missing news by @WenyXu in #4056
feat(services/koofr): set test for koofr by @suyanhanx in #4050
feat(layers/dtrace): Support User Statically-Defined Tracing(aka USDT) on Linux by @Zheaoli in #4053
feat(website): add missing news and organize the onboarding guide by @morristai in #4072
feat(services/chainsafe): setup test for chainsafe by @hoslo in #4089
docs: Add apache iceberg-rust as user by @liurenjie1024 in #4100

Changed

refactor!: avoid hard dep to tokio rt by @tisonkun in #4061
refactor: Implement IntoFuture for operator futures to remove an alloc by @Xuanwo in #4098

Fixed

fix(examples/cpp): display the results to standard output. by @SYaoJun in #4040
fix(service/icloud):Missing 'X-APPLE-WEBAUTH-USER cookie' and URL initialized failed by @bokket in #4029
fix: Implement timeout layer correctly by using timeout by @Xuanwo in #4059
fix(koofr): create_dir when exist by @hoslo in #4062
fix(seafile): test_list_dir_with_metakey by @hoslo in #4063
fix: list path recursive should not return path itself by @youngsofun in #4067
fix: Upgrade suppaftp to address build failure by @Xuanwo in #4091

Docs

docs: Remove not needed actions in release guide by @Xuanwo in #4037
docs: fix spelling errors in README.md by @SYaoJun in #4039
docs: New PMC member Liuqing Yue by @Xuanwo in #4047
docs: New Committer Yang Shuai by @Xuanwo in #4054
docs(services/sftp): add more explanation for endpoint config by @silver-ymz in #4055
docs: Add upgrade docs for core by @Xuanwo in #4095
docs: fix reference in rustdocs of FuturePresignWrite by @qrilka in #4097
docs: Add docs of options for xxx_with APIs by @Xuanwo in #4099

CI

ci(services/s3): Use minio/minio image instead by @Xuanwo in #4070
ci: Fix CI after moving out of workspacs by @Xuanwo in #4081

Chore

chore: Delete bindings/ruby/cucumber.yml by @tisonkun in #4030
chore(website): Bump download link for 0.44.2 release by @Zheaoli in #4034
chore(website): Update the release tips by @Zheaoli in #4036
chore: add doap file by @tisonkun in #4038
chore(website): Add extra artifacts check process in release document by @Zheaoli in #4041
chore(bindings/dotnet): update cargo.lock and set up ci by @suyanhanx in #4084
chore(bindings/dotnet): build os detect by @suyanhanx in #4085
chore(bindings/ocaml): pinning OCaml binding opendal version for release by @Ranxy in #4086
chore: Specify opendal version for haskell binding by @Xuanwo in #4093
chore: Bump to version 0.45.0 to start release process by @morristai in #4088

New Contributors

@SYaoJun made their first contribution in #4040
@happysalada made their first contribution in #4079
@qrilka made their first contribution in #4097
@liurenjie1024 made their first contribution in #4100

Full Changelog: v0.44.2...v0.45.0

Contributors

qrilka, liurenjie1024, and 14 other contributors

Assets 2

20 Jan 16:05

Zheaoli

v0.44.2

c5c6934

v0.44.2

What's Changed

Added

feat: add behavior tests for blocking buffer reader by @WenyXu in #3872
feat(services): add pcloud support by @hoslo in #3892
feat(services/hdfs): Atomic write for hdfs by @shbhmrzd in #3875
feat(services/hdfs): add atomic_write_dir to hdfsconfig debug by @shbhmrzd in #3902
feat: add MongodbConfig by @zjregee in #3906
RFC-3898: Concurrent Writer by @WenyXu in #3898
feat(services): add yandex disk support by @hoslo in #3918
feat: implement concurrent MultipartUploadWriter by @WenyXu in #3915
feat: add concurrent writer behavior tests by @WenyXu in #3920
feat: implement concurrent RangeWriter by @WenyXu in #3923
feat: add concurrent and buffer parameters into FuzzInput by @WenyXu in #3921
feat(fuzz): add azblob as test service by @suyanhanx in #3931
feat(services/webhdfs): Implement write with append by @hoslo in #3937
feat(core/bench): Add benchmark for concurrent write by @Xuanwo in #3942
feat(oio): add block_write support by @hoslo in #3945
feat(services/webhdfs): Implement multi write via CONCAT by @hoslo in #3939
feat(core): Allow retry in concurrent write operations by @Xuanwo in #3958
feat(services/ghac): Add workaround for AWS S3 based GHES by @Xuanwo in #3985
feat: Implement path cache and refactor gdrive by @Xuanwo in #3975
feat(services): add hdfs native layout by @shbhmrzd in #3933
feat(services/s3): Return error if credential is empty after loaded by @Xuanwo in #4000
feat(services/gdrive): Use trash instead of permanently deletes by @Xuanwo in #4002
feat(services): add koofr support by @hoslo in #3981
feat(icloud): Add basic Apple iCloud Drive support by @bokket in #3980

Changed

refactor: Merge compose_{read,write} into enum_utils by @Xuanwo in #3871
refactor(services/ftp): Impl parse_error instead of From by @bokket in #3891
docs: very minor English wording fix in error message by @gabrielgrant in #3900
refactor(services/rocksdb): Impl parse_error instead of From by @suyanhanx in #3903
refactor: Re-organize the layout of tests by @Xuanwo in #3904
refactor(services/etcd): Impl parse_error instead of From by @suyanhanx in #3910
refactor(services/sftp): Impl parse_error instead of From by @G-XD in #3914
refactor!: Bump MSRV to 1.75 by @Xuanwo in #3851
refactor(services/redis): Impl parse_error instead of From by @suyanhanx in #3938
refactor!: Revert the bump of MSRV to 1.75 by @Xuanwo in #3952
refactor(services/onedrive): Add OnedriveConfig to implement ConfigDeserializer by @Borber in #3954
refactor(service/dropbox): Add DropboxConfig by @howiieyu in #3961
refactor: Polish internal types and remove not needed deps by @Xuanwo in #3964
refactor: Add concurrent error test for BlockWrite by @Xuanwo in #3968
refactor: Remove not needed types in icloud by @Xuanwo in #4021

Fixed

fix: Bump pyo3 to fix false positive of unnecessary_fallible_conversions by @Xuanwo in #3873
fix(core): Handling content encoding correctly by @Xuanwo in #3907
fix: fix RangeWriter incorrect next_offset by @WenyXu in #3927
fix(oio::BlockWrite): fix write_once case by @hoslo in #3953
fix: Don't retry close if concurrent > 1 to avoid content lost by @Xuanwo in #3957
fix(doc): fix rfc typos by @howiieyu in #3971
fix: Don't call wake_by_ref in OperatorFuture by @Xuanwo in #4003
fix: async fn resumed after initiate part failed by @Xuanwo in #4013
fix(pcloud,seafile): use get_basename and get_parent by @hoslo in #4020
fix(ci): remove pr author from review candidates by @dqhl76 in #4023

Docs

docs(bindings/python): drop unnecessary patchelf by @tisonkun in #3889
docs: Polish core's quick start by @Xuanwo in #3896
docs(gcs): correct the description of credential by @WenyXu in #3928
docs: Add 0.44.1 download link by @Xuanwo in #3929
docs(release): how to clean up old releases by @tisonkun in #3934
docs(website): polish download page by @suyanhanx in #3932
docs: improve user verify words by @tisonkun in #3941
docs: fix incorrect word used by @zegevlier in #3944
docs: improve wording for community pages by @tisonkun in #3978
docs(bindings/nodejs): copyright in footer by @suyanhanx in #3986
docs(bindings/nodejs): build docs locally doc by @suyanhanx in #3987
docs: Fix missing word in download by @Xuanwo in #3993
docs(bindings/java): copyright in footer by @G-XD in #3996
docs(website): update footer by @suyanhanx in #4008
docs: add trademark information to every possible published readme by @PsiACE in #4014
docs(website): replace podling to project in website by @morristai in #4015
docs: Update release guide to adapt as a new TLP by @Xuanwo in #4011
docs: Add WebHDFS version compatibility details by @shbhmrzd in #4024
docs: Polish the template for add candidate as PMC member by @Xuanwo in #4027

CI

build(deps): bump actions/download-artifact from 3 to 4 by @dependabot in #3885
build(deps): bump once_cell from 1.18.0 to 1.19.0 by @dependabot in #3880
build(deps): bump napi-derive from 2.14.2 to 2.14.6 by @dependabot in #3879
build(deps): bump url from 2.4.1 to 2.5.0 by @dependabot in #3876
build(deps): bump mlua from 0.8.10 to 0.9.2 by @oowl in #3890
ci: Disable supabase tests for our test org has been paused by @Xuanwo in #3908
ci: Downgrade artifact actions until regression addressed by @Xuanwo in #3935
ci: Refactor fuzz to integrate with test planner by @Xuanwo in #3936
ci: Pick random reviewers from committer list by @Xuanwo in #4001

Chore

chore: update release related docs and script by @dqhl76 in #3870
chore(NOTICE): update copyright to year 2024 by @suyanhanx in #3894
chore: Format code to make readers happy by @Xuanwo in #3912
chore: Remove unused dep async-compat by @Xuanwo in #3947
chore: display project logo on Rust docs by @tisonkun in #3983
chore: improve trademarks in bindings docs by @tisonkun in #3984
chore: use Apache OpenDAL™ in the first and most prominent mention by @tisonkun in #3988
chore: add more information for javadocs by @tisonkun in #3989
chore: precise footer by @tisonkun in #3997
chore: use full form name when necessary by @tisonkun in #3998
chore(bindings/python): Enable sftp service by default for unix platform by @Zheaoli in #4006
chore: remove disclaimer by @suyanhanx in #4009
chore: Remove incubating from releases by @Xuanwo in https://github.c...

Contributors

gabrielgrant, Xuanwo, and 17 other contributors

Assets 2

06 Jan 08:50

Xuanwo

v0.44.1

d4aab1e

v0.44.1

What's Changed

Added

feat(service/memcached): Add MemCachedConfig by @ankit-pn in #3827
feat(service/rocksdb): Add RocksdbConfig by @ankit-pn in #3828
feat(services): add chainsafe support by @hoslo in #3834
feat(bindings/python): Build all available services for python by @Xuanwo in #3836
feat: Adding Atomicserver config by @k-aishwarya in #3845
feat(oio::read): implement the async buffer reader by @WenyXu in #3811
feat(oio::read): implement the blocking buffer reader by @WenyXu in #3860
feat: adapt the CompleteReader by @WenyXu in #3861
feat: add basic behavior tests for buffer reader by @WenyXu in #3862
feat: add fuzz reader with buffer tests by @WenyXu in #3866
feat(ofs): implement ofs based on fuse3 by @Inokinoki in #3857

Changed

refactor: simplify bindings_python.yml by @messense in #3837
refactor: Add edge test for aws assume role with web identity by @Xuanwo in #3839
refactor(services/webdav): Add WebdavConfig to implement ConfigDeserializer by @kwaa in #3846
refactor: use TwoWays instead of TwoWaysReader and TwoWaysWriter by @WenyXu in #3863

Fixed

fix: Add tests for listing recursively on not supported services by @Xuanwo in #3826
fix(services/upyun): fix list api by @hoslo in #3841
fix: fix a bypass seek relative bug in BufferReader by @WenyXu in #3864
fix: fix the bypass read does not sync the cur of BufferReader by @WenyXu in #3865

Docs

docs: Add Apache prefix for all bindings by @Xuanwo in #3829
docs: Add apache prefix for python docs by @Xuanwo in #3830
docs: Add branding in README by @Xuanwo in #3831
docs: Add trademark for Apache OpenDAL™ by @Xuanwo in #3832
docs: Add trademark sign for core by @Xuanwo in #3833
docs: Enable doc_auto_cfg when docs cfg has been enabled by @Xuanwo in #3835
docs: Address branding for haskell and C bindings by @Xuanwo in #3840
doc: add 0.44.0 release link to download.md by @dqhl76 in #3868

CI

ci: Remove workflows that not running or ready by @Xuanwo in #3842
ci: Migrate ftp to test planner by @Xuanwo in #3843

Chore

chore(bindings/java): Add name and description metadata by @tisonkun in #3838
chore(website): improve a bit trademark refs by @tisonkun in #3847
chore: Fix clippy warnings found in rust 1.75 by @Xuanwo in #3849
chore(bindings/python): improve ASF branding by @tisonkun in #3850
chore(bindings/haskell): improve ASF branding by @tisonkun in #3852
chore(bindings/c): make c binding separate workspace by @suyanhanx in #3856
chore(bindings/haskell): support co-log-0.6.0 && ghc-9.4 by @silver-ymz in #3858
chore: Bump to version 0.44.1 to start release process by @Xuanwo in #3869

New Contributors

@ankit-pn made their first contribution in #3827
@k-aishwarya made their first contribution in #3845
@kwaa made their first contribution in #3846
@Inokinoki made their first contribution in #3857

Full Changelog: v0.44.0...v0.44.1

Contributors

messense, Xuanwo, and 10 other contributors

Assets 2

31 Dec 03:00

dqhl76

v0.44.0

b0c6ea2

v0.44.0

Upgrade Note

Rust core

Public API

Moka Service Configuration

The thread_pool_enabled option has been removed.

List Prefix Supported

After RFC: List Prefix landed, we have changed the behavior of list a path without /. OpenDAL used to return NotADirectory error, but now we will return the list of entries that start with given prefix instead.

Nodejs binding

Public API

Now, the list operation returns Array<Entry> instead of a lister.
Also, we removed scan, you can use list('some/path', {recursive: true})/listSync('some/path', {recursive: true}) instead of scan('some/path')/scanSync('some/path').

What's Changed

Added

feat(core): service add HuggingFace file system by @morristai in #3670
feat(service/moka): bump moka from 0.10.4 to 0.12.1 by @G-XD in #3711
feat: add operator.list support for OCaml binding by @Young-Flash in #3706
feat(test): add Huggingface behavior test by @morristai in #3712
feat: Add list prefix support by @Xuanwo in #3728
feat: Implement ConcurrentFutures to remove the dependences on tokio by @Xuanwo in #3746
feat(services): add seafile support by @hoslo in #3771
RFC-3734: Buffered reader by @WenyXu in #3734
feat: Add content range support for RpRead by @Xuanwo in #3777
feat: Add presign_stat_with support by @Xuanwo in #3778
feat: Add project layout for ofs by @Xuanwo in #3779
feat(binding/nodejs): align list api by @suyanhanx in #3784
feat: Make OpenDAL available under wasm32 arch by @Xuanwo in #3796
feat(services): add upyun support by @hoslo in #3790
feat: Add edge test s3_read_on_wasm by @Xuanwo in #3802
feat(services/azblob): available under wasm32 arch by @suyanhanx in #3806
feat: make services-gdrive compile for wasm target by @Young-Flash in #3808
feat(core): Make gcs available on wasm32 arch by @Xuanwo in #3816

Changed

refactor(service/etcd): use EtcdConfig in from_map by @G-XD in #3703
refactor(object_store): upgrade object_store to 0.7. by @youngsofun in #3713
refactor: List must support list without recursive by @Xuanwo in #3721
refactor: replace ftp tls impl as rustls by @oowl in #3760
refactor: Remove never used Stream poll_reset API by @Xuanwo in #3774
refactor: Polish operator read_with by @Xuanwo in #3775
refactor: Migrate gcs builder to config based by @Xuanwo in #3786
refactor(service/hdfs): Add HdfsConfig to implement ConfigDeserializer by @shbhmrzd in #3800
refactor(raw): add parse_header_to_str fn by @hoslo in #3804
refactor(raw): refactor APIs like parse_content_disposition by @hoslo in #3815
refactor: Polish http_util parse headers by @Xuanwo in #3817

Fixed

fix(oli): Fix cp -r command returns invalid path error by @kebe7jun in #3687
fix(website): folder name mismatch by @suyanhanx in #3707
fix(binding/java): fix SPECIAL_DIR_NAME by @G-XD in #3715
fix(services/dropbox): Workaround for dropbox limitations for create_folder by @Xuanwo in #3719
fix(ocaml_binding): sort actual & expected to pass ci by @Young-Flash in #3733
fix(ci): Make sure merge_local_staging handles all subdir by @Xuanwo in #3788
fix(services/gdrive): fix return value of get_file_id_by_path by @G-XD in #3801
fix(core): List root should not return itself by @Xuanwo in #3824

Docs

docs: add maturity model check by @suyanhanx in #3680
docs(website): show maturity model by @suyanhanx in #3709
docs(website): only VOTEs from PPMC members are binding by @G-XD in #3710
doc: add 0.43.0 release link to download.md by @G-XD in #3729
docs: Add process on nominating committers and ppmc members by @Xuanwo in #3740
docs: Deploy website to nightlies for every tags by @Xuanwo in #3739
docs: Remove not released bindings docs from top level header by @Xuanwo in #3741
docs: Add dependencies list for all packages by @Xuanwo in #3743
docs: Update maturity docs by @Xuanwo in #3750
docs: update the RFC doc by @suyanhanx in #3748
docs(website): polish deploy to nightlies by @suyanhanx in #3753
docs: add event calendar in community page by @dqhl76 in #3767
docs(community): polish events by @suyanhanx in #3768
docs(bindings/ruby): reflect test framework refactor by @tisonkun in #3798
docs(website): add service Huggingface to website by @morristai in #3812
docs: update release docs to add cargo-deny setup by @dqhl76 in #3821

CI

build(deps): bump cacache from 11.7.1 to 12.0.0 by @dependabot in #3690
build(deps): bump prometheus-client from 0.21.2 to 0.22.0 by @dependabot in #3694
build(deps): bump github/issue-labeler from 3.2 to 3.3 by @dependabot in #3698
ci: Add behavior test for b2 by @Xuanwo in #3714
ci(cargo): Add frame pointer support in build flag by @Zheaoli in #3772
ci: Workaround ring 0.17 build issue, bring aarch64 and armv7l back by @Xuanwo in #3781
ci: Support CI test for s3_read_on_wasm by @Zheaoli in #3813

Chore

chore: bump aws-sdk-s3 from 0.38.0 to 1.4.0 by @memoryFade in #3704
chore: Disable obs test for workaround by @Xuanwo in #3717
chore: Fix bindings CI by @Xuanwo in #3722
chore(binding/nodejs,website): Replace yarn with pnpm by @suyanhanx in #3730
chore: Bring persy CI back by @Xuanwo in #3751
chore(bindings/python): upgrade pyo3 to 0.20 by @messense in #3758
chore: remove unused binding feature file by @tisonkun in #3757
chore: Bump governor from 0.5.1 to 0.6.0 by @G-XD in #3761
chore: Split bindings/ocaml to separate workspace by @Xuanwo in #3792
chore: Split bindings/ruby to separate workspace by @ho-229 in #3794
chore(bindings/php): bump ext-php-rs to support latest php & separate workspace by @suyanhanx in #3799
chore: Address comments from hackernews by @Xuanwo in #3805
chore(bindings/ocaml): dep opendal point to core by @suyanhanx in #3814
chore: Bump to v0.44.0 to start release process by @dqhl76 in #3819

New Contributors

@kebe7jun made their first contribution in #3687
@memoryFade made their first contribution in #3704
@ho-229 made their first contribution in #3794

Full Changelog: http...

Contributors

messense, Xuanwo, and 16 other contributors

Assets 2

08 Dec 10:43

G-XD

v0.43.0

146e960

v0.43.0

Upgrade Note

Rust Core

Public API

List Recursive

After RFC-3526: List Recursive landed, we have changed the list API to accept recursive instead of delimiter:

Users will need to change the following usage:

op.list_with(path).delimiter("") -> op.list_with(path).recursive(true)
op.list_with(path).delimiter("/") -> op.list_with(path).recursive(false)

delimiter other than "" and "/" is not supported anymore.

Stat a dir path

After RFC: List Prefix landed, we have changed the behavior of stat a dir path:

Here are the behavior list:

Case	Path	Result
stat existing dir	`abc/`	Metadata with dir mode
stat existing file	`abc/def_file`	Metadata with file mode
stat dir without `/`	`abc/def_dir`	Error `NotFound` or metadata with dir mode
stat file with `/`	`abc/def_file/`	Error `NotFound`
stat not existing path	`xyz`	Error `NotFound`

Services like s3, azblob can handle stat("abc/") correctly by check if there are objects with prefix abc/.

Raw API

Lister Align

We changed our internal lister implementation to align with the list public API for better performance and readability.

trait Page => List
struct Pager => Lister
trait BlockingPage => BlockingList
struct BlockingPager => BlockingLister

Every call to next will return an entry instead a page of entries. Also, we changed our async list api into poll based instead of async_trait.

Java binding

Breaking change

Because of a TLS lib issue, we temporarily disable the services-ftp feature.

Node.js binding

Breaking change

Because of a TLS lib issue, we temporarily disable the services-ftp feature.

Python binding

Breaking change

Because of a TLS lib issue, we temporarily disable the services-ftp feature.

C binding

There are no API changes.

What's Changed

Added

feat(bindings/C): Add opendal_operator_rename and opendal_operator_copy by @jiaoew1991 in #3517
feat(binding/python): Add new API to convert between AsyncOperator and Operator by @Zheaoli in #3514
feat: Implement RFC-3526: List Recursive by @Xuanwo in #3556
feat(service): add alluxio rest api support by @hoslo in #3564
feat(bindings/python): add OPENDAL_DISABLE_RANDOM_ROOT support by @Justin-Xiang in #3550
feat(core): add Alluxio e2e test by @hoslo in #3573
feat(service): alluxio support write by @hoslo in #3566
feat(bindings/nodejs): add retry layer by @suyanhanx in #3484
RFC: Concurrent Stat in List by @morristai in #3574
feat(service/hdfs): enable rename in hdfs service by @qingwen220 in #3592
feat: Improve the read_to_end perf and add benchmark vs_fs by @Xuanwo in #3617
feat: Add benchmark vs aws sdk s3 by @Xuanwo in #3620
feat: Improve the performance of s3 services by @Xuanwo in #3622
feat(service): support b2 by @hoslo in #3604
feat(core): Implement RFC-3574 Concurrent Stat In List by @morristai in #3599
feat: Implement stat dir correctly based on RFC-3243 List Prefix by @Xuanwo in #3651
feat(bindings/nodejs): Add capability support by @suyanhanx in #3654
feat: disable ftp for python and java binding by @ZutJoe in #3659
feat(bindings/nodejs): read/write stream by @suyanhanx in #3619

Changed

refactor(services/persy): migrate tot test planner by @G-XD in #3476
refactor(service/etcd): Add EtcdConfig to implement ConfigDeserializer by @Xuxiaotuan in #3543
chore(service/tikv): rename Backend to TikvBackend by @caicancai in #3545
refactor(services/azblob): add AzblobConfig by @acehinnnqru in #3553
refactor(services/cacache): migrate to test planner by @G-XD in #3568
refactor(services/sled): migrate to test planner by @G-XD in #3569
refactor(services/webhdfs): migrate to test planner by @G-XD in #3578
refactor(core): Rename all Page to List by @Xuanwo in #3589
refactor: Change List API into poll based and return one entry instead by @Xuanwo in #3593
refactor(services/tikv): migrate to test planner by @G-XD in #3587
refactor(service/redis): Migrate task to new task planner by @sunheyi6 in #3374
refactor(oio): Polish IncomingAsyncBody::bytes by @Xuanwo in #3621
refactor(services/rocksdb): migrate to test planner by @G-XD in #3636
refactor(services/azfile): Check if dir exists before create by @ZutJoe in #3652
refactor: Polish concurrent list by @Xuanwo in #3658

Fixed

fix(bindings/python): Fix the test command in doc by @Justin-Xiang in #3541
docs(bindings/python): Fix the test command in doc by @Justin-Xiang in #3561
fix(ci): try enable corepack before setup-node action by @suyanhanx in #3609
fix(service/hdfs): enable hdfs append support by @qingwen220 in #3600
fix(ci): fix setup node by @suyanhanx in #3611
fix(core): Path in remove not normalized by @Xuanwo in #3671
fix(core): Build with redis features and Rust < 1.72 by @vincentdephily in #3683

Docs

docs: Add questdb in users list by @caicancai in #3532
docs: Add macos specific doc updates for hdfs by @shbhmrzd in #3559
docs(bindings/java): add basic usage in README by @caicancai in #3534
doc: add 0.42.0 release link to download.md by @silver-ymz in #3598

CI

ci(services/libsql): add rust test threads limit by @G-XD in #3540
ci(services/redb): migrate to test planner by @suyanhanx in #3518
ci: Disable libsql behavior test until we or upstream address them by @Xuanwo in #3552
ci: Add new Python binding reviewer by @Zheaoli in #3560
ci(bindings/nodejs): add aarch64 build support by @suyanhanx in #3567
ci(planner): Polish the workflow planner code by @Zheaoli in #3570
ci(core): Add dry run for rc tags by @Xuanwo in #3624
ci: Disable persy until it has been fixed by @Xuanwo in #3631
ci: Calling cargo to make sure rust has been setup by @Xuanwo in #3633
ci: Fix etcd with tls and auth failed to start by @Xuanwo in #3637
ci(services/etcd): Use ALLOW_NONE_AUTHENTICATION as workaround by @Xuanwo in #3638
ci: dry run publish on rc tags for python binding by @everpcpc in #3645
ci: Add java linux arm64 build by @Xuanwo in #3660
ci(java/binding): Use zigbuild for glibc 2.17 support by @Xuanwo in #3664
ci(bindings/python): remove aarch support by @G-XD in #3674

Chore

chore(servies/sftp): Upgrade openssh-sftp-client to 0.14 by @sd44 in https://g...

Contributors

vincentdephily, jtescher, and 25 other contributors

Assets 2

16 Nov 04:17

silver-ymz

v0.42.0

9b423ae

v0.42.0

Upgrade Note

Rust Core

MSRV Changed

OpenDAL bumps it's MSRV to 1.67.0.

S3 Service Configuration

The enable_exact_buf_write option has been deprecated and is superseded by BufferedWriter, introduced in version 0.40.

Oss Service Configuration

The write_min_size option has been deprecated and replaced by BufferedWriter, also introduced in version 0.40.
A new setting, allow_anonymous, has been added. Since v0.41, OSS will now return an error if credential loading fails. Enabling allow_anonymous to fallback to request without credentials.

Ghac Service Configuration

The enable_create_simulation option has been removed. We add this option to allow ghac simulate create empty file, but it's could result in unexpected behavior when users create a file with content length 1. So we remove it.

Wasabi Service Removed

wasabi service native support has been removed. Users who want to access wasabi can use our s3 service instead.

Java binding

There are no API changes.

Node.js binding

There are no API changes.

Python binding

Breaking change for layers

Operator and BlockingOperator won't accept layers anymore. Instead, we provide a layer API:

op = opendal.Operator("memory").layer(opendal.layers.RetryLayer())

We removed not used layers ConcurrentLimitLayer and ImmutableIndexLayer along with this change.

File and AsyncFile

OpenDAL removes Reader and AsyncReader classes, instead, we provide file-like object File and AsyncFile.

Open a file for reading in blocking way:

with op.open(filename, "rb") as r:
    content = r.read()

Open a file for reading in async way:

async with await op.open(filename, "rb") as r:
    content = await r.read()

Breaking change for Errors

We remove the old error classes and provide a couple of Exception based class for the error handling.

opendal.Error is based class for all the exceptions now.
opendal.exceptions.Unexpected is added.
opendal.exceptions.Unsupported is added.
opendal.exceptions.ConfigInvalid is added.
opendal.exceptions.NotFound is added.
opendal.exceptions.PermissionDenied is added.
opendal.exceptions.IsADirectory is added.
opendal.exceptions.NotADirectory is added.
opendal.exceptions.AlreadyExists is added.
opendal.exceptions.IsSameFile is added.
opendal.exceptions.ConditionNotMatch is added.
opendal.exceptions.ContentTruncated is added.
opendal.exceptions.ContentIncomplete is added.
opendal.exceptions.InvalidInput is added.

C binding

The naming convention for C binding has been altered.

Struct Naming

Renaming certain struct names for consistency.

opendal_operator_ptr => opendal_operator
opendal_blocking_lister => opendal_lister
opendal_list_entry => opendal_entry

API Naming

We've eliminated the blocking_ prefix from our API because C binding doesn't currently support async. In the future, we plan to introduce APIs such as opendal_operator_async_write.

opendal_operator_blocking_write => opendal_operator_write
opendal_operator_blocking_read => opendal_operator_read

What's Changed

Added

feat(binding/java): add rename support by @G-XD in #3238
feat(prometheus): add bytes metrics as counter by @flaneur2020 in #3246
feat(binding/python): new behavior testing for python by @laipz8200 in #3245
feat(binding/python): Support AsyncOperator tests. by @laipz8200 in #3254
feat(service/libsql): support libsql by @G-XD in #3233
feat(binding/python): allow setting append/buffer/more in write() call by @jokester in #3256
feat(services/persy): change blocking_x in async_x call to tokio::task::blocking_spawn by @Zheaoli in #3221
feat: Add edge test cases for OpenDAL Core by @Xuanwo in #3274
feat(service/d1): Support d1 for opendal by @realtaobo in #3248
feat(services/redb): change blocking_x in async_x call to tokio::task::blocking_spawn by @shauvet in #3276
feat: Add blocking layer for C bindings by @jiaoew1991 in #3278
feat(binding/c): Add blocking_reader for C binding by @jiaoew1991 in #3259
feat(services/sled): change blocking_x in async_x call to tokio::task::blocking_spawn by @shauvet in #3280
feat(services/rocksdb): change blocking_x in async_x call to tokio::task::blocking_spawn by @shauvet in #3279
feat(binding/java): make Metadata a POJO by @G-XD in #3277
feat(bindings/java): convey backtrace on exception by @tisonkun in #3286
feat(layer/prometheus): Support custom metric bucket for Histogram by @Zheaoli in #3275
feat(bindings/python): read APIs return memoryview instead of bytes to avoid copy by @messense in #3310
feat(service/azfile): add azure file service support by @dqhl76 in #3312
feat(services/oss): Add allow anonymous support by @Xuanwo in #3321
feat(bindings/python): build and publish aarch64 and armv7l wheels by @messense in #3325
feat(bindings/java): support duplicate operator by @tisonkun in #3330
feat(core): Add enabled for Scheme by @Xuanwo in #3331
feat(bindings/java): support get enabled services by @tisonkun in #3336
feat(bindings/java): Migrate behavior tests to new Workflow Planner by @Xuanwo in #3341
feat(layer/prometheus): Support output path as a metric label by @Zheaoli in #3335
feat(service/mongodb): Support mongodb service by @Zheaoli in #3355
feat: Make PrometheusClientLayer Clonable by @flaneur2020 in #3352
feat(service/cloudflare_kv): support cloudflare KV by @my-vegetable-has-exploded in #3348
feat(core): exposing Metadata::metakey() api by @G-XD in #3373
feat(binding/java): add list & remove_all support by @G-XD in #3333
feat: Add write_total_max_size in Capability by @realtaobo in #3309
feat(core): service add DBFS API 2.0 support by @morristai in #3334
feat(bindings/java): use random root for behavior tests by @tisonkun in #3408
feat(services/oss): Add start-after support for oss list by @wcy-fdu in #3410
feat(binding/python): Export full_capability API for Python binding by @Zheaoli in #3402
feat(test): Enable new test workflow planner for python binding by @Zheaoli in #3397
feat: Implement Lazy Reader by @Xuanwo in #3395
feat(binding/nodejs): upgrade test behavior and infra by @eryue0220 in #3297
feat(binding/python): Support Copy operation for Python binding by @Zheaoli in #3454
feat(bindings/python): Add layer API for operator by @Xuanwo in #3464
feat(bindings/java): add layers onto ops by @tisonkun in #3392
feat(binding/python): Support rename API for Python binding by @Zheaoli in #3467
feat(binding/python): Support remove_all API for Python binding by @Zheaoli in #3469
feat(core): fix token leak in OneDrive by @morristai in #3470
feat(core): service add OpenStack Swift support by @morristai in #3461
feat(bindings/python)!: Implement File and AsyncFile to replace Reader by @Xuanwo in #3474
feat(services): Implement ConfigDeserializer and add S3Config as example by @Xuanwo in #3490
feat(core): add OpenStack Swift e2e test by @morristai in #3493
feat(doc): add OpenStack Swift document for the website by @morristai in #3494
feat(services/sqlite): add SqliteConfig by @hoslo in #3497
feat(bindings/C): imp...

Contributors

flaneur2020, jokester, and 32 other contributors

Assets 2

16 Oct 08:50

suyanhanx

v0.41.0

779750a

v0.41.0

Upgrade Note

Rust Core

There are no public API and raw API changes.

Java binding

Breaking change for constructing operators

PR-3166 changes the API for constructing operators:

new BlockingOperator(scheme, config);
new Operator(scheme, config);

Current:

BlockingOperator.of(scheme, config);
Operator.of(scheme, config);

Now, there is no public constructor for operators, but only factory methods. In this way, the APIs are free to do arbitrary verifications and preparations before constructing operators.

Node.js binding

There are no API changes.

Python binding

There are no API changes.

What's Changed

Added

feat: allow using prometheus-client crate with PrometheusClientLayer by @flaneur2020 in #3134
feat(binding/java): support info ops by @G-XD in #3154
test(binding/java): add behavior test framework by @G-XD in #3129
feat: Include starting offset for GHAC upload Content-Range by @huonw in #3163
feat(bindings/cpp): make ReaderStream manage the lifetime of Reader by @silver-ymz in #3164
feat: Enable multi write for ghac by @Xuanwo in #3165
feat: Add mysql support for OpenDAL by @Zheaoli in #3170
feat(service/postgresql): support connection pool by @Zheaoli in #3176
feat(services/ghac): Allow explicitly setting ghac endpoint/token, not just env vars by @huonw in #3177
feat(service/azdls): add append support for azdls by @dqhl76 in #3186
feat(bindings/python): Enable BlockingLayer for non-blocking services that don't support blocking by @messense in #3198
feat: Add write_can_empty in Capability and related tests by @Xuanwo in #3200
feat: Add basic support for bindings/go using CGO by @jiaoew1991 in #3204
feat(binding/java): add copy test by @G-XD in #3207
feat(service/sqlite): Support sqlite for opendal by @Zheaoli in #3212
feat(services/sqlite): Support blocking_get/set/delete in sqlite service by @Zheaoli in #3218
feat(oay): port WebdavFs to dav-server-fs-opendal by @Young-Flash in #3119

Changed

refactor(services/dropbox): Use OpWrite instead of passing all args as parameters by @ImSingee in #3126
refactor(binding/java): read should return bytes by @tisonkun in #3153
refactor(bindings/java)!: operator jni calls by @tisonkun in #3166
refactor(tests): reuse function to remove duplicate code by @zhao-gang in #3219

Fixed

fix(tests): Create test files one by one instead of concurrently by @Xuanwo in #3132
chore(ci): fix web identity token path for aws s3 assume role test by @everpcpc in #3141
fix(services/s3): Detect region returned too early when header is empty by @Xuanwo in #3187
fix: making OpenDAL compilable on 32hf platforms by @ClSlaid in #3188
fix(binding/java): decode Java’s modified UTF-8 format by @G-XD in #3195

Docs

docs(release): describe how to close the Nexus staging repo by @tisonkun in #3125
docs: update release docs for cpp and haskell bindings by @silver-ymz in #3130
docs: Polish VISION to make it more clear by @Xuanwo in #3135
docs: Add start tracking issues about the next release by @Xuanwo in #3145
docs: Add download link for 0.40.0 by @Xuanwo in #3149
docs(bindings/cpp): add more using details about cmake by @silver-ymz in #3155
docs(bindings/java): Added an example of adding dependencies using Gradle by @eastack in #3158
docs: include disclaimer in announcement template by @Venderbad in #3172
docs: Add pants as a user by @huonw in #3180
docs: Add basic readme for go binding by @Xuanwo in #3206
docs: add multilingual getting started by @tisonkun in #3214
docs: multiple improvements by @tisonkun in #3215
docs: Add verify script by @Xuanwo in #3239

CI

ci: Align tags with semver specs by @Xuanwo in #3136
ci: Migrate obs to databend labs sponsored bucket by @Xuanwo in #3137
build(bindings/java): support develop with JDK 21 by @tisonkun in #3140
ci: Migrate GCS to Databend Labs sponsored bucket by @Xuanwo in #3142
build(bindings/java): upgrade maven wrapper version by @tisonkun in #3167
build(bindings/java): support explicit cargo build target by @tisonkun in #3168
ci: Pin Kvrocks docker image to 2.5.1 to avoid test failure by @git-hulk in #3192
ci(bindings/ocaml): add doc by @Ranxy in #3208
build(deps): bump actions/checkout from 3 to 4 by @dependabot in #3222
build(deps): bump korandoru/hawkeye from 3.3.0 to 3.4.0 by @dependabot in #3223
build(deps): bump rusqlite from 0.25.4 to 0.29.0 by @dependabot in #3226

Chore

chore(bindings/haskell): add rpath to haskell linker option by @silver-ymz in #3128
chore(ci): add test for aws s3 assume role by @everpcpc in #3139
chore: Incorrect debug information by @OmAximani0 in #3183
chore: bump quick-xml version to 0.30 by @Venderbad in #3190
chore: Let's welcome the contributors from hacktoberfest! by @Xuanwo in #3193
chore(bindings/java): simplify library path resolution by @tisonkun in #3196
chore: Make clippy happy by @Xuanwo in #3229
chore: Bump to v0.41.0 to start release process by @suyanhanx in #3241

New Contributors

@ImSingee made their first contribution in #3126
@eastack made their first contribution in #3158
@Venderbad made their first contribution in #3172
@OmAximani0 made their first contribution in #3183
@git-hulk made their first contribution in #3192
@zhao-gang made their first contribution in #3219

Full Changelog: v0.40.0...v0.41.0

Contributors

flaneur2020, zhao-gang, and 20 other contributors

Assets 2

21 Sep 05:36

Xuanwo

v0.40.0

9a775bd

v0.40.0

Checkout our OwO #1 to know more about this release!

Upgrade Note

Public API

RFC-2578 Merge Append Into Write

RFC-2578 merges append into write and removes append API.

For writing a file at once, please use op.write() for convenience.
For appending a file, please use op.write_with().append(true) instead of op.append().

The same rule applies to writer() and writer_with().

RFC-2774 Lister API

RFC-2774 proposes a new lister API to replace current list and scan. And we add a new API list to return entries directly.

For listing a directory at once, please use list() for convenience.
For listing a directory recursively, please use list_with().delimiter("") or lister_with().delimiter("") instead of scan().
For listing in streaming, please use lister() or lister_with() instead.

RFC-2779 List With Metakey

RFC-2779 proposes a new op.list_with().metakey() API to allow list with metakey and removes op.metadata(&entry) API.

Please use op.list_with().metakey() instead of op.metadata(&entry), for example:

// Before
let entries: Vec<Entry> = op.list("dir/").await?;
for entry in entris {
  let meta = op.metadata(&entry, Metakey::ContentLength | Metakey::ContentType).await?;
  println!("{} {}", entry.name(), entry.metadata().content_length());
}

// After
let entries: Vec<Entry> = op
  .list_with("dir/")
  .metakey(Metakey::ContentLength | Metakey::ContentType).await?;
for entry in entris {
  println!("{} {}", entry.name(), entry.metadata().content_length());
}

RFC-2852: Native Capability

RFC-2852 proposes new native_capability and full_capability API to allow users to check if the underlying service supports a capability natively.

native_capability returns true if the capability is supported natively.
full_capability returns true if the capability is supported, maybe via a layer.

Most of time, you can use full_capability to replace capability call. But if to check if the capability is supported natively for better performance design, please use native_capability instead.

Buffered Writer

OpenDAL v0.40 added buffered writer support!

Users don't need to specify the content_length() for writer anymore!

- let mut w = op.writer_with("path/to/file").content_length(1024).await?;
+ let mut w = op.writer_with("path/to/file").await?;

Users can specify the buffer() to control the size we call underlying storage:

let mut w = op.writer_with("path/to/file").buffer(8 * 1024 * 1024).await?;

If buffer is not specified, we will call underlying storage everytime we call write. Otherwise, we will make sure to call underlying storage when buffer is full or close is called.

Raw API

RFC-3017 Remove Write Copy From

RFC-3017 removes copy_from API from the oio::Write trait. Users who implements services and layers by hand should remove this API.

What's Changed

Added

feat(service/etcd): support list by @G-XD in #2755
feat: setup the integrate with PHP binding by @godruoyi in #2726
feat(oay): Add read_dir by @Young-Flash in #2736
feat(obs): support loading credential from env by @everpcpc in #2767
feat: add async backtrace layer by @dqhl76 in #2765
feat: Add OCaml Binding by @Ranxy in #2757
feat(bindings/haskell): support logging layer by @silver-ymz in #2705
feat: Add FoundationDB Support for OpenDAL by @ArmandoZ in #2751
feat(oay): add write for oay webdav by @Young-Flash in #2769
feat: Implement RFC-2774 Lister API by @Xuanwo in #2787
feat(bindings/haskell): enhance original OpMonad to support custom IO monad by @silver-ymz in #2789
feat: Add into_seekable_read_by_range support for blocking read by @Xuanwo in #2799
feat(layers/blocking): add blocking layer by @yah01 in #2780
feat: Add async list with metakey support by @Xuanwo in #2803
feat(binding/php): Add basic io by @godruoyi in #2782
feat: fuzz test support read from .env by different services by @dqhl76 in #2824
feat(services/rocksdb): Add scan support by @JLerxky in #2827
feat: Add postgresql support for OpenDAL by @Xuanwo in #2815
feat: ci for php binding by @godruoyi in #2830
feat: Add create_dir, remove, copy and rename API for oay-webdav by @Young-Flash in #2832
feat(oli): oli stat should show path as specified by users by @sarutak in #2842
feat(services/moka, services/mini-moka): Add scan support by @JLerxky in #2850
feat(oay): impl some method for WebdavMetaData by @Young-Flash in #2857
feat: Implement list with metakey for blocking by @Xuanwo in #2861
feat(services/redis): add redis cluster support by @G-XD in #2858
feat(services/dropbox): read support range by @suyanhanx in #2848
feat(layers/logging): Allow users to control print backtrace or not by @Xuanwo in #2872
feat: add native & full capability by @yah01 in #2874
feat: Implement RFC-2758 Merge Append Into Write by @Xuanwo in #2880
feat(binding/ocaml): Add support for operator reader and metadata by @Ranxy in #2881
feat(core): replace field _pin with !Unpin as argument by @morristai in #2886
feat: Add retry for Writer::sink operation by @Xuanwo in #2896
feat: remove operator range_read and range_reader API by @oowl in #2898
feat(core): Add unit test for ChunkedCursor by @Xuanwo in #2907
feat(types): remove blocking operation range_read and range_reader API by @oowl in #2912
feat(types): add stat_with API for blocking operator by @oowl in #2915
feat(services/gdrive): credential manage by @suyanhanx in #2914
feat(core): Implement Exact Buf Writer by @Xuanwo in #2917
feat: Add benchmark for buf write by @Xuanwo in #2922
feat(core/raw): Add stream support for multipart by @Xuanwo in #2923
feat(types): synchronous blocking operator and operator's API by @oowl in #2924
feat(bindings/java): bundled services by @tisonkun in #2934
feat(core/raw): support stream body for mixedpart by @silver-ymz in #2936
feat(bindings/python): expose presign api by @silver-ymz in #2950
feat(bindings/nodejs): Implement presign test by @suyanhanx in #2969
docs(services/gdrive): update service doc by @suyanhanx in #2973
feat(bindings/cpp): init cpp binding by @silver-ymz in #2980
feat: gcs insert object support cache control by @fatelei in #2974
feat(bindings/cpp): expose all api returned by value by @silver-ymz in #3001
feat(services/gdrive): implement rename by @suyanhanx in #3007
feat(bindings/cpp): expose reader by @silver-ymz in #3004
feat(bindings/cpp): expose lister by @silver-ymz in #3011
feat(core): Avoid copy if input is larger than buffer_size by @Xuanwo in #3016
feat(service/gdrive): add gdrive list support by @Young-Flash in #3025
feat(services/etcd): Enable etcd connection pool by @Xuanwo in #3041
feat: Add buffer support for all services by @Xuanwo in #3045
feat(bindings/ja...

Contributors

jbampton, fatelei, and 34 other contributors

Assets 2

Releases: apache/opendal

v0.46.0

Upgrade to v0.46

Public API

MSRV Changed to 1.75

Services Feature Flag

Range Based Read

Buffer Based IO

Bridge API

Raw API

Async in IO trait

What's Changed

Added

Changed

Contributors

v0.45.1

Pacakages

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

Contributors

v0.45.0

Upgrade to v0.45

Core

Public API

BlockingLayer is not enabled by default

TimeoutLayer deprecated with_speed

Raw API

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Contributors

v0.44.2

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

Contributors

v0.44.1

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Contributors

v0.44.0

Upgrade Note

Rust core

Public API

Moka Service Configuration

List Prefix Supported

Nodejs binding

Public API

What's Changed

Added

Changed

Fixed

Docs

CI

Chore

New Contributors

Contributors

v0.43.0

Upgrade Note

Rust Core

TimeoutLayer deprecated `with_speed`