Skip to content

3.22
Compare
Choose a tag to compare
@alex-aizman alex-aizman released this 25 Feb 18:14
· 250 commits to main since this release
v1.3.22
939b4d3

Highlights

  • Blob downloader
  • Multi-homing: support multiple user-facing network interfaces
  • Versioning and remote sync
    • execute in presence of out-of-band changes/deletions
    • support latest version: the capability to check in-cluster metadata and, possibly, GET, download, prefetch, and/or copy the latest remote (object) version
    • remote synch: same as above, plus: remove in-cluster object if its remote counterpart is not present (any longer)
    • both latest version and remote sync are supported in a variety of APIs (including GET primitive) and tools (CLI, aisloader)
  • Intra-cluster n-way mirroring
    • to withstand a loss of node(s) erasure coding is now optional
  • AWS S3 (frontend) API
    • multipart V2 (major upgrade); other productization
    • listing very large S3 datasets
    • support presigned S3 requests (beta)
  • List objects (job): show diff: in-cluster vs. remote
  • Prefetch (job): V2 (major upgrade)
  • Copy/transform (jobs): V2 (major upgrade)
  • AWS S3: migrate AWS backend to AWS SDK V2
  • Azure Blob Storage: transition to latest stable native SDK

See also: aistore features and brief overview.

Core

  • NVMe multipathing: pick alternative block-stats location !6432
  • rotate logs; remove redundant interfaces, other refactoring !6433
  • cold GET: add stats !6435
  • http(s) clients: unify naming, construction; reduce code !6438, !6439
  • don't escape URL paths; up cli !6441
  • dsort: sort records (minor) !6445
  • core: micro-optimize copy-buffer !6447
  • list-objects utilities and helpers; rerun list-objects code-gen: refactor and optimize; cleanup !6450, !6451
  • intra-cluster transport: zero-copy header !6455
  • Go API: (object, multi-object): ref !6456
  • add 'read header timeout'; docs: aistore environment variables !6459
  • core: support target multi-homing - comma-separated IPs (part one) !6464
  • package 'ais': continued refactoring; up cli !6466
  • support multiple user-facing network interfaces (multi-homing) !6467, !6468
  • when setting backend two (or more) times a row !6469
  • core: (begin, abort, commit) job - corner cases !6470
  • in-cluster K8s environment: prune and cleanup, comment, and document !6471
  • multi-object PUT - variations !6473, !6474
  • unify PUT and PROMOTE destination naming !6475
  • APPEND (verb) to append if exists; amend metadata (major) !6476
  • EC: refactor and simplify erasure-coding datapath; docs: remove all gitlab references !6477
  • list-objects: enforce intra-cluster access, validate !6480
  • EC: remove redundant state; simplify !6481
  • Go API get-bmd; follow-up !6483
  • EC: cleanup manager: remove rlock and unused map - micro-optimize !6490
  • copy bucket: extend the command to sync remote bucket !6491
  • extend 'copy bucket' to sync remote !6494, !6495, !6497, !6498, !6499
  • don't compare checksums of different (checksum) types !6496
  • when deleting non-present (remote) object !6502
  • move transform/copy-bucket from 'mirror' package to 'xs' !6503
  • don't create data mover in a single-node cluster !6504
  • multi-object transform/copy (job): add missing cleanup !6506
  • multi-object transform & copy !6507
  • core: abort all (jobs) of a given kind; CLI 'ais stop'; strings: Damerau-Levensthein !6508
  • revamp target initialization !6509
  • copy/transform remote, non-present !6510
  • revamp target initialization !6512, !6513
  • [API change] get latest version (feature) !6516
  • amend Prefetch; flush atime cache when shutting down !6517
  • amend metadata cache flushing logic (atime, prefetch, is-dirty) !6518
  • core: remote reader to support 'latest version' !6519
  • extend config ROM; follow-up !6520
  • Prefetch v2 !6521
  • backend error formatting; notification-listener name !6522
  • [API change] Prefetch v2; multi-object operations !6523
  • Prefetch v2; cold-get stats; put size !6524
  • [config change] versioning vs remote version changed or deleted !6525, !6526
  • add 'remote-deleted' stats counter; Prefetch: test more !6528
  • AWS backend not-found; job status; other cleanup !6529
  • core: refactor 'copy-object' interface, prep to sync remote => in-cluster !6531
  • [Cluster Config change] versioning vs remote version: remote changed, deleted !6532
  • copy/transform (bucket | multi-object); intra-cluster notifications !6533
  • revise/simplify 'is-not-exist' check; ldp.reader to honor sync-remote option !6537
  • pre-parse (log-modules, log-level); micro-optimize !6538
  • amend error handling: not-found vs list iterator; OOS !6539
  • jobs ("xactions"): add and log non-critical errors; join(error) and fiends !6540
  • [API change] list-objects to report 'version-changed' (new) !6541
  • list-objects to report 'version-changed' (new) !6543, !6545
  • list-objects to report: 'version-changed', 'deleted' !6546
  • list-objects to support (in-cluster <=> remote) diff !6547, !6548
  • copy/transform with an option to sync remote: prune destination !6549
  • copy/transform --sync: add stress test, extract "pruning" logic !6550
  • revise and refine object write transaction (OWT) !6554, !6555
  • Go API: amend 'wait-idle' helper method !6558
  • copy/transform '--sync': use probabilistic filtering !6559
  • refactor list-range-prefix iterator !6560
  • multi-object copy/transform with '--sync' option !6561
  • S3 API (on the front): fix list-objects !6562, !6563
  • multi-object copy/transform with '--sync' option !6564
  • core: reset idle timer; xaction names (micro-optimizations) !6565
  • core: ETag in response headers !6569
  • S3 API (frontend): validate object names; multipart pathnames !6570
  • copy/transform with '--sync' option: add scripted test !6571, !6573
  • backend: special case to return 404 instead of 403 !6575
  • productize Azure backend !6576, !6578, !6580
  • S3 multipart: write-through all parts !6585
  • multipart upload: write-through all parts !6586
  • multipart upload: add extended error message; add stress test !6587
  • all supported backends: revisit range read (make it consistent across) !6589
  • introduce blob downloader (new) !6592
  • xaction (job) descriptor: remove unused specifiers !6593
  • blob downloader: add dedicated (non-generic) control path !6595
  • blob downloader (new) !6596, !6599, !6603
  • multipart upload: fix s3cmd to run elsewhere !6600, !6601
  • blob downloader (new) !6605, !6606, !6608
  • blob downloader (new); remote AIS cluster !6613
  • silent HEAD(bucket) !6614
  • leverage erasure coding to provide intra-cluster mirroring (new) !6615, !6616
  • blob downloader (new) !6618
  • S3 (frontend): support presigned S3 requests (new) !6621
  • intra-cluster mirroring: add integration test (no limit) !6622
  • blob downloader (new) !6628, !6629, !6631, !6632, !6633, !6639
  • add target's get-cold-blob interface; refactoring !6634
  • AWS backend: nil client !6636
  • Prefetch via blob-downloader: add 'blob-threshold' option !6637, !6638
  • blob-downloader: user abort; expected checksum !6646
  • Azure: ETag as object version; build !6647
  • Azure: transition from preview to stable 1.x (major) !6648
  • AWS backend: use sync.Map instead !6649, !6651
  • (AWS, GCP) backend: log extended error info; RC5 !6653
  • S3: presigned S3 requests; bucket config: add max-page-size !6657

Python

  • v1.4.17 release !6431
  • add support for self-signed certificates with or without verification !6465
  • add 'latest' flag for GET !6536
  • latest flag for prefetch and copy !6542
  • release 1.4.19 !6544
  • stress test for copy w/ '--sync' !6552
  • fix pylint to pass !6556
  • test multi-object copy with '--sync' flag !6567
  • fix black formatter issues in github CI !6582
  • github-CI lint - follow up !6583
  • support range read (offset, length) !6588
  • update common requirements !6609
  • bump SDK version !6610
  • lint: add more !6454

Bench

  • aisloader-composer: install docker alongside latest cri-o on CentOS !6436
  • aisloader-composer: fix install-docker and update OCI inventory !6446, !6449
  • aisloader-composer: update OCI inventory; avoid using reserved variables in playbooks !6452
  • aisloader-composer: update dashboard with k8s only networking visualization !6453
  • aisloader: support latest-version !6581
  • aisloader: add '--cached' flag !6623

Build, CI

  • refactor common 'k8s' package; up cli mod; docs !6434
  • build/minikube: skip making cli !6437
  • gitlab-CI: scheduled pipeline changes !6442
  • upgrade OSS packages !6443
  • lint: enable gocritic "huge-param" !6457
  • lint: add gosec linter !6462
  • gitlab: add etl label & rule !6488
  • github-CI: publish pypi package for aistore !6492
  • build: upgrade all minors !6501
  • rename 'cluster' package !6514
  • 'api' package not to import 'core' !6515
  • tests, tests, and more tests !6530
  • CI: fix HDFS docker image !6566
  • CI: remove HDFS build and tests !6572
  • deployment: add jq to init container for parsing JSON in Bash scripts !6577
  • CI: update tgt cnt for test short !6579
  • gitlab CI: add short test for cloud providers and long test for Azure !6584
  • build: new linter !6624
  • add github issue templates !6630
  • build: release candidate 4 (rc4) !6640
  • build: rc7; fixes !6658

Documentation

  • blog: aistore fast tier cache !6444
  • blog: maximizing cluster bandwidth with multihoming !6642
  • document aistore environment variables !6459
  • aistore environment variables !6461
  • packages (meta, ais, cmn); docs !6463
  • in-cluster K8s environment: prune and cleanup, comment, and document !6471
  • erasure coding: remove all gitlab references from the docs !6477
  • update bucket.md, out_of_band.md, and CLI docs !6527
  • fix LRU config docs !6598
  • add blob downloader (readme) !6604
  • update etl-init usage instructions !6611
  • update main readme !6644
  • blob-downloader, Prefetch !6645
  • new arch/block diagrams !6655
  • v3.22: pre-release updates !6659

CLI

  • add iterate-fields separator; 'config log.modules' multi-choice !6458
  • 'put source-directory' vs. 'archive put source-directory': usability !6472
  • support '--template' option with no ranges - prefix only !6482
  • universal command line for multi-object operations (major update) !6500
  • copy (bucket | multi-object) !6534
  • invalid bucket (usability) !6535
  • fix transform bucket argument !6557
  • jobs: non-verbose x-start; wait for job-id !6614
  • set bucket props (usability) !6652
  • 'ais' with no arguments - show one page of the most important information; update docs !6660
Assets 8