Bot releases are hidden (Show)

aistore - 3.22 Latest Release

Published by alex-aizman 8 months ago

Highlights

Blob downloader
Multi-homing: support multiple user-facing network interfaces
Versioning and remote sync
- execute in presence of out-of-band changes/deletions
- support latest version: the capability to check in-cluster metadata and, possibly, GET, download, prefetch, and/or copy the latest remote (object) version
- remote synch: same as above, plus: remove in-cluster object if its remote counterpart is not present (any longer)
- both latest version and remote sync are supported in a variety of APIs (including GET primitive) and tools (CLI, aisloader)
Intra-cluster n-way mirroring
- to withstand a loss of node(s) erasure coding is now optional
AWS S3 (frontend) API
- multipart V2 (major upgrade); other productization
- listing very large S3 datasets
- support presigned S3 requests (beta)
List objects (job): show diff: in-cluster vs. remote
Prefetch (job): V2 (major upgrade)
Copy/transform (jobs): V2 (major upgrade)
AWS S3: migrate AWS backend to AWS SDK V2
Azure Blob Storage: transition to latest stable native SDK

See also: aistore features and brief overview.

Core

NVMe multipathing: pick alternative block-stats location !6432
rotate logs; remove redundant interfaces, other refactoring !6433
cold GET: add stats !6435
http(s) clients: unify naming, construction; reduce code !6438, !6439
don't escape URL paths; up cli !6441
dsort: sort records (minor) !6445
core: micro-optimize copy-buffer !6447
list-objects utilities and helpers; rerun list-objects code-gen: refactor and optimize; cleanup !6450, !6451
intra-cluster transport: zero-copy header !6455
Go API: (object, multi-object): ref !6456
add 'read header timeout'; docs: aistore environment variables !6459
core: support target multi-homing - comma-separated IPs (part one) !6464
package 'ais': continued refactoring; up cli !6466
support multiple user-facing network interfaces (multi-homing) !6467, !6468
when setting backend two (or more) times a row !6469
core: (begin, abort, commit) job - corner cases !6470
in-cluster K8s environment: prune and cleanup, comment, and document !6471
multi-object PUT - variations !6473, !6474
unify PUT and PROMOTE destination naming !6475
APPEND (verb) to append if exists; amend metadata (major) !6476
EC: refactor and simplify erasure-coding datapath; docs: remove all gitlab references !6477
list-objects: enforce intra-cluster access, validate !6480
EC: remove redundant state; simplify !6481
Go API get-bmd; follow-up !6483
EC: cleanup manager: remove rlock and unused map - micro-optimize !6490
copy bucket: extend the command to sync remote bucket !6491
extend 'copy bucket' to sync remote !6494, !6495, !6497, !6498, !6499
don't compare checksums of different (checksum) types !6496
when deleting non-present (remote) object !6502
move transform/copy-bucket from 'mirror' package to 'xs' !6503
don't create data mover in a single-node cluster !6504
multi-object transform/copy (job): add missing cleanup !6506
multi-object transform & copy !6507
core: abort all (jobs) of a given kind; CLI 'ais stop'; strings: Damerau-Levensthein !6508
revamp target initialization !6509
copy/transform remote, non-present !6510
revamp target initialization !6512, !6513
[API change] get latest version (feature) !6516
amend Prefetch; flush atime cache when shutting down !6517
amend metadata cache flushing logic (atime, prefetch, is-dirty) !6518
core: remote reader to support 'latest version' !6519
extend config ROM; follow-up !6520
Prefetch v2 !6521
backend error formatting; notification-listener name !6522
[API change] Prefetch v2; multi-object operations !6523
Prefetch v2; cold-get stats; put size !6524
[config change] versioning vs remote version changed or deleted !6525, !6526
add 'remote-deleted' stats counter; Prefetch: test more !6528
AWS backend not-found; job status; other cleanup !6529
core: refactor 'copy-object' interface, prep to sync remote => in-cluster !6531
[Cluster Config change] versioning vs remote version: remote changed, deleted !6532
copy/transform (bucket | multi-object); intra-cluster notifications !6533
revise/simplify 'is-not-exist' check; ldp.reader to honor sync-remote option !6537
pre-parse (log-modules, log-level); micro-optimize !6538
amend error handling: not-found vs list iterator; OOS !6539
jobs ("xactions"): add and log non-critical errors; join(error) and fiends !6540
[API change] list-objects to report 'version-changed' (new) !6541
list-objects to report 'version-changed' (new) !6543, !6545
list-objects to report: 'version-changed', 'deleted' !6546
list-objects to support (in-cluster <=> remote) diff !6547, !6548
copy/transform with an option to sync remote: prune destination !6549
copy/transform --sync: add stress test, extract "pruning" logic !6550
revise and refine object write transaction (OWT) !6554, !6555
Go API: amend 'wait-idle' helper method !6558
copy/transform '--sync': use probabilistic filtering !6559
refactor list-range-prefix iterator !6560
multi-object copy/transform with '--sync' option !6561
S3 API (on the front): fix list-objects !6562, !6563
multi-object copy/transform with '--sync' option !6564
core: reset idle timer; xaction names (micro-optimizations) !6565
core: ETag in response headers !6569
S3 API (frontend): validate object names; multipart pathnames !6570
copy/transform with '--sync' option: add scripted test !6571, !6573
backend: special case to return 404 instead of 403 !6575
productize Azure backend !6576, !6578, !6580
S3 multipart: write-through all parts !6585
multipart upload: write-through all parts !6586
multipart upload: add extended error message; add stress test !6587
all supported backends: revisit range read (make it consistent across) !6589
introduce blob downloader (new) !6592
xaction (job) descriptor: remove unused specifiers !6593
blob downloader: add dedicated (non-generic) control path !6595
blob downloader (new) !6596, !6599, !6603
multipart upload: fix s3cmd to run elsewhere !6600, !6601
blob downloader (new) !6605, !6606, !6608
blob downloader (new); remote AIS cluster !6613
silent HEAD(bucket) !6614
leverage erasure coding to provide intra-cluster mirroring (new) !6615, !6616
blob downloader (new) !6618
S3 (frontend): support presigned S3 requests (new) !6621
intra-cluster mirroring: add integration test (no limit) !6622
blob downloader (new) !6628, !6629, !6631, !6632, !6633, !6639
add target's get-cold-blob interface; refactoring !6634
AWS backend: nil client !6636
Prefetch via blob-downloader: add 'blob-threshold' option !6637, !6638
blob-downloader: user abort; expected checksum !6646
Azure: ETag as object version; build !6647
Azure: transition from preview to stable 1.x (major) !6648
AWS backend: use sync.Map instead !6649, !6651
(AWS, GCP) backend: log extended error info; RC5 !6653
S3: presigned S3 requests; bucket config: add max-page-size !6657

Python

v1.4.17 release !6431
add support for self-signed certificates with or without verification !6465
add 'latest' flag for GET !6536
latest flag for prefetch and copy !6542
release 1.4.19 !6544
stress test for copy w/ '--sync' !6552
fix pylint to pass !6556
test multi-object copy with '--sync' flag !6567
fix black formatter issues in github CI !6582
github-CI lint - follow up !6583
support range read (offset, length) !6588
update common requirements !6609
bump SDK version !6610
lint: add more !6454

Bench

aisloader-composer: install docker alongside latest cri-o on CentOS !6436
aisloader-composer: fix install-docker and update OCI inventory !6446, !6449
aisloader-composer: update OCI inventory; avoid using reserved variables in playbooks !6452
aisloader-composer: update dashboard with k8s only networking visualization !6453
aisloader: support latest-version !6581
aisloader: add '--cached' flag !6623

Build, CI

refactor common 'k8s' package; up cli mod; docs !6434
build/minikube: skip making cli !6437
gitlab-CI: scheduled pipeline changes !6442
upgrade OSS packages !6443
lint: enable gocritic "huge-param" !6457
lint: add gosec linter !6462
gitlab: add etl label & rule !6488
github-CI: publish pypi package for aistore !6492
build: upgrade all minors !6501
rename 'cluster' package !6514
'api' package not to import 'core' !6515
tests, tests, and more tests !6530
CI: fix HDFS docker image !6566
CI: remove HDFS build and tests !6572
deployment: add jq to init container for parsing JSON in Bash scripts !6577
CI: update tgt cnt for test short !6579
gitlab CI: add short test for cloud providers and long test for Azure !6584
build: new linter !6624
add github issue templates !6630
build: release candidate 4 (rc4) !6640
build: rc7; fixes !6658

Documentation

blog: aistore fast tier cache !6444
blog: maximizing cluster bandwidth with multihoming !6642
document aistore environment variables !6459
aistore environment variables !6461
packages (meta, ais, cmn); docs !6463
in-cluster K8s environment: prune and cleanup, comment, and document !6471
erasure coding: remove all gitlab references from the docs !6477
update bucket.md, out_of_band.md, and CLI docs !6527
fix LRU config docs !6598
add blob downloader (readme) !6604
update etl-init usage instructions !6611
update main readme !6644
blob-downloader, Prefetch !6645
new arch/block diagrams !6655
v3.22: pre-release updates !6659

CLI

add iterate-fields separator; 'config log.modules' multi-choice !6458
'put source-directory' vs. 'archive put source-directory': usability !6472
support '--template' option with no ranges - prefix only !6482
universal command line for multi-object operations (major update) !6500
copy (bucket | multi-object) !6534
invalid bucket (usability) !6535
fix transform bucket argument !6557
jobs: non-verbose x-start; wait for job-id !6614
set bucket props (usability) !6652
'ais' with no arguments - show one page of the most important information; update docs !6660

aistore - 3.21

Published by alex-aizman 12 months ago

Highlights

cold GET: extract and micro-optimize the flow
sync Cloud bucket
- leverage validate-warm-GET bucket config, and
- extend it to support non-versioned Cloud buckets, and
- optionally, delete (remotely deleted) objects
bucket sizing and counting:
- support very large buckets that are not necessarily present in the cluster;
- unify ais ls --summary and ais storage summary to utilize the same control message and flags
list, summarize, and lookup the properties of remote buckets without adding them to cluster's BMD
HTTPS:
- support TLS configuration to authenticate clients
- switch cluster from HTTP to HTTPS, and vice versa
optimize metadata cache
optimize capacity management
bug fixes, performance improvements

Core

set prime-time to amend local generation of globally unique IDs !6325
multi-object (archive, copy, transform) jobs: transport endpoint !6326
core: (maintenance, decommission, shutdown) transition w/ rebalancing !6327
core: (maintenance, decommission, shutdown) transition w/ rebalancing !6328
intra-cluster transport: make receive-side stats optional !6329
intra-cluster transport: reduce receive side contention !6330
fix channel full condition; rebalance-cluster; transport !6331
feature flags: add limited-coexistence; transport: track closed endpoints !6334
fix prime-time: add caller-is-primary; up cli module !6335
switch existing cluster between HTTPS and HTTP !6336
Go 1.21: use built-in min and max functions !6337
list-objects(remote-bucket-and-only-remote-props); Go 1.21 clear built-in !6339
Go 1.20: use typed atomic pointer, remove unsafe !6343
core: assorted micro-optimizations; remove read locks !6346
tweak multi-error join-err, remove error channel (minor) !6347
[API change] capacity management !6348
xxhash; field-align vol package !6349
bucket: new-query help; silent GET; test tools !6350
etl: adding fqn param to spec templates !6351
low-level control structs: bucket, namespace !6352
etl: Keras template fix !6355
etl: fix hello-world ais-etl tests !6356
core: don't recompute uname hash !6359
repackage HRW methods !6361
core: lom cache v2 (major update) !6362
refactor: downloader's diff resolver; control plane (receive BMD) !6363
core: lom metadata cache (cont-ed) !6365
dsort: error handling, assorted cleanups, more scripted tests !6366
core transactions: concurrency !6368
downloader: throttle; wait !6369
optimize cold GET !6370
global rebalance: log; minor edits !6373
core: update backend 'get-reader' API (all supported backends) !6374
core: validate-warm-get to support non-versioned buckets, and more !6375
validate-warm-get to support non-versioned buckets !6376
[API change] silent HEAD(object) request !6378
core: add load-unsafe (the faster way to load local metadata) !6382
total disk size: compute at startup, recompute on change !6383
[API change] new bucket summary; unify list-objects and summary !6384, !6386, !6387
add config.Rom to consolidate assorted "read-mostly" config values; refactor and unify !6388
[API change] new bucket summary (major update) !6390
mountpath jogger: support bucket query !6392
backend providers: do not include (checksum, version) if not asked to !6394
python: updated bucket info API !6395
feature flags: don't-add-remote & don't-head-remote; log: add s3 module; verbosity; !6398
support listing remote buckets without adding them to cluster's BMD !6399
concurrent HEAD(object) vs evict/create bucket - fix the race !6400
[API change] list and summarize remote buckets without adding remote buckets to cluster's BMD !6401
datapath query (dpq) !6402
Go-based API: response header to error message !6403
[API change] new bucket summary !6405, !6406
downloader: streamline and cleanup initialization sequence !6409
HTTPS: support TLS configuration !6410, !6411, !6412, !6413, !6414, !6415, !6416
assorted minor fixes !6417, !6418
core: cold GET: fast path & slow path !6419
cluster configuration: flip validate-cold-get !6420
downloader (major update); [API change]: xaction registry !6422
validate-warm-get: add scripted test utilizing remote ais cluster !6423
core: cold GET: fast path & slow path !6424, !6427
feature flags: add disable-fast-cold-get; show performance latency; up cli module !6425
refactor ais/utils !6429

Bench: `aisloader` and `aisloader-composer`

skip list objects for 100% put load !6332
composer: add playbook and script for intial aisloader copy !6333
composer: add support for aisloader --filelist option !6345
default value for duration should be infinite if num-epochs value is defined !6353
composer: add epochs option for GET workloads !6354
composer: add cluster name prefix to netdata sources for easier filtering !6357
new bucket not to be listed; usability !6358

CLI

typed does-not-exist error; misc !6358
always print dsort job description !6367
show cluster to report total num disks !6371
show performance: usability fixes, improvements !6379
show performance not to filter regex-selected zero columns !6380
attempt to copy/transform an empty remote bucket !6393
new bucket summary; evict multiple buckets in one shot; pretty print !6396, !6397
ais show bucket with an option to add remote bucket to cluster's BMD (effectively, create bucket) !6404
ais search: CLI command search results to include idiomatic extensions !6428

Build, CI, Deployment

tests: upon node shutdown: wait for the node to stop (tcp) listening !6338
CI: add gather-logs template for K8s tests !6340
deploy: ais with HTTPS in minikube !6364
build: bump urllib version !6372
tests: validate-warm-get (scripted) !6423
K8s playbooks: update kill aisloader command !6385
docs: validate-warm-get; assorted !6377
docs: add performance.md; inline help; rm all-columns flag (redundant) !6381
build: upgrade all minors !6389
CI: add checkmarx scan !6391
build: upgrade golangci-lint, add linters !6407, !6408

aistore - 3.20

Published by alex-aizman about 1 year ago

Core

tweak stop-maintenance logic; rebalance: cleanup log messages; assorted minor fixes !6288
do not timestamp err-aborted message !6290
[API change] dsort: remove extended metrics; add new counters; revise and refactor !6297
list-objects; house-keeper; aisloader, logger (assorted fixes) !6298
core stats: remove mutex and work channel - speed up !6299
slab allocator: remove stats mutex, do not sort !6300
consolidate and revise OOM handling !6301
ETL: require admin access to create & delete; add feature flag !6302
remove unused heartbeat tracker w/ minor ref !6308
reimplement keep-alive mechanism (major) !6309
keep-alive v2 (major update) !6312
keep-alive v2: remove timeout stats (control structure and code) !6317
keep-alive v2: add fast path !6320
micro-optimize get-all-running (jobs); atomic heard-from/timed-out !6321
node-restarted: remove 'lsof', use net dialer; fix node-decommissioning tests !6322

Tools and tests

CI: update fspath (aka mountpath) config for minikube-based aistore deployments !6289
aisloader: list and read s3 buckets directly !6291
aisloader: list, read, and write s3 buckets directly !6292
tests: K8s long tests (EchoGolang) fix !6293
aisloader: fix cleanup option for s3 bucket benchmarks !6294
aisloader: reimplement direct get from s3 - use SDK !6295
aisloader: show progress when listing s3 directly !6296
CLI: add show details param to etl !6304
tools: add check for ais etl deployment !6305
tools: add ETL_NAME var for CLI tests !6310
aisloader-composer: add playbook and script for clearing Linux Page Cache on all AIS targets !6311
aisloader-composer: add playbook for copying aws credentials !6314
tools: update check for aistore Kubernetes deployment !6315
CI: update github action version (all modules) !6316
CLI/ETL: support enumerated arg-type !6287, !6323

Build

upgrade all OSS packages (minor versions) !6313
transition to Go 1.21 !6318

aistore - 3.19

Published by alex-aizman about 1 year ago

Core

[API change] archive and download logs (feature) !6172, !6175
[API change] dsort: extend input format !6181
[API change] dsort spec; CLI: print job spec !6204
[API change] revise request spec (major upd) !6217
[API change] dsort: is now 'xaction' as well !6253
(downloader, dsort, ETL): disallow to run when out of space !6235
handle "DNS lookup fail" as one of the unreachable err types; nlog flush-exit !6164
when electing new primary; when joining nodes at startup !6165
k8s: Change prod k8s and docker default to not log all to stderr !6166
revise GFN !6167
stats runner is now responsible to periodically flush logs !6170
core: fail user attempt to abort global rebalance when !6184
new Go API; assorted fixes !6189
metasync BMD; up modules !6190
downloader: return not-found when not found !6196
start using scripted integration tests; CLI: 'dsort src dst spec' !6198
support S3 AWS profiles with alternative creds (feature) !6214
core: state transition => rebalance => (point of no return) !6216
amend low-level Go API check-response routine; add error type-code !6228, !6229
control plane: deserialize original error from call result !6230
xactions: when checking inactivity ("is idle") !6242 !6243
primary readiness vs cluster shutdown !6244
Go API: wait for xaction-related conditions !6245
assorted tuneups: space cleanup; housekeeping (HK) callback; log !6246
access control: when copying/transforming/dsorting to non-existing 'ais://' destination !6255
core: a call to update stats should never block !6257
core stats: add fast counters !6258 !6259 !6261
sparsify latency stats !6260
ETL: refactor and cleanup construction !6267
deploy/dev: updated minikube scripts !6272
new option to add Cloud bucket to aistore without checking accessibility !6275, !6277
un-throttle PUT mirroring; assorted changes !6278
feature: local generation of global (job) IDs !6280 !6282

Performance

Add distributed loader scripts and playbooks for using aisloader with multiple hosts !6156
pyaisloader: usability improvements !6215
Update Grafana dashboard to include latency statistics !6249
Reorganize benchmarks and related tools !6254
aisloader: no need to call rand for 100% or 50% read/write workloads !6256
aisloader-composer: add dashboard for DC network and disk !6266
aisloader: add an option to randomize gateways !6279
aisloader-composer: fix output files for GET bench !6283

Python

sdk: update ETL templates (docker migration) !6168
sdk: Release version 1.4.1 !6169
sdk: ETL templates (compress + ffmpeg decode) !6185
sdk: ETL templates (imagepullpolicy as always) !6191
sdk: adding keras_transform template !6200
sdk: ETL templates fix !6201
sdk: ETL templates (ffmpeg decode transformer) !6205
sdk: compress ETL template (updated usage) !6211
sdk: torchvision sample transformer ETL template !6221
sdk: fix comments (minor) !6240
sdk: update version !6248
sdk: increase timeout for torchvision transformer template (large image) !6252
sdk: updated torchvision transform ETL !6262
sdk: update dsort job info query and related tests !6265
sdk: switch ETL init code 'transform_url' boolean flag to 'arg_type' string !6269
docs: update ETL dev deployment for macOS !6163
ETL: keras template minor fix !6213
ETL: remove incorrect reference !6268
ETL: add 'arg-type=FQN' (new) !6271

Datasets (resize, resort, and shuffle)

[API change] dsort: extend input format !6181
dsort input format: iterate list, iterate range !6186 !6187
start using scripted integration tests; CLI: 'dsort src dst spec' !6198
add test scripts; memsys: init gmm only once !6192
refactoring and renaming !6193
move/consolidate error types; continued refactoring !6202
Go API change; add dsort/api.go; CLI: print job spec !6203
[API change]: dsort spec; CLI: print job spec !6204
CLI/dsort: extend inline help, pretty-print job spec; update docs !6206
dsort: continued refactoring (major update) !6208, !6209, !6210
free sgl on error; feature: any extension !6212
[API change] revise request spec (major upd) !6217
create destination on the fly !6218
record content path to retain full shard name !6219
output shard size estimation (rewrite) !6223
add is-compressed; refactor dsort-mem !6227
compressable shards (major) !6231
output ext; rcb buffer; fixes !6232
duplicated records (full coverage & stress); fixes !6233
fix tests; add stress !6234
rename subpackage, fix comments, refactor !6237
remove dsort-context, rewrite initialization !6238
static/stateless shard readers/writers; refactor and simplify !6239
two goroutines per each shard-distributing request !6241
[API change]: dsort: is now 'xaction' as well !6253
dsort: support generic abort-xaction API !6264
no need to block when sending shard records !6286

CLI

archive and download logs (feature) !6180
clarify "copying" vs "transforming" and "cached" vs "present" !6183
start using scripted integration tests; CLI: 'dsort src dst spec' !6198
dsort: extend inline help, pretty-print job spec; update docs !6206
dsort: Go API change; add dsort/api.go; CLI: print job spec !6203
'archive get' is now a shortcut (an alias) !6222

Build, test, and tools

add test scripts; memsys: init gmm only once !6192
tests and tools: cleanup around stop-maintenance, wait-rebalance" !6194
deployment: update local deployment script to allow target-only deployment with defined primary host !6195
deployment: optionally, skip deploying primary proxy !6197
start using scripted integration tests; CLI: 'dsort src dst spec' !6198
tools/generate shards: optimize buffer allocation !6224
deploy/dev: Add ansible deployment scripts for deploying locally on multiple nodes !6199
aistorage/CI docker image (lzma libraries) !6220
tests: init with cleanup and without !6226
CI: Retry stuck Python ETL tests in GitLab CI pipeline !6270
remove aisfs (FUSE) !6273
dev tools: readers; handle read from corrupted arch or non-arch !6250

Documentation

update getting started !6161
updated python sdk readme !6162
update ETL dev deployment for macOS !6163
update documentation with recent ETL changes !6173
CLI/dsort: extend inline help, pretty-print job spec; update docs !6206

aistore - 3.18

Published by alex-aizman over 1 year ago

Core

add htext to track restarted state; target run and misc !5966
cluster rebalance (scenarios) !5969, !5971, !5973, !5974, !5975, !5977, !5980, !5983, !5986, !5987, !5989, !5991, !5992, !5993, !5995, !6002
add 'cluster-ready' helper; use it to reinforce !5976
cleanup better when decommissioning; previous BMD at startup !5979
fs: reliable remove-all !5981, !5982, !5984
yet another buf pool !5985
do not modify cluster map when starting up; always skip logging idle disks !5988
rebalance (scenarios, major update) !5992
[API change]: core: rebalance (scenarios) !5993
rebalance (major update); when receiving new cluster map !5995
up modules; handle housekeeper registration race !5994
'not present in the loaded cluster map' and similar startup validation !5996
shutdown or decommission a node that's already in maintenance !5998
transport: never establish a streaming connection to the peer that's in maintenance (or will be) !5999
metasync just-in-time; assorted refactoring (minor) !6001
maintenance mode: pre & post vs keepalive & metasync; CLI: more colored cues !6004
shutdown is also 'maintenance'; docs: adding-removing intro !6005
add meta package !6006, !6007
ETL: add arg-type parameter when initializing with code !6008
archive v2: support empty template (tar entire bucket); atime !6013
keep poi.atime in nanoseconds !6015
archive v2: append to arch; refactoring !6017
archive v2: up modules !6018
archive v2: part four (major) !6019
archive v2: detect an empty tar when appending, and handle !6020
archive v2: part six !6022
archive v2: mime detection !6024
archive v2: extend 'append-to-arch' to support tar.gz !6025, !6027
archive v2: tar and tgz append; fixes !6028
log filenames; overlapping run vs node-restarted !6029
archive v2: multi-object append-to-arch !6030
archive v2: multi-object append-to-arch !6033
archive v2: multi-object append-to-arch !6034
cleanup disk utils (minor) !6035
ios startup: run the command only once !6036
hide AuthN secret !6038
archive v2: append to zip !6041
archive v2: append to msgpack !6043
add cmn/archive package !6044
archive v2: write and copy via new 'cmn/archive' !6045
archive v2: append via new 'cmn/archive' !6046
[API change] archive v2: MIME vs file extensions !6047
ios: cleanup lsblk cache; CLI: refactor get-node-arg; up modules !6048
archive v2: remove msgpack; refactor !6051
archive v2: add '.tar.lz4' serialization (new) !6053
archive v2: tar.lz4 cont-d !6054
archive v2: lz4 features; checksum !6055
s3 compat: run E2E tests with correct HTTP/HTTPS mode !6057
[API change]: append to arch if exists !6062
[API change] append to arch if doesn't exist; CLI cont-d !6064
checksumming and buffering vs reader-from !6074
core: content-length universally; revise write-json and friends !6075, !6076
archive v2: [API change] put (files, dirs) with an option to append !6081, !6082
archive v2: quiesce faster, refine continue-on-error logic !6083
core: double-check target-in-maintenance, quiesce faster !6084
archive v2: finalize cmn/archive package !6085
archive v2: finalize cmn/archive package !6086
log verbosity: core and modules !6087
http client: disable compression; core: undefer & micro-optimize !6066
append to (non-existing) arch: an option to create !6068
mem-pool alloc/free symmetry: copy/transform & archive !6069
copy/transform, multi-archive: refactor Rx logic and error handling !6071
log verbosity: core and modules; remote cluster !6089
log verbosity: core and modules; remote cluster !6091
ec: minor refactoring !6092
archive v2: WD basename; get with extraction; Range !6093
archive v2: WD basename; get with extraction; Range !6094
archive v2: tools/archive utils !6095
archive v2: tools/archive utils !6096
compile-out asserts; super-verbose logging; log module 'mirror' (ref) !6097
log verbosity at runtime; log modules; remove glog; unify (major update) !6099
fields iterator; size converter; log rotation (fixes) !6101
[API change] get-bucket-info to count remote objects !6102
[API change] get-bucket-info to count remote objects !6103
[API change] get-bucket-info (part three); docs and CLI !6106
list-objects vs buckets: revise and refactor, add validation, clarify !6107
list-objects: introduce optional args (ref, cleanup) !6108
list-objects: mem-pool msgpack buffers !6109
kvdb: remove redundant err-not-found; amend dsort, downloader, authn !6110
x-lso must idle more time !6111
log modules (part three) !6112
add nlog (new logger) !6113, !6122, !6124
do not log perf counters when there's no change; sort the names !6119
fix disk usage call for clusters on mac OS !6121
log etl events: spec parsed, pod ready, hpull/hpush !6125
cleanup fs.PathError; add object name validation !6127
extend dsort to support .tar.lz4 !6128
aisloader: cleanup output when running with json option !6129
xactions to return extended error info !6130
xactions: add and return errors (major upd) !6131, !6132
xactions: error handling cont-d; bucket-scope multi-object op-s !6133
jobs error handling and reporting; tests !6134, !6135, !6136
use Go 1.20 join-err; reduce default pre-election interval !6141
http request multiplexer !6142
dsort: use common cmn/archive pkg (major upd) !6143
archive v2: tar formats (USTAR, etc.) !6153
archive v2: extend list-objects and GET to operate inside archives !6155
archive v2: extend list-objects and GET to operate inside archives !6157
control plane: consistently propagate cluster map !6158
old EC metadata shouldn't terminate cluster-wide rebalance !6159
core: when primary goes down it notifies !6160

CLI

performance tabs: always show 'cluster idle' if idle !5997
stop-maintenance w/ no rebalance; stats idle-ness (minor) !6000
add transform-url flag (used when initializing ETL with code) !6010
misc. improvements !6011
mountpath completions; disable/detach; minor ref !6056
de-spaghettify put handler !6059
archive multi-object (cont-d) !6060
archive v2: CLI put, append, alias, docs !6072
move 'gen-shards' and extend it to support all formats !6073
archive v2: CLI: APPEND-to-arch is now on-par with multi-PUT !6077
archive v2: CLI: dry-run option is back !6078
archive v2: CLI: destination naming; dry-run; tips and examples !6079
list-objects: extended help, template with no ranges !6104
error and warning verbosity (major update) !6105
log modules (part two); CLI archive: pre-parse and add tip, advice !6100
archive help !6126
archive v2: CLI and docs !6149
archive v2: multiple CLI updates and improvements; is-archive bit !6150
remove k8s apimachinery pkg (minor) !6151

Python

sdk/python: Add option to pre-import modules when initializing ETL with code !5963
sdk/python: Refactor ETL to provide name once on object creation !5968
sdk/python: Release version 1.2.0 !5970
sdk/python: Fix python package links !5978
sdk/python: improve SDK testing, fix bucket eviction keep_md parameter !5990
sdk/python: Release version 1.2.2 !6012
sdk/python: Add dSort support in SDK !6032
sdk/python: Add wait to dsort abort test !6037
sdk/python: Fix dsort abort test on faster systems !6039
sdk/python: Add get_url option to bucket, object group, and objects !6042
sdk/python: Add interface for AIS sources that can be accessed via list of URLs !6052
PyTorch: Add new PyTorch DataPipe to iterate over URLs from various AIS sources !6058
update lint-tests to work with Python 3.11 (pytorch unsupported) !6061
sdk/python: Include user agent in SDK requests !6067
sdk/python Use msgpack content type when listing objects in bucket !6070
sdk/python: Add msgspec to pyproject dependencies !6088
sdk/python: Release SDK version 1.3.0 !6090
sdk/python: bucket summary + bucket info !6098
sdk/python: bck info integration test fix !6115
sdk/python: bck summary integration test fix !6117
sdk/python: bck summ + bck info fixes (revert and cleanup) !6120
sdk/python: pyaisloader !6123
pyaisloader: bucket utils (minor fixes) !6139
sdk/python: bucket summary and info (minor fixes) !6140
pyaisloader: total size/count fix !6146
sdk/python: Release version 1.4.0 !6152

Docs

add lifecycle.md !6009
WebDataset Blog Post part 1: Storing WebDataset in AIS !6014
WebDataset Blog Post part 2: AIS ETL on WebDataset shards !6016
update docs/tools and cmd/README !6050
WebDataset blog post pt 3 -- PyTorch Datatpipe with ETL !6065
archive v2: CLI put, append, alias, docs !6072
archive v2: CLI: destination naming; dry-run; tips and examples !6079
archive v2: CLI and docs !6149

Build, CI

add local build options to k8s scripts and fix local registry !6021
build: upgrade all minors !6040
add pyaisloader stage !6148
k8s: fix deployment scripts for compatibility with the latest aisnode image !6154

aistore - 3.17

Published by alex-aizman over 1 year ago

CLI v1.2
Python SDK v1.1.2
S3 compatibility and Botocore
API changes
Tests and Documentation
Core: bug fixes and improvements
Build and Continuous Integration
Extensions: Downloader, dSort, ETL

CLI

show all jobs !5645
start/stop job/xaction !5660
refresh rate and countdown; long-running 'show job' and friends !5651
'show log node-name' to mimic 'tail -f' !5652, !5654
add custom duration flag and logic !5655
'ais config (cluster|node|cli)', 'ais config reset', and friends !5656
bucket completions !5657
set-config to show all updates; tweak iter-fields reflection !5658
'show job' to aggregate all categories and support all selections !5661
transition to using job display names (major) !5663
start, stop, show jobs and xactions (cont-d) !5665
amend and restructure jobs !5666
running xactions (completions) !5672
tweak config json printout; get-config from memory !5673
update backend config !5674
update backend config (part two) !5677
add footnote, marshal message only once !5678
remove xaction term and subcommand (everything is job now) !5692
suggest (targets, proxies, nodes) !5694, !5696
revise bash completions script !5697
remove 'xaction' (term and subcommand) !5698, !5699, !5700
'show cluster': separate cluster nodes from all other (tab-tab) completions !5701
consolidate and refactor cluster map access !5704
tweak ais create bucket --props & ais bucket props set !5706
extend 'job start' to support (resilver, copy-bucket, rename-bucket) !5715
tweak listed props !5719
remove (cleanup) download and dsort jobs !5721
extend 'ais stop' to support --all|--regex !5722
'show job' verbose option; unify usage args; ref PUT/APPEND !5723
rewrite command-not-found logic; add similar commands !5724
show jobs (major) !5726, !5727
bash autocomplete ordering improvements !5728
improvements (usability) !5729
add bucket cp alias !5730
flag printable name; split 'show job' in parts; usability !5736
further unify stopping, waiting-for, and showing jobs !5744
revise & amend 'show rebalance' - all permutations !5761
universal start-end formatting; template refactoring !5762
jobs grouping by name and, within name, by UUID !5764
complete etl-name transition !5767
ETL tools, UUID (part one) !5745, !5746, !5749, !5753, !5754, !5763
fix download/dsort progress !5769
new table to show target statistics !5788
'ais show performance' (new) !5791, !5793, !5800, !5802, !5803, !5809, !5810, !5811, !5812, !5816
IEC, SI, and raw (bytes, nanoseconds) formatting (major) !5820
reduce code, simplify, cleanup !5821
IEC, SI, and raw (bytes, nanoseconds) formatting (major) !5823
disk stats: add average read/write sizes !5824
amend existing mountpath tab and add a new one !5833, !5834
expect node unreachable when iterating '--refresh' !5837
assorted usability; add 'no-color' config !5839
'ais show performance': average (GET, PUT, etc.) sizes on the fly !5840
support new API to reset stats !5841, !5843
'ais show performance': refactor throughput, add latency !5844
'ais show performance': finalize latency tab !5847
'ais show performance' cont-d !5848
'ais show performance': finalize top-level tab !5850
'ais show performance': add cluster-level throughput, beautify !5852
'ais show performance': alias 'stats' and remove older code !5853
'ais show performance': disk table v2 !5855
'ais show performance': finalize disk table !5857
'ais show performance': new mountpaths/disks/capacity table !5858, !5859
'ais show performance': finalize capacity table !5861
refactor and cleanup multi-object put !5862
multi-object PUT: source dir, list/range; matching pattern !5865
fix concatenation logic, refactor progress bar !5867
copy bucket: support progress bars (copied objects and size) !5870
consistent timeout management !5871
copy/transform a list or range of objects: add progress bar !5873
copy/transform with progress bar: style, reuse !5874
multi-object PUT !5876
rogress bar: all multi-object operations; universal 'wait-for' !5879
PUT multi-object - all flavors !5880
get multiple objects in one shot (""multi-object GET"") !5884
GET destination & assorted fixes !5882
copy-bucket: prepend prefix, command helps, examples !5889, !5891
more inline help !5892
assorted improvements (minor) !5900
fix downloading with progress bar enabled !5903
how-to text: how to reconfigure remote ais cluster !5932
add CLI compatibility warning (new CLI vs old cluster and vice versa) !5952
cluster membership-changing operations (shutdown, decommission, et al): improve usability !5956

Python: SDK library and ETL

Add unit tests for cluster class !5675
Add unit test for api client class !5676
Fix python ETL test workflow !5680
Add unit tests for Xaction class !5681
Add unit test for object class, fix object put data from a filepath !5682
Add unit tests for ETL class !5683
Add unit tests for sdk utils !5685
Restructure python subdirectories and containment !5686
Run python unit tests as part of default python test make option, include python ETL tests in all python test runs !5688
Only run python ETL tests when python labels are added !5702
update test utils to support running Python tests on Windows !5716
Add multi-object functionality !5720
Add tests and validation to object ranges, support leading zeros !5731
Add unit test for object group class !5732
Update documentation for python multi-object ops !5734
Use aws bucket for python sdk CI tests to test caching functions !5739
Add string template support to object groups !5741
Refactor all references from xaction to job !5742
Update ETL runtime defaults and add new python 3.11 option !5743
Fix remote bucket tests to avoid collisions !5748
Increment cloudpickle version and update github action to use 3.11 for tests !5756
Fix python test dependencies !5780
Refactor to simplify typing and tests !5789
Bump python SDK to 1.0.5 !5790
Update python sdk version !5792
Update pytorch integration README with compatibility issue !5794
Patch sdk to support torchdata integration !5795
Bump aistore package version !5798
Fix pylint and formatting !5799
Add ObjAttr type for returning additional object metadata !5805
Address lint warnings, general improvements !5808
Add writer option to object get !5813
Split README for different python projects !5814
Add PROMOTE functionality to objects !5818
Standardize pylint version, fix lint errors !5822
Improve object put behavior and add directory put options !5826
Use Pathlib over os.path !5827
Refactor multi-file put to bucket class !5830
Improve job interface and update promote options !5832
Update python package build tools, increase version to 1.0.9 !5835
Improve example documentation !5842
Improve input validation !5846
Set up proper logging, update constants !5849
Add job wait for idle status and fix job bucket filter !5863
Add multi-object copy !5866
Fix remote test fixture !5877
Add multi-object ETL !5878
Release 1.0.10 !5881
Expand multi-object examples !5883
Improve usability !5888
Follow-up for copy/transform prepend !5890
Improve object interface !5893
Add prefix_filter option to bucket copy !5899
Add flags and target options to bucket list_objects methods !5901
Release 1.1.0 !5910
Release 1.1.1 !5911
Improve python sdk examples !5913
Add cluster list_jobs, require id for individual job status query !5916
Add support for multi-object archive !5920
Update bucket params to use Bucket object, support namespaces !5925
Add prefix filter to bucket transform !5949
Release 1.1.2 !5950
PyTorch: add support for etl-name in AISDataset and AISFileLoaderIterDataPipe !5957

S3 compatibility and Botocore

Add botocore monkey patch alongside python SDK !5684
Move botocore and pytorch packages to python top-level, separate from sdk !5691
Add s3 compatibility testing with boto3 !5703
list-objects vs HEAD !5705
compute multipart md5 and set etag !5708
Improve s3 compat test documentation and update validated tests !5747
Pass S3 delete a list of objects !5854
fix infinite loop when listing objects; add bucket name to list object response !5864
return bucket creation date in UTC !5869
new flag NoRecursion to support S3 delimiter feature !5930

API changes and new APIs

add API to query multiple xactions via any IC proxy !5670, !5671
yet another API to query xactions (new) !5687
list-objects API: tweak listed props !5718
[API change] flatten xaction-snap control structure (major upd) !5740
[API change] ETL: tools, UUID (part nine) !5754
[API change] ETL: new query parameter to specify transform name !5755, !5765
[API change] init/start ETL to return xaction ID !5768
[API change] GET(object) !5770, !5772, !5773, !5775
[API change] remove xaction 'query-msg' (deprecated) !5783, !5784, !5785
get-object API: amend comments !5769
refactor api package (major) !5776
[new API] query metric names and kinds ('counter', 'latency', 'throughput', et al.) !5796
[API change] get node status !5811
[API change] core stats: consistency between stats-querying APIs !5817
[API change] PUT(object) !5828
[API change] amend capacity-disks-filesystems control !5831
[API change] add API to reset stats !5841
[API change] copy bucket: add prefix option !5894
[API change] copy/transform bucket: add 'prefix' option !5895
[API change] bucket summary: add 'prefix' option (major) !5896, !5898
[API change] return comma-separated list of job UUIDs !5927
[API change] extend & refactor system info, add load averages !5951

Core

xactions: display name, global & bucket scope, abort by kind (core) !5662
xaction snaps V2 (major) !5667
tweak reversing-to-remais logic !5668
bucket summary !5678
bucket summary: handle failure to start !5679
offline access to remote ais cluster buckets !5674
refactor cluster.Target interface !5689
unify time formatting; log: current date periodically !5705
fix HEAD(remote-ais-bucket) when the latter is not present !5707
xaction registry: cleanup list-objects separately, revise house-keeping !5725
copy-bucket & transform-bucket: revise generic stats !5730
global rebalance: fix cleanup upon (re)transmit failures; handle err-stream-terminated !5733
extend xaction snap: add source and destination buckets !5735
always return atime and checksum when listing 'cached' content !5738
assorted fixes and ref (cont-d) !5759
add has-active-peers; refactoring !5760
special circumstance to retry delete-object; lesser assert !5782
consolidate xaction API messages !5786
rebalance upon cold reboot, refactor join/kalive !5787
GET(object): add version-changed stats counter !5801
core stats: lockless (counter, gauge, special) !5804
core stats: revise error counting (major) !5806, !5807
core stats: introduce KindSize !5815
IEC and IS units of measurement (ref) !5819
core: micro-optimizations and cleanup !5825
rename mountpath and filesystem (ref pass) !5829
stats: serialize to SGL !5834
stats: rename for consistency !5836
config features: add fsync-put; don't destroy bucket when creating one !5845
core: micro-optimize BMD update !5851
core: refactor 'ios' !5856
extend the minimal/default subset of list-objects queried props !5868
refactor packages: 'nl', 'api', 'xact' !5872
add 'idles-before-finishing' state !5875
parse IEC and SI units !5882
multi-object GET (part two) !5885
streaming-x: multi-bucket matching, transport name !5886
long timeout to query all xactions !5887
copy-bucket: prepend prefix !5889
LOM as a remote data provider (DP) !5905
copy/transform remote objects that may or may not be present (major) !5906
copy/transform remote objects that may or may not be present !5908
copy/transform remote objects (part three) !5909
end-to-end protection: compute a given checksum type only once !5912
copy/transform remote objects (part four) !5914
core/list-objects: always primary !5915
copy/transform remote objects (part four) !5917
canonical bucket/object name, and more !5918
copy/transform remote objects (part five); canonical names !5919, !5921
store cleanup: delay deleting bucket directories when in doubt !5923
copy/transform remote objects (part six) - tests !5924
remote ais cluster: fix API forwarding logic; transform/copy test !5926
when node is in maintenance don't try to query it's state-and-status !5956
revise waiting/polling logic, reset probe frequency !5960

Build and Continuous Integration (CI)

add workflow job to build ais-init image !5646
build: move CI (image, make); move/update testing-in-docker content !5710
build: upgrade all minors !5711
Update minikube deploy script, remove from test run !5774
Update docker/ci image to include docker io !5778
Add uuid dependency to docker/ci image !5779
build: clear golang/x/net security alerts (two commits) !5838
up modules; cleanup redundant err helpers, simplify; objhead !5907
lint with new linter !5929
upgrade linter !5931
build: remove 3rd party atomic package (use standard) !5934, !5935, !5936
math/rand vs crypto/rand ref !5937
build: transition to Go 1.20 !5938
up modules; ""end-to-end protection"" and other comments !5939
add k8s pipeline to run manually !5941, !5943

Tests and Documentation

tests: amend all-xaction-status; golangci-lint deadline !5781
rename 'ais/tests => ais/test' !5860
tests: fix copy/transform eviction logic, and misc !5928
extend ETL tests, add from/to permutations !5940
docs: 'show job' !5946
docs: 'ais job' !5947
docs: list objects and buckets, copy buckets, and more; add help.md !5953
tests: fix test-promote !5763
tests: remote bucket versioning !5766
main readme, cluster.md, storage.md, config.md updated, and more...

Extensions: Downloader, dSort, ETL

downloader: do not renew xaction !5645
amend and refactor dloader package (formerly, 'downloader') !5647, !5648, !5649
downloader and dsort: list-active !5650
downloader: cluster-wide xaction ID (major) !5659
downloader: continued refactoring !5660
move ais extensions: etl, downloader, dsort !5664
k8s: remove unused ExecCmd !5712
default bucket props vs cluster config !5713
dsort: add support for EKM with JSON structure !5717
ETL: tools, UUID (part one) !5745
ETL: tools, UUID (part two) !5746
ETL: tools, UUID (parts three & four) !5749
ETL: CLI, tools, UUID (parts three & four) !5751
ETL: use (logical) "etl_name" instead of (unique) "etl_id" and "uuid" (major update) !5752
ETL: tools, UUID (part eight) !5753
Add pod health API endpoint for ETL !5771
ETL: assorted renames for consistency !5777
ETL stats; xaction base notif abstraction (minor) !5902
downloader: reduce "wrapping" and deferring !5904

aistore - 3.16.rc2

Published by alex-aizman over 1 year ago

aistore - 3.12

Published by alex-aizman almost 2 years ago

Remote AIS clusters
List objects
List and summarize buckets
S3 API
CLI
Documentation
Tests
Python SDK & ETL
Core
Refactoring & fixes
Tools
Build & CI

Remote AIS clusters

list remote ais buckets; fix remote namespace containment !5567, !5621
aliasing and bucket namespaces (major revision) !5622, !5623
ais gateways to cache remote ais clusters (information) !5630
extend remote ais errors - add cluster UUID !5631
remote ais aliasing !5632
[API change]: remote ais clusters (major) !5633, !5634
[API change] add HealthUptime; use it to show remote clusters !5636
remove support for multi-aliasing !5637
alias vs bucket metadata !5638

List objects

CLI to show CACHED (column) only when it's informative (and not when user asks ais ls ... --cached) !5554, !5555
rename list-objects structures; regenerate message pack !5542, !5543
amend and re-enforce single target rule for remote buckets !5556
introduce location property, phase-out target URL (property) !5557, !5559
add custom metadata, regenerate message pack; refactor !5564, !5565
list-objects version 2 (major): !5583, !5584, !5586, !5589, !5590, !5591, !5601, !5602, !5605, !5607
[backend API change] mem-pool 'list-objects' pages !5604
list-objects flow with a single target listing remote props (name, size, version) !5608
list remote pages v2: flow controls; ais gateways will now select single target to execute remote call !5609, !5610
list-objects to include (in listing result) custom object properties !5614
consistency in custom props across: get, put, head object, and list-objects !5615, !5616

List & summarize buckets

easy URL to support listing buckets via /gs, /az, and /ais endpoints !5486
list-buckets and get-bucket-info API; CLI summary table !5521, !5522
list-buckets version 2 (major update) !5524
unify 'list-buckets' and 'bucket-summary' !5530, !5531
shorter and faster list-buckets version: w/ presence but wo/ summary !5535, !5536- [API change] add presence filter; eliminate bucket-info control structure (is redundant); refactor !5540
list-buckets and bucket-summary cont-d !5541
bucket sizing: always compute two sizes: on-disk and sum objs !5544
bucket-summary on-disk (apparent) vs sum-all-objects, remote vs cached !5545
bucket-summary and list-buckets cont-d unification !5546, !5547
list-buckets by target: remote ais; hdfs and http; cloud; other !5567
rename/move bucket summary; align control structures !5568
list remote ais buckets; fix remote namespace containment !5621
[API change] list-buckets correctness for named bucket queries !5626

S3 API

multipart upload !5458, !5460, !5467, !5468
extend s3 API to support remote buckets (major) !5462
persistent multipart state (major) !5466
multipart GET !5472
multipart list active uploads and upload parts (major) !5474
return xml-formatted errors (compliance) !5480
multipart: refine and simplify !5482
revise list-buckets to include all providers !5485
always set header Server when redirecting !5495
amend s3 compatibility readme; add multipart example !5496
multipart: sha256 fix; md5 not checked but computed; aws s3api example !5500
xml-format errors (parts two and three) !5501
revise HEAD object implementation; multipart ETag; miscellaneous fixes !5504
compatibility: configurable root '/' access to AIS cluster !5572, !5573, !5574
compatibility: assorted error codes !5620

CLI

'ais etl' to replace extension in transformed objects !5418
standardize error and warning coloring !5493
don't log and don't show backend secret !5506
improve unknown-command messaging, provide hints !5512
add ais object ls command; add present flags !5514
support updated HEAD(bucket) API; 'ais ls' and permutations to show obj props !5517
static linkage with cgo disabled !5526
rename aws, et al. backend provider constants !5528, !5529
option to display sizes either in bytes or KiB, MiB etc. units !5532
rename sub-packages; show-unmatched & hide-header !5534
bucket display-name helper (and consistent usage) !5539
flip bucket listing default to 'present' (was 'all accessible') !5550
option --all to list both buckets and objects (in remote buckets) !5551
multi-feature completions; action-warn and action-done !5575
show config subsections in JSON; multi-feature completions !5576
amend config JSON printout !5577
extend and amplify: (config | section | json) !5578
is-time-not-set, is-object-not-cached (for consistency) !5611
atime & ctime !5612
throughput must be averaged over an interval of time since the previous request !5643

Documentation

amend main readme; add Reuse section !5375, !5376, !5378
blog: Python SDK: Getting Started !5373
add curl example to set backend bucket property !5380
python/sdk: ETL !5397
amend main readme: add PyTorch integration section !5406
remote backend; native integration !5407
revise ulimit, reference checked-in version of limits.conf !5429
update CONTRIBUTING.md !5459
blog: AIStore: Data Analysis w/ DataFrames demo !5469, !5473
update s3cmd readme - add multipart example !5483
restructure and extend getting-started; revise and amend Local Playground option !5484
easy URL: add readme with extended examples and comments !5487
REST API querying information !5641
REST API listing buckets (remote namespace, presence and existence) !5644
update ETL main readme, CLI readme, and examples !5471
amend s3 compatibility readme, add multipart example !5496
main readme: Installing from release binaries (add section) !5640

Tests

python ETL !5410, !5461, !5476
python IO communicator !5415
tests: skipping short (minor) !5416
pyhton/sdk: ETL stress test !5420
add s3 multipart test !5482
add ETL test for streaming with python runtime !5513
re-enable and amend object-properties tests across all providers !5515
amend list-buckets: add cases; up CLI !5569, !5570
reinforce multi-object tests, fix bucket-exists helper !5606
amend and fix multi-object range/list copy test !5619
amend list-buckets test that checks expected result !5631

Python SDK & ETL

ETL API part 1 !5370
sdk version !5374
code (re)formatting and misc !5381
reorg Makefile dependencies !5390
ETL APIs part 2 !5394
bucket ETL transform API !5405
bump sdk version to 1.0.3 !5408
fix IO communicator fixes !5413
fix transform bucket !5421, !5428
add examples !5441
add streaming runtime w/ refactoring and cleanup !5444
streaming init-code with chunk-size param (major) !5446, !5447
with temp debug !5448
init_code changes for before and after functions !5449
remove debug w/ minor ref !5450
functions filter (new), transform, and before/after !5451
runtime v2 fixes !5455
bootstrapper; by reference !5457
ETL API fixes !5464
update readme (etl init_code) !5475
extend hpush yaml with bitwise flags !5479
remove unused ETL functions; add test for streaming with python runtime !5513
update Jupyter notebook to reflect sdk changes !5548

Core

global rebalance: initiating new vs receiving delayed ACKs !5426
etl md: remove sgl, read-all !5470
error handling: introduce type codes (major upgrade) !5477, !5481
easy URL: support listing buckets via /gs, /az, and /ais endpoints !5486
easy URL: add readme with extended examples and comments !5487
user agent, internet browser: pretty-print JSON response !5492
add User-Agent to http request headers !5494
backward compatibility; remote backends !5498
consolidate http headers; add type-cast fail !5503
introduce error types unsupported and not-implemented-yet !5508
refactor GET(object) !5507
add lom.rename-file w/cleanups !5509
simplify lom.persist & recache/refresh !5510
update atime on PUT !5511
HEAD(bucket): extend the API in re: cluster metadata !5516
[API change] add api.GetBucketInfo !5519
enumerate and unify bucket and object 'presence' vs 'existence' !5520
fs: add size-bucket and refactor x-bucket-summary !5523
backend providers as simple KVs (minor) !5525
on the fly and (new) upon request remote bucket creation/addition !5538
object props: remove get-node, add location and helpers !5561, !5562, !5563
creating/adding buckets on the fly: further clarifications !5566
debug pkg: assert-msg is redundant; minor ref !5579
cos-assert; align (minor ref) !5587
periodic stats log: tweak idleness logic !5588
x-archive to always close data mover !5594
key-value driver; probabilistic filter !5595, !5596
embed stop channel !5597
transport: let individual streams define max header size !5598
intra-cluster transport: grow receive header, simplify rx-loop !5599
intra-cluster transport: within-read retries !5603
aws backend: rewrite list-objects versioning part, speed-up working with versioned buckets !5612, !5613
consistency in custom props across: get, put, head object, and list-objects !5615, !5616
simplify bucket try-init logic to add buckets on the fly !5628
notifications, control plane (ref) !5629
cluster and node 'started-up' is now uptime (was bool) !5635
[API change] add HealthUptime and use it to ais show remote clusters !5636

Refactoring & fixes

make deploy: mark aws secret optional to unblock startup !5379
consolidate xaction, move xs package !5383
move dbdriver, simplify !5384
GKE, cluster restart: amend false-positive SIE (storage integrity error) detection !5412
defer freeing etl-md sgl !5414
fieldalignment analyzer !5431, !5432, !5433, !5434, !5435, !5437, !5438, !5440, !5442
fix missing curl in aisnode k8s image !5427
http REST: parsing etc. helpers !5488
rename aws, etc. provider constants !5527
gofmt to replace interface{} with any !5533
refactor and revise objwalk pkg !5551, !5553
delete object vs 503 status; duplicated trname (transport endpoint name) !5614

Tools

scripts: revise github-release: tar.gz binaries, sha256(resulting tar.gz); naming convention !5411
add devtools/trand for tests; use strong randomization for product !5465
tools: revise get-primary, wait-for-cluster-state, wait-smap-sync !5571, !5580, !5581
aisloader: writing non-ais buckets !5600
tools: add buckets-contain helper !5627
scripts: amend local multi-cluster deployment to --override-backends and more !5617, !5618
scripts: add install_from_binaries !5640

Build & CI

github workflow: setting go version to 1.18.3 !5369
undo lint and Go version downgrades !5371, !5372
add .dockerignore !5382
cluster-minimal image: use alpine latest, remove azure !5385
deploy/k8s: fixes; minimal minikube deployment !5386
pull latest minikube image in GitHub workflow !5387, !5388
fully transition to Ubuntu 22; apt-get changes !5389
add limits.conf to all docker images !5391
deploy/prod: docker Makefile !5392
gitlab pipeline to build and push docker images !5393
add github workflow to build docker images !5395
two-stage Dockerfile to produce aisnode for minikube !5398
add Makefile for dev/k8s scripts !5399
add short CI stage to the k8s yaml template !5401
cleanup deploy/dev/k8s readme and scripts !5402
add cross-compile option !5409
upgrade all minors !5417
rebuild and use new docker images !5436
fix latest lint issues !5423
fix gitlab Dockerfile !5425
golangci: re-enable nolintlint, run go1.17, comment-out disabled !5439
run python/etl manually (was part of test-short) !5502
cli module to have its own Makefile !5505
github-release; never-nil (minor) !5422
lint: re-enable nolintlint !5424
update docker github workflow !5430
pylint: fix warnings !5453
golangci v.49 !5489
fully transition to Go 1.19 (part two) !5491
docker/cluster-minimal: configure aws and gcp backends !5499
gitlab/CLI: force back e2e test suite; upgrade and build !5552
upgrade all minor versions !5560
add early return (linter) !5582
update pylint settings to skip generated files !5642

aistore - 3.11

Published by alex-aizman over 2 years ago

Highlights

Python SDK v1.0.1 with PyTorch integrations
Authentication and Access Control
GitHub Action to deploy and test local playground
Bug fixes, documentation updates, performance improvements

Authentication and Access Control

CI: fix AuthN to run !5264
replace form3tech-oss/jwt and upgrade to jwt/v4 !5266
prep for production !5267
prep for production (major update) !5268
prep for production (part three) !5269
prep for production (part four) !5272
prep for production (aisloader) !5275
prep for production (clean_deploy) !5277
extend error info, add usage examples, tweak CLI help !5279
revamp ais auth CLI !5281
rename APIs for consistency; add CLI to update existing role !5283
E2E test: added basic AuthN test !5284
make cluster optional for role add/set; randomize E2E tests; unify auth-enabled checks !5285
rename AuthN entities for consistency !5288
add api package, isolate server internals (major) !5289
continued refactoring, assorted fixes !5291
add 'tok' package to consolidate all jwt calls; continued refactoring !5292
status 'forbidden' or 'unauthorized' to accompany all errors !5310
allow 3rd party client lookup ht://bucket !5312
separate enabling HTTPS for AuthN and AIS cluster when deploying !5313
oad-token: if user-given location is empty then !5314
secret handshake when adding/updating AIS clusters !5316
revisit AIS calling and retrying; unify errors !5317
login by non-superusers: require cluster (to be explicitly specified) !5326
assorted fixes, improvements !5333
introduce 'show-cluster' permission; reinforce admin access !5350

Python SDK

add copy_bucket method !5225
add method xact_status !5226
move test to cluster ops file !5232, !5233
add xaction start !5237
restructuring (adding pytorch) !5238
PyTorch plugin (dataset/aisio) implementation for aistore !5242
fixed bug - list empty buckets !5251
changed error handling !5258
add iterator for bucket objects !5259
improved pytorch plugin !5274
release 0.9.0 fixes !5276
cluster health api !5280
health check for non existing urls !5282
syncing aisio.py !5286
rename_bucket() port !5304
test suite changes !5305
makefile dependency fix !5307
cluster health api !5278
added aisio example !5322
restructure api (bucket) !5323
fixing aisio example !5325
aisio example fixes !5328
restructure cluster !5335
obj streamer fixes !5337
blog post for ais file loader and lister !5338
using session for requests !5341
restructure object !5342
object method name changes !5343, !5346
restructure fixes !5348
bucket api fixes !5351
object get fix !5352
restructure xaction operations !5353
fixes and release prep !5355
minor readme edits !5356
readme changes wrt pytorch !5357
api docs and make file fixes !5359
removing aistore.pytorch imports from main init !5360
makefile fix (pipx path) !5367

GitHub Action

add github action for building ais !5265
add github workflows for lint, python-tests and build !5270
add test-short github workflow (Linux) !5271
github/workflow: changed go lint version !5366

CLI

confirm assorted operations: decommission, rm bucket, rm -rf !5224
revamp ais show cluster; add K8s POD names !5243, !5244
bucket extra-props-to-update; filter by provider !5256
allow setting permission on per-bucket basis for a role !5309
revise 'show config', add nested completions !5339
revise 'set config', add local vs inherited !5344
add --cluster filter flag to 'auth show role' command !5349

Bucket Summary

revise bucket summary (major) !5230
bucket summary: micro-optimize and simplify !5262
bucket summary: multi-bucket; add min/max/avg counters; fixes !5299
bucket summary: apparent vs on-disk sizing, formatting clarity !5300, !5302

Common environment and system filenames

move and refactor common environment; AuthN version and build time !5318
filename constants; environment variables !5324
consolidate all configs under $HOME/.config/ais !5327

Misc. Features

warn when decommissioning cluster, support '--rm-user-data' option !5221
ability to decommission node and join it back with a new identity !5222
anonymous public-access buckets: list-objects anonymously (w/ API change) !5330
s3: the capability to set custom s3 endpoint !5257, !5260

Docs

tweak ETL markdown !5227
updated getting_started and etl markdown !5228
changes to standalone deployment README !5253
minor indentation fix to standalone readme !5254
standalone ec2 deployment instructions and benchmarks !5261
python api reference !5294
python_api.md fixes !5297
bucket.md to reflect 'summary' changes !5308
update 'ais ls', add anonymous and more !5336
main readme !5358
s3 compatibility, 's3cmd' configuration and getting-started !5361
main readme, index.md, python.md !5362
amend and extend cli/config.md; fix config node reset !5363
add 's3cmd.md' !5364

Tests

random page, remote bucket !5234
remote bucket, object props !5235
extend cleanup to support (prior to test, remote bucket) !5240
ensure more !5245
fix iter-fields unit for recently added config extras !5290
reverse proxy test against GCS !5329

Build

upgrade gopkg !5223
fix statsd github URL !5229
new (and supported) JWT packages !5266

Performance micro-optimizations, bug fixes, refactoring

aisloader: rewrite SGL freeing piece !5236
refactor low-level is-bucket, bucket-init, and friends !5239, !5241
BMD: speed-up bucket init; add global namespace uname (const) to avoid allocations !5246
BMD atomic update vs backend props !5247
reuse buffer when iterating range template !5248
devtools: refactor and simplify !5249
bucket extra-props-to-update: include readonly; CLI: filter by provider !5256
yet another case to skip err logging block devices !5273, !5295
cluster config: allow critical updates !5298
capacity: periodic log and refresh; version and build time; fixes !5315
local playground: loopback device sizing; fixes !5321
refactor and move docker pkg (formerly containers) !5331
list-objects flags: unify, rename, document, and reference !5332
config: restart-required selection; feature flags !5334
blog image fix !5347
assorted fixes and refactoring !5354
CI/CD: skip-ci & python-tests-only conflict fix !5365

aistore - 3.10

Published by alex-aizman over 2 years ago

Highlights

Global (cluster) config meta-version 2
Bucket metadata (BMD) meta-version 2
Python SDK
Build: Go 1.18; upgrade all packages
MessagePack archiving

Global Config v2 & Bucket Metadata v2

bump up cluster config's meta-version; handle backward compatibility !5122, !5123, !5124
redefine and extend feature flags !5132
capacity and space watermarks (cleanup, low/high, OOS) !5143, !5144
BMD meta-v2 with backward-compatible logic to load v1 !5145, !5150
BMD meta-v2: simplify mirroring configuration !5151
add transport section !5153
support loading previous meta-version !5154
readonly knobs; backward compatibility !5155
introduce cos.size type and "kb", "mb", etc. suffixes !5156
cos.size vs backward compatibility !5157
add memsys section and tunables !5158
log flush time interval !5159
join-cluster-at-startup timeout !5161
add log-stats time, add validators !5171
continued refactoring !5172
add tcb (transform-and-copy-bucket) section; stream bundle multiplier; compression !5173
revise docs/configuration.md !5180

`msgpack` archiving; list, create, and read `msgpack` formatted shards

msgp (generic) shard !5100, !5101
fix preambles in the msgp generated files !5086
alternative msgp; sgl as io.ByteScanner; Python unpack to test !5103
assorted text fixes; more msgpack !5104
archiving formats: add msgpack (in addition to tar, tgz, zip) !5107, !5108
add integration tests !5100, !5101

Python SDK

basic bucket and object operations !5109
introduce pydantic; refactor list-buckets !5120
CI: add Python SDK test !5112
improve list-objects API !5163
refactor API calls for usability: !5128
move direct requests call out of list objects, create and delete bucket, cluster map !5130
put-object & head-bucket API !5141
improve list object API !5163
fix return type in _request_raw method !5164
add head-object method !5166
add setUp and tearDown in tests !5167
improve creating temporary file !5168
add evict-bucket and delete-object; add props argument for list-objects !5175
get-object returns a stream instead of the entire content !5184
update README, docs !5178, !5190
fix naming; add read-all method for streams !5196
list_object to support pagination !5198
improve object streaming; rename msg.py -> types.py !5203
Python SDK tutorial !5218

CLI

show cluster config feature flags in human-readable format !5133
fix setting backend bucket props !5152
add ais object rm --all to remove all in one shot !5195
verbose and non-verbose multi-object operations !5197

Tests

re-enable smoke-test for Cloud buckets !5098
msgp (generic) shard !5100, !5101
add missing wait for put-copies; BMD string !5204
failback; devtools: wait-cluster; transport !5209
dummy keepalive tracker !5211

Docs

website: tech blog on promoting local and shared files !5082
explain msgp usage; add msgp-update to Makefile !5085
update msgp generator usage !5094
msgpack !5110
revise and amend configuration.md !5180
performance tuning !5115, !5116
development, first-time usage, production deployments (summary) !5127
getting started (recommendations) !5129
local playground, tracking access time (more edits) !5139
fix references, add a reference !5140
erasure coding (space utilization ratio, IO performance) !5142

Build

upgrade fasthttp package !5092
upgrade direct dependencies !5096, !5097
transition to Go 1.18 !5106
upgrade all dependencies except 1.18 !5113
upgrade all dependencies except 1.18; v3.10-rc2 !5181
upgrade all dependencies including 1.18; v3.10-rc3 !5186
upgrade all minors !5200

Bug fixes and performance improvements; continuous refactoring

skip bucket name, etc. validation when initializing lom !5083
follow-up !5084
unsafe string: inline and use it consistently !5087
write-msg-pack (ref) !5088
control-plane broadcast & unicast: result value factories and decoders !5089, !5090, !5091, !5093, !5095
global rebalance: dynamic reg/unreg streams; housekeeper name suffix !5099
fix sending http request body on redirect: recompute content-length !5102
not restoring atime when !5105
multi-object operations !5111
local from-scratch deployment: add user-friendly tips !5114
follow-up !5117, !5118
write-policy: both metadata and data; bump up config's meta-version; backward compatibility (major) !5121
deploy: fix Dockerfiles and related scripts !5126
assorted usability tweaks !5134
local playground: print ports AIS listens on !5135
remove bench/soaktest; rewrite devtools/readme !5136
config: write policy for data must be 'immediate' !5137
docker + LVM deployment: no disks fix !5146
move mock-bmd => cluster/mock (refactoring) !5147
docker + LVM deployment: no disks !5148
local playground: pgrep to check; quiet run-cmd !5160, !5162
fix deploy.sh script for older bash versions !5165
target startup: skip docker union mounts when enumerating drives !5169, !5170
dedup fix HEAD(obj) system attributes !5174
get-object lock/unlock (refactoring) !5176
URI parsing, config validations, and assorted ref !5177, !5179
making "rlock" exception to forcefully remove corrupted !5182
dont-lookup-remote-bucket = true: proceed silently, log-wise !5183, !5185
remove +build directive (obsolete) !5188, !5189
get random proxy (ref) !5191
cluster map (ref); node equality !5192
PUT object vs remote backend returning status=503 !5193
multi-object operation, range template (ref) !5194
aisloader works !5199
when shutting down warn with details !5201
api: xaction wait-IC min waiting time !5202
compute-checksum and clone-md (ref) !5205
memsys: mem-free to include buff/cache !5206
memsys: revise sys/mem (major) !5207
lif to lom conversion: free on error; EC error handling !5208
revise xs/rename-bucket !5210
deferred unlock via lightweight lif (ref) !5212
lom as a reader, lif as unlocker !5213
get-from-neighbor (gfn): reg hk housekeepr under lock; abort to self-expire !5215
lom copy: allocate with name !5216
lom xattr (ref) !5217

aistore - 3.9

Published by alex-aizman over 2 years ago

AIS v3.9 is substantial productization and performance-improving upgrade. Much of the codebase has been refactored for consistency, with numerous micro-optimization and stabilization fixes across the board.

Highlights

promote: redefine to handle remote file shares; collaborate when promoting via entire cluster; add usability options; productize;
xmeta: extend to also dump in a human-readable format: a) erasure-coded metadata and b) object metadata;
memory usage and fragmentation: consistently use mem-pooling (via sync.Pool) for all control structures in the datapath;
optimistic concurrency when running batch prefetch jobs; refactor and productize;
optimize PUT datapath;
core logic to deconflict running concurrent xactions (asynchronous jobs): bucket rename vs bucket copy, put a node into maintenance mode vs offline ETL, and similar;
extend and reinforce resilvering logic to withstand simultaneous disk losses/attachments - at runtime and with no downtime;
stabilize global rebalance to successfully pass multiple hours of random node "kills" and restarts - node-left and node-joined events - in presence of stressful data traffic;
self-healing: object metadata cache to support recovery upon mountpath events (e.g., drive failures);
error handling: phase out generic fmt.Errorf and consistently use assorted error types instead;
additional options to speedup listing of very large buckets (list-objects);
numerous micro-optimizing improvements: fast datapath query (DPQ) and many more.

Promote files and directories

refactor as a 2-phase transaction and auto-detect file share (initial) !4929
auto-detect file share and distribute the work between target nodes !4945
add test; add target node => IC notifications !4975
extend test coverage; reinforce global UUID when promoting via entire cluster !4976
rename api.Promote; add test permutations and checks !4985
remove redundant control structures; cleanup !4987
add API options delete-src (delete source) and overwrite-dst (overwrite destination)!4988
fix extra-copy optimization with full refactoring !4989
revise/optimize HEAD(object) implementation and utilize it when promoting with overwrite-dst=false (major) !4991
extend object write transaction (OWT) to support the flow !4992
support in (i.e., transmitted), out (i.e., received) and locally-promoted stats counters - files/objects and bytes !4993
introduce confirmed file share; add user option not to auto-detect file share !5019
CLI: add overwrite-dst and delete-src E2E tests !5024
consolidate control, eliminate ambiguity !5045
increase test coverage !5047, !5063
add all test permutations to cover (ais | cloud | remote-ais) bucket vs. (non-redundant | EC | n-way mirror) !5068

ETL

add CLI to view stored ETL code and specification !4925
handle target-down; test !4933
redefine and improve ETL API (!4947, !4966, !5022), including:
- manage (CRUD wise) persistently stored ETL definitions
- eliminate redundant URL path parameters
- enforce uniqueness of the user-provided ETL name
remove (obsolete) embedded ETL-specific annotations from the init spec (pod template)
support stopping and restarting ETLs !5005, !5056
update ETL docs and fix minor bugs !4984, !5022

Global Rebalance

global rebalance status: always respond with total (cumulative) stats counters !4905
generic fs.Walk for global rebalance (refactoring) !4889, !4930
get-status & health !4934
global rebalance status: reimplement to optimize !4936
devtools: merge WaitForRebalanceToStart and WaitForAllResilver !4937
tweak/optimize receive logic !4994
abort via stage notifications from other target nodes (major) !5015
transport streams vs receive errors; assorted fixes !5040
tweak preemption logic (when rebalance triggering events arrive back to back) !5057
assorted fixes: global rebalance vs n-way mirroring & resilvering !5071, !5072
consistent renames and continuous refactoring !5075

Resilvering (in presence of drive failures and attachments)

tools and stats: wait-for-all-resilvers, multi-snap API !4888
resilvering vs copy management (major) !4865, !4866, !4867
resilvering: tweak is-active time interval !4882
support losing multiple disks (mountpaths) simultaneously !4884
multiple overlapping add/remove disk operations: fixes !4894
resilvering as scrubbing: recover objects to their expected (default) locations !4900
resilvering: interval-of-inactivity multiplier !4974
resilvering under stress in presence of lost mountpath(s) !5058

Asynchronous Jobs (aka `xactions`)

when aborting and propagating abort to the control-plane caller, make sure not to lose the original cause for the abort !4886
fix put-xaction finishing logic !4887
aborting jobs: propagate the original cause through channels and APIs !4890, !4891
revise lookup by only-running and/or by UUID !4897
move xaction and xreg packages with refactoring !4898
clarify running vs not finished xaction !4908
registry: fix matching logic, remove redundant code !4911, !4912
registry: amend housekeeping !4913
registry: continued refactoring and cleanup !4914
"limited coexistence" between running and about-to-run services (new) !4915, !4916, !4917
xact package: revise and optimize abort-checking concurrency !4923
registry: continued simplification and cleanup !4940
reinforce global UUID for all cluster-wide xactions !4978
IC notifications vs transactional xactions: same rules for all !4982
more stateful info: propagate xaction reference all the way into local PUT flow (major) !4995
copy-object -- xaction -- promote: continued refactoring !5002
registry: micro-optimizations and cleanup !5028

CLI

PUT: add an option not to load (skip loading) object metadata; amend docs; refactor and cleanup !4859
add a command to view ETL code/spec !4925
fix: do not add --help flag to the subcommands of subcommands !4926
amend 'show config' to include CLI config (in addition to cluster and local configs); fix cluster-unreachable error !4983
revise 'flag-is-set' for Boolean flags !5021, !5023, !5030
copy bucket: add --force option !5042
add start etl command !5056
ais show cluster: add support for refresh=<time-interval> and count options !5076
update CLI docs !5078
enable 'ais show storage' and friends to run continuously and refresh periodically !5079

Testing

test fixes to align with changes in the core !4861
add ensure-num-mountpaths helper, and reinforce !4892
use api.WaitForXaction instead of tutils.WaitForXactionByID !4893
re-enable one fs-checker test, allow more time for mountpaths !4903
add more checks when downloading object !4910
extend CLI e2e promote test !4932
WaitFor follow-up !4943
fix e2e AuthN messages !4952
retry upon failure to recover a damaged erasure-coded object !4986
amend and extend EC tests !4990
revise and enable bucket-rename-and-copy test !5060

CI/CD (continuous integration)

add CI job that runs on multiple cloud buckets !5027
add 1.18-rc1 version to build check !5044
add test-short-minimal to test a single-node cluster !5046
make AWS_REGION global env variable !5050
update test-long stage !5059

Bug fixes, performance improvements; continuous refactoring

LOM load to return distinct types: syscall-error and corrupted-error !4849
LOM vs n-way mirroring: fix and revise caching of the metadata !4850
list-objects: add fast mode --only-names !4851
introduce permission to overwrite disconnected backend !4852
api: refactor PUT API; fix devtools !4853
optimize PUT latency by allowing not to load object metadata !4854
aisloader bench: do not run goroutine per each PUT request !4855
reinforce access time atime (major) !4856
reintroduce no-metadata error; fix n-way stress; refactor !4857
build: fix deprecation warning on MacOS !4858
when copying objects differentiate between copying == mirroring and all other scenarios !4860
simplify LOM from-fs logic !4862
general: don't use regex to validate names and UUIDs !4863, !4864
assorted fixes !4868
preserve atime across LOM caches !4869
storage cleanup: leftover copies, corrupted and missing metadata !4870
refactor cmn and api packages !4871
list-objects: use-cache option !4872
consistently use HTTP status 507 throughout; assorted fixes !4875
eliminate redundant mirroring !4876
get-cold (aka cold-GET) follow-up !4877
object write transaction (OWT) fusion !4878
prefetch: support optimistic concurrency (major) !4879
name locker: fit two structures into 24 bytes !4880, !4881
disable/detach mountpath: graceful (admin request) and immediate (FSHC) !4883
move health package under fs/health !4885
bucket summary and obj-list query: move, refactor, and simplify !4895, !4896
control-plane: always free call-results back to pool !4899
api: eliminate code duplication !4901
general: deprecate and remove query objects !4902
refactor ais package !4904
control-plane transactions: refactor, reduce code !4906
initial ETL get API implementation !4907
control-plane transactions: follow-up !4909
introduce read-only (but still configurable) timeouts: cplane-operation and max-keepalive !4918
intra-cluster transport streams: tweak termination logic !4919
slab allocator: amend pooling of the SGL control structures !4920
transport streams: tweak termination logic !4921
transport streams: revise transmit <=> terminate concurrency, optimize !4922
refactor transaction types (minor) !4927
list-objects: name-only option (follow-up) !4928
fs: non-recursive walk in lexical order !4931
fs: refactor bucket-traversing logic, eliminate nested closures" !4935
list-objects and bucket-summary: refactor target side !4938
LRU & storage cleanup: clarify when-previous-is-running !4939
transport: redefine client callback to return error !4941
transport: consistent drain-and-free cleanup on the receive side !4942
lint: gofumpt & gocritic !4944
list-objects buffering, caching: refactor, optimize !4946
etl: intuitive RESTful URL paths - Part 1 !4947
feature flags: refactor, enforce intra-cluster requests via API endpoints !4948
API-level JSON messaging: uniformity & consistency !4949
core: fast URL query parsing (major) !4950, !4953, !4954, !4955, !4956, !4957, !4958
tools: add support for EC metafile to xmeta !4959
bench: revamp lstat !4960
api package: memory pooling !4961
follow-up: DPQ; t.Helper !4962
api package: memory pooling !4963
api package: memory pooling !4964
fs and cos packages: alternative slightly faster fstat to check existence !4965
etl: change init-code and init-spec; intuitive RESTful APIs - part 2 !4966
fix the logic to attach remote cluster during early startup !4967
api package: memory pooling !4968
api/object, aisloader: continued refactoring !4969
core/cluster: rewrite max-version decision logic !4970
core: call-args control structure !4971, !4972, !4973
tools: xmeta support for LOM !4977
IC notifications: eliminate redundant aborts !4980
etl: implement ETL Stop & Delete APIS; intutive RESTful API part 3 !4981
docs: updates to reflect all ETL API changes !4984
alloc/free put-obj; mem-pool !4994
cos package: inline usage of one-time constants !4996
ais target: refactor copy-object, copy-reader logic !4997
double transactions begin timeout; send-remote and `OWT !4998
target-to-target copy object: remove local-only option !4999
URL query parameter (ref) !5000
go-vet xmeta !5003
intra-cluster PUT vs user PUT: further differentiate and account for !5004
etl: add API to start ETL !5005
error processing: wrap errors to retain the types (major) !5006, !5007
tools: fix linting when golangci-lint is installed differently !5008
cmn package: remove LogLevel and Vmodule fields from config !5009
simplify error reporting and attribution (major ref) !5010
lint-1.44.2 !5011
error processing and attribution !5012
keep-alive cluster nodes: major refactoring, cleanup, code reduction !5013
transport: transmit/receive unsized objects resulting from streaming transformation !5017
etl: delete operation must be controlled by primary gateway !5022
apc: new package for API constants (major ref) !5026
copy/transform bucket: add 'force' option !5029
apc package: move metadata-write and refactor bucket-props validators !5032
apc package: move access control (ref) !5033
apc package: move action message (ref) !5034
apc package: move list-objects control message !5035
apc package: move copy/transform bucket message (ref) !5036
common bucket structure (major) !5037, !5038
fix: do not use jogger-group bucket inside individual jogger's visit-object callbacks !5048
core: two distinct methods to initialize LOM (major) !5049
backend bucket: fix returned AWS error to include details !5051
assorted fixes !5052, !5053
cloning LOM (metadata): follow-up !5065
self-healing: object recovery in the GET path !5069
assorted fixes: object recovery vs runtime metadata extension !5070
avoid duplicate FQN parsing when traversing buckets and visiting content types !5073

aistore - 3.8

Published by alex-aizman almost 3 years ago

Highlights

ETL: !4621, !4624, !4633, !4649, !4681, !4702, !4780, !4790, !4802, !4831, !4833
storage cleanup: !4632, !4741, !4748, !4753, !4754
custom user-defined object metadata, system Cloud-specific metadata: !4655, !4657, !4659, !4661, !4663, !4667, !4668, !4670, !4673
reinforce and protect ais volume and associated metadata: !4683, !4684, !4698, !4699, !4701, !4703, !4813, !4814, !4834, !4837
safely add and remove mountpaths (disks) at runtime !4721, !4722, !4728, !4729, !4735, !4736, !4740, !4744, !4789, !4818
"easy URL": support gs/bucket/object, s3/bucket/object, and similar (easy) URLs across all backends !4711
introduce node standby mode !4688, !4689, !4691
performance monitoring with a scope node | batch job | cluster !4792, !4793, !4794, !4798, !4800, !4810, !4812
support ais targets with no disks (feature) !4825
Kubernetes Operator v0.9

CLI

fix file name and message when extracting a file from an archive !4602
templates: circumvent index wrap around for aliases !4603
multi-object transform and copy: continue-on-err; add similar keywords !4605
show object (and its variations): improve usability and fix minors !4674
show config improvements; follow-up !4676
add command to immediately unregister node from cluster map !4693
new command ais storage cleanup !4704
attach mountpath and detach mountpath (ref) !4705
show storage subcommands !4713
--wait option for storage cleanup command !4773
report human-readable stats in show job xaction !4788
add commands to manage AIS CLI configuration !4791
improve alias command and docs !4795
follow-up: introduce xaction snapshot (aka "snap") into CLI templates !4806
add ais cluster show stats subcommand !4807
refresh for stats !4811
improve job xaction show command; minor refactoring !4815, !4816
node IDs are very special completions (provide visual cue) !4823
fix archive ls; set and show config usability; log_level !4824
remove glog's vmodule (obsolete) !4826
improve ais cluster rebalance command, allow ais archive ls to list entire bucket (of tarballs, etc.) !4828
do not print hint if JSON format is on !4839
introduce unknown-value and not-set-value, and use consistently; fix stop-maintenance logic !4842
downloader: fix error handling, improve usability, !4844

Integration and unit testing; CI/CD

add Go 1.17 shuffling for short tests !4615
restore original primary in TestForwardCP !4619
use new t.Setenv instead of os.Setenv and defer os.Unsetenv !4620
CI: fix lint errors on MacOS !4622
wait for xaction to finish when enabling mirroring !4628
tests/cleanup: ignore "server closed idle connection" !4631
fix failing TestGetAfterReregisterWithMissedBucketUpdate !4640
dev-tools/tassert: timestamp failure message, customize print-stack !4647
dev-tools and tests: continued refactoring and cleanup !4654, !4656
CI: add minimal cluster job that runs with 1 proxy and 1 target !4643
save/check cluster state; log: memory reporting !4677
CI: skipping tests in short mode !4776
test cluster config persistence and mountpath add/remove across restarts !4821
massive read operation vs detach/attach mountpath (non-redundent bucket) !4822
wait for mirroring, re-enable mountpaths !4835

Documentation & blog; website aiatscale.org

transition to Go 1.17 (add a note) !4623
remove empty reference to Python client !4636
remote AIS cluster: examples and references !4665
remote buckets (naming notation, CLI examples, cross-refs) !4669
bucket property inheritance, LRU note; E2E tests: set-custom !4673
configuration.md; transport/Rx: PDU header validation !4679
website: AIStore & ETL: Using AIS/PyTorch connector to transform ImageNet (post #2) !4694
remote ais cluster, global namespace !4697
cleanup http_api docs and add missing json for list buckets operation !4715
website: AIStore & ETL: Introduction (post #1) !4716
update command-line usage !4727
ais cluster show stats !4809
update ais show job CLI !4817
website: copying existing file datasets in two easy steps !4829
update main v3.8 README !4830
touch cluster.md; bump version to 3.8 !4838
What's new in AIS v3.8

Build and toolchain; dependencies

transition to Go 1.17 and use new Go features !4627
make: add run and restart make targets !4707
reinitialize go-modules and upgrade all minor versions !4724, !4726

Bug fixes and improvements

multi-object copy/transform: handle errors; ref common (part two) !4601
multi-object archive/copy/transform: aborting logic and cleanup !4606
general: use Go 1.17 enhancements !4607
health-checker: more efficient method to read directory entries !4608
bucket copy/transform: aborting logic, cleanup, and test !4609
transport/streams: fix broadcast in a single-target cluster !4610
revise err-aborted, refactor common errors, downloader !4611
cluster config vs bucket props vs LRU !4612
erasure-coding: fix bucket-encoding xaction hang on errors and upon abort !4613
deployment: add registry URL in docker build and docker push commands !4614
add atomic counted err value !4616, !4617
on-demand xaction: mutex to protect compound state !4618
etl: add more details when K8s pod times out on being ready; add debug log !4621, !4624
intra-cluster notifications: minor refactoring !4625
xaction: add immutable (original) bucket !4626
revise xaction aborting logic !4629
lru pkg: trash non-existing buckets; health check: docs and follow-up ref-s !4630
storage cleanup: remove artifacts of erasure-coding (unfinished slices, redundant replicas and metafiles) !4632
etl: fix waiting for a ready condition when starting K8s pod !4633
fix global-rebalance can-start & must-run helpers !4634, !4635
log: add backend module; AWS: ignore "unknown region" most of the time !4637
single-target cluster: support bucket renaming !4638
node name vs 1) nodes joining the cluster and 2) early startup !4639
xaction pkg: consistent naming and refactoring !4641
refine health check: add 'ready to rebalance'; fix test to wait for cluster state !4642
default bucket props (ref) !4644
glog: set-node, reduce header; transport streams: add destination ID !4645
copy configuration values for default bucket props !4646
retriable connection errors; transport: do retry !4648
etl: remove single-pod limitation (i.e., the capability to run multiple ETLs) !4649
retriable connection errors: use the same condition consistently throughout !4650
single-target cluster shutdown and assorted fixes !4651
global rebalance: refine preemption logic, abort associated streams (major ref) !4652, !4653
custom object metadata: set/replace vs add/update !4655
consistent custom versioning, checksumming, ETag (major refactoring) !4657
default buffer sizes (TCP, HTTP) !4658
custom metadata: unify downloader; unify version comparison !4659, !4661
downloader: make object comparison much more rigorous !4662
object metadata: to/from header converters !4663
unify the code that checks local/remote equality, change backend.HeadObj() API, simplify cold-GET !4664
erasure-coding config and bucket props: add batch size back for compatibility !4666
object metadata: system attributes (as opposed to user-defined) !4667
object metadata: unset custom keys on PUT, resolve bucket copy vs inc-version dichotomy !4668
unify object props and system object attributes (major revision) !4670
get-bucket-props: remove redundant code, simplify !4671, !4672
configuration: add transport-idle-teardown !4675
memsys: tracking memory stats and responding to OOM (major rev) !4678
xmeta (tool): add VMD, refactor !4680
etl: add new (non-HTTP) communicator and stop relying on the K8s API Server !4681
SizeBytes() and AtimeUnix() methods for CT !4682
volume: revise, refactor, and reinforce operations on metadata !4683, !4684
downloader: refactoring !4685
cluster membership: shutdown, decommission, maintenance !4686
target standby mode !4688, !4689, !4691
dev scripts: deploy remote ais cluster with the same build tags !4690
refactor and simplify admin-join/auto-join logic !4692
follow-up: self-join, admin-join, immediate removal from the cluster map !4695
phase out "unregister" & further consistent renames !4696
add volume package (ref) !4698
volume: consolidate loading and initialization; bootstrap in two passes !4699
cmn/context => cmn/cos/context; rename "attach/detach remote ais" (minor ref) !4700
volume: keep it sync with local config !4701
etl: add more Python runtimes !4702
volume: keep it sync with local configuration !4703
api to attach/detach/enable/disable mountpaths: pass node ref consistently !4706
make: add run and restart targets !4707
error formatting (ref) !4708
api package (major ref) !4709, !4710
"easy URL": accommodate list-objects with no json msg !4711
deploy script: fix getting the last parameter !4712
intra-cluster reverse proxying vs content-length !4714
improve error formatting and content !4717, !4718
cluster/hrw and cmn/err: continued refactoring !4719
graceful mountpath (disk) removal - major revision !4721, !4722
fs: more validation and naming consistency !4725
api to attach mountpath with force !4728
transactional (begin -- commit) to remove mountpath, with committing after resilvering !4729
lint and refactoring, housekeeping !4730
follow-up: add wait-for-resilver, sort mountpaths; increase CI timeout !4732
add/remove disks: make it transactional with resilvering in-between !4735, !4736, !4740, !4744
refactor and split reb pkg, add res (resilver) pkg !4737
revise GFN (state): simplify global, move/rewrite local !4738
follow-up: move global GFN state to reb pkg !4739
separate storage cleanup xaction from LRU one !4741
xaction registry vs housekeeper: tweak initializations !4742
housekeeper: run first & reinforce; x-factories: ordered initialization !4743
cos pkg: make ExitLogf print file and line of the caller !4745
housekeeper: add wait for housekeeper readiness when registering !4746
fs: get and get-available; test-init; refactoring !4747
storage cleanup (feature, major upgrade) !4748
rename API action constants for consistency; tweak OOS logic !4749
dsort: fix panic when creating shard !4750
new pkg to combine both cleanup and eviction logic !4751
memsys: reduce free-mem requirement for tests & minor fixes !4752
storage cleanup: repackage, rewrite, and refactor !4753, !4754
memsys for tests (follow-up) !4755
parse request query only once (core) !4757
create and lookup buckets on the fly (major rev) !4758
memsys and housekeeper: further clarification !4759
follow-up: do not lookup remote bucket; stopping hk !4760
revise and refactor logic to handle deleted (formerly trash) !4761
memsys vs mock target: more orderly initialization !4762
move mock target out of production code !4763
memsys: tune-up low-watermark !4764
delete/undelete: retain bucket names when deleting !4766
memsys: two distinct pairs of slab allocators !4767
general: rename mountpoint to mountpath for consistency !4768
memsys: tune-up for OOM !4769
memsys: reintroduce max-depth !4770
reduce glog and tweak panic-OOM condition !4771
memsys: fix swapping state !4772, !4774
fs: make mountpath flags atomic !4775
xreg pkg: remove obsolete RebalanceArgs struct !4777
ais pkg: split etl.go into tgtetl.go and prxetl.go for consistency !4778
cmn pkg: sort actions alphabetically !4779
etl: integrate etl communicator with xaction !4780
set-name when joining nodes; keepalive logging; test tweaks !4781
not aborting xactions anymore when adding/removing mountpaths !4782
fs: refactor fqn parsing !4783
fs: refactor fqn parsing, update unit tests and docs !4784, !4785
log rotation: uniformity of the headers !4786
fs: minor fixes; tests !4787
add an option to remove mountpaths without resilvering !4789
etl: manage etl metadata !4790
refactor xaction stats: remove redundant interfaces, rename !4792
xactions: introduce snapshot, add in and out counters !4793, !4794
integrate: streams and target stats, rebalance and xaction snapshots !4796
dsort: wait for all in-flight stream requests !4797
data mover to support xaction stats; in, out, and locally processed !4798
transport: complete transition to using opcodes !4799
stats pkg: add get-daemon-stats API, remove redundant code and refactor !4800
etl: fix panic in unit test !4801
etl: change signature of InitSpec function to require options !4802
memsys: tweak init-once logic yet again !4803
consolidate all mock objects in cluster/mock !4804
xactions: complete transition to snaps (ref) !4805
dsort: fix placement of aborted check !4808
fix streams received-bytes count; bump CLI and aisloader versions !4810
remove logic to compute min/max k-alive latency; simplify & refactor !4812
capability to recover lost or corrupted node ID at startup !4813, !4814
detach/disable mountpath: actually wait for resilvering !4818
uuid/gid: consolidate and refactor generation and validation !4819, !4820
aistore with no disks (feature) !4825
http provider: ETag check and prefetch prefix !4827
etl metadata: minor fixes and ref !4831
rebalance: simplify-out Tx semaphore !4832
always re-encode etl metadata; improvements and ref !4833
volume: when creating new volume (and associated VMD), make sure to have a record of configured fspaths !4834
cmn pkg: add is-err-not-found !4836
option to force starting up with a lost or missing mountpath !4837
change one api constant to support rolling upgrade !4840
fix ais advanced remove-from-smap command !4841
minikube: RBAC v1 !4843

aistore - 3.7.1

Published by alex-aizman about 3 years ago

AIStore K8s Operator v0.8
review TODOs, fix comments, remove unused bits !4600
multi-object streaming xactions: ref common, eliminate duplication !4599
K8s dev scripts: fix checking minikube version !4598
multi-proxy stress test: randomize sleep, minor cleanups !4597

aistore - 3.7

Published by alex-aizman about 3 years ago

Highlights

Reading, writing, and listing archives
Multi-object transactions; multi-object ETL (in addition to previously supported full bucket-to-bucket transforms)
"Easy URL"
New and improved CLI
New documentation website
Bug fixes and performance improvements

TAR, TGZ, and ZIP archives

unify/reuse list-range and archiving logic !4483
revise xaction renewal process; propagate archival UUID back to client !4488
unify archiving with existing multi-object operations !4484
CLI: extend PUT to support archive creation !4485
archive: support multiple mime types; add zip test !4491
naming iniside archives: by default, do not include source bucket name !4495
archive: ref count targets on a per archival transaction basis !4497
archive: error processing; end-of-iteration !4500
list objects API: new option to list archived content !4536
archive: support TGZ !4541
support full path via archpath query parameter !4543
archive: nest (tar, tgz) writers !4544
archive: add checksumming for all supported types !4547
add checksum-sizer; archive: add basic stats; eliminate fseek !4548
multi-obj archiving: transactions vs reusable xaction !4550
archive: add a file to a TAR archive !4553
archive: API naming, message flags !4556
archive CLI: make it work; docs: add multi-object operations !4561
CLI: add top level archive command !4570
Blog: an article on how to properly append to an existing TAR !4571
archive: finalize and cleanup logic; consistent renames !4575
archive: transactions, control flags, and error handling !4576
archive: allow multiobject operation to append to existing archive !4580

Multi-object Transform and Copy

multi-object transform & copy and archive: ref-count targets, unify quiescence logic !4511
multi-object transform (initial); consistent renames !4555
add multi-object (list | range) transformation !4565
multi-object copy & transform: multiple transactions, single xaction !4569
transform & copy, bucket & multi-obj: consistent naming !4572
transform & copy, bucket & multi-obj: continued ref !4573
transform & copy multi-obj: ref-count transactions !4574
multi-object copy & transform: refactor API, add stress !4581
multi-object copy & transform: add utils !4583
ETL: forward Init call to the primary !4591

Multi-object operations (common)

move/rename list and range xops (ref) !4459
list and range xops (refactoring part two) !4461
revise list and range operations !4464
list and range ops: continued refactoring !4469
more rigorous idleness check; unique transport endpoint (major) !4595

eXtended Actions (xactions)

xaction registry: remove finished (ref) !4442
enforce unique xaction ID !4449
consolidate xactions (ref) !4462
consolidate xactions (part two) !4463
remove/simplify cluster.XactID (ref) !4474
remove xaction.Args - simplify (ref) !4475
xactions: continued refactoring !4476
renewable xaction (w/ code reduction) !4515
renewable xaction (part two) !4516
renewable xaction (part three: WPR) !4517
rename on-demand xaction !4518
renewable xaction (part four) !4520
renewable xaction (part five) !4521
renewable xaction (part six) !4522
renewable xaction (part seven) !4523
renewable xaction (part eight) !4524
renewable xaction (part nine) !4526
rename/clarify xaction scope !4527
renewable xaction (part ten) !4528
simplify initialization of on-demand xactions !4529
EC: xaction renewals !4533
renewable xaction (part eleven) !4534
improve synchronicity when starting to run (distributed) xactions !4535
wait for rebalance to start (in re: improve synchronicity) !4538
xactions: improve synchronicity (LRU, resilver) !4546

Intra-cluster Transport

data mover stages; reduce unreg-recv delay !4545
consistency in copying obj attrs to/from http and transport headers !4452
transport streams: send completion vs freeing obj headers !4496
transport/streams: reduce/optimize header size, add SID/opcode; archive: send-done !4505
transport: refactor receive logic; use slab for headers; monotime !4507
transport Rx: simplify eof-ok !4508

Documentation

docs: add a section on increasing disk priority for aisnode !4470
docs: setting CPU governor to performance value !4471
CLI docs: update bucket props command !4506
docs: major restructuring to place all documentation in docs directory !4514
docs: bucket properties and property inheritance !4551
docs: make documentation a website !4554
website: replace images links with embedded youtube videos !4557
docs: move 'info for developers' out from user cli documentation !4558
website: various visual improvements !4559
website: remove posts header !4560
website: refactore and fix redirections and images !4563
website: add author field to blog post !4564
website: fix tables and enumerations !4568
docs: revamp RESTful API doc !4590

Tests

tests: allow larger number of idle connections so we can reuse them !4445
make archive test more stressful !4562
tests: fix PrefetchList and PrefetchRange !4466
add multi-objects test !4567
update archiveListRange test, make generated obj names more predictable !4468
fix object props test for aws and a bucket with disabled versioning !4478
test: remove hidden object path augmentation !4486
archive stress-test improvements !4499
add functional test for transforming object with GET request !4531
fix LRU unit test; check capacity consistently with existing code !4539

Bug fixes and improvements

revise/refactor LRU; xaction-being-renewed; misc renames !4537
revise housekeeper !4443
GFN: remove locks, revise, simplify !4444
unify object metadata, transport and ETL headers (part one) !4446
unify object metadata, transport and ETL headers (part two) !4447
checksum utils: add accessors, optimize !4451
object custom MD: send/receive and store upon migration (part one) !4453
not-found error: add source node for context !4454
revise head-object: target (incl. S3) and api package !4455
cleanup metafile if a bucket is deleted while EC is saving metadata !4456
s3 compatibility: reuse target.getObject, eliminate copy-paste !4457
set custom object properties (part one) !4458
move EC file type constants to fs package !4460
add lom.HrwTarget helper (ref) !4465
handle bucket-already-exists when creating one !4467
consistency in handling unmarshal errors !4473
gcp client singleton !4479
aws client per region !4480
backend API changes: context (part one) !4481
backend API changes: context (part two) !4482
CLI: new options for PUT object operation !4487
assorted TODO fixes !4490
xreg args by value !4492
lint/revive: enable unused-parameter !4493
"easy URL" - aka alternative AIS API mapping (feature) !4494
tweak initializing remote backend (ref, race) !4498
ETL executor implementation (initial) !4501
ETL: revise bucket copy&transform (major) !4502
CLI: add subcommand 'ais bucket prop' !4503
follow-up: aborting prior to starting to run !4504
slab-allocator: add AllocSize !4509
CLI: TAB-TAB to list only active cloud providers !4510
EC: use Hdr.SID instead of internal EC.intraReq.sender !4512
EC: use transport.Hdr.Opcode instead of intraReq.action !4519
dSort: remove redundant field in remoteRequest msg !4525
CVE fix: replace JWT library !4530
LRU: remove on-demand logic; archive: no need to quiesce data mover !4540
notifications: getting status and err vs rlocks !4542
CLI: minor refactoring and cleanup; error handling !4549
CLI: add bucket lru subcommand; show bucket props by default !4552
EC recv meta: fix error handling !4566
debug modules; logs; persistent markers !4577
node name !4578
glog not to flush in a separate goroutine; minor ref !4579
CI: add newest Go version to be used in GitHub actions !4586
revise quiescence logic; on-demand stats !4587
build: upgrade fuse pkg !4588
CLI ais bucket cp: support --list and --template options !4589
CLI ETL: support --list and --template options !4593

aistore - 3.6

Published by alex-aizman over 3 years ago

Highlights

observability: revise StatsD integration, add disk metrics, support Prometheus
EC version 2: introduce new (packed) metadata format; speed-up erasure-coding and optimize rebalancing of erasure-coded buckets
API for reading and writing (tar, tgz, zip) archives - initial, experimental

Observability

revise StatsD integration (part one: major update) !4340
revise StatsD integration (part two: cleanup, refactoring) !4342
stats/ref: move core-stats to common, close StatsD !4345
revise StatsD integration (part three) !4346
prepare metric names at init time !4355
implement Prometheus exporter !4356
make StatsD naming and units consistent with Prometheus !4357
preallocate disk stats !4358
add disk metrics, publish disk metrics !4372
tests: fix disk utilization computation !4374
track average read/write sizes; log improvements; refactoring !4376
read-lock when collecting Prometheus stats !4378

AIS CLI

CLI: do not validate config received from a daemon !4339
CLI: add an option --validate for bucket summary to show the number of 'not-ok' objects !4341
CLI: introduce new top-level command: storage !4350
CLI: fix counting missing copies in 'ais storage' output !4353
CLI: docs for ais storage command !4362
CLI: add 'ais storage disk' sub-command !4375

EC

EC: extend metafile with version and a list of slices !4373
EC: use packed binary format for metadata !4382
EC: flatten EC props for object HEAD response !4386
EC: save slice locations in metafile !4387
EC: protect metadata with checksum !4395
EC: rebalance v2 (major upd) !4426
EC: optimize network usage when deleting and restoring an object !4419

CI

CI: fix linter on Darwin platform !4337
CI, Jenkins: run cloud ETL tests !4370, !4384
CI, Jenkins: correctly read and apply AWS environment for minikube !4397
CI: fix dev/k8s template !4403
CI: push credentials in secret !4408

Bug fixes and improvements

aisloader: require -cleanup option !4335
general: use Go 1.16 embed package !4336
test: add reader test for SGL !4338
config ports, json: remove duplication and parsing !4343
config, etc. !4347
config: add custom duration type and parsing (part one) !4348
revisit json duration (type) !4349
config: eliminate string durations, use custom type (part two) !4351
docs: "Monitoring AIStore with Prometheus" !4352
comments !4354
network call with retry: client and server flavors !4344
cmn: rename BucketNames struct to Bcks to better describe content !4359
fs, ios, ais: minor refactoring & cleanup !4360
mountpath, fs disks (ref) !4361
fs: revise init and validation logic !4363
fs/vmd minor ref !4364
transport, streams: tweak global defaults !4365
metasync: not using nil SGL for payload (fix) !4366
metasync: delayed cleanup (fix) !4367
deployment: remove unused aisloader container !4368
minor changes in documentation and code !4369
ETL: collect stats for communicator !4371
metasync: remove CoW debug validation; fixes !4377
cos/utils: create-file now always creates write-only !4379
dsort: replace section reader !4380
ETL: pull docker images only if not present !4381
ais: minor ref (get, extract) !4383
metasync, BMD, SGL !4385
reuse SGL when range-reading w/checksum; DEADBEEF; miscellaneous !4389
compute redirect latency every so often (perf) !4393
GET and PUT: remove dry-run option !4399
Jenkins: upgrade minikube !4400
override backends at cluster startup !4401
golint is deprecated: using revive !4406
memsys: tune-up and refactoring !4409
lint: revive fixes (part two) !4410
transport/streams: Rx callbacks do not need (or depend on) http !4411
comments fixing and minor ref ATB !4412
list/range API; CLI comments/examples !4413
tests: limit the number of cleanup-generated errors !4414
ref: move "connection-refused" and similar exceptions !4415
PUT into mirrored bucket !4416
more lint !4417
rename/refactor xaction providers !4420
ref: xaction args !4421
xaction factory (ref, consistency) !4422
read-json: prevent superfluous response !4423
revise xaction renewal (internal) API !4424
lint/revive: enable unused-receiver rule !4428
revise xaction registry (part one) !4430
proxy: mem-pool API args; original bucket url !4431
run-provider tests must cleanup after themselves !4432
bucket lookups vs a missing backend provider !4433
introduce fullname() shortcut for: lom, transport header, archive !4436
api/object: close response body, close source reader !4438
list-buckets corner cases with no authentication and/or compiled-in backend !4441

aistore -

Published by alex-aizman over 3 years ago

Bug fixes and improvements

metasync: dynamic SGL sizing !4282
http-common: serialize shutdown and initialization !4283
streams: intra-data endpoint !4284
ETL/target: intra-control endpoint !4285
s3 compatibility: stubs for unsupported features; s3cmd !4286
CI: add missing build configurations !4287
upgrade Go1.15 to Go1.16 !4288
enable intra-cluster networks for functional testing !4289
GODEBUG=madvdontneed=1 vs Go 1.16 !4290
deploy: remove obsolete charts and scripts !4293
tests: re-enable conditions for dSort tests with memory and disk !4294
CI: add more linters + chores around existing linters !4295
metasync: reuse jsp, optimize serialization (part one: cluster map) !4296
metasync: just-in-time revs !4297
CLI: use default config instead of empty one if the configuration file is missing !4298
fix minikube based deployment for tests !4299
target: serialize receiving BMD vs graceful termination !4300
extend is-unreachable helper with more !4301
updating nil BMD !4302
CLI: sort disk output by target_ID and disk_name !4303
fflush upon close prior to rename !4304, !4305, !4307
no metasync when shutting down !4306
tests: wait cluster state - use reverse proxy !4308
metasync: reuse jsp, optimize serialization (part two: BMD) !4309
metasync: avoid re-marshaling received cluster map (part four) !4312
metasync: optimize-out remarshaling (part five: Rx BMD) !4313
metasync: encode first, persist and distribute second !4315
metasync: reuse jsp, optimize serialization (part six: config) !4316
use direct I/O and sync for FSHC file read and write !4317
debug panic must leave a trace !4318
build: fix MacOS build !4319
rejoin cluster during startup if primary gets re-elected (major) !4320
metasync: optimize-out remarshaling (part seven: Rx Config) !4321
bootstrap: use joining targets info to discover BMD; strict UUID checking !4322
bootstrap, joining targets and RMD; fixes !4323
tweak primary election !4325
bootstrap: refactor reg-pool processing; handle global config !4326
metasync: revise receive logic; structured error type (part eight) !4327
metasync: handle cluster-info scenarios (part nine) !4328
starting up with outdated or missing cluster maps !4329
build/deps: upgrade all minors !4330
CLI: make order of columns stable !4331
target: refactor BMD Rx logic !4332
aisloader: support bucket URI in the command line !4333

aistore - 3.4.1

Published by alex-aizman over 3 years ago

Kubernetes Operator

https://github.com/NVIDIA/ais-k8s/releases

Bug fixes and improvements

bench: add FIO benchmark configuration for repeatable FS benchmarking !4228
refactor EC rebalance test !4255
improve deploy script !4256
docs: Use absolute links !4260
lint update v1.32 to v1.39 !4258, !4261, !4262, !4263
better checking for rebalance abort !4264
revise vmd loading and error handling !4265
rlock ios !4266
fix keepalive reconfig !4267
ios: minor optimizations; vmd: refactor init/load !4268
assign Snode flags only when changed !4269
fix build for net/http !4270
rebalance: mem-pool and fix transport/header !4271, !4272
build: 1) upgrade fasthttp and 2) replace x/crypto (deprecated) !4273
memsys: refactor sgl allocation !4274
ais/s3 API: respond via sgl !4275
mem-pool SGL (list, not buffers) !4276

CLI

allow user to set aliases !4244
move cluster configure to top-level commands !4248
show extended EC stats for 'ais show job xaction ecput/ecget' !4249
replace 'daemon id' with 'node' in titles !4251
don't show all configs by default !4252
don't use append() to initialize commands !4253
show daemon local config separately !4254
revise alias; add list, reset !4257
add completion for 'cluster detach' !4259
add link to autocompletion scripts to app description !4279
add basic k8s support as top level command 'kubectl' !4280
ais show log [node]: API and CLI !4277, !4278, !4281

aistore - 3.4

Published by alex-aizman over 3 years ago

Highlights

Kubernetes Operator: a separate repository;
Cluster life-cycle management: maintenance (node), decommission and shutdown (node and cluster);
CLI: implement (category, verb, subject) auto-completions; numerous improvements;
Global Cluster Configuration - versioned, replicated, protected; per-node capability to override global defaults;
System Metadata (5 types): unified CCS formatting, instance and meta-versioning, backward compatibility;
Authentication and ACLs: add users and roles, docs, tests, bucket and cluster management, CLI;
ETL: AIS transport streams, GCP and minikube, CI and Jenkins, stress tests (*);
PDU-based intra-cluster transport, streaming objects of unknown size;
Performance: memory pooling, in-memory metadata;
HDFS on the back – in total, supporting 6 different backends;
DNS names - supporting hostnames and/or IPv4 in public and intra-cluster networks;
Resilver: resume upon reboot, run on a selected node;
Cloud: improve AWS versioning, improve error handling;
Erasure Coding: improved stability for Cloud buckets; handle low memory and OOM;
Distributed shuffle (dSort): improve stability, optimize memory and CPU usage;
All subsystems and core: productization, stabilization, bug fixing, refactoring across board

Core

unify bucket(name, provider, namespace) transfer - part one - !3340
general: refactor and simplify cold Get - !3345
keepalive: do not decrease timeout on connection error - !3444
(new) metadata write policy - !3513, !3596
LOM minification - !3521
fs/cluster: generalize fs.GetContentFQN with interface - !3548
cluster: remove some redundancy in CT and LOM structures - !3549
cmn: return ReadOpenCloser on ReadOpenCloser.Open - !3562
memsys: remove unused SliceReader - !3563
LOM: remove one field - !3570
LOM: enforce rlock - !3615
support (DNS) hostnames for the public and intra-cluster networks - !3616
fix passing origURLBck in bucket init - !3720
LOM In Flight (LIF) - !3765
mem-pool LOM - !3722
mem-pool GET, PUT, and COPY runtime - !3825
mem-pool LOM (part two) - !3839
mem-pool LOM (part three - rebalance) - !3878
copy & transform flows: more mem-pooling and refactoring - !3879
extend core structs to support possible within-meta-version extensions - !4239
on-disk meta-versioning and backward compat (part one) - !4023
meta-versioning and backward compatibility (part two) - !4037
meta-versioning and backward compatibility (part three) - !4061
meta-versioning and backward compatibility (part four) - !4075
refactor http-common - !3639, !3641, !3649, !3656, !3660, !3788, !3793, !3816
transport: remove roff - !3661
transport: pool obj-reader struct - !3662
transport: always execute send-completion logic - !3936
transport: use interface{} instead of unsafe.Pointer as callback argument - !3953
minor refactor of proxy election method - !3582
ignore TLS handshake errors - !3586
refactor bucket initialization and permission check - !3590
add missing unlock in GET object - !3681
fix rename to accept bucket with backend buckets - !3684
send props on create bucket in message instead of query - !3692
pre-decide which type of lock should be taken on CopyObject - !3738
ensure that remote object was correctly fetched after UpgradeLock - !3761
generalize creating default bucket props - !3775
fail early when trying to explicitly create a cloud or HTTP bucket - !3778
correctly handle creation of a bucket with backend bucket - !3782
fix sending error when part of the object was already sent - !3799
determine internally if we should skip validate in defaultBckProps - !3807
correctly skip bmd modify when terminate flag is set - !3889
revise/unify error handling - !3929
fix unlock panic when copying an object - !3935
unmarshal HTTPError on call - !3608
do not append redundant call frame to error - !3821
correctly send error on bucket summary - !4100
add new function for extracting QueryBcks from request - !4105
unify listing and getting summary - !4106
remove err from bckInitArgs struct and rename queryBck - !4123
add stronger validation for bucket in bckInitArgs - !4124
return error on missing required query parameter - !4126
ensure latest bmd when doing list and summary - !4135
fix returning error when begin phase in bucket creation fails - !4138
correctly handle init remote ais cluster bucket - !4139
remove vmd creation on user register - !4152
pass error when querying daemon info - !4154
initialize bucket on bucket summary - !4164
start resilver if the new mountpath was added - !4180
revise startup sequence - !4191
earlystart: fix possible disconnect between rmd and smap - !3984
earlystart: resume global rebalance (part two) - !4000
synchronize remote AIS attachments when target joins (part two) - !4202
general: fix head object on remote AIS bucket - !4204
revise metasync receive - !4206
Sync RMD when a node joins - !4193
wait for metasync when decommissioning a target - !4087

CLI

revamp 'show rebalance' - !3551
reorganize proxy and targets templates - !3552
do not show nodes status when all online - !3554
fix put progress bar report for large uploads - !3556
show deployment type of nodes in ais show cluster - !3557
ais rename issues - !3560
introduce templates table - !3566
use colors in messages - !3578
add wait flag for copy and etl bucket - !3583
correctly handle errors from HTTP server - !3584
fix error messages - !3658
change the way ais displays daemon configs - !3666
fix AuthN errors and a little refactoring - !3677
Change the rename command to mv (part two) - !3675
panic when setting an invalid bucket property - !3695
Improve 'ais show config nonexistent' error - !3711
fix matching fail checks in tests - !3715
completion for bucket permissions - !3733
Update 'show object' to display properties vertically - !3743
fix makePairs to accept values with = characters - !3747
Remove cp objects from help message - !3764
Rebalances cannot be complete if none exist - !3766
Add a wait option for mv bucket - !3767
fix printing download ID on the start - !3599
small ETL improvements - !3603
Imply obj name as out_file in get command - !3640
change the rename command to mv - !3668
highlight bucket properties that differ from default ones - !3686
complete option values - !3699
Add support for graceful shutdown - !3805
improve AuthN UX - !3842
revoke token command - !3880
AuthN user permissions - !3887
TAB-TAB completion and parsing for user-friendly permission when adding role - !3888
Introduce object and bucket top level commands - !4029
Move rm download/dsort inside job - !4043
improve gen-shards input and allow specifing provider - !4048
Add ais show auth and ais show bucket - !4062
use standard name for no-color flag - !4068
Various grammar/wording changes - !3998
Introduce cluster and disk top level command - !4006
force flag for node maintenance - !4007
Introduce job top level command - !4017
Introduce advanced top level command - !4018
fix setting cluster/daemon config - !4076
Fix panic when aliases have subcommands - !4077
fix duplicated ais auth show command - !4082
refactor parsing bucket or bucket + objName URIs - !4085
Get daemon ID from daemon itself instead of API call - !4091
add parse functions tests - !4095
require bucket only for bucket xactions - !4104
show targets in maintenance - !4133
allow viewing cluster config - !4147
scope based props validation in configure command - !4148
Make verbose more consistent for show cluster - !4156
Fix ordering in show cluster - !4182
add config option to set default provider - !4189
Update version number - !4192
make object name optional for evict command - !4203
local and cluster config displaying - !4230
docs improvements and more polish - !4240
add 'ais cluster show bmd' command - !4241

Distributed Shuffle (dSort)

general: use new optimized msgp method - !3553
Use job instead of manager in dSort errors - !3667
access lom size only when put has been successful - !3843
re-load lom under lock when sending to another target - !3844
use cmn.Bck instead of bare name and provider - !4066
improve and add more logging - !4093
correctly cleanup resources when ignoring duplicated records - !4109
ignore targets in maintenance - !4166

Authentication & User management

Allow changing bucket permissions for read-only buckets - !3561
fix panic when checking restricted user's permissions - !3841
correct checking user's role permissions - !3857
do not cache generated tokens - !3858
AuthN docs - !3881
Authn docs and code review - !3892
API to change configuration on the fly (1 of 2) - !3947
fix returning info for a single cluster, update tests - !3987
AuthN refactoring: move to its own package - !4171
revise access permissions (ACL) - !3744
Improve bucket ACL check - !3757

ETL

use user defined ID as name - !3571
fix minikube playground etl + cloud providers - !3565
minor improvements and refactoring - !3671
add tests for target down - !3674
use a separate GCP bucket for cloud tests - !3701
add better cloud warm/cold ETL object tests - !3703
add large bucket test - !3679
health observability (part one) - !3683
run health tests - !3763
wait longer for pods metrics - !3770
add tests for occasionally failing transformation - !3785
broadcast list request to all targets - !3787
make failing ETL test less flacky - !3796
fixes around k8s.Detect() usage - !3801
better observability of ETLBigBucketTest - !3818
add objects and bytes count check - !3823
don't run big bucket test on single target deployment - !3836
test abort in the middle of running - !3852
tests improvements, better cleanups, fail faster - !3884
remove already fixed TODO comment - !3896
new StopCh for each etl big bucket test case - !3899
introduce single request timeout for ETL Bucket - !3900
set python runtime image pull policy to Always - !3909
set request timeout for TestETLBigBucket - !3910
minor tests improvements - !3913
more verbose Pod ready failure message - !3980
ETL: failfast if we know starting an ETL will fail - !3990
fix health status code regression - !4003
don't skip TestETLObjectCloud - !4004
more precise health error message - !4111
propagate whole bucket info to etl requests - !4140
wait for rebalance to complete after target down test - !4153

K8s

dev: support redeploy k8s; and datascience container - !3577
k8s: add fallback env variable when HOSTNAME is not reliable - !3567
k8s: decrease default curl timeouts on bootstrap - !3594
k8s: simplify and rewrite building and pushing aistore/aisnode image - !3610
k8s: prod docker image build node with all providers - !3659
fs: run mountpaths integration tests on minikube - !4012
deploy: increase default memory limit on minikube - !4051
deploy: add metrics-server to minikube deployment - !3689
minikube: update docker image tag - !3726
minikube: update image to aisnode:minikube - !3746
minikube: optionally deploy metrics collection - !3749
general: rename minikube testing build tag - !3786
general: waiting for primary to become ready (minikube + tests) - !3808
minikube: minor fixes in deploy scripts - !3812
minikube: allow multitarget deployment and use it on jenkins - !3835
minikube: build with debug on jenkins - !3938
minikube: disable cgo for node build - !3942
minikube: fix fspaths config - !4177
k8s: remove wait for service logic from container start-up script - !3815
k8s: update start up script - !3959
k8s: AIS_HELM_DEPLOYMENT variable as string - !3985
jenkins: update RE variable for K8s tests - !4040
skip public net hostname validation on K8S environment - !3637
k8s: add Dockerfile for initContainer - !3651

EC

allows setting Data:Parity for a bucket if the number of nodes is between... - !3708
use LIF where possible - !3789
load recovered LOM - !3806
configuration option 'disk_only' - !3827
correct LOM loading when PUTting an object - !3838
remove unused cksumType parameter - !3847
do not call lom.Load while deleting an encoded object - !3856
mem-pool cluster nodes - !3973
TestDestoryBucket - !4057
add slice locking when receiving, sending, and deleting - !4079
encode and decode xactions refactoring - !4089
EC xactions - refactoring on-demand - !3607
EC cloud test fix - !3966
EC putjogger refactoring - !3906
EC getjogger refactoring - !3919

Downloader

fix bucket backend provider check - !3810
refactor of CompareObjects test - !3845
remove special parameter for lom.Size - !3851
synchronize requests on dispatcher startup - !4019
correctly calculate download latency - !4070
listen on abort when waiting on throttler - !4071
improve checking if the task still exists - !4094
improve context cancel logic + minor refactor - !4121
Downloader: fix uploading to Cloud bucket - !3598
allow downloading to cloud bucket - !3587
return first error on download start request - !3592

API

add http error message when doing HEAD request on bucket - !3600
cloud: add API to create buckets for cloud providers - !3670
return bucket props in single header - !3682
move HEAD header error check to checkResp func - !3688
cmn: remove unused API constants - !3690
return 404 on wrong node id for config request - !3712
API change: bucket props-to-update - !3755
Add support for graceful node shutdown and related CLI - !3828
CLI and Single node resilver - !3893
set cluster config using cmn.ConfigToUpdate struct - !3894
fix closing reader in DoReqWithRetry function - !3926
add an option to evict only data for remote buckets - !3958
pass node as parameter to GetDaemonConfig function - !4036
fix 'get remote cluster list' API - !4059
cmn: cleanup api constants - !4083
Reset cluster or daemon to cluster configs - !4150
Workaround for remote ais bucket create - !4208
Refactor httpcluget GetWhatRemoteAIS - !4210

General

xact: remove unused return error parameter from Run methods - !3574
xact: always call Finish on the end of xaction - !3576
cloud providers: validations, docs (updates) - !3579
dl: auto create destination bucket if not exists - !3580
lru: minor refactor of main Run function - !3581
xact: print list objects error when not aborted - !3585
Allow cloud provider aliases (as in CLI) - !3588
metadata write policy (part two) - !3589
general: remove unnecessary query append - !3593
Use correct RecvType for each DataMover - !3597
LOM: assert locking - !3604
control plane: update forwardCP to utilize original req body - !3606
transfer bucket: correctly set objects versions - !3609
debug: remove Enabled, add Func() - !3611
fix issue with cached backend bucket props; tests evict backend bucket - !3612
bckInitArgs remove origURLBck field - !3613
fix rename bucket with backend and add test - !3614
Idle property for XactDemand - !3617
test fixes: cloud-mirror & rename-with-backend - !3618
bucket uname query (rename & doc) - !3619
fix bucket props when copying and creating on the fly - !3620
do not clone config; refactoring - !3621
fix intra control/data ports - !3622
Improving checkRESTItems - !3623
make target health handler public - !3624
Replace variadic path item list in checkREST calls with "constant" slices - !3625
optimize CoW config saving when updating cluster maps - !3626
handlers use flags for network access - !3627
fix smap-modifier - !3628
keepalive: revise cluster map update - !3629
sys: refactor & simplify - !3630
limit intra-cluster broadcast - !3631
health: refactor proxy/target health handlers - !3632
Remove ActRecoverBck - !3633
fix predeclared identifier usage - !3635
Reuse apiRequest.parse in proxy - !3636
fix setting intra-call headers - !3638
config: rename ipv4 to hostname - !3642
general: correctly register health handler on proxy - !3644
general: add missing space in log message - !3645
general: refactor join words with URLPath - !3647
health: external health check types - !3648
cloud: add support for HDFS provider - !3653
general: change format of stack printout in logs - !3654
fs: move markers when mountpath is removed or disabled - !3663
general: update copyright statement - !3664
txn: simplify alloc and free of bcast args - !3669
cloud: change the order of the methods to match interface - !3672
general: add better validation for bucket rename - !3673
config: Put and Begin/Commit/Discard - !3676
cmn: split fields in Extra by provider - !3680
Put Error to Header response if a request is a HEAD one - !3685
config: add daemon role field; fix fspath config validation - !3687
cmn: move code to match order of providers - !3691
cmn: validate HDFS config addresses - !3693
general: add HDFS build tag and allow selecting it during make deploy - !3694
scripts: ipv4 => hostname - !3696
add text: data redundancy - multiple options - !3697
metadata write policy default (minor) - !3700
deploy: easier configuration for cloud providers - !3702
Security: Reduce file permissions to user and group only - !3705
remove OpenAPI-based client generator (swagger-codegen) - !3706
revise lom create-file - !3709
cache bucket dirs with mountpaths - !3710
deploy: fix deployment of next tier cluster in cleanup script - !3713
general: fix grammar mistakes - !3717
general: remove implicit cloud provider (cloud://) leftovers - !3718
general: remove node name prefix from bucket errors - !3719
change in terminology: backend provider - !3730
general: add update config option - !3732
general: correctly handle lookup of HDFS bucket - !3737
general: fixes around HDFS support - !3739
change in terminology: backend provider (part two) - !3741
Using *ToUpdate for updating config - !3742
general: fix implementation of UpgradeLock - !3745
general: fixes around HDFS provider - !3751
general: rename cloud package to backend - !3752
cluster: fix and improve implementation of UpgradeLock - !3754
extend providers tests; more test-skipping options - !3756
general: rename CloudVersion to RemoteVersion - !3762
Use correct permissions for every initAndTry - !3768
backend: remove only extra info from AWS error - !3769
general: rename cloud_mock to backend_mock - !3771
general: fix resetting bucket props for HDFS bucket - !3772
simplify lom.bucket (ref) - !3779
PUT(mirror): optimize-out one channel from the datapath - !3780
reduce wack-target memory footprint - !3792
backend: relax address connectivity checking for HDFS backend provider - !3795
backend: correctly set ContinuationToken for HDFS's list objects - !3797
cmn: remove | character from HTTP error string - !3798
makefile: better AIStore running check - !3802
general: make skip_verify_crt option configurable in aisfs and cli - !3803
backend: correctly handle prefix in HDFS's list objects - !3809
general: rename cloud to broader terms remote or backend - !3811
backend: correctly skip HDFS files which do not match prefix - !3817
reinforce LOM loading - !3824
fix/revise delete-obj; evict to return status - !3829
unloaded deleted object - !3830
general: remove special marker for cases when the LOM is loaded - !3831
general: use consistent names and order in DeleteObject method - !3832
tutils: cleanup temporary directory created in PrepareObjects - !3846
transfer bucket: remove unused function - !3848
backend: fix PutObj implementation for remote ais - !3849
general: return copied bytes count in targets copy* methods - !3850
revise PUT versioning logic - !3859
backend: remove retry in the AIS remote cluster - !3862
obj: remove unused uncache field in coi structure - !3863
backend: simplify PutObj implementation for AIS remote cluster - !3864
copy-object-info: remove unnecessary lom.Loaded() asserts - !3865
copy&transform flow: revise and simplify put-remote - !3869
xaction must call Finish as soon as everything is cleaned up - !3870
revise copy-object: remote and local parts - !3871
extract cluster-wide operation into a separate source - !3891
review maintenance, shutdown, and decommission oper-s - !3895
transactions: simplify handler and fix double write bug - !3897
make kill timeout - !3901
node shutdown vs self-initiated unregister - !3904
revise node self-initiated removal (part two) - !3912
makefile: option to run tests only from specified directory - !3914
unify target and proxy termination - !3916
tutils: refactor cluster init - !3917
refactor invalmsghdlr and friends - !3918
general: login Jenkins to Docker Hub - !3920
general: rename cloud field and struct to backend - !3921
general: embed cleanup callback in the reader Close method - !3923
backend: refactor PutObj usage and delegate closing reader to implementation - !3924
Add configuration for automatic resilvering - !3928
revise/unify error handling (part two) - !3933
cmn: better OOS error message - !3934
revise/unify error handling (part three) - !3937
cmn: correctly set custom HTTP status code - !3939
cmn: use DurationJSON type in Bck2BckMsg struct - !3941
aisfs: build aisfs in test:long - !3943
err handler for "405 method not allowed" - !3945
Make complex tree of calls to IncPending and decPending a simpler one - !3948
jenkins: clean docker images cache - !3949
cmn: add comment about closing BodyR - !3951
general: use unlock lom on close whenever possible - !3952
cmn: fix parsing 'Range' HTTP header - !3955
LOM: strict caching and locking policy - !3956
revise the logic to resume rebalance - !3960
tutils: simplify DeployNode and CleanupNode logic - !3962
tutils: add CheckResp for checking response status code - !3963
assorted fixes - !3964
general: separate global and daemon config - !3965
graceful shutdown; can start rebalance; remove dont_run_time config - !3967
simplify fs.CreateBuckets - !3968
tutils: allow custom intervals for WaitNodeReady - !3970
reinforce naming convention for custom errors and error types - !3971
general: update build scripts to use local config - !3972
Use constant format strings to return common errors - !3975
universal quiescence - !3977
general: remove stray debugging message - !3979
general: use ErrNotFound across the codebase - !3981
build: replace non-existing /dev/nil with /dev/null - !3982
docker: revert startup wait removal for helm charts ! - !3815 - !3983
cmn: add Caller field to HTTPErr - !3986
xaction: eliminate copy-paste construction - !3988
general: refactor and simplification around http errors - !3994
on-demand stopping logic (minor) - !3997
Use GET for ListObjects instead of POST - !3999
Do not remove an object after it is rebalanced - !4001
general: move fields from global to local config - !4002
create config owner and maintain versioning - !4008
support software version and build time; show via API and CLI - !4009
HTTP header and query - !4010
config: fix config Validation returned from API - !4011
metasync global config - !4015
cluster: fix debug message which caused infinite recursion - !4020
general: use modifier to update config - !4022
Remove force option from maintenance - !4024
general: update attach/detach remote AIS to use config modifier - !4025
fix mutex assertion race - !4026
Auto rebalance settings - !4027
general: allow override global config - !4028
general: sync config on node join - !4030
general: fix setting global config from non-primary - !4031
cmn: correctly validate config confdir value - !4041
cmn: set ais as default provider when parsing bucket - !4042
general: using tags to restrict config updates - !4047
cmn: rename StringSet.Keys to ToSlice - !4053
devtools: fix wait-start-re - !4056
general: separate global and local config - !4063
Use cmn.Bck to initailize CT instead of strings - !4064
devtools: return pid as int instead of string - !4072
devtools: fix assert in PutRandObjs - !4073
action-message (minor ref) - !4074
general: refactor config management implementation - !4078
jenkins: force docker image prune - !4081
cmn: return invalid provider error from validate function - !4084
flatten devtools directory structure - !4088
redistribute VMD under lock - !4090
new common (part one) - !4092
cluster: remove unused argument - !4097
cluster: refactor around bck provider validation - !4098
Quick-fix EC test for new LOM pool - !3781
EC tests decrease the number of goroutines - !4102
Keep node in smap during shutdown - !4103
new common (part two) - !4107
config: tweak validator - !4108
debug: add http debug handlers (pprof and vars) - !4110
debug: fix pprof handler path - !4112
jsp authn (config and tokens) - !4114
general: allow setting transient cluster config - !4115
general: fix config change notifier - !4116
Get snode flags from smap - !4117
general: discover latest config - !4118
sign and version config; xmeta - !4119
Stop aisnode when decommissioning - !4120
cmn: improve QueryBcks validation methods - !4125
cmn: cleanup around provider validation - !4127
general: move metadata filenames to cmn package - !4128
general: store cluster config only on disk - !4129
general: *ToUpdate omitempty fields in json - !4130
general: remove json local extension - !4131
fs: support decommissioning, rewrite moving-markers logic - !4132
general: move log dir to local config - !4134
general: refactoring around bucket parsing and validation - !4136
general: explicitly require bucket provider - !4137
general: make fspath config representation - !4144
revise GET and mirror-copy load balancing - !4145
Improve CLI UX and documentation for configuration - !4146
make runtime stats expvar-observable in debug mode - !4149
fs: refactor around vmd - !4151
general: fix attach detach remote AIS - !4157
RmNode optional data cleanup - !4158
general: include config in nodeRegMeta - !4160
global config: add uuid - !4161
remove node from smap in removal-by-test case - !4162
unify target/proxy receive Smap logic - !4169
join-cluster: metasync vs node-reg-meta - !4170
fs: extend vmd structure with more filesystem information - !4172
stats: simplify-out listening for config updates and other refactoring - !4173
keepalive: refactor & optimize - !4174
support selecting remote backends at deployment time - !4175
cmn: add writeup about bucket validation - !4178
fs: add versioning to vmd - !4179
global config: provide for within meta-version extensions - !4181
Remove hardcoding in clean_deploy.sh - !4183
AWS: fix reading object versions for not-cached objects - !4184
target startup sequence: init fs first, snode second - !4187
cmn: move test fspaths init to config validation - !4188
general: remove unused code and code cleanup - !4190
backend: fix initialization of objMeta map - !4197
unify cluster-wide meta exchange (part one) - !4198
keep remote AIS attachments in sync when new targets join - !4199
general: implement decommission cluster - !4200
general: use proxy/target runner Stop on decommission - !4207
general: ensure intra(Primary) call - !4209
Step 2: cluMeta usage - !4211
Add more options for clean_deploy.sh - !4212
general: correct evit/destroy of remote AIS buckets - !4213
general: better error logging for hostname selection - !4214
set-bucket-props: backend - !4215
general: remove go routine shutdown/decommission request - !4218
rename http header constants - !4219
intra-cluster control: micro-optimizations and refactoring - !4220
refactor Smap out of ais msg - !4221
general: fix node shutdown - !4225
build: change default compiled providers for aisnode docker image - !4229
general: add shutdown marker on daemon shutdown/decommission - !4233
Do not panic on unexpected EOF when draining a reader - !4234
Add a build tag to use rproxy for all s3 requests - !4235
Allow disabling xattrs feature - !4237
Add alias ais show config cluster for consistency - !4242
Relax checking for Smap reliability when private network is used - !4245
reinforce cluster UUID when joining and starting up - !4246

Tests and CI

skip flaky TestDistributedSortKillTargetDuringPhases test - !3550
ci: bring back skip-ci feature - !3569
add bytes limit to TestDownloadJobLimitConnections - !3602
use t.Cleanup to destroy and evict test buckets - !3605
fix s3 tests to work with HTTPS - !3643
do not require AIS_ENDPOINT when testing on K8s - !3646
ci: fix tags on k8s jobs - !3655
ci: update Docker image - !3725
test: correct check for target count in EC rebalance test - !3728
ci: add HDFS provider job - !3734
extend the scope of tests from cloud to remote buckets - !3736
better ETL health check - !3748
detect cluster deployment type, not test suite deployment type (k8s) - !3758
run ETL tests on jenkins - !3750
ci: mark all the jobs with aistore specific tags - !3652
ci: add timeouts for all jobs - !3716
ci: use native solution for downloading dependecies - !3773
ci: fix permissions in HDFS job - !3794
ci: increase mountpath count and target count in k8s tests - !3822
ci: ignore lint for blocks which do not contain new variables - !3833
ci: allow running manual long tests on master branch - !3872
ci: re-enable running jobs in scheduled pipeline - !3925
ci: fix setup in HDFS job - !3927
ci: allow non-escaped pipe characters in RE variable - !4034
ci: add simple github workflow - !4035
ci: deploy metrics server on kube pipelines - !4052
ci: fix branch name in github workflow - !4065
ci: show correct path to the jenkin logs - !4069
ci: remove test:soak and test:bench - !4099
ci: rename and standardize jobs names - !4165
ci: use clean_deploy script to deploy cluster - !4194
add bytes limit to TestDownloadJobConcurrency - !3591
Tests: fix TestObjectPrefix filter check - !3698
wait for resilver after manipulating mountpaths - !3774
increase kube long tests timeout - !3777
extend ETL big bucket tests with new cases - !3783
add Python2 runtime for build test - !3784
Tests: improve EC tests' slice lookup procedure - !3791
skip failing EC tests for bad checksum - !3800
FailNow vs goroutines - !3813
Tests: replace full /tmp/ais traversing with fs.WalkBck - !3814
add new, faster and better implementation of PutRandObjs - !3819
do not stop ETL when it wasn't correctly started - !3820
skip tests with HDFS as backend bucket - !3834
make copy bucket tests long - !3853
use new version of PutRandObjs - !3866
remove PutObjsFromList function - !3867
make tutils init explicit - !3868
simplify TestSmoke implementation - !3873
show AIS logs for minikube tests on jenkins - !3874
increase allowed log streams for jenkins minikube - !3875
fix passing cksum type - !3876
Tests: fix ListCloudGetTargetURL test - !3882
fix map version sync - don't update higher version smap - !3885
show ETL logs when a test has failed - !3898
add flag to skip local tests - !3902
retry failed ETL YAML get from github - !3915
remove destination bucket after each ETL test - !3922
Tests: fix fshc test - !3930
fix evicting bucket/objects for HDFS backend provider - !3940
correctly set SkipArgs for local tests - !3950
fix bucket props for HDFS bucket - !3954
Tests: fix waiting for node is gone in shutdown test - !3969
fix restoring with diff IP - !3976
fix expected message error in AuthN tests - !3978
fix cleanup node error - !3989
minor refactoring around EC tests - !3995
verify that cliBck exists - !4005
Tests: correct initialization of local config in MultiProxy - !4014
Tests: add waiting for node is shut down in TestNodeShutdown - !4016
CI - collect failed Kube tests artifacts - !4021
Tests: TestECDisableEnableDuringLoad fix - !4033
fix rename-bucket typo - !4038
revert rename-bucket wait change - !4039
make sure e2e fails gracefully when there are no mountpaths - !4049
add test to check backend providers in list buckets - !4050
fix fs package tests on deployments with loopback devices - !4055
Tests: Add a more rigorous shutdown test - !4058
refactor devtools; clarify decommission - !4086
add config management tests - !4101
config tests wait for node to be ready - !4113
fix displayed TestETLObjectCloud name - !4141
remove unused test config; update log dir path - !4142
FSHC use number of object based on the number of mountpaths - !4143
Tests: use StartMaintenance instead of Decommission - !4155
Tests: Add tests for configuration resets - !4159
Tests: skip MOCK nodes when saving node command-lines for restore - !4163
re-enable AddNodeDuplicateIP tests - !4167
fix invalid url in TestRProxyGCS - !4176
retry starting aisfs binary - !4195
skip TestRWStress on HDFS backend - !4196
Tests: stressi rebalance - !4224

Documentation

close tar in imagenet ETL example - !3555
fix etl commands in k8 dev docs - !3558
update video thumbnail images and add favicon for future web use - !3559
revisit and update bucket documentation - !3704
add HDFS provider documentation - !3723
describe --force option for set-props - !3729
add HDFS tutorial - !3759
reword and add link to HDFS provider section - !3760
Remove inconsistencies in console examples - !3932
update options in clean_deploy script - !4054
Clean up dead links - !4060
rework bucket provider document - !4067
fix cli output - !4096
Minor polish for CLI reference - !4185
Update CLI demo gifs - !4222
minor cleanup - !4231
clean up for ais show and ais job - !4243
readme-3.4: new pics - !4238
Update node decommission docs - !4186

aistore -

Published by alex-aizman almost 4 years ago

Highlights

ETL - inline and offline dataset transformations, custom user-defined transformations via both user-provided containers and Python scripts, simplified ETL initialization, ETL directly to and from Cloud buckets;

Multi-Cloud capability supporting co-existence and management of datasets originating from (or hosted by) different Cloud storages - !2736, !2737, !2748, !2792, !2793;
Maintenance and decommission - the capability to put a clustered node in maintenance mode and/or safely and permanently remove it from the cluster - #947, !2935, !2957, !2983, !2990, !3094;
Volume metadata (VMD) - persistent information that describes each clustered node's storage configuration (including data drives, local filesystems, mountpaths) further used to reinforce data integrity and protection - #939, #941, !3118, !3198;
New protocol prefixht:// - uniform access to "vanilla" HTTP(S) based datasets - #882, #889;
Terraform integration - easy and automated deployment via Terraform - there's a separate repository (of scripts, charts, and documentation) that we use for production deployments;
Intra-cluster communications - the transport we use to rebalance user data, transfer erasure-coded slices, copy and transform datasets - a major upgrade !2860, !2895, !2984, !3053, !3055, !3066, !3084, !3085, !3097, !3112, !3181, !3183, !3184, !3187, !3189, !3201, !3265, !3268, !3274, !3286, !3303, !3356, !3357, !3396, !3403, !3409, !3415, !3417.

And also:

performance optimizations, CLI usability improvements, refactoring, cleanup, and stability fixes across the board.

Multi-Cloud

A new protocol prefix ht:// (in addition to s3://, gs://, and azure://) for seamless integration and uniform access to "vanilla" HTTP(S) based datasets.

Multi-Cloud via a single deployed runtime. Improved access to public Cloud buckets (from different Cloud providers). Bucket copying and transformations (see ETL below) extended to supports Cloud buckets.

New HTTP provider (ht://) - #882, #889
Multi-Cloud - added runtime support for bucket management of multiple Cloud providers - !2736, !2737
Support multiple regions for AWS buckets - #778, !2804
Improve Google provider error handling - !2792, !2793
Public GCP buckets can be use without setting PROJECT_ID - !2723
Remove default Cloud provider option (provider no must be set explicitly) - 2748
Support Cloud-based source/destination in a bucket copy operation - !2975
Prefetch performance improvement: keep cached object properties longer - #969

Core

Improve cluster stability in the presence of exceptional events, optimize cluster operation under heavy workloads, introduce maintenance mode, support permanent decommissioning of nodes from the cluster, improve the reliability of bucket destroy operation, optimize and further stabilize cluster rebalancing logic.

Node maintenance feature - #947, !2935, !2990, !3094
Improved out-of-space (out of capacity) handling - #822
Backend buckets vs bucket initialization - !2841
Improve cluster stability while it is in transition (when the primary changes) - #945, #968, #960
If cluster restarts during rebalancing we will now resume the rebalance - #913
Optimize copy-bucket and other bucket-traversing workloads - #917
Make promote consistent with other object operations - !2763, !2765
Add transfer statistics for resilvering - !2926
Configuration option Rebalance. Enabled now; affects only automatic rebalance (manual one can always be started - !2915
Reduce resource usage by StatsD (Grafana, Graphite) client - !3240
New CLI option --daemon-id to join a node with user-predefined ID - !3255
Fix object rename operation to work across different mountpaths - !3329
Make destroy bucket operation transactional - !3315
Volume meta data (VMD) - persistent information about a node and its storage configuration, used on startup when running node integrity checks - #939, #941, !3118, !3198
No metasync when shutting down - !2844
Not ignoring errors when listing multiple Cloud providers - !2845
Refactor reb (rebalance) package - !2857
Refactor target handlers and fix transactions' housekeeping logic - !2869
Refactor copy-object interface - !2879
Revise and refactor PROMOTE (command and API) - !2880
Refactor target copy-object and put-remote interfaces - !2881
Use data mover to copy buckets - !2893
LOM: fix CopyObject - !2908
cmn.JoinWords and friends - !2913
Always allow manual rebalance (even if automatic one is disabled) - !2915
Mountpath resilvering now counts moved objects and their total size - !2926
Copy buckets to return correct total size of copied content - !2919
Revise and optimize intra-cluster broadcasting - !2943
Improve HrwTargetList performance - !2945
Fix zero-size objects scenario - !3531

ETL

Multiple improvements and enhancements to the capability (introduced first with v3.2) to easily run user-defined custom dataset transformations - and scale the performance linearly with each added storage server. This release adds offline (dataset-to-dataset) transformation.

For ETL documentation (that now also includes animated presentations), please refer to docs/etl.md and etl/README.md

Add offline, local and cloud, bucket transformation - !2827, !2854, !2898, !3445
ETL for objects in the Cloud - !3399
ETL build operation - easy initialization based on the function definition - !2873, !2884, !2918, !3369
Remove kubectl (shell) calls, use K8s client-go instead - !2896, !2907
Support retrieving ETL logs - !2947
Stability and performance improvements, bug fixes - !2955, !2977, !3330, !3369, !3374, !3411
Add and improve labels in Pods and Services - !3445
Improve waiting for the Pod/Service to be ready - !3332, !3397
Add extension, prefix, and suffix flags for offline ETL - !2846
Support aborting offline ETL - !2850
Add dry run option for offline ETL - !2854
Simplify flow to initialize ETL - !2853
Consistent naming of API constants - !2861
ETL build: remove unnecessary annotations - !2871
Update skeleton docker images used to run custom Python-based transforms - !2870
Install dependencies in initContainer - !2873
POD spec: add volume mount - !2883
Unify offline ETL with copy-bucket - !2898, !2933
Improve waiting for POD-ready - !2912
Adddry-run capability - !2939
K8s client: pod namespace & refactoring - !2948
The capability to throttle ETL (transforms) depending on disk utilizations - !2998

Terraform integration

Dramatically simplified deployment of AIStore cluster on the Cloud via Terraform. This release delivers GKE but can be easily extended to support any Cloud that provides Kubernetes (service). It is now possible to start a fully functional AIStore cluster with a single command - for details, please refer to AIStore Kubernetes repository.

Add scripts for easy deployment and shutdown of the AIStore cluster on the cloud - !16, !56-!68, #14, #17
Add admin container image - !3079, !3195, !3359
Remove requirement for K8S_HOST_NAME environment variable - !3451

Information Center (IC)

More reliable extended action (xaction) status management and reporting, automatic cluster-wide xaction abort, xaction progress notifications (new). In AIS, xaction is a long-lived asynchronous operation, a job.

Notify all participating nodes when any one of them aborts xaction - !2928
Improve IC status reporting by polling xaction status from targets that have not reported xaction status yet - !2953
Fix xaction registration for newly added targets - !2924
Support both transactional and non-transactional xactions - !2734
Replace target polling with notifications when waiting for xaction to complete - !2868
xactions to return user-friendly status - !2865

Downloader

Integration with IC, more robust downloader job handling.

Downloader naming; fix mountpath register/unregister - !2842
Better job aborting; improved completion mechanisms - #902, !2960
Progress Bar: report periodic status and stats to IC (see above) - !2911

Distributed Shuffle (`dSort`)

Performance improvements, resource usage optimizations.

Performance: decrease resource usage - #938
Better data transport streams handling - #936, !3307

Erasure Coding (EC)

Resource usage optimizations, better slice checksum handling.

Fix checksum when sending constructed slices to other targets - !3073, !3132
Improve operation over data transport streams - #916, !3311
Fix receiving object slices when the bucket is being destroyed - #887
Add support for nodes in maintenance mode - !3404

Intra-cluster communications

The transport that we use to rebalance user data (e. g., when adding/removing nodes), transfer erasure-coded slices, copy and transform datasets has undergone a major upgrade:

Add data mover layer - !2860, !2895, !2899
Support for short messages and message streams - !2984, !3055, !3066, !3084, !3085, !3097, !3112, !3181, !3183, !3184, !3187, !3189, !3201, !3265, !3268, !3274, !3303
Revise and optimize transport stream multiplexing - !3141
When done transmitting, wait for data mover quiescence - !2903
Support streaming unsized objects - objects of unknown size - the functionality in particular useful when ETL-transforming objects on the fly (that is, inline) - !3356, !3357, !3396, !3403, !3409, !3415, !3417
Optimize memory management and debug unlikely races: !3053, !3189, !3286, !3298, !3309, !3314, !3319
Data mover: is-open vs quiescent - !2941

CLI (tool)

New command ais show mountpath, new option --keep for PROMOTE operation, allow running certain commands without accessing a cluster, redesigned ais rm node command, automatic progress indicator for long ais ls <bucket> operations, many fixes for various show commands.

Display EC xaction extra information for ais show xaction command - #823
Improve user experience: commands that do not need a cluster do not require the cluster is running - #878, !2914
Listing bucket objects with the flag --all displays all objects (including temporarily misplaced) - #964
Command ais cat now prints only object content, trailing object size information line is removed - !2729
Cloud bucket can be downloaded without setting backend bucket - !2803
Added progress indicator when listing a huge bucket - #884, !2786
Unify --all sub-option for all commands - !2843, !3264
New option for PROMOTE command: --keep original files after promoting them to objects - !2880
New command ais show mountpath to display target mountpath info - !2900, !3387
Fix displaying rebalance statistics - !3264
Fix ais show xaction rebalance to show the last xaction - !3250
Fix ais show cluster smap - !3243
Revise ais rm node command: add mandatory option --mode (to choose between node decommission and putting node in maintenance), and optional --no-rebalance (to skip rebalance and execute removal immediately) - !2965
An option to remove all finished download jobs - !2849
Wait option (flag) - !2876
New command ais show mountpath - !2900
Fix 'show rebalance' showing rebalance stats - !2954
Refactor CLI cat/get top-level commands - !2972

Other

aisloader (benchmark): add progress indicator when listing very large buckets - !2821
aisfs: APPEND operation is now checksum-protected - #780
build: use custom image for faster CI, enable more linters, switch to Go 1.15, add memory and CPU profiling options via make, upgrade third-party packages - !3235, !3121, !2949, !2916, !2993, !3050
CI/CD: fix k8s development scripts, run many more tests in minikube CI, add terraform GCP playground - !2851, !2858, !2980.
S3 compatibility: support AIS buckets with Cloud backend - !3532, #67, #68

aistore - 3.2.1

Published by alex-aizman about 4 years ago

bug fixes and improvements:
- multi-cloud
- Cloud backend
- HTTP(s) datasets
- CLI usability
- AWS regions
- docs
- development playground (docker and Kubernetes)
add offline ETL (in-dataset => out-dataset, experimental)

Package Rankings

Top 2.79% on Proxy.golang.org

Top 6.79% on Pypi.org

Related Projects

mc

Simple | Fast tool to manage MinIO clusters

16 Jan 2015 2,832

web-service-dlw

Restful Golang Microservice

22 Dec 2021 4

cc-azurelocal-csi-driver

01 Apr 2024 0

meta-sca

Layer for static code analysis and security hardening

23 Jan 2019 89

aistore

Highlights

Core

Python

Bench

Build, CI

Documentation

CLI

Highlights

Core

Bench: aisloader and aisloader-composer

CLI

Build, CI, Deployment

Core

Tools and tests

Build

Core

Performance

Python

Datasets (resize, resort, and shuffle)

CLI

Build, test, and tools

Documentation

Core

CLI

Python

Docs

Build, CI

Table of Contents

CLI

Python: SDK library and ETL

S3 compatibility and Botocore

API changes and new APIs

Core

Build and Continuous Integration (CI)

Tests and Documentation

Extensions: Downloader, dSort, ETL

Table of Contents

Remote AIS clusters

List objects

List & summarize buckets

S3 API

CLI

Documentation

Tests

Python SDK & ETL

Core

Refactoring & fixes

Tools

Build & CI

Highlights

Authentication and Access Control

Python SDK

GitHub Action

CLI

Bucket Summary

Common environment and system filenames

Misc. Features

Docs

Tests

Build

Performance micro-optimizations, bug fixes, refactoring

Highlights

Global Config v2 & Bucket Metadata v2

msgpack archiving; list, create, and read msgpack formatted shards

Python SDK

CLI

Tests

Docs

Build

Bug fixes and performance improvements; continuous refactoring

Highlights

Promote files and directories

ETL

Global Rebalance

Resilvering (in presence of drive failures and attachments)

Asynchronous Jobs (aka xactions)

CLI

Testing

CI/CD (continuous integration)

Bench: `aisloader` and `aisloader-composer`

`msgpack` archiving; list, create, and read `msgpack` formatted shards

Asynchronous Jobs (aka `xactions`)

Distributed Shuffle (`dSort`)