AIStore: scalable storage for AI applications
MIT License
Bot releases are visible (Hide)
GET
, download, prefetch, and/or copy the latest remote (object) versionGET
primitive) and tools (CLI, aisloader
)See also: aistore features and brief overview.
GET
: add stats !6435dsort
: sort records (minor) !6445list-objects
utilities and helpers; rerun list-objects
code-gen: refactor and optimize; cleanup !6450, !6451PUT
- variations !6473, !6474PUT
and PROMOTE
destination naming !6475APPEND
(verb) to append if exists; amend metadata (major) !6476list-objects
: enforce intra-cluster access, validate !6480get-bmd
; follow-up !6483atime
cache when shutting down !6517atime
, prefetch
, is-dirty
) !6518notification-listener
name !6522not-found
; job status; other cleanup !6529ldp.reader
to honor sync-remote
option !6537log-modules
, log-level
); micro-optimize !6538not-found
vs list iterator; OOS !6539join(error)
and fiends !6540list-objects
to report 'version-changed' (new) !6543, !6545list-objects
to report: 'version-changed', 'deleted' !6546list-objects
to support (in-cluster <=> remote) diff !6547, !6548list-range-prefix
iterator !6560list-objects
!6562, !6563ETag
in response headers !6569s3cmd
to run elsewhere !6600, !6601HEAD(bucket)
!6614get-cold-blob
interface; refactoring !6634nil
client !6636ETag
as object version; build !6647sync.Map
instead !6649, !6651max-page-size
!6657pylint
to pass !6556aisloader-composer
: install docker alongside latest cri-o
on CentOS !6436aisloader-composer
: fix install-docker and update OCI inventory !6446, !6449aisloader-composer
: update OCI inventory; avoid using reserved variables in playbooks !6452aisloader-composer
: update dashboard with k8s only networking visualization !6453aisloader
: support latest-version
!6581aisloader
: add '--cached' flag !6623gocritic
"huge-param" !6457gosec
linter !6462etl
label & rule !6488pypi
package for aistore !6492jq
to init container for parsing JSON in Bash scripts !6577meta
, ais
, cmn
); docs !6463bucket.md
, out_of_band.md
, and CLI docs !6527etl-init
usage instructions !6611iterate-fields
separator; 'config log.modules' multi-choice !6458(bucket | multi-object)
!6534job-id
!6614Published by alex-aizman 12 months ago
ais ls --summary
and ais storage summary
to utilize the same control message and flagsprime-time
to amend local generation of globally unique IDs !6325limited-coexistence
; transport: track closed endpoints !6334prime-time
: add caller-is-primary
; up cli module !6335min
and max
functions !6337list-objects
(remote-bucket-and-only-remote-props); Go 1.21 clear
built-in !6339join-err
, remove error channel (minor) !6347vol
package !6349fqn
param to spec templates !6351bucket
, namespace
!6352uname
hash !6359dsort
: error handling, assorted cleanups, more scripted tests !6366validate-warm-get
to support non-versioned buckets, and more !6375validate-warm-get
to support non-versioned buckets !6376HEAD(object)
request !6378load-unsafe
(the faster way to load local metadata) !6382list-objects
and summary
!6384, !6386, !6387config.Rom
to consolidate assorted "read-mostly" config values; refactor and unify !6388checksum
, version
) if not asked to !6394dpq
) !6402validate-cold-get
!6420validate-warm-get
: add scripted test utilizing remote ais cluster !6423disable-fast-cold-get
; show performance latency
; up cli module !6425aisloader
and aisloader-composer
composer
: add playbook and script for intial aisloader
copy !6333composer
: add support for aisloader --filelist
option !6345composer
: add epochs option for GET workloads !6354composer
: add cluster name prefix to netdata
sources for easier filtering !6357does-not-exist
error; misc !6358dsort
job description !6367show cluster
to report total num disks !6371show performance
: usability fixes, improvements !6379show performance
not to filter regex-selected zero columns !6380ais show bucket
with an option to add remote bucket to cluster's BMD (effectively, create bucket) !6404ais search
: CLI command search results to include idiomatic extensions !6428urllib
version !6372validate-warm-get
(scripted) !6423aisloader
command !6385checkmarx
scan !6391golangci-lint
, add linters !6407, !6408Published by alex-aizman about 1 year ago
err-aborted
message !6290aisloader
, logger (assorted fixes) !6298aisloader
: list and read s3 buckets directly !6291aisloader
: list, read, and write s3 buckets directly !6292aisloader
: fix cleanup option for s3 bucket benchmarks !6294aisloader
: reimplement direct get from s3 - use SDK !6295aisloader
: show progress when listing s3 directly !6296ETL_NAME
var for CLI tests !6310aisloader-composer
: add playbook and script for clearing Linux Page Cache on all AIS targets !6311aisloader-composer
: add playbook for copying aws credentials !6314arg-type
!6287, !6323Published by alex-aizman about 1 year ago
rand
for 100% or 50% read/write workloads !6256aisfs
(FUSE) !6273Published by alex-aizman over 1 year ago
htext
to track restarted state; target run and misc !5966metasync
just-in-time; assorted refactoring (minor) !6001meta
package !6006, !6007cmn/archive
package !6044nlog
(new logger) !6113, !6122, !6124fs.PathError
; add object name validation !6127.tar.lz4
!6128join-err
; reduce default pre-election interval !6141transform-url
flag (used when initializing ETL with code) !6010mountpath
completions; disable/detach; minor ref !6056dry-run
option is back !6078pyaisloader
: total size/count fix !6146pyaisloader
stage !6148aisnode
image !6154Published by alex-aizman over 1 year ago
See also:
xaction
term and subcommand (everything is job
now) !5692ais create bucket --props
& ais bucket props set
!5706show jobs
(major) !5726, !5727bucket cp
alias !5730etl-name
transition !5767etl-name
in AISDataset
and AISFileLoaderIterDataPipe
!5957NoRecursion
to support S3 delimiter feature !5930xaction-snap
control structure (major upd) !5740api
package (major) !5776prefix
option !5894cluster.Target
interface !5689err-stream-terminated
!5733has-active-peers
; refactoring !5760delete-object
; lesser assert !5782KindSize
!5815ais-init
image !5646golang/x/net
security alerts (two commits) !5838test-promote
!5763dloader
package (formerly, 'downloader') !5647, !5648, !5649etl
, downloader
, dsort
!5664notif
abstraction (minor) !5902Published by alex-aizman over 1 year ago
Published by alex-aizman almost 2 years ago
See also:
HealthUptime
; use it to show remote clusters !5636CACHED
(column) only when it's informative (and not when user asks ais ls ... --cached
) !5554, !5555list-objects
structures; regenerate message pack !5542, !5543list-objects
version 2 (major): !5583, !5584, !5586, !5589, !5590, !5591, !5601, !5602, !5605, !5607list-objects
flow with a single target listing remote props (name, size, version) !5608list-objects
to include (in listing result) custom object properties !5614get
, put
, head
object, and list-objects
!5615, !5616/gs
, /az
, and /ais
endpoints !5486list-buckets
and get-bucket-info
API; CLI summary table !5521, !5522list-buckets
version 2 (major update) !5524list-buckets
version: w/ presence but wo/ summary !5535, !5536- [API change] add presence
filter; eliminate bucket-info control structure (is redundant); refactor !5540list-buckets
and bucket-summary
cont-d !5541bucket-summary
on-disk (apparent) vs sum-all-objects, remote vs cached !5545bucket-summary
and list-buckets
cont-d unification !5546, !5547list-buckets
by target: remote ais; hdfs and http; cloud; other !5567list-buckets
correctness for named bucket queries !5626list-buckets
to include all providers !5485Server
when redirecting !5495aws s3api
example !5500HEAD object
implementation; multipart ETag
; miscellaneous fixes !5504ais object ls
command; add present
flags !5514cgo
disabled !5526display-name
helper (and consistent usage) !5539--all
to list both buckets and objects (in remote buckets) !5551atime
& ctime
!5612curl
example to set backend bucket property !5380ulimit
, reference checked-in version of limits.conf
!5429Local Playground
option !5484IO communicator
!5415list-buckets
: add cases; up CLI !5569, !5570bucket-exists
helper !5606list-buckets
test that checks expected result !5631IO communicator
fixes !5413init-code
with chunk-size param (major) !5446, !5447init_code
changes for before and after functions !5449filter
(new), transform
, and before
/after
!5451hpush
yaml with bitwise flags !5479/gs
, /az
, and /ais
endpoints !5486User-Agent
to http request headers !5494lom.rename-file
w/cleanups !5509lom.persist
& recache/refresh
!5510atime
on PUT !5511api.GetBucketInfo
!5519fs
: add size-bucket and refactor x-bucket-summary !5523cos-assert
; align (minor ref) !5587x-archive
to always close data mover !5594rx-loop
!5599list-objects
versioning part, speed-up working with versioned buckets !5612, !5613get
, put
, head
object, and list-objects
!5615, !5616try-init
logic to add buckets on the fly !5628bool
) !5635HealthUptime
and use it to ais show
remote clusters !5636xaction
, move xs
package !5383dbdriver
, simplify !5384etl-md
sgl !5414fieldalignment
analyzer !5431, !5432, !5433, !5434, !5435, !5437, !5438, !5440, !5442curl
in aisnode k8s image !5427gofmt
to replace interface{}
with any
!5533objwalk
pkg !5551, !5553trname
(transport endpoint name) !5614devtools/trand
for tests; use strong randomization for product !5465get-primary
, wait-for-cluster-state
, wait-smap-sync
!5571, !5580, !5581aisloader
: writing non-ais buckets !5600buckets-contain
helper !5627--override-backends
and more !5617, !5618.dockerignore
!5382limits.conf
to all docker images !5391aisnode
for minikube !5398golangci
: re-enable nolintlint
, run go1.17, comment-out disabled !5439cli
module to have its own Makefile !5505nolintlint
!5424golangci
v.49 !5489Published by alex-aizman over 2 years ago
ais auth
CLI !5281copy_bucket
method !5225test-short
github workflow (Linux) !5271rm -rf
!5224ais show cluster
; add K8s POD names !5243, !5244$HOME/.config/ais
!5327config node reset
!5363iter-fields
unit for recently added config extras !5290list-objects
flags: unify, rename, document, and reference !5332Published by alex-aizman over 2 years ago
cos.size
type and "kb", "mb", etc. suffixes !5156cos.size
vs backward compatibility !5157memsys
section and tunables !5158join-cluster-at-startup
timeout !5161log-stats
time, add validators !5171tcb
(transform-and-copy-bucket) section; stream bundle multiplier; compression !5173msgpack
archiving; list, create, and read msgpack
formatted shardsmsgp
(generic) shard !5100, !5101msgp
generated files !5086msgp
; sgl as io.ByteScanner
; Python unpack to test !5103msgpack
!5104msgpack
(in addition to tar, tgz, zip) !5107, !5108list-buckets
!5120list-objects
API !5163put-object
& head-bucket
API !5141_request_raw
method !5164head-object
method !5166setUp
and tearDown
in tests !5167evict-bucket
and delete-object
; add props argument for list-objects
!5175get-object
returns a stream instead of the entire content !5184read-all
method for streams !5196list_object
to support pagination !5198msg.py
-> types.py
!5203show cluster config
feature flags in human-readable format !5133ais object rm --all
to remove all in one shot !5195msgp
(generic) shard !5100, !5101devtools
: wait-cluster; transport !5209msgp
usage; add msgp-update
to Makefile !5085msgp
generator usage !5094msgpack
!5110fasthttp
package !5092lom
!5083content-length
!5102atime
when !5105write-policy
: both metadata and data; bump up config's meta-version; backward compatibility (major) !5121bench/soaktest
; rewrite devtools/readme
!5136mock-bmd
=> cluster/mock (refactoring) !5147deploy.sh
script for older bash versions !5165get-object
lock/unlock (refactoring) !5176+build
directive (obsolete) !5188, !5189aisloader
works !5199api
: xaction
wait-IC min waiting time !5202compute-checksum
and clone-md
(ref) !5205memsys
: mem-free to include buff/cache !5206memsys
: revise sys/mem (major) !5207lif
to lom
conversion: free on error; EC error handling !5208lif
(ref) !5212lom
as a reader, lif
as unlocker !5213gfn
): reg hk
housekeepr under lock; abort to self-expire !5215lom
copy: allocate with name !5216lom
xattr (ref) !5217Published by alex-aizman over 2 years ago
AIS v3.9 is substantial productization and performance-improving upgrade. Much of the codebase has been refactored for consistency, with numerous micro-optimization and stabilization fixes across the board.
sync.Pool
) for all control structures in the datapath;prefetch
jobs; refactor and productize;xactions
(asynchronous jobs): bucket rename vs bucket copy, put a node into maintenance mode vs offline ETL, and similar;mountpath
events (e.g., drive failures);fmt.Errorf
and consistently use assorted error types instead;DPQ
) and many more.delete-src
(delete source) and overwrite-dst
(overwrite destination)!4988HEAD(object)
implementation and utilize it when promoting with overwrite-dst=false
(major) !4991overwrite-dst
and delete-src
E2E tests !5024ais
| cloud
| remote-ais
) bucket vs. (non-redundant | EC
| n-way mirror
) !5068init
spec (pod template)fs.Walk
for global rebalance (refactoring) !4889, !4930devtools
: merge WaitForRebalanceToStart
and WaitForAllResilver
!4937wait-for-all-resilvers
, multi-snap API !4888xactions
)put-xaction
finishing logic !4887only-running
and/or by UUID !4897xaction
and xreg
packages with refactoring !4898xaction
!4908xact
package: revise and optimize abort-checking concurrency !4923xactions
!4978xactions
: same rules for all !4982copy-object
-- xaction
-- promote
: continued refactoring !5002--help
flag to the subcommands of subcommands !4926cluster-unreachable
error !4983--force
option !5042start etl
command !5056ais show cluster
: add support for refresh=<time-interval>
and count
options !5076ensure-num-mountpaths
helper, and reinforce !4892api.WaitForXaction
instead of tutils.WaitForXactionByID
!4893fs-checker
test, allow more time for mountpaths !4903WaitFor
follow-up !4943AuthN
messages !4952bucket-rename-and-copy
test !5060test-short-minimal
to test a single-node cluster !5046AWS_REGION
global env variable !5050test-long
stage !5059LOM
load to return distinct types: syscall-error
and corrupted-error
!4849LOM
vs n-way mirroring: fix and revise caching of the metadata !4850list-objects
: add fast mode --only-names
!4851api
: refactor PUT API; fix devtools
!4853aisloader
bench: do not run goroutine per each PUT
request !4855atime
(major) !4856no-metadata
error; fix n-way stress; refactor !4857LOM
from-fs
logic !4862atime
across LOM
caches !4869cmn
and api
packages !4871list-objects
: use-cache
option !4872OWT
) fusion !4878health
package under fs/health
!4885obj-list
query: move, refactor, and simplify !4895, !4896call-results
back to pool !4899api
: eliminate code duplication !4901ais
package !4904cplane-operation
and max-keepalive
!4918list-objects
: name-only option (follow-up) !4928fs
: non-recursive walk
in lexical order !4931fs
: refactor bucket-traversing logic, eliminate nested closures" !4935list-objects
and bucket-summary
: refactor target side !4938transport
: redefine client callback to return error !4941transport
: consistent drain-and-free cleanup on the receive side !4942gofumpt
& gocritic
!4944list-objects
buffering, caching: refactor, optimize !4946lstat
!4960api
package: memory pooling !4961DPQ
; t.Helper
!4962api
package: memory pooling !4963api
package: memory pooling !4964fs
and cos
packages: alternative slightly faster fstat
to check existence !4965init-code
and init-spec
; intuitive RESTful APIs - part 2 !4966api
package: memory pooling !4968api/object
, aisloader
: continued refactoring !4969xmeta
support for LOM
!4977put-obj
; mem-pool !4994cos
package: inline usage of one-time constants !4996copy-object
, copy-reader
logic !4997xmeta
!5003golangci-lint
is installed differently !5008cmn
package: remove LogLevel
and Vmodule
fields from config !5009transport
: transmit/receive unsized objects resulting from streaming transformation !5017apc
: new package for API constants (major ref) !5026apc
package: move metadata-write
and refactor bucket-props validators !5032apc
package: move access control (ref) !5033apc
package: move action message (ref) !5034apc
package: move list-objects
control message !5035apc
package: move copy/transform bucket message (ref) !5036jogger-group
bucket inside individual jogger's visit-object
callbacks !5048LOM
(major) !5049LOM
(metadata): follow-up !5065Published by alex-aizman almost 3 years ago
mountpaths
(disks) at runtime !4721, !4722, !4728, !4729, !4735, !4736, !4740, !4744, !4789, !4818gs/bucket/object
, s3/bucket/object
, and similar (easy) URLs across all backends !4711node
| batch job
| cluster
!4792, !4793, !4794, !4798, !4800, !4810, !4812show object
(and its variations): improve usability and fix minors !4674show config
improvements; follow-up !4676ais storage cleanup
!4704attach mountpath
and detach mountpath
(ref) !4705show storage
subcommands !4713--wait
option for storage cleanup command !4773show job xaction
!4788alias
command and docs !4795xaction snapshot
(aka "snap") into CLI templates !4806ais cluster show stats
subcommand !4807job xaction show
command; minor refactoring !4815, !4816vmodule
(obsolete) !4826ais cluster rebalance
command, allow ais archive ls
to list entire bucket (of tarballs, etc.) !4828unknown-value
and not-set-value
, and use consistently; fix stop-maintenance
logic !4842TestForwardCP
!4619t.Setenv
instead of os.Setenv
and defer os.Unsetenv
!4620xaction
to finish when enabling mirroring !4628TestGetAfterReregisterWithMissedBucketUpdate
!4640tassert
: timestamp failure message, customize print-stack !4647mountpath
add/remove across restarts !4821mountpath
(non-redundent bucket) !4822mountpaths
!4835set-custom
!4673http_api
docs and add missing json for list buckets
operation !4715ais cluster show stats
!4809ais show job
CLI !4817run
and restart
make targets !4707err-aborted
, refactor common errors, downloader !4611docker build
and docker push
commands !4614xaction
: mutex to protect compound state !4618etl
: add more details when K8s pod times out on being ready; add debug log !4621, !4624xaction
: add immutable (original) bucket !4626lru
pkg: trash non-existing buckets; health check: docs and follow-up ref-s !4630etl
: fix waiting for a ready condition when starting K8s pod !4633can-start
& must-run
helpers !4634, !4635backend
module; AWS: ignore "unknown region" most of the time !4637xaction
pkg: consistent naming and refactoring !4641etl
: remove single-pod limitation (i.e., the capability to run multiple ETLs) !4649backend.HeadObj()
API, simplify cold-GET !4664inc-version
dichotomy !4668transport-idle-teardown
!4675memsys
: tracking memory stats and responding to OOM (major rev) !4678xmeta
(tool): add VMD
, refactor !4680etl
: add new (non-HTTP) communicator and stop relying on the K8s API Server !4681SizeBytes()
and AtimeUnix()
methods for CT !4682volume
: revise, refactor, and reinforce operations on metadata !4683, !4684downloader
: refactoring !4685volume
: consolidate loading and initialization; bootstrap in two passes !4699cmn/context
=> cmn/cos/context
; rename "attach/detach remote ais" (minor ref) !4700volume
: keep it sync with local config !4701etl
: add more Python runtimes !4702volume
: keep it sync with local configuration !4703mountpaths
: pass node ref consistently !4706run
and restart
targets !4707cluster/hrw
and cmn/err
: continued refactoring !4719mountpath
(disk) removal - major revision !4721, !4722fs
: more validation and naming consistency !4725api
to attach mountpath
with force
!4728mountpath
, with committing after resilvering !4729mountpaths
; increase CI timeout !4732reb
pkg, add res
(resilver) pkg !4737GFN
(state): simplify global, move/rewrite local !4738xaction
registry vs housekeeper: tweak initializations !4742housekeeper
: run first & reinforce; x-factories: ordered initialization !4743cos
pkg: make ExitLogf
print file and line of the caller !4745housekeeper
: add wait for housekeeper readiness when registering !4746fs
: get
and get-available
; test-init
; refactoring !4747dsort
: fix panic when creating shard !4750memsys
: reduce free-mem requirement for tests & minor fixes !4752memsys
for tests (follow-up) !4755memsys
and housekeeper
: further clarification !4759deleted
(formerly trash
) !4761memsys
vs mock target: more orderly initialization !4762memsys
: tune-up low-watermark !4764memsys
: two distinct pairs of slab allocators !4767mountpoint
to mountpath
for consistency !4768memsys
: tune-up for OOM !4769memsys
: reintroduce max-depth !4770panic-OOM
condition !4771memsys
: fix swapping state !4772, !4774fs
: make mountpath
flags atomic !4775xreg
pkg: remove obsolete RebalanceArgs
struct !4777ais
pkg: split etl.go
into tgtetl.go
and prxetl.go
for consistency !4778cmn
pkg: sort actions alphabetically !4779etl
: integrate etl communicator with xaction
!4780set-name
when joining nodes; keepalive logging; test tweaks !4781xactions
anymore when adding/removing mountpaths
!4782fs
: refactor fqn parsing !4783fs
: refactor fqn parsing, update unit tests and docs !4784, !4785fs
: minor fixes; tests !4787mountpaths
without resilvering !4789etl
: manage etl metadata !4790xaction
stats: remove redundant interfaces, rename !4792xactions
: introduce snapshot, add in and out counters !4793, !4794xaction
snapshots !4796dsort
: wait for all in-flight stream requests !4797transport
: complete transition to using opcodes !4799stats
pkg: add get-daemon-stats
API, remove redundant code and refactor !4800etl
: fix panic in unit test !4801etl
: change signature of InitSpec
function to require options !4802memsys
: tweak init-once logic yet again !4803xactions
: complete transition to snaps (ref) !4805dsort
: fix placement of aborted check !4808volume
: when creating new volume (and associated VMD), make sure to have a record of configured fspaths
!4834cmn
pkg: add is-err-not-found !4836ais advanced remove-from-smap
command !4841minikube
: RBAC v1 !4843Published by alex-aizman about 3 years ago
minikube
version !4598Published by alex-aizman about 3 years ago
UUID
back to client !4488archpath
query parameter !4543checksum-sizer
; archive: add basic stats; eliminate fseek !4548archive
command !4570Init
call to the primary !4591xaction ID
!4449cluster.XactID
(ref) !4474xaction.Args
- simplify (ref) !4475xaction
(w/ code reduction) !4515WPR
) !4517eof-ok
!4508aisnode
!4470performance
value !4471bucket props
command !4506docs
directory !4514cli
documentation !4558PrefetchList
and PrefetchRange
!4466archiveListRange
test, make generated obj names more predictable !4468api
package !4455target.getObject
, eliminate copy-paste !4457lom.HrwTarget
helper (ref) !4465unused-parameter
!4493AllocSize
!4509Hdr.SID
instead of internal EC.intraReq.sender
!4512transport.Hdr.Opcode
instead of intraReq.action
!4519remoteRequest
msg !4525bucket lru
subcommand; show bucket props by default !4552ais bucket cp
: support --list
and --template
options !4589--list
and --template
options !4593Published by alex-aizman over 3 years ago
--validate
for bucket summary to show the number of 'not-ok' objects !4341storage
!4350ais storage
command !4362--extract
option; refactoring !4390archpath
keyword; add test; refactoring !4391GET
from tar, tgz, and zip !4394aisloader
: require -cleanup
option !4335embed
package !4336cmn
: rename BucketNames
struct to Bcks
to better describe content !4359fs
, ios
, ais
: minor refactoring & cleanup !4360mountpath
, fs disks
(ref) !4361fs
: revise init and validation logic !4363fs/vmd
minor ref !4364metasync
: not using nil SGL for payload (fix) !4366metasync
: delayed cleanup (fix) !4367aisloader
container !4368cos/utils
: create-file now always creates write-only !4379dsort
: replace section reader !4380ais
: minor ref (get, extract) !4383metasync
, BMD
, SGL !4385DEADBEEF
; miscellaneous !4389golint
is deprecated: using revive
!4406memsys
: tune-up and refactoring !4409xaction
providers !4420xaction
args !4421xaction
factory (ref, consistency) !4422read-json
: prevent superfluous response !4423xaction
renewal (internal) API !4424xaction
registry (part one) !4430fullname()
shortcut for: lom
, transport header, archive !4436GODEBUG=madvdontneed=1
vs Go 1.16 !4290dSort
tests with memory and disk !4294jsp
, optimize serialization (part one: cluster map) !4296revs
!4297target_ID
and disk_name
!4303fflush
upon close prior to rename !4304, !4305, !4307reg-pool
processing; handle global config !4326Published by alex-aizman over 3 years ago
deploy
script !4256ios
: minor optimizations; vmd
: refactor init/load !4268Snode
flags only when changed !4269cluster configure
to top-level commands !4248append()
to initialize commands !4253alias
; add list, reset !4257Published by alex-aizman over 3 years ago
fs.GetContentFQN
with interface - !3548CT
and LOM
structures - !3549ReadOpenCloser
on ReadOpenCloser.Open
- !3562SliceReader
- !3563rlock
- !3615origURLBck
in bucket init - !3720LOM
(part two) - !3839roff
- !3661interface{}
instead of unsafe.Pointer
as callback argument - !3953CopyObject
- !3738UpgradeLock
- !3761defaultBckProps
- !3807terminate
flag is set - !3889HTTPError
on call
- !3608QueryBcks
from request - !4105err
from bckInitArgs
struct and rename queryBck
- !4123bckInitArgs
- !4124metasync
when decommissioning a target - !4087put
progress bar report for large uploads - !3556ais rename
issues - !3560makePairs
to accept values with =
characters - !3747cp objects
from help message - !3764rename
command to mv
- !3668object
and bucket
top level commands - !4029rm download/dsort
inside job
- !4043gen-shards
input and allow specifing provider - !4048ais show auth
and ais show bucket
- !4062cluster
and disk
top level command - !4006job
top level command - !4017advanced
top level command - !4018show cluster
- !4182msgp
method - !3553cmn.Bck
instead of bare name and provider - !4066TestETLBigBucket
- !3910TestETLObjectCloud
- !4004HOSTNAME
is not reliable - !3567aistore/aisnode
image - !3610RE
variable for K8s tests - !4040initContainer
- !3651Data:Parity
for a bucket if the number of nodes is between... - !3708LOM
- !3806cksumType
parameter - !3847CompareObjects
test - !3845lom.Size
- !3851checkResp
func - !3688DoReqWithRetry
function - !3926GetDaemonConfig
function - !4036bucket create
- !4208GetWhatRemoteAIS
- !4210error
parameter from Run
methods - !3574Finish
on the end of xaction - !3576Run
function - !3581smap-modifier
- !3628apiRequest.parse
in proxy - !3636Extra
by provider - !3680make deploy
- !3694backend provider
- !3730UpgradeLock
- !3745cloud
package to backend
- !3752UpgradeLock
- !3754CloudVersion
to RemoteVersion
- !3762cloud_mock
to backend_mock
- !3771lom.bucket
(ref) - !3779wack-target
memory footprint - !3792ContinuationToken
for HDFS's list objects - !3797|
character from HTTP error string - !3798skip_verify_crt
option configurable in aisfs and cli - !3803cloud
to broader terms remote
or backend
- !3811DeleteObject
method - !3832tutils
: cleanup temporary directory created in PrepareObjects
- !3846PutObj
implementation for remote ais - !3849uncache
field in coi
structure - !3863PutObj
implementation for AIS remote cluster - !3864copy-object-info
: remove unnecessary lom.Loaded() asserts - !3865put-remote
- !3869copy-object
: remote and local parts - !3871invalmsghdlr
and friends - !3918cloud
field and struct to backend
- !3921cleanup
callback in the reader Close
method - !3923PutObj
usage and delegate closing reader to implementation - !3924DurationJSON
type in Bck2BckMsg
struct - !3941BodyR
- !3951tutils
: simplify DeployNode
and CleanupNode
logic - !3962tutils
: add CheckResp
for checking response status code - !3963fs.CreateBuckets
- !3968tutils
: allow custom intervals for WaitNodeReady - !3970ErrNotFound
across the codebase - !3981/dev/nil
with /dev/null
- !3982Caller
field to HTTPErr
- !3986xaction
: eliminate copy-paste construction - !3988confdir
value - !4041ais
as default provider when parsing bucket - !4042StringSet.Keys
to ToSlice
- !4053PutRandObjs
- !4073devtools
directory structure - !4088aisnode
when decommissioning - !4120QueryBcks
validation methods - !4125expvar
-observable in debug mode - !4149fs
first, snode
second - !4187objMeta
map - !4197clean_deploy.sh
- !4212ais show config cluster
for consistency - !4242TestDistributedSortKillTargetDuringPhases
test - !3550TestDownloadJobLimitConnections
- !3602RE
variable - !4034test:soak
and test:bench
- !4099clean_deploy
script to deploy cluster - !4194TestDownloadJobConcurrency
- !3591/tmp/ais
traversing with fs.WalkBck
- !3814PutRandObjs
- !3819PutRandObjs
- !3866PutObjsFromList
function - !3867TestSmoke
implementation - !3873fs
package tests on deployments with loopback devices - !4055TestETLObjectCloud
name - !4141AddNodeDuplicateIP
tests - !4167TestRProxyGCS
- !4176aisfs
binary - !4195TestRWStress
on HDFS backend - !4196--force
option for set-props - !3729clean_deploy
script - !4054ais show
and ais job
- !4243VMD
) - persistent information that describes each clustered node's storage configuration (including data drives, local filesystems, mountpaths
) further used to reinforce data integrity and protection - #939, #941, !3118, !3198;ht://
- uniform access to "vanilla" HTTP(S) based datasets - #882, #889;And also:
A new protocol prefix ht://
(in addition to s3://
, gs://
, and azure://
) for seamless integration and uniform access to "vanilla" HTTP(S) based datasets.
Multi-Cloud via a single deployed runtime. Improved access to public Cloud buckets (from different Cloud providers). Bucket copying and transformations (see ETL below) extended to supports Cloud buckets.
ht://
) - #882, #889PROJECT_ID
- !2723Improve cluster stability in the presence of exceptional events, optimize cluster operation under heavy workloads, introduce maintenance mode
, support permanent decommissioning
of nodes from the cluster, improve the reliability of bucket destroy
operation, optimize and further stabilize cluster rebalancing logic.
maintenance
feature - #947, !2935, !2990, !3094Backend
buckets vs bucket initialization - !2841copy-bucket
and other bucket-traversing workloads - #917resilvering
- !2926Rebalance
. Enabled now; affects only automatic rebalance (manual one can always be started - !2915StatsD
(Grafana, Graphite) client - !3240--daemon-id
to join a node with user-predefined ID - !3255object rename
operation to work across different mountpaths
- !3329destroy bucket
operation transactional - !3315VMD
) - persistent information about a node and its storage configuration, used on startup when running node integrity checks - #939, #941, !3118, !3198metasync
when shutting down - !2844reb
(rebalance) package - !2857copy-object
interface - !2879PROMOTE
(command and API) - !2880copy-object
and put-remote
interfaces - !2881LOM
: fix CopyObject
- !2908cmn.JoinWords
and friends - !2913Mountpath resilvering
now counts moved objects and their total size - !2926HrwTargetList
performance - !2945Multiple improvements and enhancements to the capability (introduced first with v3.2) to easily run user-defined custom dataset transformations - and scale the performance linearly with each added storage server. This release adds offline (dataset-to-dataset) transformation.
For ETL documentation (that now also includes animated presentations), please refer to docs/etl.md and etl/README.md
build
operation - easy initialization based on the function definition - !2873, !2884, !2918, !3369kubectl
(shell) calls, use K8s client-go
instead - !2896, !2907initContainer
- !2873copy-bucket
- !2898, !2933dry-run
capability - !2939Dramatically simplified deployment of AIStore cluster on the Cloud via Terraform. This release delivers GKE but can be easily extended to support any Cloud that provides Kubernetes (service). It is now possible to start a fully functional AIStore cluster with a single command - for details, please refer to AIStore Kubernetes repository.
admin
container image - !3079, !3195, !3359K8S_HOST_NAME
environment variable - !3451More reliable extended action (xaction
) status management and reporting, automatic cluster-wide xaction
abort, xaction
progress notifications (new). In AIS, xaction
is a long-lived asynchronous operation, a job.
xaction
- !2928IC
status reporting by polling xaction
status from targets that have not reported xaction
status yet - !2953xaction
registration for newly added targets - !2924xactions
- !2734xaction
to complete - !2868xactions
to return user-friendly status - !2865Integration with IC
, more robust downloader job handling.
mountpath
register/unregister - !2842IC
(see above) - !2911dSort
)Performance improvements, resource usage optimizations.
Resource usage optimizations, better slice checksum handling.
The transport that we use to rebalance user data (e. g., when adding/removing nodes), transfer erasure-coded slices, copy and transform datasets has undergone a major upgrade:
Data mover
: is-open vs quiescent - !2941New command ais show mountpath
, new option --keep
for PROMOTE
operation, allow running certain commands without accessing a cluster, redesigned ais rm node
command, automatic progress indicator for long ais ls <bucket>
operations, many fixes for various show
commands.
xaction
extra information for ais show xaction
command - #823--all
displays all objects (including temporarily misplaced) - #964ais cat
now prints only object content, trailing object size information line is removed - !2729--all
sub-option for all commands - !2843, !3264PROMOTE
command: --keep
original files after promoting them to objects - !2880ais show mountpath
to display target mountpath
info - !2900, !3387ais show xaction rebalance
to show the last xaction
- !3250ais show cluster smap
- !3243ais rm node
command: add mandatory option --mode
(to choose between node decommission and putting node in maintenance), and optional --no-rebalance
(to skip rebalance and execute removal immediately) - !2965ais show mountpath
- !2900cat
/get
top-level commands - !2972aisloader
(benchmark): add progress indicator when listing very large buckets - !2821aisfs
: APPEND
operation is now checksum-protected - #780build
: use custom image for faster CI, enable more linters, switch to Go 1.15, add memory and CPU profiling options via make
, upgrade third-party packages - !3235, !3121, !2949, !2916, !2993, !3050CI/CD
: fix k8s development scripts, run many more tests in minikube
CI, add terraform GCP playground - !2851, !2858, !2980.S3 compatibility
: support AIS buckets with Cloud backend - !3532, #67, #68Published by alex-aizman about 4 years ago