AIStore: scalable storage for AI applications
MIT License
Bot releases are visible (Hide)
tar2tf
ETL offload that handles on-the-fly TFRecord/tf.Example conversion;list-objects
to greatly reduce response times;list-objects
with advanced filtering capabilities;And also:
Skip already downloaded/existing objects, limit download speed, support Azure Cloud, option to synchronize Cloud into AIS bucket, numerous CLI improvements.
prefix
-ed objects - !2204Reduce/optimize CPU and memory usage. Refactor and stabilize.
Replace old basic authentication with a role-based one. Allow a single AuthN server to manage any number of AIS clusters. Add support for both HTTP and HTTPS AIS clusters. More API endpoints require a token issued by AuthN when AuthN is enabled (before this all GET requests worked without any authentication)
add
/remove
/show user
/show cluster
- !2153Revised and fast list-objects
. Reduce memory usage. Use MessagePack. Employ bigger pages to speed up listing operations.
Experimental support for the caching - list-objects
result can now be used across multiple users/requests.
list-objects
API is now always paged; remove -fast
option as obsolete - !2539list-objects
content: only-cached
, include-misplaced
- !2613only-cached
option - !2645New API that extends list-objects
with added support for filtering and selection (a so-called inner and outer* SELECT).
init
and next
API - #754, !2399No more hardcoded xxhash
as AIS checksum for objects: any checksum can be selected from a list that currently also includes MD5, SHA, CRC, and can be easily extended.
Numerous improvements and bug fixes. In particular, new command-line options, shorter commands, better readable output, improved TAB-TAB
support.
show cluster
- #744ais put object bck/obj -
- #748s3://
and gs://
are aliases for aws://
and gcp://
- !1789register
as join
(as in: join new cluster node) - !1988TAB-TAB
and output improvements - #649, #772, !1888, !1857show cluster
to display a single JSON output - #810--chunk-size
option for PUT object - !2164show object command
- !2185search
command - !2400ais start xaction <name>
are now ais start <name>
- !2448set props
command to show what's actually changed - !2479--parity-slices
and --data-slices
for ais ec-encode
command` - !2387start xaction
or set props
CLI, for instance) - !2377To efficiently and optimally monitor asynchronous operations (jobs), AIStore employs what we call Information Center (IC) - a group of gateways that “own” all the currently running (as well as already finished) jobs in the cluster. Those jobs, codenamed eXtended actions, or xactions, include global rebalance, n-way mirroring, erasure coding, ETL-type distributed workload, and more. IC continuously monitors all async by coordinating with other clustered nodes.
list-
and query-objects
caching - !2570aisloader
: add support for ETL (for benchmarking) - !2573aisloader
)Support TAR generating and reading. Support ETL benchmarking via included echo
(at https://hub.docker.com/repository/docker/aistore/transformer_echo), md5
, and tar2tf
ETL containers.
AIS_ENDPOINT
environment variable (options--port
and --ip
are still supported) - !2642minikube
based Kubernetes development environment - !2456, !2558, !2508aisnode
(AIS target and AIS proxy) Alpine Linux-based minimal-footprint docker image - !2709Make names of used environment variables consistent. Introduce $trash
directory to keep deleted buckets for a while. Safer and better node startup: assorted APIs are now accessible only after the node is up and running.
Extend Local Playground for developers: add K8s minikube .
ais
/aisloader
/cli
for consistency - !2133$trash
directory to put deleted buckets to it - !2351minukube
dev deployment - !2456AIStore v3.1 is a significant upgrade with new capabilities that include:
And also:
AIS_PRIMARY_ID
: #706, !2033creation_date
: !2010backend_bck
for AIS bucket connected to a Cloud one - it contains a name of a parent cloud bucket: !2096rebalance.dont_run_time
): !2048batch_size
: !1922client
): !1901--provider
flag in favor of provider://bucket_name
syntax: !1763ls
command by moving subcommands to show
command: !1786wait
to wait for xaction/dSort job/download job finishes: #645cat
to show object's content: #646attach remote
and detach remote
(cluster): !1968attach mountpath
and detach mountpath
: !1986set primary
: !2053compose
command as concat
: !1745--dry-run
flag for put
, evict
, delete
, and prefetch
commands: #636, !1828ais put
more intuitive when generating object names from file paths: #640TAB-TAB
completion improvements: #649, !1745, !1786, !1763, !1818, !1988, !2006ls
) for large buckets: #644show xaction
(!1704)make
and build: enhancements and improvements to consolidate most of (and most often used) build, run, and test operations (#564, !1466, !1483, !1498, !1512)aisnode
container start script (!1725)single-node-aistore
: docker image for easy and fast turn-key single-host deployments