dstack is an open-source orchestration engine for cost-effectively running AI workloads in the cloud as well as on-premises. Discord: https://discord.gg/u8SmfwPpMd
MPL-2.0 License
Bot releases are hidden (Show)
Published by peterschmidt85 about 1 year ago
For the past six weeks, we've been diligently overhauling dstack
with the aim of significantly simplifying the process of configuring clouds and enhancing the functionality of the API. Please take note of the breaking changes, as they necessitate careful migration.
Previously, the only way to configure clouds for a project was through the UI. Additionally, you had to specify not only the credentials but also set up a storage bucket for each cloud to store metadata.
Now, you can configure clouds for a project via ~/.dstack/server/config.yml
. Example:
projects:
- name: main
backends:
- type: aws
creds:
type: access_key
access_key: AIZKISCVKUKO5AAKLAEH
secret_key: QSbmpqJIUBn1V5U3pyM9S6lwwiu8/fOJ2dgfwFdW
Regions and other settings are optional. Learn more on what credential types are supported
via Clouds.
The earlier introduced Python API is now greatly refined.
Creating a dstack
client is as easy as this:
from dstack.api import Client, ClientError
try:
client = Client.from_config()
except ClientError:
print("Can't connect to the server")
Now, you can submit a task or a service:
from dstack.api import Task, Resources, GPU
task = Task(
image="ghcr.io/huggingface/text-generation-inference:latest",
env={"MODEL_ID": "TheBloke/Llama-2-13B-chat-GPTQ"},
commands=[
"text-generation-launcher --trust-remote-code --quantize gptq",
],
ports=["80"],
)
run = client.runs.submit(
run_name="my-awesome-run",
configuration=task,
resources=Resources(gpu=GPU(memory="24GB")),
)
The dstack.api.Run
instance provides methods for various operations including attaching to the run,
forwarding ports to localhost
, retrieving status, stopping, and accessing logs. For more details, refer to
the example and reference.
dstack gateway
command, and their usage requires you to specify a domain.dstack start
command is now dstack server
.dstack
package to dstack.api
.Unfortunately, when upgrading to 0.12.0, there is no automatic migration for data.
This means you'll need to delete ~/.dstack
and configure dstack
from scratch.
pip install "dstack[all]==0.12.0"
~/.dstack
~/.dstack/server/config.yml
(see the new guide)dstack server
The documentation and examples are updated.
Getting started with dstack
takes less than a minute. Go ahead and give it a try.
$ pip install "dstack[all]" -U
$ dstack server
Questions and requests for help are very much welcome in our Discord server.
Published by peterschmidt85 about 1 year ago
For the past six weeks, we've been diligently overhauling dstack
. Today, we're excited to introduce the first release candidate of this major update. 🥳🎊
While this update enhances dstack
, it also brings significant ⚠️ breaking changes, necessitating careful migration.
dstack
server. No need to specify a storage bucket for configuring a cloud backend.~/.dstack/server/config.yml
) instead of using the UI.0.12.0
, you'll need to delete ~/.dstack/server/data
and configure dstack
from scratch.dstack start
command is now dstack server
.dstack
package to dstack.api
.Secrets and private Docker registry support coming in 0.12.0
or 0.12.1
.
The updated documentation can be found in the repo. It will be published once 0.12.0
is released (within a week or so).
pip install "dstack[all]==0.12.0rc1"
~/.dstack
~/.dstack/server/config.yml
(see the new guide)dstack server
0.12.0
to be released within the next week or so.0.12.0
and 0.12.1
.Published by peterschmidt85 about 1 year ago
The 0.11.3 update fixes several bugs related to the Python API.
Here's how to install the update:
pip install "dstack[all]==0.11.3"
Please report any issues to the tracker or on Discord.
Full Changelog: https://github.com/dstackai/dstack/compare/0.11.2...0.11.3
Published by peterschmidt85 about 1 year ago
Previously, dstack
offered the CLI as the primary method for submitting and managing tasks, services, and dev environments. However, the latest release of dstack
now allows for programmatic submission and management of runs via Python API.
Here's what the API enables you to do:
You can learn more about the API at https://dstack.ai/docs/reference/api/python.
To showcase how this API functions, we've created a simple example: a Streamlit app that enables deploying an LLM of your choice as a task.
Learn more about the example (and view its source code) at https://dstack.ai/examples/python-api/.
Here's how to install the update:
pip install "dstack[all]==0.11.2"
We encourage you to try the update and report any issues either through the tracker or by writing directly on Slack.
Published by peterschmidt85 about 1 year ago
The upcoming 0.11.2
release brings the long-requested Python API that enables you to submit and manage tasks and services programmatically from your Python code.
You can test this new API using this 0.11.2rc1
pre-release build.
For more information on how to install it and how to use the Python API, refer to the documentation.
Published by peterschmidt85 about 1 year ago
Services is a preview feature that allows serving LLMs via public endpoints. The latest update of dstack
makes the use of services much easier.
Previously, to run a service, you had to create a gateway using the dstack gateway create
command and pass its address via the gateway
property in the service configuration file.
Now, you don't need to use the gateway
property anymore, as you can create a gateway via the UI and mark it as default.
Once the gateway is created (and assigned an external IP), you can set up an A record with your DNS provider to map *.<your domain name>
to the gateway's IP and specify this wildcard domain in the gateway's settings.
If a wildcard domain is configured, dstack automatically enables HTTPS and runs services at https://<run name>.<your domain name>
.
In other news, the update fixes a few bugs with the --retry-limit
argument in dstack run
. Now, it works again, allowing you to schedule tasks even if there is no required capacity at the moment.
Last but not least, we've updated the entire documentation and examples.
Here's how to install the update:
pip install "dstack[all]==0.11.1"
We encourage you to try the update and report any issues either through the tracker or by writing directly on Slack.
Published by peterschmidt85 about 1 year ago
The 0.11 version of dstack
introduces automatic discovery of the best GPU price and availability across multiple configured cloud providers and regions.
Now, dstack run
leverages price data from multiple configured cloud providers and regions to automatically suggest the most cost-effective options.
To implement this change, we have modified the way projects are configured. You can now configure multiple clouds and regions within a single project.
Also, it is now possible to deploy services using HTTPS. All you need to do is configure a wildcard domain (e.g., *.mydomain.com
), point it to the gateway IP address, and then pass the subdomain you want to use (e.g., myservice.mydomain.com
) to the gateway property in YAML (instead of the gateway IP address).
.dstack/profiles.yml
project
property is no longer supported.max_price
to set the maximum price per hour in dollars.dstack run
dstack run
command, you are now able to utilize options such as --gpu
, --memory
, --env
, --max-price
, and several other arguments to override the profile settings.Lastly, the local backend is no longer supported. Now, you can run everything using only a cloud backend.
The documentation is updated to reflect the changes in the release.
The dstack
version 0.11 update brings significant changes that break backward compatibility. If you used prior dstack
versions, after updating to dstack==0.11
, you'll need to log in to the UI and reconfigure clouds. We apologize for any inconvenience and aim to ensure future updates maintain backward compatibility.
Here's how to install the update:
pip install "dstack[all]==0.11"
We encourage you to try the update and report issues to the tracker, or write directly on Slack.
Published by peterschmidt85 about 1 year ago
The upcoming major release, dstack 0.11
, introduces multi-cloud projects. The first release candidate, dstack 0.11rc
, allows you to test this new functionality.
Now, dstack run
leverages price data from multiple configured cloud providers and regions to automatically suggest the most cost-effective options.
.dstack/profiles.yml
: The project
property is no longer supported.dstack run
: The dstack run command now supports --gpu, --memory, --env, --max-price, and other arguments.Version 0.11 brings significant changes that break backward compatibility. You'll need to reconfigure project and cloud backends. We apologize for any inconvenience and aim to maintain backward compatibility in future updates.
Note
The documentation will be updated within a week after the finaldstack 0.11
release. In case of questions, write us in the Slack chat.
In the meantime, we encourage you to try dstack 0.11rc1
and report issues to the tracker or write on Slack.
Published by peterschmidt85 about 1 year ago
This minor update addresses a crucial issue that was causing services to malfunction. If you're utilizing services, be sure to update.
Changelog: https://github.com/dstackai/dstack/compare/0.10.7...0.10.8
Published by peterschmidt85 about 1 year ago
Until now, dstack
has supported dev-environment
and task
as configuration types. With the 0.10.7
update, we introduce
service
, a dedicated configuration type for serving.
type: service
gateway: ${{ secrets.GATEWAY_ADDRESS }}
image: ghcr.io/huggingface/text-generation-inference:0.9.3
port: 8000
commands:
- text-generation-launcher --hostname 0.0.0.0 --port 8000 --trust-remote-code
The gateway
property represents the address of a special cloud instance that wraps the running service with a public endpoint.
Before you can run a service, you have to configure a gateway.
First, you have to create a gateway in a project of your choice using the dstack gateway create
command:
dstack gateway create
Once the gateway is up, the command will print its address. Go ahead and create a secret with this address.
dstack secrets add GATEWAY_ADDRESS <gateway address>
That's it! Now you can run your service using the dstack run
command, which deploys the service and forwards the traffic to the gateway, thereby providing you with a public endpoint.
This initial support for services is the first step towards providing multi-cloud and cost-effective inference. In the near future, we plan to make it more functional and easier to use.
A detailed guide on how to try services can be found in our documentation.
As always, your feedback and questions are very much welcome in our Slack chat.
dstack init
should handle invalid Git credentials by @peterschmidt85 in https://github.com/dstackai/dstack/pull/594
dstack
doesn't work if the repo contains a submodule with SSH URL by @peterschmidt85 in https://github.com/dstackai/dstack/pull/598
code
is not configured in PATH
by @peterschmidt85 in https://github.com/dstackai/dstack/pull/613
dstack init
doesn't work if .git/config
or ~/.gitconfig
doesn't have the user
section by @peterschmidt85 in https://github.com/dstackai/dstack/pull/617
dstack config
doesn't work if ~/.dstack
doesn't exist by @peterschmidt85 in https://github.com/dstackai/dstack/pull/618
Changelog: https://github.com/dstackai/dstack/compare/0.10.6...0.10.7
Published by peterschmidt85 about 1 year ago
The latest release of dstack
brings numerous improvements in many areas.
Any task that is running on dstack
can expose ports. Here's an example:
type: task
ports:
- 7860
commands:
- pip install -r requirements.txt
- gradio app.py
When you run it with dstack run
, by default, dstack
forwards the traffic from the specified port to the same port on your local machine.
With this update, you now have the option to override the local machine's port for traffic forwarding.
$ dstack run . -f serve.dstack.yml --port 3000:7860
This command forwards the traffic to port 3000
on your local machine.
If you specify a port on your local machine already taken by another process, dstack will notify you before provisioning cloud resources.
Previously, when running a dev environment or task with dstack and forgetting about it, it would continue indefinitely. Now, you can use the max_duration
property in .dstack/profiles.yml
to set a maximum time for workloads.
Example:
profiles:
- name: gcp-t4
project: gcp
resources:
memory: 24GB
gpu:
name: T4
max_duration: 2h
With this profile, dstack
will automatically stop the workload after 2 hours.
If you don't specify max_duration
, dstack
defaults to 6h
for dev environments and 72h
for tasks.
To disable max duration
, you can set it to off
.
Imagine the amount of money your team can save with this minor configuration.
With the CUDA version updated to 11.8, dstack
now supports additional GPU types, including NVIDIA T4
and NVIDIA L4
. These GPUs are highly efficient for LLM development, offering excellent performance at low costs!
If you are using a custom Docker image, you can now utilize a CUDA version up to 12.2.
Last but not least, the K80 GPU is no longer supported.
Make sure to check the new page with examples.
The documentation is updated to reflect the changes in the release.
Getting started with dstack
takes less than a minute. Go ahead and give it a try.
pip install "dstack[aws,gcp,azure,lambda]" -U
dstack start
We warmly welcome your feedback in our Slack chat.
dstack run
by @Egor-S in https://github.com/dstackai/dstack/pull/573
11.8
by @peterschmidt85 in https://github.com/dstackai/dstack/pull/584
Changelog: https://github.com/dstackai/dstack/compare/0.10.5...0.10.6
Published by peterschmidt85 over 1 year ago
The Lambda Cloud integration has significantly improved with this release. Additionally, we've added support for custom Docker images.
We've added the possibility to create Lambda Cloud projects via the user interface.
All you need to do is provide your Lambda Cloud API key and specify an S3 bucket and AWS credentials for storing state and artifacts.
Check the docs for more details.
By default, dstack uses its own base Docker images to run dev environments and tasks. These base images come pre-configured with Python, Conda, and essential CUDA drivers. However, there may be times when you need additional dependencies that you don't want to install every time you run your dev environment or task.
To address this, dstack now allows specifying custom Docker images. Here's an example:
type: task
image: ghcr.io/huggingface/text-generation-inference:0.9
env:
- MODEL_ID=tiiuae/falcon-7b
ports:
- 3000
commands:
- text-generation-launcher --hostname 0.0.0.0 --port 3000 --trust-remote-code
Note
Dev environments require the Docker image to haveopenssh-server
pre-installed.
Getting started with dstack takes less than a minute. Go ahead and give it a try.
pip install "dstack[aws,gcp,azure,lambda]" -U
dstack start
Changelog: https://github.com/dstackai/dstack/compare/0.10.4...0.10.5
Published by peterschmidt85 over 1 year ago
dstack 0.10.4 is a bugfix update that addresses two major issues related to AWS and Lambda Cloud.
Feel free to install the update, give it a try, and share your feedback in our Slack community.
Changelog: https://github.com/dstackai/dstack/compare/0.10.3...0.10.4
Published by peterschmidt85 over 1 year ago
With the 0.10.3 update, dstack
now allows provisioning infrastructure in Lambda Cloud while storing state and artifacts in an S3 bucket.
See the Reference for detailed instructions on how to configure a project that uses Lambda Cloud.
Note, there are a few limitations in the preview:
In other news, we have pre-configured the base Docker image with the required Conda channel, enabling you to install additional CUDA tools like nvcc
using conda install cuda
. Note that you only need it for building a custom CUDA kernel; otherwise, the essential CUDA drivers are already pre-installed and not necessary.
The documentation and examples are updated to reflect the changes.
Go ahead, and install the update, give it a spin, and share your feedback in our Slack community.
nvidia/label/cuda-11.4.3
channel for conda
in the CUDA image by @peterschmidt85 in https://github.com/dstackai/dstack/pull/541
Changelog: https://github.com/dstackai/dstack/compare/0.10.2...0.10.3
Published by peterschmidt85 over 1 year ago
With the dstack 0.10.2
release, we've added two major new features that make running dev environments and ML tasks in the cloud 10-x easier.
Consider the following configuration:
type: dev-environment
build:
- apt-get update
- apt-get install -y ffmpeg
- pip install -r requirements.txt
ide: vscode
If you run it using dstack run
and there is no pre-built image available for this configuration, it will fail and prompt you to either use the dstack build
command or add the --build
flag to the dstack run
command. If the pre-built image is available, it will be used automatically.
Now, if your setup is complex or time-consuming, you don't necessarily have to build and publish your own Docker images. You can simply define the build property and use the dstack build command or add the --build
flag to the dstack run
command.
If you have specific commands that need to be executed before the dev environment starts, you can now use the init property in YAML.
You can find more details on the updated syntax of .dstack.yml
in the Reference section.
Another new major feature is support for spot_policy
and retry_policy
in .dstack/profiles.yml
.
The spot_policy
determines if dstack
should use spot or on-demand cloud instances for running dev environments and
tasks. It can be set to the following values:
spot
– Always uses spot cloud instances onlyon-demand
– Always uses on-demand cloud instances onlyauto
– Try to use spot cloud instances if available and on-demand if notIf you don't specify spot_policy
, it is set to on-demand
by default for dev environments, while for tasks,
the default is auto
.
The retry_policy
determines if dstack
should retry when there is no cloud capacity for running the dev environment or
task. It can have the following nested properties:
retry
– true
if dstack
should retry, and false
if notlimit
– The duration within which dstack
should retry. The default is 1d
(one day).If you don't specify the retry_policy
, it is enabled for tasks by default, while for dev environments, it is disabled.
You can find more details on the updated syntax of .dstack/profiles.yml
in the Reference section.
With these introduced policies, using spot cloud instances becomes easier. Additionally, you can now schedule tasks that
utilize on-demand instances. This is particularly useful when there is limited capacity in a specific region and you
want the task to automatically run once the capacity becomes available.
Other improvements in this release:
dstack
correctly configuring your Git credentials.The documentation and examples are updated to reflect the changes.
Go ahead, and install the update, give it a spin, and share your feedback in our Slack community.
Changelog: https://github.com/dstackai/dstack/compare/0.10.1...0.10.2
Published by peterschmidt85 over 1 year ago
The new update brings several significant improvements.
First and foremost, dstack
now enables running dev environments, workflows, and apps with Azure.
All you need to do is create the corresponding project via the UI and provide your Azure credentials.
For detailed instructions on setting up dstack
for Azure, refer to the documentation.
Secondly, you can now browse the logs and artifacts of any run through the user interface.
Last but not least, with the update, we have reworked the documentation to provide a greater emphasis on specific use cases.
Please note that when installing dstack
via pip
, you now need to specify the exact list of cloud providers you intend to use:
pip install "dstack[aws,gcp,azure]" -U
This requirement applies only when using the CLI to start the server locally. If you wish to connect to a server hosted elsewhere, you can still use the previous syntax:pip install dstack
.
If you have any feedback, including issues or questions, please share them in our Slack chat or file it as a GitHub issue.
Published by peterschmidt85 over 1 year ago
A couple of weeks ago, we introduced Hub, an extension of dstack
that makes it easier for teams to use dstack
. Because Hub not only helps teams, but also has other valuable features, such as a user interface, more secure management of cloud credentials, and better workflow scheduling (e.g., job queues), we decided to make Hub the default and only way to use dstack
.
Here's what specifically changed:
To use dstack
, you now always need Hub, even if you run workflows locally. You can start Hub by running the dstack start
command. On startup, this command creates the default project with the backend type Local
, which means that workflows run locally.
To run workflows in the cloud, you need to log in to Hub, create the corresponding project, and configure the CLI to use it via dstack config
.
The dstack config
command now allows you to configure Hub projects only. However, you can configure multiple projects and always have one as the default.
The --remote
flag in dstack run
is now dropped. Instead, you can use the --project NAME
argument to specify the project. The --project
argument is now supported by all commands, including dstack init
, dstack ps
, etc.
Why this matters:
dstack
, making it simpler to set up and configure.Note that this change is not backward compatible. In case of any issues, please write in Slack or raise GitHub issues.
To try the new experience, follow our updated Quick start guide.
While the dstack start
command is the most recommended way to run the Hub application locally, Docker is the recommended way to deploy Hub for remote access.
The updated Docker image is now available here, and you will also find the updated documentation there.
Note:
By default, the Hub application stores the state locally (via SQLite) in the ${HOME}/.dstack/hub folder.To help persist the state, the Hub application bundles Litestream, a tool that can automatically persist the state in the object store in your cloud account.
You'll find instructions on how to do it in the README section of the Docker image.
If you want to run Hub on Hugging Face, it's now very easy to do so.
All you have to do is to duplicate this template and configure your projects.
The admin token can be found in the logs.
Note:
Because Hugging Face doesn't persist data, feel free to configure Litestream settings via Hugging Face secrets. For example, you can set your space to automatically persist data in your AWS, GCP, or Azure object storage.
Previously, dstack
always required the folder where you run it to be a Git repo with a remote branch configured. Now, this requirement is dropped. Now you can run dstack
on any folder.
Last but not least, we've changed the behavior of dstack
when you press Ctrl+C
. If earlier it aborted the run immediately, now it stops it gracefully, allowing your workflow to stop, and save output artifacts.
To install the update, simply run the following command:
pip install dstack==0.8
If you have any feedback, including issues or questions, please share them in our Slack chat or file it as a GitHub issue.
Published by peterschmidt85 over 1 year ago
dstack 0.7
is a massive update that introduces a major new feature called Hub. Hub is a new extension of dstack
specifically designed for teams, with the purpose of providing a unified interface that enables teams to effectively manage their model development workflows on any cloud. To learn more about Hub, please read the Hub announcement in our blog.
Apart from introducing Hub, the update also adds the "Cache" feature which enables caching of files across runs of the same workflow. You can learn more about this feature in our documentation.
To install the latest version of dstack
, simply run the following command
pip install dstack --upgrade
If you have any feedback, including issues or questions, please share them in our Slack chat or file it as a GitHub issue.
Published by peterschmidt85 over 1 year ago
The release of dstack 0.2.6
addresses various bugs, extends the providers with the ssh
property, and introduces Playground. Please read on for additional information.
The bash
, code
, lab
, and notebook
providers now support the ssh
property. By setting it to true
, you can connect to the running workflow via SSH.
See the example below:
workflows:
- name: hello-ssh
provider: bash
commands:
- tail -f /dev/null
ssh: true
Run this workflow:
$ dstack run hello-ssh
RUN WORKFLOW SUBMITTED STATUS TAG BACKENDS
moody-emu-1 hello-ssh now Submitted local
Provisioning... It may take up to a minute. ✓
To interrupt, press Ctrl+C.
To connect via SSH, use:
ssh -i /Users/cheptsov/.ssh/id_rsa root@localhost -p 56947
This enables connection to the workflow through SSH using the command provided in the output. Similar to other provider properties, the ssh
feature can be used for both local and remote workflows.
Moreover, this update resolves a few issues related to dstack init
. We recommend updating, regardless of whether you encountered any problems.
Lastly, we have significantly revamped our examples and introduced a Playground that can be accessed in Gitpod. This will automatically install dstack
and prepare the environment for you.
The playground is the easiest way to run examples and learn how dstack
operates.
To install the latest version of dstack
, simply run the following command
pip install dstack --upgrade
If you have any feedback, including issues or questions, please share them in our Slack chat or file it as a GitHub issue.
Published by peterschmidt85 over 1 year ago
The latest update to dstack 0.2.2
improves the developer experience of remote workflows.
Previously, when running a remote workflow that used ports (for example, to host dev environments or applications), dstack
made those ports on the remote machine public, allowing anyone within the configured VPC to access them via HTTP.
This solution was insecure and posed a security risk. Additionally, dev environments or applications that used iframes, like VS Code rendering Jupyter notebooks inside the IDE, did not work due to security reasons.
However, the dstack 0.2.2
update resolves this issue by establishing an SSH tunnel between the local and remote machine and forwarding remote ports to your local machine. This solves both issues, as workflow ports are not exposed publicly, allowing only authorized users to access them, and dev environments and applications work seamlessly even if they use iframes.
To initialize your project using dstack
, run dstack init
. The dstack
backend will use your public SSH key (which you can override by using dstack init --ssh-identity-file PATH
). Then, when you run a remote workflow via dstack run
, it establishes an SSH tunnel to the remote machine and forwards ports to your local machine.
Even though the workflow is running remotely (e.g. in GCP), it prints the URL on which the application is available on your local machine. For example, if you run the code
provider, you can use the built-in Jupyter notebooks conveniently.
This new feature makes it easy to set up remote dev environments and run remote applications securely.
In the meantime, we are working on another SSH-related feature that will allow you to attach any IDE or terminal inside the running workflow conveniently via SSH.
To install the latest version of dstack
, simply run the following command
pip install dstack --upgrade
If you have any feedback, including issues or questions, please share them in our Slack chat or file it as a GitHub issue.