The dstack 0.2.1 release introduces several significant improvements:

Private Docker registry

The docker provider now supports passing credentials to use Docker images from private registries.

Here's an example:

workflows:
  - name: private-registry
    provider: docker
    image: ghcr.io/my-organization/top-secret-image:v1
    registry_auth:
      username: ${{ secrets.GHCR_USER }}
      password: ${{ secrets.GHCR_TOKEN }}

Note
To avoid storing credentials in plain text, it is recommended to add them as secrets.

Custom Docker entrypoint

Previously, the docker provider always used bash as the entrypoint for running provided commands. With this update, the default shell has been changed to sh, which is more commonly used. Also, it is now possible to override the entrypoint by using the entrypoint property in the YAML file.

Background processes

The bash and docker providers can now run background processes. This can be achieved in the same way as in a regular shell, by adding & to the end of the command.

Here's an example:

workflows:
  - name: train-with-tensorboard
    provider: bash
    ports: 1
    commands:
      - pip install torchvision pytorch-lightning tensorboard
      - tensorboard --port $PORT_0 --host 0.0.0.0 --logdir lightning_logs &
      - python train.py
    artifacts:
      - path: lightning_logs

In this example, the tensorboard application runs in the background, allowing for browsing of training script logs while the script is still in progress.

Try it now

To install the latest version of dstack, simply run the following command

pip install dstack --upgrade

Should you have any feedback, whether it be regarding an issue with the functionality or a question, please share it in our Slack chat or report it as a GitHub issue.

With the release of version 0.2 of dstack, configuring a GCP as a remote is now possible. All the features that were previously available for AWS except real-time artifacts are now available for GCP as well.

To use GCP with dstack, you will require a service account.

For more details on how to configure GCP, refer to the documentation.

Once you have created a service account, proceed to execute the dstack config command. After that, you're good to go! Use the --remote flag with the dstack run command to execute workflows in GCP, and dstack will automatically create and destroy cloud instances based on resource requirements, including cost strategies like using spot instances.

How to install the update

To install the update, run the following command:

pip install dstack --upgrade

Join our Slack

Have a question, need help, or just want to stay updated? Make sure to join our Slack chat.

The dstack 0.1.2 release brings a fix to the following bug:

• Secrets are not passed to local workflow #220

To install the update, run the following command

pip install dstack --upgrade

Should you have any feedback, whether it be regarding an issue with the functionality or a question, please share it in our Slack chat or report it as a GitHub issue.

The dstack 0.1.1 release brings a number of important bug fixes:

• Run failed if the user is using an RSA key for SSH #197
• Run logs arrive corrupted #185
• The dstack ls command fails on macOS and Windows #189
• Allow creating local tags by uploading local files via dstack tags add #173

Try it now

To install the latest version of dstack, simply run the following command

pip install dstack --upgrade

Should you have any feedback, whether it be regarding an issue with the functionality or a question, please share it in our Slack chat or report it as a GitHub issue.

Local and remote backends

Previously, dstack required a cloud account for use, even when running workflows locally, as the artifacts would automatically be pushed to cloud storage. The new update resolves this by introducing two distinct backends: local and remote.

By default, all workflows now run locally. If you prefer to run a workflow in the cloud, simply add the --remote flag:

dstack run train-mnist --remote

RUN        WORKFLOW    SUBMITTED  STATUS      TAG BACKENDS
chicken-0  mnist-data  now         Submitted      aws

Provisioning... It may take up to a minute. ✓

To interrupt, press Ctrl+C.

When a workflow is run locally, the resulting artifacts are also stored locally.

A cloud account is no longer mandatory to use dstack locally.

Pushing and pulling artifacts

Reusing output artifacts across workflows is a core feature of dstack.

The latest update allows for local workflows to reuse the output artifacts of other local workflows, and for remote workflows to reuse the output artifacts of other remote workflows.

To reuse the output artifacts of a remote workflow locally, use the dstack pull command to retrieve the remote run locally.

dstack pull --help     
usage: dstack pull [-h] (RUN | :TAG)

positional arguments:
  (RUN | :TAG)  A name of a run or a tag

To reuse the output artifacts of local workflows in the cloud, use the dstack push command to upload the local run to the configured cloud.

Other changes

Below is a list of other notable changes:

• The dstack artifacts list command was renamed to dstack ls
• The dstack artifacts download command was discontinued and replaced with dstack pull
• The before_run property in YAML was renamed to setup

The changes have been incorporated into the updated documentation and examples.

Try it now

To install the latest version of dstack, simply run the following command

pip install dstack --upgrade

Should you have any feedback, whether it be regarding an issue with the functionality or a question, please share it in our Slack chat or report it as a GitHub issue.

Meanwhile, stay tuned, everyone! We're hard at work on some really cool things that we'll be sharing with you soon.

dstack 0.1 RC1 is a preview of the new upcoming major release.

In this release, we've entirely reworked how workflows run locally and remotely.

TL;DR:

1. You don't need AWS to use dstack anymore. By default, all workflows run locally. To run in AWS, you need to tell it explicitly.
2. Running workflows locally is a lot easier now. The output artifacts of local workflows stay locally (unless you'll decide to push them to AWS explicitly).
3. Local and remote runs do not interfere anymore.

Here's a more detailed changelog:

• By default, now all workflows run locally. The flag --local in the dstack run CLI command (as well as the property local: true in the YAML) is dropped.
• dstack config is not required anymore. If you want to try dstack, and for example run workflows only locally, you can use dstack without AWS at all.
• If you configured AWS via dstack config and want to run a workflow there, you need to use the --remote flag in the dstack run CLI command (or the property remote: true in the YAML).
• If you run a workflow locally, its output artifacts stay locally and are not uploaded to AWS. If you want to push the output artifacts of a local run to AWS (e.g. to reuse it from another workflow that you run in AWS), you can use the dstack push command (followed by the name of the run).
• If you run a workflow remotely (in AWS) but would like to reuse its output artifacts locally, you would need to use the dstack pull command (followed by the name of the run). This command will download the artifacts of a remote run to local storage (~/.dstack/artifacts).
• The command dstack artifacts list was renamed to dstack ls.
• The command dstack artifacts download was replaced with dstack pull
• The before_run property in YAML was renamed to setup

While these changes may theoretically sound not as major, they actually are. They make local and remote runs a lot more convenient from the user experience point of view, and also prepare dstack for the upcoming even more significant changes (which we’ll write about pretty soon).

How to install the preview:

pip install dstack --upgrade --pre

Note
The documentation has not been updated yet and doesn't reflect the listed changes. The docs will be updated right after the release will go public (within less than a week).

dstack 0.0.14 is a minor update that brings several improvements.

Features

• Allow defining workflows in separate YAML files inside the .dstack/workflows directory #133
• Allow parametrizing workflows and passing workflow arguments via dstack run #132
• Use more granular AWS cost allocation tags for AWS billing reports #113

Bugfixes

• Pre-select the AWS region configured for the selected AWS profile in dstack config #128
• Fixed the problem of not-working secrets (the issue was introduced in 0.0.13)

Examples

Last but not least, thanks to #133, we've done a massive refactoring of the dstack-examples repository.

• We've added there the source code of the Examples.
• The source code of the Quickstart was refactored: workflows were moved to a separate YAML file, and the source files were moved to a separate folder.
• The documentation was updated to reflect the changes above.

dstack 0.0.13 is a minor update that fixes two important bugs:

• Allow workflow containers to execute files from artifacts #124
• Artifacts are uploaded incorrectly on Windows using dstack tags create -a PATH #121

The dstack 0.0.12 update is here. Inside, you'll find several major new features and a couple of important bug-fixes.

GitLab, GitHub Enterprise, and other Git repos

Now you can use dstack with GitLab, GitHub Enterprise, and any other Git repos (regardless of where they are hosted).

Just make sure the hostname of the repo is accessible from AWS (be it the default VPN or any other selected one).

Built-in Conda-Forge and Miniforge support

All built-in providers now come with Miniforge pre-installed.

Now you can use all the benefits Conda provides without worrying about the license.

Moreover, you can use conda env create and conda activate within your workflows to create custom Conda environments, save them as artifacts, and reuse them from other workflows.

Here's an example of two workflows, where one creates an environment, and another one reuses it:

workflows:
  - name: setup-conda
    help: Prepares an environment
    provider: bash
    commands:
      - conda env create --file environment.yml
    artifacts:
      - path: /opt/conda/envs/myenv

  - name: use-conda
    provider: bash
    deps:
      - workflow: setup-conda
    commands:
      - conda activate myenv

If your environment is sophisticated and has a lot of dependencies, downloading it as an artifact will be a lot faster than preparing in inside the workflow.

Colourful and interactive output

The workflow output is now fully interactive and colorful.

Multiple users

The dstack ps command, for every run, now shows who submitted it.

dstack ps -a
 RUN                 WORKFLOW  SUBMITTED   OWNER           STATUS   TAG
 cuddly-dragonfly-1  code      6 days ago  olgenn          Aborted
 terrible-bear-1     bash      6 days ago  peterschmidt85  Stopped
 mighty-chicken-1    download  yesterday   ivan-bokov      Done     mnist_data
 bitter-zebra-1      train     yesterday   peterschmidt85  Aborted

This feature will be helpful if multiple users are working on the same repo together.

Bugfixes

The full changelog, including bugfixes, can be found here.

Feedback

In case you have feedback, if either something doesn't work as expected, or you have a question, please share it in our Slack chat, or report as a GitHub issue.

The dstack 0.0.11 update brings support for local workflows.

Run locally

If you want, now you can run workflows on your local machine instead of the cloud. This is helpful if you want to quickly test something locally before spinning resources in the cloud.

Here's an example of how to define such a workflow:

workflows:
  - name: hello
    provider: bash
    commands:
      - echo "Hello world"
    resources:
      local: true

Alternatively, you can use the -l (or --local) argument with the dstack run command:

dstack run hello -l

This will run the hello workflow locally.

Note
Running workflows locally requires Docker or NVIDIA Docker to be installed locally.

Last but not least, the update brings updates to the documentation. For example, we've added the list of AWS permissions that the dstack requires.

dstack 0.0.9 is a minor update that brings the following improvements to the dstack config command:

• It allows interactively choosing an AWS profile (from the list of locally configured profiles).
• It allows interactively choosing an AWS region (from the list of supported regions).
• It allows interactively choosing an EC2 subnet. (from the list of available subnets in the chosen region).

Besides that, we've added support for the Singapore region (ap-southeast-1).

Finally, the source code has the runner's code.

dstack 0.0.8 is a bugfix update that brings the following improvements:

• Support for interactive outputs in dstack run (e.g. tqdm and alike)
• Now it's possible to create a tag right with the dstack run command (via -t TAG argument)
• Fixed the issue causing artifact files (from deps) to be downloaded in a broken state (#99)
• Significantly improved performance of the dstack ps command

dstack 0.0.7 is the first public beta release of the dstack CLI.