Examples of using Terraform to deploy Databricks resources
OTHER License
This repository contains the following:
There are two ways to use this repository:
examples
folder for individual examples.modules
folder.Code in the repository is organized into the following folders:
modules
- implementation of specific Terraform modules.examples
- specific instances that use Terraform modules.cicd-pipelines
- Detailed examples of implementing CI/CD pipelines to automate your Terraform deployments using Azure DevOps or GitHub Actions.Note For detailed information about the examples, modules, or CI/CD pipelines, refer to
README.md
file inside the corresponding folder for a detailed guide on setting up the CI/CD pipeline.
The folder examples
contains the following Terraform implementation examples :
Cloud | Example | Description |
---|---|---|
Azure | adb-lakehouse | Lakehouse terraform blueprints |
Azure | adb-with-private-link-standard | Provisioning Databricks on Azure with Private Link - Standard deployment |
Azure | adb-vnet-injection | A basic example of VNet injected Azure Databricks workspace |
Azure | adb-exfiltration-protection | A sample implementation of Data Exfiltration Protection |
Azure | adb-external-hive-metastore | Example template to implement external hive metastore |
Azure | adb-kafka | ADB - single node kafka template |
Azure | adb-private-links | Azure Databricks Private Links |
Azure | adb-squid-proxy | ADB clusters with HTTP proxy |
Azure | adb-teradata | ADB with single VM Teradata integration |
Azure | adb-uc | ADB Unity Catalog Process |
Azure | adb-unity-catalog-basic-demo | ADB Unity Catalog end-to-end demo including UC metastore setup, Users/groups sync from AAD to databricks account, UC Catalog, External locations, Schemas, & Access Grants |
Azure | adb-overwatch | Overwatch multi-workspace deployment on Azure |
AWS | aws-workspace-basic | Provisioning AWS Databricks E2 |
AWS | aws-workspace-with-firewall | Provisioning AWS Databricks E2 with an AWS Firewall |
AWS | aws-exfiltration-protection | An implementation of Data Exfiltration Protection on AWS |
AWS | aws-workspace-with-private-link | Coming soon |
AWS | aws-databricks-flat | AWS Databricks simple example |
AWS | aws-databricks-modular-privatelink | Deploy multiple AWS Databricks workspaces |
AWS | aws-workspace-uc-simple | Provisioning AWS Databricks E2 with Unity Catalog in a single apply |
AWS | aws-databricks-uc | AWS UC |
AWS | aws-databricks-uc-bootstrap | AWS UC |
AWS | aws-remote-backend-infra | Simple example on remote backend |
AWS | aws-workspace-config | Configure workspace objects |
GCP | gcp-sa-provisionning | Provisionning of the identity with the permissions required to deploy on GCP. |
GCP | gcp-basic | Workspace Deployment with managed vpc |
GCP | gcp-byovpc | Workspace Deployment with customer-managed vpc |
The folder modules
contains the following Terraform modules :
Cloud | Module | Description |
---|---|---|
All | databricks-department-clusters | Terraform module that creates Databricks resources for a team |
Azure | adb-lakehouse | Lakehouse terraform blueprints |
Azure | adb-lakehouse-uc | Provisioning Unity Catalog resources and accounts principals |
Azure | adb-with-private-link-standard | Provisioning Databricks on Azure with Private Link - Standard deployment |
Azure | adb-exfiltration-protection | A sample implementation of Data Exfiltration Protection |
Azure | adb-with-private-links-exfiltration-protection | Provisioning Databricks on Azure with Private Link and Data Exfiltration Protection |
Azure | adb-overwatch-regional-config | Overwatch regional configuration on Azure |
Azure | adb-overwatch-mws-config | Overwatch multi-workspace deployment on Azure |
Azure | adb-overwatch-main-ws | Main Overwatch workspace deployment |
Azure | adb-overwatch-ws-to-monitor | Overwatch deployment on the Azure workspace to monitor |
Azure | adb-overwatch-analysis | Overwatch analysis notebooks deployment on Azure |
AWS | aws-workspace-basic | Provisioning AWS Databricks E2 |
AWS | aws-databricks-base-infra | Provisioning AWS Infrastructure to be used for the deployment of a Databricks E2 workspace |
AWS | aws-databricks-unity-catalog | Provisioning the AWS Infrastructure and setting up the metastore for Databricks Unity Catalog |
AWS | aws-databricks-workspace | Provisioning AWS Databricks E2 Workspace using pre-created AWS Infra |
AWS | aws-workspace-with-firewall | Provisioning AWS Databricks E2 with an AWS Firewall |
AWS | aws-exfiltration-protection | An implementation of Data Exfiltration Protection on AWS |
AWS | aws-workspace-with-private-link | Coming soon |
GCP | gcp-sa-provisionning | Provisions the identity (SA) with the correct permissions |
GCP | gcp-workspace-basic | Provisions a workspace with managed VPC |
GCP | gcp-workspace-byovpc | Workspace with customer-managed VPC. |
The cicd-pipelines
folder contains the following implementation examples of pipeline:
Tool | CI/CD Pipeline |
---|---|
GitHub Actions | manual-approve-with-github-actions |
Azure DevOps | manual-approve-with-azure-devops |
When contributing the new code, please follow the structure described in the Repository content section:
modules
directory to be easily included when it's published to the Terraform registry. Prefer to implement the modular design consisting of multiple smaller modules implementing a specific functionality vs. one big module that does everything. For example, a separate module for Unity Catalog objects could be used across all clouds, so we won't need to reimplement the same functionality in cloud-specific modules.examples
directory - it should show how to use the given module(s).