This is the code repository for Terraform Cookbook, Second Edition, published by Packt.
Provision, run, and scale cloud architecture with real-world examples using Terraform
The author of this book is -Mikael Krief
HashiCorp Configuration Language (HCL) has changed how we define and provision data center infrastructure with the launch of Terraform, a top-tier product for building Infrastructure as Code (IaC). Terraform Cookbook shows you how to leverage Terraform to manage complex infrastructure with ease.
This new edition has been updated to include real-world examples for provisioning Azure, AWS and GCP infrastructure with Terraform. You'll delve into manual and automated testing with Terraform configurations, creating and managing a balanced, efficient, and reusable infrastructure with Terraform modules. You'll learn how to automate the deployment of Terraform configuration with continuous integration and continuous delivery (CI/CD). Besides that, several new chapters have been added that describe the use of Terraform for Docker and Kubernetes, examine advanced topics on GitOps practices, and explain how to test Terraform configurations using different tools to check code and security compliance. The final chapter covers troubleshooting common Terraform issues and provides solutions for frequently encountered errors.
By the end of this book, you'll have developed the skills needed to get the most value out of Terraform and to effectively manage your infrastructure.
This new edition has been updated to include real-world examples for provisioning Azure, AWS and GCP infrastructure with Terraform. You'll delve into manual and automated testing with Terraform configurations, creating and managing a balanced, efficient, and reusable infrastructure with Terraform modules. You'll learn how to automate the deployment of Terraform configuration with continuous integration and continuous delivery (CI/CD). Besides that, several new chapters have been added that describe the use of Terraform for Docker and Kubernetes, examine advanced topics on GitOps practices, and explain how to test Terraform configurations using different tools to check code and security compliance. The final chapter covers troubleshooting common Terraform issues and provides solutions for frequently encountered errors.
This chapter delves into the diverse methods of manually installing Terraform, encompassing manual installations, script-based installations, and utilization within Docker containers. The chapter commences by advocating comprehension of Infrastructure as Code (IaC) principles for effective configuration composition. Subsequently, it elucidates the creation of a localized developmental environment enabling the crafting of Terraform configuration files and their subsequent application. The chapter progresses to offer pivotal insights into both IaC and Terraform best practices. It commences with elucidating the acquisition and manual installation of Terraform on Windows systems. It extends the discourse to encompass scripted installations for both Windows and Linux systems. Furthermore, the chapter elaborates on the utilization of Terraform within Docker containers, providing practical guidance for its integration. The chapter culminates with a focus on enhancing the Terraform ecosystem through the upgrading of Terraform providers, thereby enveloping a comprehensive array of practices pertinent to Terraform's establishment and utilization.
Key Insights:
The chapter delves into the intricate process of crafting effective Terraform configurations by addressing a range of essential components and techniques. The Terraform language's versatility becomes evident from the outset, enabling robust manipulation for various scenarios. Through practical recipes, readers are guided in utilizing Terraform's language to its fullest potential in real-world business applications. The recipes systematically cover diverse aspects. Initial attention is given to configuring Terraform and specifying provider versions. Techniques for enhancing flexibility are explored, such as employing aliases to utilize multiple instances of a single provider and leveraging variables and outputs for dynamic code. The integration of built-in functions and conditional expressions into configurations is examined, empowering users to craft more adaptive and intelligent setups. Furthermore, the chapter delves into resource interdependencies, imparting strategies to manage these relationships effectively. An in-depth exploration of custom pre- and postconditions offers insight into rigorous checks that can be applied to ensure robust provisioning, and readers are introduced to validation techniques for the provisioned infrastructure.
Key Insights:
This chapter delves into the remarkable advantages of Infrastructure as Code (IaC), notably its swift provisioning of extensive infrastructure in contrast to manual methods. Emphasizing the synergy between IaC and established software development principles, the chapter underscores the application of principles like Don't Repeat Yourself (DRY). This principle advocates against code duplication, urging developers to streamline their codebase. A thorough exploration of the DRY principle is provided through an external link, shedding light on its essence and practicality. The chapter transitions to the heart of its content by unveiling the efficacy of leveraging expressions from the Terraform language. Through techniques like count, maps, collections, arrays, and dynamic blocks, the chapter elucidates how these tools empower the creation of concise Terraform configurations. This, in turn, facilitates the deployment of diverse infrastructure across multiple environments, eradicating the need for redundant code and fostering efficiency. The chapter culminates by introducing a set of recipes that include provisioning infrastructure across varied environments, utilizing count for multiple resource provisioning, harnessing maps, iteratively navigating map objects, generating dynamic blocks, and implementing map filters, collectively providing a robust framework for scalable infrastructure deployment with Terraform.
Key Insights:
This chapter delves into leveraging Terraform configurations to address scenarios where information about pre-existing external resources or infrastructures is necessary. The focal point involves utilizing data sources to extract data from external systems and probing these resources via queries. This enhances Terraform's capabilities beyond its conventional provisioning scope. Additionally, the chapter elucidates how Terraform can engage with local operations, including managing local files and orchestrating the execution of locally installed programs. A comprehensive exploration of the Terraform Shell provider is also undertaken, exemplifying the execution of shell scripts through Terraform. The chapter encompasses a diverse array of topics, encapsulating data retrieval from external systems, local file manipulation, and versatile script execution. The chapter unfolds through an assortment of practical recipes, each serving to broaden the reader's toolkit. Starting with the acquisition of external data through data sources, the text advances into the realm of querying external data using Terraform's capabilities. Subsequently, it navigates the terrain of local operations, elucidating how Terraform can adeptly oversee local file manipulations and the execution of locally available programs. The culmination of the chapter is the introduction of the Terraform Shell provider, a mechanism that facilitates the execution of shell scripts orchestrated by Terraform. By encompassing a spectrum of techniques, from sourcing external data to orchestrating local and shell operations, this chapter equips readers with a well-rounded proficiency in expanding Terraform's purview beyond conventional provisioning.
Key Insights:
This chapter delves into the significance of the Terraform state file, a crucial artifact within the Terraform ecosystem. Serving as a repository for all executed configurations, this file underpins the entire Terraform workflow. While direct interaction with the state file is typically unnecessary, instances may arise where manipulation becomes imperative—whether for resource exploration, removal, import, or relocation. While one might be tempted to manually edit this JSON file, such an approach is ill-advised due to the potential introduction of errors that could disrupt future Terraform operations. To mitigate these risks, HashiCorp offers dedicated tools like the Terraform CLI and configuration blocks tailored to the Terraform system. The chapter comprehensively guides readers through secure methods of manipulating Terraform state. The Terraform CLI takes center stage in this process, enabling a spectrum of tasks including resource listing, deletion, synchronization, importing, and movement. Additionally, the text explores the integration of external resources from disparate state files, highlighting the versatility of the Terraform ecosystem. The chapter culminates in the presentation of practical scenarios, or "recipes," elucidating diverse aspects of Terraform state manipulation. These recipes span from leveraging local Terraform state instances to refining configuration through the Terraform move block, thereby equipping readers with a nuanced understanding of effective state management.
Key Insights:
The chapter delves into the practical utilization of the Terraform CLI to navigate the fundamental workflow of Terraform. It begins by emphasizing the significance of presenting a well-structured configuration and elucidates methods to validate its syntax. The chapter proceeds with a comprehensive exploration of pivotal Terraform commands and options. This includes elucidating the process of efficiently destroying infrastructure resources and revealing the providers employed within the configuration. Additionally, the utilization of workspaces as a means to manage diverse environments is expounded upon. The chapter further elucidates diverse techniques for enhanced interaction with Terraform, such as generating a single cross-compatible lock file for both Windows and Linux. It expounds upon strategies for copying Terraform modules and introduces the concept of "tainting" resources. The practical aspects of producing a dependency graph are laid out, along with insights into managing distinct Terraform configuration directories. Importantly, the chapter provides in-depth guidance on testing and evaluating Terraform expressions, facilitating users in optimizing their configurations. It culminates in a comprehensive guide to debugging Terraform execution, equipping readers with tools to address and resolve potential issues effectively.
Key Insights:
The chapter addresses the recurring challenge developers confront regarding effective and seamless code reuse in various programming languages. To tackle this, the software industry has witnessed the rise of reusable language constructs, frameworks, and software packages, like NuGet, NPM, Bower, PyPI, RubyGems, and more, that enable code sharing across applications and teams. The application of this principle extends to Infrastructure as Code (IaC), where similar concerns about code structure, standardization, and sharing within organizations arise.
Key Insights:
This chapter delves into the practical utilization of Terraform's versatile array of providers, specifically focusing on provisioning cloud infrastructure within the Azure environment. The preceding chapters of this book laid the foundation by exploring fundamental Terraform concepts and command-line intricacies, along with module-based configuration sharing. Though the configuration examples revolved around the "azurerm" provider, their applicability extends universally to all Terraform providers. Here, the chapter commences by elucidating the integration of Terraform with Azure Cloud Shell and addressing security concerns through robust authentication mechanisms and safeguarding the Terraform state file in an Azure storage account. The narrative progresses to intricate scenarios wherein Azure Resource Manager (ARM) templates and Azure CLI scripts are executed seamlessly through Terraform. Moreover, the chapter navigates through acquiring Azure resource listings with Terraform and adeptly shielding sensitive data within Azure Key Vault. Two pragmatic case studies further exemplify Terraform's prowess in Azure: one outlining the provisioning and configuration of an Infrastructure as a Service (IaaS) architecture comprising Virtual Machines (VMs), and another elucidating the deployment of a Platform as a Service (PaaS) infrastructure. The chapter culminates with advanced topics, encompassing the estimation of Azure resource expenses based on Terraform configurations, and the innovative application of the AzAPI provider for generating Terraform configurations from pre-existing infrastructures.
Key Insights:
In this chapter, the spotlight turns towards familiarizing readers with fundamental techniques for utilizing Terraform to provision infrastructure within the realms of both AWS and GCP. For each respective cloud provider, the chapter delves into critical aspects such as the establishment of Terraform credentials, configuring providers, and safeguarding the state file through remote backends like AWS S3 and GCP Storage. Particularly noteworthy is the comprehensive exploration of Google Cloud Shell, an integrated command-line interface, enabling users to execute Terraform operations devoid of local machine setup. The chapter concludes by presenting an array of instructive recipes that span across various topics, including initial steps with Terraform on AWS, integration of S3 backend functionality within AWS, initiation of Terraform usage on GCP, implementation of Google Cloud Storage (GCS) backend within the GCP context, and adept execution of Terraform via Google Cloud Shell.
Key Insights:
This chapter delves into the diverse capabilities of Terraform beyond its conventional usage for Infrastructure as Code on major cloud platforms like Azure, AWS, and GCP. While Terraform is commonly associated with cloud resource provisioning, it encompasses a wide array of providers that extend its automation potential to various domains, including non-cloud entities like files, Docker, and Kubernetes. The preceding chapter established a foundation for Terraform application in cloud contexts; however, this chapter transitions to an alternative domain: employing Terraform for orchestrating Docker and Kubernetes. This transition is driven by Terraform's straightforward approach, positioning it as a pivotal tool for unifying the automation of resource and component deployment, all under the umbrella of a consistent Infrastructure as Code language. Throughout this chapter, the focus shifts towards harnessing Terraform's capabilities to effectively manipulate Docker images and facilitate application deployment within Kubernetes environments. Key concepts encompass the utilization of the kubernetes and helm providers to facilitate Kubernetes resource management and Helm chart deployment, respectively. The chapter also introduces the integration of GitOps methodologies with Terraform, which is achieved through the employment of the Kubernetes Terraform controller. The chapter's scope is structured around a series of recipes that encompass pivotal topics such as the creation of Docker containers through Terraform, Kubernetes resource deployment via Terraform, Helm chart deployment within Kubernetes using Terraform, and the integration of a Kubernetes controller into Terraform workflows, functioning as a reconciliation loop. In essence, this chapter illuminates Terraform's versatility beyond conventional boundaries, positioning it as a pivotal tool for streamlined resource and application management within Docker and Kubernetes ecosystems.
Key Insights:
When crafting Terraform configurations, it is imperative to seamlessly integrate a testing phase within the workflow to ensure security protocols are adhered to, vulnerabilities are mitigated, and company standards are upheld. This necessitates the incorporation of one or more testing stages to validate the Terraform configuration. Just as in application development, various types of tests are vital for Terraform configurations, encompassing unit tests, plan analysis for compliance, contact tests, and integration tests. Unit tests, executed before the terraform apply command, validate syntax and variable consistency through commands like terraform validate or by analyzing the terraform plan output. Following this, compliance checks based on the plan outcome can be performed using tools like tflint, checkov, or tfsec, ensuring conformity with prescribed rules or custom frameworks. Additionally, integration tests are paramount for assessing accurate resource provisioning, particularly in Terraform modules. These tests simulate the configuration execution, evaluate connectivity and security aspects, and culminate in the resource destruction. This process can be automated through scripting languages or testing frameworks such as Terratest and Kitchen-Terraform. This chapter encompasses a two-part exploration. The initial segment delves into diverse methods and tools for Terraform configuration testing, involving the creation of custom rules and policies using PowerShell or Python. It encompasses a range of tests such as terraform-compliance, tfsec, and Open Policy Agent (OPA), targeting optimal validation of Terraform configurations. The second part concentrates on conducting integration tests for Terraform modules utilizing tools like Terratest and Kitchen-Terraform. Finally, the chapter introduces a novel integrated Terraform module integration test, permitting direct testing within the HCL Terraform configuration. Throughout this chapter, readers gain insights into distinct recipes, including leveraging PowerShell's Pester framework for Terraform testing, Python-based testing, OPA integration, tfsec compliance analysis, application of terraform-compliance, validation of Terraform module code using Terratest, scrutinizing Terraform configurations with Kitchen-Terraform, and harnessing the new integrated Terraform module integration test.
Key Insights:
The preceding sections of the book delved into foundational aspects of Terraform, encompassing its installation, configuration creation, CLI commands, and module-based configuration sharing, leading up to deploying infrastructure on major cloud providers. This chapter extends our Terraform proficiency even further. We explore advanced recipes that unveil nuanced capabilities of Terraform. Firstly, the chapter explores how Terraform's templates can be harnessed to generate Ansible inventory files. Techniques to safeguard resources from accidental destruction and how to accommodate zero-downtime deployments are then discussed, ensuring the stability of operational environments. We then delve into resource management, covering the use of Terragrunt for handling workspace dependencies and as a wrapper for the Terraform CLI. Also, the chapter showcases the generation of self-signed certificates and underscores the significance of Git hooks in pre-commit configuration checks. An innovative visualization tool named Rover is introduced to intuitively depict Terraform resource dependencies. The chapter culminates in spotlighting the Terraform CDK's prowess, demonstrating how to elevate Terraform configuration using high-level languages like TypeScript. Overall, this chapter takes us beyond the basics, enabling us to master advanced Terraform techniques and tools. It equips readers with an array of recipes that empower them to optimize infrastructure management, streamline workflows, and leverage Terraform's full potential.
Key Insights:
This chapter delves into the intricacies of automating Terraform operations. It underscores the significance of executing Terraform configurations automatically within the context of DevOps and automation. The conventional Terraform workflow, reliant on user intervention for apply and destroy commands, necessitates a paradigm shift to seamlessly integrate Terraform with CI/CD pipelines, facilitating unattended deployments. A prior instance of such automation was examined in a previous chapter where Terraform configurations were orchestrated within a GitOps framework, emphasizing the need for frictionless execution. This chapter advances this discourse by elaborating on the integration of Terraform execution in CI/CD pipelines, spotlighting two prominent DevOps cloud services: GitHub and Azure Pipelines. The emphasis remains on sidestepping user input and driving the entire process through automation. The chapter elucidates executing Terraform in an automated milieu and introduces strategies for invoking commands or scripts that present succint summaries of 'terraform plan' outcomes. Preparatory steps, found in recommended practices for transitioning from manual provisioning to collaborative Infrastructure as Code (IaC) workflows, are also advocated for better contextual understanding. The chapter's offerings extend to a spectrum of insights and practices, including automated execution in GitHub Actions, orchestrating Terraform modules in Azure Pipelines, and instituting workflows for publishing Terraform modules using GitHub Actions. In essence, this chapter is a comprehensive guide to harnessing the power of automation within Terraform workflows, aligning with modern DevOps principles and bolstering efficiency and reliability in infrastructure management.
Key Insights:
The chapter introduces Terraform Cloud as an essential solution for managing and collaborating on complex infrastructure projects within a corporate context. While the book has previously focused on smaller-scale projects, Terraform Cloud addresses the needs of larger endeavors by providing a centralized platform for automated and collaborative infrastructure provisioning. As part of the HashiCorp Cloud Platform, Terraform Cloud extends the functionality of Terraform, offering a suite of features beneficial to teams and organizations. These include collaboration tools, remote state management, version control integration, policy enforcement, workflow automation, secure provisioning, and monitoring capabilities. This chapter offers an in-depth exploration of Terraform Cloud's capabilities, demonstrating how to authenticate, create projects and workspaces, execute configurations, manage states, and utilize advanced features such as Open Policy Agent (OPA) for compliance checks and cost estimation. Through practical recipes, readers will grasp the significance of Terraform Cloud in facilitating efficient, secure, and scalable infrastructure management within large-scale projects.
Key Insights:
In the final chapter of the book, the focus shifts towards the practical aspects of debugging Terraform configurations. Throughout the preceding chapters, readers have gained a comprehensive understanding of using Terraform for provisioning infrastructure resources. However, the journey doesn't conclude without addressing the inevitable troubleshooting that arises when working with Terraform. This chapter delves into the meticulous process of identifying and rectifying common configuration errors. The provided link directs readers to HashiCorp's documentation, where they can access valuable insights on the troubleshooting workflow. The chapter elaborates on a series of typical issues including interpolation errors, address cycle challenges, for_each loop complications, and output errors. Each section follows a consistent structure—starting with an examination of the misconfiguration within the Terraform setup, followed by a systematic guide to reproducing the error, and culminating in a step-by-step correction procedure. It is important to note that while this chapter comprehensively tackles an array of frequently encountered problems, it does not encompass all possible Terraform errors. Notably, the chapter acknowledges the existence of other error types such as issues with Terraform state locking or corruption, along with resource duplication errors. These errors, including their remedies, have been explored in prior chapters, enriching the reader's understanding of Terraform's intricacies.
Key Insights:
If you feel this book is for you, get your copy today!
With the following software and hardware list you can run all code files present in the book.
Software required | Link to the software | Hardware specifications | OS required |
---|---|---|---|
Terraform Cli , version ≥1.5 | https://developer.hashicorp.com/terraform/downloads | Should work on any recent computer | Windows, MacOS, Linux |
Terraform Cloud | https://app.terraform.io/app/ | Should work on any recent computer | NA (Any Browser) |
Azure | https://azure.microsoft.com/en-in | Should work on any recent computer | NA (Any Browser) |
Python version ≥ 3.11 | https://www.python.org/downloads/ | Should work on any recent computer | Windows, MacOS, Linux |
PowerShell scripting | https://learn.microsoft.com/en-us/powershell/scripting/install/installing-powershell-on-windows?view=powershell-7.3 | Should work on any recent computer | Windows, MacOS, Linux |
Shell scripting | - | Should work on any recent computer | Linux / WSL / MacOS |
Golang Version ≥1.20 | https://go.dev/doc/install | Should work on any recent computer | Windows, MacOS, Linux |
Azure CLI | https://learn.microsoft.com/en-us/cli/azure/install-azure-cli | Should work on any recent computer | Windows, MacOS, Linux |
Azure DevOps | https://azure.microsoft.com/en-in/products/devops | Should work on any recent computer | NA (Any Brwser) |
GitHub | []https://desktop.github.com/](https://desktop.github.com/) | Should work on any recent computer | NA (Any Brwser) |
Git | https://git-scm.com/downloads | Should work on any recent computer | Windows, MacOS, Linux |
Ruby version ≥ 3.0.0 | https://www.ruby-lang.org/en/downloads/ | Should work on any recent computer | Windows, MacOS, Linux |
Docker | https://www.docker.com/ | Should work on any recent computer | Windows, MacOS, Linux |
Terragrunt | https://terragrunt.gruntwork.io/docs/getting-started/install/ | Should work on any recent computer | Windows, MacOS, Linux |
Jq | https://jqlang.github.io/jq/download/ | Should work on any recent computer | Windows, MacOS, Linux |
Infracost | https://www.infracost.io/ | Should work on any recent computer | Windows, MacOS, Linux |
kubectl / Helm | https://kubernetes.io/docs/tasks/tools/ | Should work on any recent computer | Windows, MacOS, Linux |
Node.js | https://nodejs.org/en/download | Should work on any recent computer | Windows, MacOS, Linux |
You can get more engaged on the discord server for more latest updates and discussions in the community at Discord
If you have already purchased a print or Kindle version of this book, you can get a DRM-free PDF version at no cost. Simply click on the link to claim your free PDF. Free-Ebook
We also provide a PDF file that has color images of the screenshots/diagrams used in this book at GraphicBundle
Mikael Krief is a DevOps engineer who lives in France. He believes that Infrastructure as Code is a fundamental practice in DevOps culture. He is therefore interested in HashiCorp products and specializes in the use of Terraform. Mikael loves to share his passion through various communities, such as the HashiCorp User Groups. Over the years, he has contributed to many public projects, written various blog posts, published several books, and spoken at leading conferences. For his contributions and passion, he has been nominated and selected as a HashiCorp Ambassador since 2019, and he has been awarded the Microsoft Most Valuable Professional (MVP) award for 8 years.