terraform-provider-utils

🚀 Enhancements

what

• Bump atmos to 1.26.1

why

• Update Go templates in stack configs
• If a context is not provided (as if using import with context), don't process Go templates in the imported configurations (templating can be used for Datadog, ArgoCD etc. w/o requiring Atmos to process them)

references

• https://github.com/cloudposse/atmos/releases/tag/v1.26.1

what

• Bump atmos to 1.26.0
• Update all examples and tests that atmos uses

why

• Use the new feature implemented in the latest atmos versions, in particular

  • Add imports schema with context
  • Allow using Go templates in imported configurations
  • Auto-generate Atmos components from templates

references

• https://github.com/cloudposse/atmos/releases/tag/v1.25.0

what

• Bump atmos to 1.15.0
• Update atmos.yaml
• Update example and tests

why

• Update atmos "deps" calculation. In some cases, when more than one different YAML stack config files imported the same import (b/c some components in both derived from the same base component), then all of those stack config files were added to the component's "deps" list (resulting in Spacelift stacks having unrelated dependency labels which would cause unnecessary stack triggering)
• Keep up to date
• Keep the examples and tests the same as in atmos repo

references

• https://github.com/cloudposse/atmos/releases/tag/v1.15.0

what

• Bump GitHub actions
• Bump atmos version

why

• Remove all global vars from Go code - this fixes remote state for Terraform utils provider

  • Terraform executes a provider data source in a separate process and calls it using RPC
  • But this separate process is only one per provider, so if we call the code the get the remote state of two different components, the same process will be called
  • In the previous implementation, we used the global Go variable Config which was used by all functions to get the CLI configuration to use it in calculations of component and stack paths
  • When we tried to get the remote state of two different components from two diff repos with two diff atmos.yaml CLI config files, Terraform executed the two calls concurrently. The first call processed atmos.yaml and updated the Config global var with the component and stack paths. Then the second call also processed a different atmos.yaml (from another repo) and overrode the global Config var with values related to the second repo
  • One of the calls randomly failed with the error "Searched all paths, but could not find the component xxx in the stack yyy"
  • Removing all globals from the Go code (including the Config var) allows executing any part of the code concurrently without storing any state in global vars. The CLI config is now processed on all calls to the atmos code and is used as parameter to all Go functions that require it

references

• https://github.com/cloudposse/atmos/pull/210

🚀 Enhancements

what

• Fix atmos CLI config processing
• Improve logs.verbose

why

• Fix issues with CLI config processing introduced in https://github.com/cloudposse/atmos/pull/210
• In Go, a struct is passed by value to a function (the whole struct is copied), so if the function modifies any struct fields, the changes are not visible from outside of the functions
• The function processEnvVars was accepting the CLI config struct by value, checking the ENV var ATMOS_BASE_PATH and modifying the BasePath field - this modification was lost when the function returned resulting in the error from the utils provider "failed to find a match for the import ..." because the atmos base path was not set correctly (note that ATMOS_BASE_PATH ENV var is critical for the utils provider to work, and it's set by geodesic or by Spacelift scripts)
• Changing the function to accept a pointer to the struct fixed the issue (the modified fields are visible to the calling code)

references

• https://github.com/cloudposse/atmos/pull/211

what && why

• Re-releasing 1.3.0 as 1.4.1 while fixing some remote state issues in 1.4.0

what

• Bump GitHub actions
• Bump atmos version

why

• Remove all global vars from Go code - this fixes remote state for Terraform utils provider

  • Terraform executes a provider data source in a separate process and calls it using RPC
  • But this separate process is only one per provider, so if we call the code the get the remote state of two different components, the same process will be called
  • In the previous implementation, we used the global Go variable Config which was used by all functions to get the CLI configuration to use it in calculations of component and stack paths
  • When we tried to get the remote state of two different components from two diff repos with two diff atmos.yaml CLI config files, Terraform executed the two calls concurrently. The first call processed atmos.yaml and updated the Config global var with the component and stack paths. Then the second call also processed a different atmos.yaml (from another repo) and overrode the global Config var with values related to the second repo
  • One of the calls randomly failed with the error "Searched all paths, but could not find the component xxx in the stack yyy"
  • Removing all globals from the Go code (including the Config var) allows executing any part of the code concurrently without storing any state in global vars. The CLI config is now processed on all calls to the atmos code and is used as parameter to all Go functions that require it

references

• https://github.com/cloudposse/atmos/pull/210

what

• Update utils_component_config data source - add atmos_cli_config_path and atmos_base_path inputs
• Bump atmos version
• Update tests

why

• The atmos component processor's code is used by the provider to get the component's remote state
• We already supported the ATMOS_CLI_CONFIG_PATH and ATMOS_BASE_PATH ENV vars to specify the CLI config file (atmos.yaml) path and atmos base path to be used to get a remote state of a component from a remote repo, e.g.

module "other_repo" {
  source = "git::ssh://[email protected]/xxxx/other-repo.git"
}

locals {
  other_repo_local_path = "${path.module}/.terraform/modules/other_repo"

  env = {
    ATMOS_BASE_PATH       = local.other_repo_local_path
    ATMOS_CLI_CONFIG_PATH = "${local.other_repo_local_path}/rootfs/usr/local/etc/atmos"
  }
}

module "account_map" {
  source  = "cloudposse/stack-config/yaml//modules/remote-state"
  version = "1.0.0"

  component   = "account-map"
  env         = local.env

  context = module.always.context
}

• The problems with using the ENV vars are as follows:

  • Terraform executes a provider code in a separate process and calls it using RPC
  • But this separate process is only one per provider, so if we call the code the get the remote state of two different components from two diff repos, the same process will be called
  • When we specify the ENV vars ATMOS_BASE_PATH and ATMOS_CLI_CONFIG_PATH, the provider process gets the ENV vars set in the process space
  • Then, if we call the provider a second time from the same terraform component (e.g. to get a remote state of another component from a different repo), the initially set ENV vars ATMOS_BASE_PATH and ATMOS_CLI_CONFIG_PATH are still set in the provider process space, which prevents the provider from finding the atmos.yaml CLI config related to the current repo (since the ENV vars still point to the other/remote repo config), which in turn causes an error when searching for the component in the stack
  • Even if we unset the ENV vars in the second call to the provider, it does not help since terraform executes data sources in parallel, so one of them will get the ENV vars set, and the other call will fail during the time window when the ENV vars are still set in the same process

• We need to be able to specify atmos base path and atmos CLI config path in the utils provider w/o using ENV vars - the component processor code now supports additional parameters to specify it (and they override all other paths set by the ENV vars)

references

• https://github.com/cloudposse/atmos/pull/209

what

• Bump atmos to the latest version
• Update tests
• Use namespace context variable in the code that is used to return remote-state for a component in a stack

why

• For stacks config using multiple Orgs, we use namespace in stack names, and need to be able to find the remote state of the components provisioned in these stack

references

• https://github.com/cloudposse/atmos/pull/205

what

• Bump atmos to the latest version

why

• Use many fixes and improvements in the atmos code

what

• Update atmos to 1.4.26 and Go to 1.19
• Update tests

why

• Keep up to date
• When searching for the specified component in the specified stack (e.g. atmos describe component <component> -s <stack>), if any of the stack config files throws error (which also means that we can't find the component in that stack), print the error to the console and continue searching for the component in the other stack config files. This will allow having partial stack configs even if a partial stack config file does not provide all the required context variables (namespace, tenant, environment, stage) for all components in it (but specifies the context for some components, which we are interested in)
• Print the above misconfiguration errors to the console in logs verbose mode (export ATMOS_LOGS_VERBOSE=true)
• An example of partial stacks definition: examples/complete/stacks/orgs/cp/tenant1/dev/us-east-2-extras.yaml defines a new component new-vpc in the stack tenent1-ue2-dev; note that the same stack tenent1-ue2-dev is defined in examples/complete/stacks/orgs/cp/tenant1/dev/us-east-2.yaml but for different components

references

• https://github.com/cloudposse/atmos/releases/tag/v1.4.26

🚀 Enhancements

what

• Bump atmos version
• Update tests

why

• When a component defines inherited YAML base components in metadata.inherits and the stack imported the YAML file(s) where the inherited YAML components are defined, add the imported YAML files to the deps labels
• Before this fix, only the base terraform component was used in the deps calculation. All imported YAML files where the base YAML components were defined were not included in the deps labels (and the YAML files were missed from the Spacelift dependencies)

For example:

components:
  terraform:
    "test/test-component-override-3":
       metadata:
        component: "test/test-component"
        inherits:
          - "test/test-component-override"
          - "test/test-component-override-2"
          - "mixin/test-1"
          - "mixin/test-2"

In the above config, test/test-component-override-3" inherits test/test-component-override-2 YAML component.
Part of the config for test/test-component-override-2 YAML component is defined in service-1-override-2.yaml file:

components:
  terraform:
    "test/test-component-override-2":
      vars:
        service_1_name: "service-1-override-2"

the service-1-override-2.yaml file was not included in the deps labels because the code only checked for the base terraform component test/test-component in all imports.

After this fix, the code checks all imported files for the base terraform component and ALL base YAML components.

references

• https://github.com/cloudposse/atmos/pull/177

🚀 Enhancements

what

• Update Go module versions
• Refactor stacks to the latest pattern
• Refactor tests
• Update atmos.yaml

why

• Use latest Go modules
• Refactor stacks in the examples to use orgs->tenants->accounts->regions + catalog and mixins - this is our latest pattern of stacks configuration
• Refactor tests to reflect the new structure of the stacks folder
• Use the latest versions of atmos.yaml

references

• https://github.com/cloudposse/atmos/pull/175

🚀 Enhancements

what

• Update Golang to 1.18
• Update atmos to 1.4.22
• Add ENV vars to data sources

why

• Use the latest Golang version, it offers many new features (including the any alias instead of interface{})
• Update to the latest version of atmos to use the latest features and improvements
• Add ENV vars to data sources - this will allow controlling the execution of atmos, utils provider and terraform using externally set ENV vars

locals {
  component   = "test/test-component-override"
  stack       = "tenant1-ue2-dev"
  tenant      = "tenant1"
  environment = "ue2"
  stage       = "dev"

  env = {
    ENVIRONMENT           = local.environment
    STAGE                 = local.stage
    ATMOS_CLI_CONFIG_PATH = "."
  }
}

data "utils_component_config" "example1" {
  component     = local.component
  stack         = local.stack
  ignore_errors = false
  env           = local.env
}

• In particular, the latest atmos version supports ATMOS_CLI_CONFIG_PATH ENV var to set the path to atmos.yaml CLI config file. This ENV var will allow using the monorepo pattern by loading a remote repo and pointing to its atmos.yaml using ATMOS_CLI_CONFIG_PATH ENV var

module "monorepo" {
  source = "git::ssh://[email protected]/ACME/infrastructure.git?ref=v0.0.1"
}

locals {
  monorepo_local_path     = "${path.module}/.terraform/modules/monorepo"
  stack_config_local_path = "${local.monorepo_local_path}/stacks"
}

module "iam_roles" {
  source  = "git::ssh://[email protected]/ACME/infrastructure.git//components/terraform/account-map/modules/iam-roles?ref=v0.0.1"
  env = {
     ATMOS_CLI_CONFIG_PATH = "${path.module}/.terraform/modules/monorepo/rootfs/usr/local/etc/atmos"
  } 
}

references

• https://github.com/cloudposse/atmos/pull/168

🚀 Enhancements

what

• Update atmos 1.4.20 and tests

why

• Use the code from the latest atmos version
• Use the latest stack/component config and tests (Note that we copy the examples and all tests from atmos to the utils provider to be able to run the same tests as atmos)

references

• https://github.com/cloudposse/atmos/releases/tag/v1.4.20
• Closes https://github.com/cloudposse/terraform-provider-utils/pull/162

🚀 Enhancements

what

• Update atmos and tests

why

• Use the code from the latest atmos version
• Use the latest stack/component config and tests (Note that we copy the examples and all tests from atmos to the utils provider to be able to run the same tests as atmos)

references

• https://github.com/cloudposse/atmos/pull/149

🚀 Enhancements

what

• Update atmos and tests
• Support dashes - in the tenant, environment and stage names

why

• The old atmos supported dashes in the names (because it was filename-based, not logical stack name based)
• Some clients want to name tenants/environments/stages with dashes in the names (and some already have it, so we need to support that when converting from the old to the new atmos)

references

• https://github.com/cloudposse/atmos/pull/143

🚀 Enhancements

what

• Add utils_aws_eks_update_kubeconfig datasource

why

• Download kubeconfig from EKS clusters and save it to a file using Terraform, then use the downloaded kubeconfig in the Terraform helmfile provider to provision helm charts with helmfiles

The data source can executes 'aws eks update-kubeconfig' commands in four different ways:

1. If all the required parameters (cluster name and AWS profile/role) are provided,
   then it executes the command without requiring the CLI config ('atmos.yaml') and component/stack/context.
   'atmos.yaml' is not required/needed in this case.

2. If 'component' and 'stack' are provided,
   then it executes the command using the atmos CLI config (see atmos.yaml) and the context by searching for the following settings:
     - 'components.helmfile.cluster_name_pattern' in 'atmos.yaml' CLI config (and calculates the '--name' parameter using the pattern)
     - 'components.helmfile.helm_aws_profile_pattern' in 'atmos.yaml' CLI config (and calculates the '--profile' parameter using the pattern)
     - 'components.helmfile.kubeconfig_path' in 'atmos.yaml' CLI config
     - the variables for the component in the provided stack
     - 'region' from the variables for the component in the stack

3. If the context ('tenant', 'environment', 'stage') and 'component' are provided,
   then it builds the stack name by using the 'stacks.name_pattern' CLI config from 'atmos.yaml', then performs the same steps as example #2.

4. Combination of the above. Provide a component and a stack (or context), and override other parameters (e.g. 'kubeconfig', 'region').

If 'kubeconfig' (the filename to write the kubeconfig to) is not provided, then it's calculated by joining
the base path from 'components.helmfile.kubeconfig_path' CLI config from 'atmos.yaml' and the stack name.

Supported inputs of the 'utils_aws_eks_update_kubeconfig' data source:
  - component
  - stack
  - tenant
  - environment
  - stage
  - cluster_name
  - kubeconfig
  - profile
  - role_arn
  - alias
  - region

references

• https://github.com/cloudposse/atmos/pull/136
• https://docs.aws.amazon.com/cli/latest/reference/eks/update-kubeconfig.html

🚀 Enhancements

Bumps github.com/cloudposse/atmos from 1.4.5 to 1.4.8.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot merge will merge this PR after your CI passes on it
• @dependabot squash and merge will squash and merge this PR after your CI passes on it
• @dependabot cancel merge will cancel a previously requested merge and block automerging
• @dependabot reopen will reopen this PR if it is closed
• @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
• @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
• @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

🚀 Enhancements

what

• Update atmos
• Update docs
• Update example and tests

why

• Latest atmos has many new features, bug fixes and improvements
• Use the latest tests from atmos
• Keep the docs up to date

references

• https://github.com/cloudposse/atmos/pull/134
• https://github.com/cloudposse/atmos/pull/133
• https://github.com/cloudposse/atmos/pull/132
• https://github.com/cloudposse/atmos/pull/131
• https://github.com/cloudposse/atmos/pull/130

🚀 Enhancements

what

• Update atmos
• Add init_run_reconfigure CLI config
• Update stack_name_pattern

why

• init_run_reconfigure CLI config allows enabling/disabling the -reconfigure argument for terraform init when running it before running other terraform commands
• Don't use the default stack_name_pattern because it used {tenant} which is not available for all clients
• Running terraform plan and terraform workspace on abstract components creates terraform workspaces which is not needed