Automating variable group Creation & Update in Azure DevOps with Terraform.

When you are working in a large scale projects that requires you to create and manage variable groups of multiple pipelines in Azure DevOps, you certainly don’t want to suffer and manage it manually.

As DevOps is a culture, automating the creation and the update of your variable groups will help you move faster, improve your workflow in your project and reduce human error.

We will use Terraform in order to achieve our goal today.

Provisioning flow.

Prerequisites.

Git
Pipeline agent with Terraform installed.
GitHub / Azure repository.
Storage account.
Container data storage.

1. Create the storage account and the container.

The first thing to do is to create the storage account where we will store the state of terraform.

This state is used by Terraform to map real-world resources to your configuration, keep track of metadata, and to improve performance for large infrastructures. This state is stored by default in a local file named “terraform. tfstate”.

Once a storage account is created, you need to create a container where your tfstate file will be stored.

Once we create the container, we will have the below layout.

Now as our terraform storage account is ready, we will create our project in Azure DevOps.

2. Create the project and its pipeline

I called my project “Create variable group“.

Now, let’s create our pipeline.

trigger:
  - main

pool:
  vmImage: ubuntu-latest

steps:
  - bash: echo ${AZURE_DEVOPS_CLI_PAT} | az devops login
    env:
      AZURE_DEVOPS_CLI_PAT: $(System.AccessToken)
    displayName: 'Login Azure DevOps Extension'

  - task: TerraformCLI@0
    inputs:
      command: 'init'
      workingDirectory: '$(System.DefaultWorkingDirectory)/terraform'
      allowTelemetryCollection: true

  - task: TerraformCLI@0
    inputs:
      command: 'plan'
      workingDirectory: '$(System.DefaultWorkingDirectory)/terraform'
      allowTelemetryCollection: true

  - task: TerraformCLI@0
    inputs:
      command: 'validate'
      workingDirectory: '$(System.DefaultWorkingDirectory)/terraform'
      allowTelemetryCollection: true

  - task: TerraformCLI@0
    inputs:
      command: 'apply'
      workingDirectory: '$(System.DefaultWorkingDirectory)/terraform'
      allowTelemetryCollection: true

We can see that pipeline has 5 tasks:

1- Login to Azure DevOps extension: You can authenticate using the System.AccessToken security token is used by the running pipeline, by assigning it to an environment variable named AZURE_DEVOPS_CLI_PAT.

2- Terraform init

3- Terraform plan

4- Terraform validate

5- Terraform apply

If we check the pipeline script, we will see that the working directory is ‘$(System.DefaultWorkingDirectory)/terraform’, so all our terrafrom files are located under “/terraform” in our repository.

3. Create terraform file

In order to make it simple in this article, we will not go through terraform best practices and use modules .. etc, we will use only one TF file which is main.tf.

terraform {
  required_providers {
    azuredevops = {
      source  = "microsoft/azuredevops"
      version = "=0.1.7"
    }
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=2.75.0"
    }
  }
}

terraform {
  backend "azurerm" {
    resource_group_name   = "terraform"
    storage_account_name  = "avaxiaterraform"  # Replace with your storage account name
    container_name        = "tfstate"          # Replace with your container name
    key                   = "terraform.tfstate" # Replace with your TF state file name
    subscription_id       = "ADD YOUR SUBSCRIPTION ID HERE"
    tenant_id             = "ADD YOUR TENANT ID HERE"
    client_id             = "ADD YOUR SERVICE PRINCIPAL ID HERE"
    client_secret         = "ADD YOUR SERVICE PRINCIPAL SECRET HERE"
  }
}

provider "azurerm" {
  features {}
}

provider "azuredevops" {
  org_service_url        = "https://dev.azure.com/YOUR ORG NAME"
  personal_access_token  = "ADD YOUR PERSONAL ACCESS TOKEN HERE"
}

resource "azuredevops_serviceendpoint_azurerm" "demo-terraform" {
  project_id                = "ADD YOUR PROJECT ID HERE"
  service_endpoint_name     = "Sample AzureRM"
  description               = "Managed by Terraform"
  credentials {
    serviceprincipalid      = "ADD YOUR SVC PRINCIPAL ID HERE"
    serviceprincipalkey     = "ADD YOUR SVC PRINCIPAL KEY HERE"
  }
  azurerm_spn_tenantid      = "ADD YOUR TENANT ID HERE"
  azurerm_subscription_id   = "ADD YOUR SUBSCRIPTION ID HERE"
  azurerm_subscription_name = "ADD YOUR SUBSCRIPTION NAME HERE"
}

resource "azuredevops_variable_group" "variablegroup" {
  project_id   = "ADD YOUR PROJECT ID HERE"  # ID of the project where you will update the variable group
  name         = "Test Variable Group"
  description  = "Test Variable Group Description"
  allow_access = true

  variable {
    name         = "Account Password"
    secret_value = "p@ssword1233333"
    is_secret    = true
  }

  variable {
    name  = "key1"
    value = "testudate"
  }
}

resource "azuredevops_variable_group" "eventhub" {
  project_id   = "ADD YOUR PROJECT ID HERE"  # ID of the project where you will update the variable group
  name         = "evh-topic"
  description  = "Test Variable Group Description"
  allow_access = true

  variable {
    name         = "evh Password"
    secret_value = "p@sswor333"
    is_secret    = true
  }

  variable {
    name  = "topic"
    value = "add topic name"
  }
}

We are creating two variable groups here, the first is called “Test Variable Group” and the second is called “evh-topic”.

Now we run the pipeline and then check if the variable group are created or not.

Pipeline executed successfully.

Variable groups automatically created

evh-topic variables

Test Variable Group variables

Now, you can manage your variable group variables by updating or adding variables to terraform main.tf file.

This approach helped in so many projects save so much time and made it much more flexible for developers as managing hundreds of Azure projects variable manually is not a best practice.