A continuous Deployment Recipe consuming Azure DevOps 👉 PowerShell 👉 Terraform Cloud(API-driven Run Workflow)

8 minute read

I have been working with Terraform Cloud/Enterprise using the Version Control System VCS which is good but to my opinion does not provide enough flexibility when you want to have a proper Continuous Deployment flow where you are not just deploying Infrastructure, but you are covering the cycle from Continuous Integration and ending with a Release of verified artefacts/packages or even container images. Continuous Deployment guarantee that qualified releases are automatically deployed to Production.

So I wanted to Test-Drive this model by using the Terraform Cloud API-driven Run Workflow by not associating a workspace to a VCS repo, instead using Azure DevOps Pipeline to decide when a configuration should be changed and when runs should occur.

Environment preparation

What do I need to get started?

Azure DevOps Project, you can use the Free Tier.
Terraform Cloud Instance Free plan up to 5 users and you can leverage the Private Module registry and Remote State Storage.
Some code to deploy to the Web App, here I am deploying a Gatsby Site that can be found on the Microsoft learn modules.

My repo is organized like this to make simple

Web App Repo
        |_Terraform
        |       |_ main.tf
        |       |_ variables.tf
        |       |_ outputs.tf
        |       |_ backend.tf
        |       |_ provider.tf
        |        
        |_ src
        |    |_ Application Source Code
        |
        |_TFC.ps1

Deployment Process

I have created a pipeline with three stages as per diagram.

Builds the application source code to be deployed on the Web App and deploys the infrastructure by triggering a Terraform Cloud Workspace using the API
Deploy the artifact coming from the build to the Web App Staging slot if successful, there will be a pre-deployment approval that will be required before it goes to Production. Providing a timeout of 24hrs before it expires.
If approval given after validating the app changes in the staging slot. It will be executing the Slot Swap with the Production slot

Just in case The staging slot will now have the previous production app.

image-center

Azure DevOps Pipeline

I have setup the pipeline steps as below in order to demo the deployment. For the first step, I am triggering a PowerShell Script that will trigger the deployment on my Terraform Cloud Workspace in this case. I was using the built-in PowerShell task. However, it was using PowerShell v5.1 and not PowerShell v7, hence I used the PowerShell v2 task and explicitly declared to use PowerShell Core by using the Boolean true. Additionally, I stored the API token generated in Terraform Cloud as a secret value variable for the interaction with my workspace image-center

Infrastructure as Code Deployment Stage

- stage: IaC
  displayName: Infrastructure Deployment
  jobs:
  - job: Build
    steps:
      - task: PowerShell@2
        displayName: 'IaC TFC/TFE'
        inputs:
          filePath: '$(System.DefaultWorkingDirectory)\TFC.ps1'
          arguments: >
            -Org  "$(org)"
            -Server "$(server)" 
            -tf_token "$(tfc_token)"
            -workspace_name "$(tfc_workspace)"
          pwsh: true
          workingDirectory: '$(System.DefaultWorkingDirectory)'```

The below is the PowerShell Script that interacts with the Terraform Cloud API to carry on the taks on my workspace (already created by another TFC workspace 🐱‍👤). The script is the very least you will need to make the right calls and deploy successfully the infrastructure is the Terraform Plan is successful. However, it will require refinenment to be more productionalized e.g. Logging and more error handling due to the multiple responses from Terraform Cloud.

The Script goes through the following process:

Create a configuration to upload: this needs to be on a .tar.gz file
Parse the Terraform Workspace ID: where the config is to be uploaded, planned and applied to the Cloud platform in this case Azure. The workspace already has the variables required to authenticate via AzureRM Provider
Create Configuration Version: this configuration-version is created to associate uploaded content with the workspace. This API call performs two tasks: it creates the new configuration version and it extracts the upload URL to be used in the next step.
Upload Config: it will upload the config and provided I have setup the auto-queue-runs = true it will start a run with a plan
Terraform Plan: it will run the plan and go through a logic to get to the next step. If plan is successful, the plan output will print out and will trigger an Apply Run. In case there are no changes to be made then it will not execute an Apply and the task in Azure DevOps will be successful to continue with the Code build task. The plan status will be checked.
Terraform Apply: it will apply the configuration changes and save the state file in Terraform Cloud workspace. The Apply status will be checked and after the apply run is finished it will print out the explicit outputs created on screen from the outputs.tf.

[cmdletBinding()]
Param(
    [Parameter(Mandatory = $true)]
    [string] $Org,
    [Parameter(Mandatory = $true)]
    [string] $Server,
    [Parameter(Mandatory = $true)]
    [string] $TF_TOKEN,
    [Parameter(Mandatory = $true)]
    [string] $WORKSPACE_NAME
)

#Configuration files to upload as .tar.gz. This is required as we are not fetching files from version control directly to the workspace
$UPLOAD_FILE_NAME = "content-$((get-date).ToString("yyyyMMdd")).tar.gz"
cd Terraform
tar -cvzf $UPLOAD_FILE_NAME .\*.tf 

$headers = @{
    Authorization = "Bearer $TF_TOKEN"
}

$body = @"
            {
                "data": {
                    "type": "configuration-versions",
                    "attributes": {
                      "auto-queue-runs": true
                        }
                  }
            }
"@

$apply_on = @"
              {
            "comment": "apply via  CeRoCooL API"
              }
"@

#parsing Workspace ID
$WORKSPACE_ID = (Invoke-RestMethod  -Uri "https://$Server/api/v2/organizations/$Org/workspaces/$WORKSPACE_NAME" -Method Get -ContentType "application/vnd.api+json" -Headers $headers).data.id

#Create configuration Version
$UPLOAD_URL = (Invoke-RestMethod  -Uri "https://$Server/api/v2/workspaces/$WORKSPACE_ID/configuration-versions" -Method POST -ContentType "application/vnd.api+json" -Headers $headers -Body $body).data.attributes."upload-url"

#Upload Configuration File
$UPLOAD_FILE = Invoke-RestMethod  -Uri $UPLOAD_URL -Method Put -ContentType "application/octet-stream"  -InFile $UPLOAD_FILE_NAME

$id = @"
        {
    "data": {
        "attributes": {
            "is-destroy": false
        },
        "type": "runs",
        "relationships": {
            "workspace": {
                "data": {
                    "type": "workspaces",
                    "id": "$WORKSPACE_ID"
                }
            }
        }
    }
}
"@

#Parse Run ID 
$RUN_ID = (Invoke-RestMethod  -Uri "https://$Server/api/v2/runs" -Method Get -ContentType "application/vnd.api+json" -Headers $headers -Body $id).data.id | Select-Object -First 1
$continue = 1
while ($continue -ne 0) {
    foreach ($RUN in $RUN_ID) {
        $RESULT = Invoke-RestMethod  -Uri "https://$Server/api/v2/runs/$RUN" -Method Get -ContentType "application/vnd.api+json" -Headers $headers -Body $id
        $STATUS = $RESULT.data.attributes.status
        $CONFIRMABLE = $RESULT.data.attributes.actions."is-confirmable"
        #Verifies plan has succesfully finished
        if ($STATUS -eq "planned" -and $CONFIRMABLE -eq "False") {
            $PLAN = Invoke-RestMethod  -Uri ("https://$Server/api/v2/runs/$RUN" + "?include=plan") -Method Get -ContentType "application/vnd.api+json" -Headers $headers
            $PLAN_LOG = $PLAN.included.attributes."log-read-url"
            #print out Plan Log for verification
            Invoke-RestMethod  -Uri $PLAN_LOG
            $continue = 0
            #start Apply Process after succesful Plan
            $APPLY = Invoke-RestMethod  -Uri "https://$Server/api/v2/runs/$RUN/actions/apply" -Method Post -ContentType "application/vnd.api+json" -Headers $headers -Body $apply_on

            $RESULT = Invoke-RestMethod  -Uri ("https://$Server/api/v2/runs/$RUN" + "?include=apply") -Method Get -ContentType "application/vnd.api+json" -Headers $headers

            # Get apply ID
            $APPLY_ID = $RESULT.included.id

            $continue = 1
            while ($continue -ne 0) {
                $RESULT = Invoke-RestMethod  -Uri "https://$Server/api/v2/applies/$APPLY_ID" -Method Get -ContentType "application/vnd.api+json" -Headers $headers
                $STATUS = $RESULT.data.attributes.status
                Write-Output $STATUS
                if ($STATUS -eq "finished") {
                    Write-Host "Apply finished"
                    $APPLY_LOG = $RESULT.data.attributes.'log-read-url'
                    $STATE_ID = $RESULT.data.relationships.'state-versions'.data.id
                    $STATE_LOG = Invoke-RestMethod  -Uri ("https://$Server/api/v2/state-versions/$STATE_ID" + "?include=outputs") -Method Get -ContentType "application/vnd.api+json" -Headers $headers
                    $OUTPUTS = ($STATE_LOG.included).Count
                    #Get all Outputs on screen 
                    for ($OUTPUT = 0; $OUTPUT -lt $OUTPUTS; $OUTPUT++) {
                        $STATE_LOG.included[$OUTPUT].attributes
                    }
                    $continue = 0
                }
                elseif ($STATUS -eq "errored") {
                    Write-Host "Terraform Apply errored"
                    $APPLY_LOG = $RESULT.data.attributes.'log-read-url'
                    Invoke-RestMethod  -Uri $APPLY_LOG
                    $continue = 0
                }
                elseif ($STATUS -eq "canceled") {
                    Write-Host "Terraform Apply canceled"
                    $APPLY_LOG = $RESULT.data.attributes.'log-read-url'
                    Invoke-RestMethod  -Uri $APPLY_LOG
                    $continue = 0
                }
                else {
                    Write-Host "Terraform Apply in progress"
                }
            }
        }
        if ($STATUS -eq "planned_and_finished") {
            $PLAN = Invoke-RestMethod  -Uri ("https://$Server/api/v2/runs/$RUN" + "?include=plan") -Method Get -ContentType "application/vnd.api+json" -Headers $headers
            $PLAN_LOG = $PLAN.included.attributes."log-read-url"
            Invoke-RestMethod  -Uri $PLAN_LOG
            $continue = 0
            break
        }
        else { Write-Host "Terraform Plan in Progress" }
    }
}

Infrastructure as Code deployment in Action

The below is a quick demo of the Azure DevOps and Terraform Cloud interaction triggered via API calls to successfully deploy the Infrastructure.

image-center

Application Code build

The below describes the minimal steps required to build and generate an artefact to be deployed in the Azure Web App staging slot.

stage: build
  displayName: App Build and Staging Deployment
  jobs:
  - job: Build
    steps:
      - task: NodeTool@0
        displayName: 'Node.js version'
        inputs:
          versionSpec: 12.x

      - task: Npm@0
        displayName: 'npm install'
        inputs:
          arguments: '--force'

      - task: Npm@1
        displayName: 'npm build'
        inputs:
          command: custom
          verbose: false
          customCommand: 'run build'

      - task: ArchiveFiles@1
        displayName: 'Archive files '
        inputs:
          rootFolder: public

      - task: CopyFiles@2
        displayName: 'Copy Files'
        inputs:
          SourceFolder: '$(Build.ArtifactStagingDirectory)'
          Contents: '$(Build.BuildId).zip'
          TargetFolder: '$(Build.ArtifactStagingDirectory)\ArtifactsToBePublished'

      - task: PublishBuildArtifacts@1
        displayName: 'Publish Artifact: drop'
        inputs:
          PathtoPublish: '$(Build.ArtifactStagingDirectory)\ArtifactsToBePublished'
          artifact: Webapp

Staging slot deployment

The below describes the task required to deploy the artefact generated from the step before to be deployed in the Azure Web App staging slot.

      - task: AzureRmWebAppDeployment@4
        displayName: 'Staging deployment'
        inputs:
          azureSubscription: $(sub)
          appType: webAppLinux
          WebAppName: $(WebAppName)
          deployToSlotOrASE: true
          ResourceGroupName: '$(RG)'
          SlotName: staging
          package: $(Build.ArtifactStagingDirectory)\ArtifactsToBePublished\*.zip 

Web App Deployment to Production

In this stage, I have two different tasks to ensure there is a manual approval before the code in the staging slot is swapped against the production one. I could potentially just deployed the code directly to the Prod slot. However, I find the swapping feature a nice way to ensure zero-downtime deployments and a way to validate and/or test new features before pushing it into Production.

- stage: production
  displayName: App Deployment Production
  jobs:
  - job: waitForValidation
    displayName: Approval   
    pool: server    
    timeoutInMinutes: 4320 # job times out in 3 days
    steps:   
    - task: ManualValidation@0
      timeoutInMinutes: 1440 # task times out in 1 day
      inputs:
        notifyUsers: |
          email@gmail.com
        instructions: 'Please validate the build configuration and resume'
        onTimeout: 'resume' 
  - job: Production_Release
    steps:
      - task: AzureAppServiceManage@0
        displayName: 'Production deployment'
        inputs:
          azureSubscription: $(sub)
          WebAppName: $(WebAppName)
          ResourceGroupName: '$(RG)'
          SourceSlot: staging

Summary

My focus on this post was to demo and show a way using PowerShell to interact with Terraform Cloud and/or Terraform Enterprise using the API-driven Run Workflow in an Azure DevOps pipeline. However, this will work with any CI/CD tool you may be working with.

I did have some fun and removed some rust from my Azure DevOps days. If you find yourselves already invested in TFC/TFE this is definitely a way that will allow you to have more flexibility while taking advantage of the features offered by enterprise products.

I do hope this helps someone and that you find it informative,, so please let me know as constructive feedback is always important🕵️‍♂️,, That’s it for now,,, Hasta la vista🐱‍🏍!!!

🚴‍♂️ If you enjoyed this blog, you can empower me with some caffeine to continue working in new content 🚴‍♂️.

John Alfaro

A continuous Deployment Recipe consuming Azure DevOps 👉 PowerShell 👉 Terraform Cloud(API-driven Run Workflow)

Environment preparation

Deployment Process

Azure DevOps Pipeline

Infrastructure as Code Deployment Stage

Infrastructure as Code deployment in Action

Application Code build

Staging slot deployment

Web App Deployment to Production

Summary

Comments

You May Also Enjoy

Terraforming the Azure Open AI service Deployment via AzAPI and AzureRM providers for the Enterprise… and chat GPT model

Managing Azure NetApp Files Volume Backup feature with Terraform AzAPI Provider

Deploying Azure NetApp Files Preview features using Terraform AzAPI Provider

Federating Google Cloud Identities with Azure Active Directory