Run you Azure DevOps pipelines in Kubernetes pods as Self-Hosted Agents

Prerequisites for this pipeline to run successfully :

1. Associate the correct Azure Subscription to your Azure DevOps Organization.
2. Create/have an ACR or Docker registry and make sure all configs are set up.
3. Create/have an Azure Kubernetes Cluster .

Resource group resources.

Now navigate to Azure DevOps and to your Project, and create a service connection for Azure Kubernetes Service:

Project Settings → Service Connections → Create New Service Connection → Choose Kubernetes → Choose Azure Subscription → Select Subscription , Cluster Name and Namespace → Give it a name and click [Save]

If you are creating a new ACR , don’t forget to create its service connection.

Now that we have finished creating the pre requisites, let’s move on to creating the pipeline in Azure DevOps.

I have provided all the files you need to create the pipeline in the following GitHub Repository :

Let’s quickly go through the main files.

1. Dockerfile — This uses a base image of the microsoft/vsts-agent.If you need more softwares you can add the commands to download and install inside the dockerfile. By default standard softwares like nodeJS, java, python, etc. are available. To view this Dockerfile click on this link .
2. start.sh — This file is used inside the Dockerfile and is responsible for starting up the agents. It requires the name of the Agent Pool which we created above in step 4. It also needs the URL of your Azure DevOps Organization : https://dev.azure.com/<name of your organization>. Finally it requires your Personal Access Token (PAT) with which it can attach the agents to your organization. You can create a PAT following this. To view this start.sh file click here.
3. deployment.yml — This is the Kubernetes manifest file which is responsible to deploy the agents in the Pods. The number of agents to be spun up is determined by the replicas : n tag. We also need to specify the imagePullSecrets in this file which is needed by Kubernetes to authenticate the pull from the ACR. Creation of this secret is a one time activity and can be achieved using this command from Azure CLI. To see this file click on this link.

You can find the required manifests here.

Credentials for ACR

kubectl create secret docker-registry acr-secret --docker-server=<Registry URL> --docker-username=<your acr username> --docker-password=<your acr password> -n selfhosted-agent

acr-secret is the name of the secret that we created and it is used deployment.yml manifest

4. azure-pipelines.yml — This file is to start the azure pipeline which will perform all the activities above. View this file here.

This build pipeline is divided into 2 stages :

Pipeline execution stages.

The first stage is to build the docker image of the agents and push it to ACR. The second stage is to pull the images and deploy them in AKS, hence spinning up the Self-hosted agents inside Kubernetes cluster.

Self-hosted agent creation flow

In the first stage, there is only one task :

Build the docker image and push to Azure Container Registry.

- task: Docker@2
    inputs:
      containerRegistry: 'selfhosted-demo'
      repository: 'agent'
      command: 'buildAndPush'
      Dockerfile: '**/Dockerfile'
      tags: 'latest'

In the second stage, there are primarily 3 tasks :

1. Login to Azure Kubernetes Service using the Service Connection

- task: Kubernetes@1
  inputs:
    connectionType: 'Kubernetes Service Connection'
    kubernetesServiceEndpoint: '$(AKSserviceConnectionName)'
    command: 'login'

2. Replace variables AZP_POOL_NAME, AZP_URL_VALUE, AZP_TOKEN_SECRET in Pipeline Variables. We can do that by creating some pipeline variables like below.

3. The third task is to deploy the manifest deployment.yml

- task: CmdLine@2
  displayName: 'Deploy'
  inputs:
   script: kubectl apply -f deployment.yml -n selfhosted-agent

After the pipeline execution is completed, you should see new Agents created in the Agent Pool that you had created earlier.

Self hosted agents running on Kubernetes Cluster

Now to use these agents in any pipeline, you can simply add the pool name in your pipeline.

We will just install python package as sample task.

trigger:
- main pool: selfhosted-agent
steps:
- script: |
    python -m pip install --upgrade pip
    pip install pytest
    if [ -f requirements.txt ]; then pip install -r requirements.txt; 
    fi
  displayName: 'Install dependencies'

Conclusion

Running a self-hosted agent in Kubernetes is a cool solution that helps you save costs of running jobs in Virtual machines but we should not forget that if you are using self-hosted agents in your projects , then you are fully responsible of their maintenance and capabilities.

I hope this article helps some of you in order to improve their DevOps/GitOps flow.