Skip to content

Instantly share code, notes, and snippets.

@terma

terma/ec2-spot-github-self-hosted-runners.md

Created January 28, 2020 09:32
Show Gist options
  • Save terma/32e0787217bc21deaa7c70fa3b3c06c9 to your computer and use it in GitHub Desktop.
Save terma/32e0787217bc21deaa7c70fa3b3c06c9 to your computer and use it in GitHub Desktop.
Download ZIP
AWS EC2 Spot Instances for GitHub Self-hosted runners
Raw
ec2-spot-github-self-hosted-runners.md

AWS EC2 Spot Instances for GitHub Self-hosted runners

Below CloudFormation stack which runs GitHub Self-hosted runners on EC2 Spot Instances managed by AutoScalingGroup. It automatically register new instances as self-hosted runners and removes them when Spot is interrupted.

How to use?

  1. Download YML below into file
  2. Create AWS CloudFormation stack
    • Provide values for parameters
      • If you need to install smt in UserData fill stack parameter AdditionalUserData
      • For Git yum install git -y
      • For Git + Mvn yum install git -y && wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo && sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo && yum install -y apache-maven
  3. Add required Action into GitHub Repositry
    • Make sure to specify runs-on: self-hosted
  4. Check Settings > Actions > Self-hosted runners for configured GitHub repo

Implementation Details

  • Create AutoScalingGroup to run EC2 Spot Instances
  • Use UserData script to register EC2 instance in GitHub Self-hosted runners API
  • Use AutoScalingGroup termination CloudWatch Event to run Lambda to remove instance from GitHub Self-hosted runners API

Cloud Formation YML

Parameters:
  Size:
    Type: Number
    Default: 1
    Description: "Will run required amount of spot instances"
  ImageId:
    Type: String
    Default: ami-062f7200baf2fa504
    Description: "Default is public Amazon"
  InstanceType:
    Type: String
    Default: t2.micro
  AdditionalUserData:
    Type: String
    Default: ""
    Description: "If you need to run smt in UserData put it here"
  Repository:
    Type: String
    Description: "GitHub repository name"
  Owner:
    Type: String
    Description: "GitHub repository owner"
  PersonalAccessToken:
    Type: String
    Description: "Authorization for GitHub API, Personal Access Tokens"
  SpotPrice:
    Type: Number
    Default: 0.004

Resources:
  LaunchConfiguration:
    Type: AWS::AutoScaling::LaunchConfiguration
    Properties:
      ImageId:
        Ref: ImageId
      InstanceType:
        Ref: InstanceType
      SpotPrice:
        Ref: SpotPrice
      UserData:
        Fn::Base64:
          !Sub |
            #!/bin/bash -xe
            echo "Run additional user data"
            ${AdditionalUserData}
            echo "Download runner"
            mkdir actions-runner && cd actions-runner
            curl -O -L https://github.com/actions/runner/releases/download/v2.164.0/actions-runner-linux-x64-2.164.0.tar.gz
            tar xzf ./actions-runner-linux-x64-2.164.0.tar.gz
            echo "Install jq"
            yum install jq -y
            echo "Prepare configuration for self-runner"
            export RUNNER_ALLOW_RUNASROOT=true
            export INSTANCE_ID=$(curl http://169.254.169.254/latest/meta-data/instance-id)
            export TOKEN_JSON=$(curl -vvv -u ${Owner}:${PersonalAccessToken} -X POST https://api.github.com/repos/${Owner}/${Repository}/actions/runners/registration-token)
            export TOKEN=$(echo ${!TOKEN_JSON} | jq -r ".token")
            echo "Token ${!TOKEN}"
            echo "Configure self-runner"
            printf "${!INSTANCE_ID}\n\n" | ./config.sh --url https://github.com/${Owner}/${Repository} --token ${!TOKEN}
            echo "Run self-runner"
            ./svc.sh install
            ./svc.sh start

  ASG:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      AvailabilityZones:
        Fn::GetAZs: ""
      LaunchConfigurationName:
        Ref: LaunchConfiguration
      MaxSize:
        Ref: Size
      DesiredCapacity:
        Ref: Size
      MinSize: 0

  LambdaPermission:
    Type: AWS::Lambda::Permission
    Properties:
      Action: lambda:InvokeFunction
      FunctionName:
        Ref: Lambda
      Principal: events.amazonaws.com
      SourceArn:
        Fn::GetAtt:
          - Event
          - Arn

  LambdaRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        {
          "Version": "2012-10-17",
          "Statement": [
          {
            "Effect": "Allow",
            "Principal": {
              "Service": "lambda.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
          }
          ]
        }
      Policies:
        - PolicyDocument:
            {
              "Version": "2012-10-17",
              "Statement": [
              {
                "Effect": "Allow",
                "Action": "logs:CreateLogGroup",
                "Resource": "*"
              },
              {
                "Effect": "Allow",
                "Action": [
                  "logs:CreateLogStream",
                  "logs:PutLogEvents"
                ],
                "Resource": ["*"]
              }
              ]
            }
          PolicyName: String

  Lambda:
    Type: AWS::Lambda::Function
    Properties:
      Handler: index.handler
      Role:
        Fn::GetAtt:
          - LambdaRole
          - Arn
      Runtime: nodejs12.x
      Environment:
        Variables:
          Owner: 
            Ref: Owner
          Repository: 
            Ref: Repository
          PersonalAccessToken:
            Ref: PersonalAccessToken
      Code:
        ZipFile: |
          const https = require('https');

          async function getBody(options) {
              // Return new promise
              return new Promise(function (resolve, reject) {
                  const request = https.request(options.url, options, function (response) {
                      let data = '';

                      response.on('data', (chunk) => {
                          data += chunk;
                      });

                      response.on('end', () => {
                          if (data.length > 0) {
                              resolve(JSON.parse(data));
                          } else {
                              resolve(null);
                          }
                      });
                  });

                  request.on('error', (e) => {
                      reject(e);
                  });

                  request.end();
              })
          }

          exports.handler = async (event) => {
              console.log(JSON.stringify(event));
              if (event['detail-type'] !== 'EC2 Instance Terminate Successful') {
                  return {
                      statusCode: 200,
                      body: `no action for event type ${event['detail-type']}`,
                  };
              }

              const instanceId = event.detail.EC2InstanceId;
              console.log(`Removing GitHub self-hosted running from EC2 instance ${instanceId}`);

              const owner = process.env.Owner;
              const repo = process.env.Repository;
              const password = process.env.PersonalAccessToken;
              const auth = 'Basic ' + Buffer.from(username + ':' + password).toString('base64');

              const runners = await getBody({
                  method: 'GET',
                  url: `https://api.github.com/repos/${owner}/${repo}/actions/runners`,
                  headers: {'Authorization': auth, 'User-Agent': owner}
              });

              const runner = runners.find(r => r.name === instanceId);

              if (runner) {
                  await getBody({
                      method: 'DELETE',
                      url: `https://api.github.com/repos/${owner}/${repo}/actions/runners/${runner.id}`,
                      headers: {'Authorization': auth, 'User-Agent': owner}
                  });
                  console.log(`GitHub self-hosted running from EC2 instance ${instanceId} removed`);
                  return {
                      statusCode: 200,
                      body: `self-hosted runner remove for ${instanceId}`,
                  };
              } else {
                  console.log(`No GitHub self-hosted running for EC2 instance ${instanceId} skip`);
                  return {
                      statusCode: 200,
                      body: `no action for unknown instance ${instanceId}`,
                  };
              }
          };


  Event:
    Type: AWS::Events::Rule
    Properties:
      Targets:
        - Arn:
            Fn::GetAtt:
              - Lambda
              - Arn
          Id: Lambda
      EventPattern:
        Fn::Sub:
          - '{
              "source": ["aws.autoscaling"],
              "detail-type": [
                "EC2 Instance Launch Successful",
                "EC2 Instance Terminate Successful",
                "EC2 Instance Launch Unsuccessful",
                "EC2 Instance Terminate Unsuccessful",
                "EC2 Instance-launch Lifecycle Action",
                "EC2 Instance-terminate Lifecycle Action"
              ],
              "detail": {
                "AutoScalingGroupName": ["${AsgName}"]
              }
            }'
          - AsgName:
              Ref: ASG
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment