I recently came across a Medium post that claimed "Even though we really like AWS’ services we admit that CodeCommit (Git) and CodePipeline (CI/CD) are not ready for production use, yet. The problem of CodePipeline is that you need to create a pipeline for every branch. You cannot create it easily automatically without running CloudFormation." Which is half true. The part that is true is that CodePipeline is terrible at doing Code Builds that aren't meant to be deployed. If only there was some service in the Code Suite that allowed you to Build Code. If they ever make such a service, I hope it would be name CodeBuilder or BuildCoder, I would hope that this service is capable of running unit/integration tests for any commit on any branch.

Fast forward -835 days and AWS Announces the launch of AWS CodeBuild.

In this post, I will show you how to build a CI/CD pipeline with AWS CodeCommit, CodeBuild, CodePipeline, Lambda, and SNS that will ensure your build process runs for every commit. Here is the high-level overview of what we're going to build.

CFN Template

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: CICD Pipeline that builds every commit on every branch and only merges passing builds to mainline

Parameters:
  ProjectName:
    Description: Name for this project
    Type: String
    Default: "myProject"
Resources:

    CodeCommitRepo:
      Type: AWS::CodeCommit::Repository
      Properties:
        RepositoryDescription: my super awesome repo
        RepositoryName: !Ref ProjectName
        Triggers:
          - DestinationArn: !GetAtt [HelloWorldFunction, Arn]
            CustomData: !Sub ${ProjectName}
            Events:
              - all
            Name: SendToBuild
    CloudWatchEventRule:
      Type: AWS::Events::Rule
      Properties:
        Description: "Rule to Fire SNS Message"
        EventPattern: !Sub |
            {
              "source": [
                "aws.codebuild"
              ],
              "detail-type": [
                "CodeBuild Build State Change"
              ],
              "detail": {
                "build-status": [
                  "IN_PROGRESS",
                  "SUCCEEDED",
                  "FAILED",
                  "STOPPED"
                ],
                "project-name": [
                  "${ProjectName}"
                ]
              }
            }
        State: ENABLED
        Targets:
         - Arn: !Ref DeveloperNotification
           Id: "SNSTopic"
    DeveloperNotification:
      Type: AWS::SNS::Topic
      Properties:
        DisplayName: "Build Notifier"

    SNSPolicy:
      Type: AWS::SNS::TopicPolicy
      Properties:
        PolicyDocument:
          Statement:
            Sid: "MYSID001"
            Effect: "Allow"
            Principal:
              Service: events.amazonaws.com
            Action: sns:Publish
            Resource: !Ref DeveloperNotification
        Topics:
          - !Ref DeveloperNotification

    HelloWorldFunction:
        Type: AWS::Serverless::Function
        Properties:
            CodeUri: ../commit_builder/
            Handler: app.lambda_handler
            Runtime: python3.7
            Timeout: 30
            Policies:
              - AWSLambdaBasicExecutionRole
              - Version: '2012-10-17'
                Statement:
                  - Effect: Allow
                    Action: "*"
                    Resource: "*"
            Environment:
              Variables:
                BUILD_PROJECT_NAME: !Ref ProjectName
                ARTIFACT_BUCKET_NAME: !Ref ArtifactBucket
                SOURCE_LOCATION: !Sub https://git-codecommit.${AWS::Region}.amazonaws.com/v1/repos/${ProjectName}

    InvokePermission:
      Type: AWS::Lambda::Permission
      Properties:
        FunctionName: !Ref HelloWorldFunction
        Action: 'lambda:InvokeFunction'
        Principal: apigateway.amazonaws.com
        SourceArn: !GetAtt [CodeCommitRepo, Arn]

    ArtifactBucket:
      Type: AWS::S3::Bucket
      DeletionPolicy: Retain

    ToolsCodeCommitRole:
      Type: AWS::IAM::Role
      Properties:
        RoleName: !Sub ${ProjectName}-ToolsAcctCodePipelineCodeCommitRole
        AssumeRolePolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Principal:
                AWS:
                  - !Ref AWS::AccountId
              Action:
                - sts:AssumeRole
        Path: /

    Policy:
        Type: AWS::IAM::Policy
        Properties:
          PolicyName: !Sub ${ProjectName}-ToolsAcctCodePipelineCodeCommitPolicy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - codecommit:BatchGetRepositories
                  - codecommit:Get*
                  - codecommit:GitPull
                  - codecommit:List*
                  - codecommit:CancelUploadArchive
                  - codecommit:UploadArchive
                  - s3:*
                Resource: "*"
          Roles:
            - !Ref ToolsCodeCommitRole
    BuildProjectRole:
      Type: AWS::IAM::Role
      Properties:
        RoleName: !Sub ${ProjectName}-CodeBuildRole
        AssumeRolePolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - codebuild.amazonaws.com
              Action:
                - sts:AssumeRole
        Path: /

    BuildProjectPolicy:
      Type: AWS::IAM::Policy
      Properties:
        PolicyName: !Sub ${ProjectName}-CodeBuildPolicy
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action:
                - codecommit:BatchGetRepositories
                - codecommit:Get*
                - codecommit:GitPull
                - codecommit:List*
                - codecommit:CancelUploadArchive
                - codecommit:UploadArchive
                - s3:PutObject
                - s3:GetBucketPolicy
                - s3:GetObject
                - s3:ListBucket
              Resource:
                - !Join ['',['arn:aws:s3:::',!Ref ArtifactBucket, '/*']]
                - !Join ['',['arn:aws:s3:::',!Ref ArtifactBucket]]
                - !GetAtt [CodeCommitRepo, Arn]
            - Effect: Allow
              Action:
                - logs:CreateLogGroup
                - logs:CreateLogStream
                - logs:PutLogEvents
              Resource: arn:aws:logs:*:*:*
            - Effect: Allow
              Action:
                - "*"
              Resource: "*"
        Roles:
          - !Ref BuildProjectRole
    BuildProject:
      Type: AWS::CodeBuild::Project
      Properties:
        Name: !Ref ProjectName
        Description: !Ref ProjectName
        ServiceRole: !GetAtt BuildProjectRole.Arn
        Artifacts:
          Type: CODEPIPELINE
        Environment:
          Type: linuxContainer
          ComputeType: BUILD_GENERAL1_SMALL
          Image: aws/codebuild/python:3.7.1-1.7.0
          EnvironmentVariables:
            - Name: S3Bucket
              Value: !Ref ArtifactBucket
        Source:
          Type: CODEPIPELINE
          BuildSpec: |
            version: 0.1
            phases:
              install:
                commands:
                  - printenv
                  - ls -R
                  - pip install -r requirements.txt -t "$PWD"
              build:
                commands:
                  - aws cloudformation package --template-file templates/databases.yaml --s3-bucket $S3Bucket --s3-prefix databases/codebuild --output-template-file databases.yaml
                  - aws cloudformation package --template-file templates/functions.yaml --s3-bucket $S3Bucket --s3-prefix functions/codebuild --output-template-file functions.yaml
                  - aws cloudformation package --template-file templates/api.yaml --s3-bucket $S3Bucket --s3-prefix api/codebuild --output-template-file api.yaml
            artifacts:
              files:
                - templateConfigurations/*
                - databases.yaml
                - functions.yaml
                - api.yaml
              discard-paths: yes

        TimeoutInMinutes: 10
        Tags:
          - Key: Name
            Value: !Ref ProjectName
    PipeLineRole:
      Type: AWS::IAM::Role
      Properties:
        RoleName: !Sub ${ProjectName}-codepipeline-role
        AssumeRolePolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - codepipeline.amazonaws.com
              Action:
                - sts:AssumeRole
        Path: /

    PipelinePolicy:
      Type: AWS::IAM::Policy
      Properties:
        PolicyName: !Sub ${ProjectName}-codepipeline-policy
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action:
                - codepipeline:*
                - iam:ListRoles
                - cloudformation:Describe*
                - cloudFormation:List*
                - codecommit:List*
                - codecommit:Get*
                - codecommit:GitPull
                - codecommit:UploadArchive
                - codecommit:CancelUploadArchive
                - codebuild:BatchGetBuilds
                - codebuild:StartBuild
                - cloudformation:CreateStack
                - cloudformation:DeleteStack
                - cloudformation:DescribeStacks
                - cloudformation:UpdateStack
                - cloudformation:CreateChangeSet
                - cloudformation:DeleteChangeSet
                - cloudformation:DescribeChangeSet
                - cloudformation:ExecuteChangeSet
                - cloudformation:SetStackPolicy
                - cloudformation:ValidateTemplate
                - iam:PassRole
                - lambda:*
                - s3:ListAllMyBuckets
                - s3:GetBucketLocation
              Resource:
                - "*"
            - Effect: Allow
              Action:
                - s3:PutObject
                - s3:GetBucketPolicy
                - s3:GetObject
                - s3:ListBucket
              Resource:
                - !Join ['',['arn:aws:s3:::',!Ref ArtifactBucket, '/*']]
                - !Join ['',['arn:aws:s3:::',!Ref ArtifactBucket]]

        Roles:
          - !Ref PipeLineRole
    Pipeline:
        Type: AWS::CodePipeline::Pipeline
        Properties:
          RoleArn: !GetAtt PipeLineRole.Arn
          Name: !Ref AWS::StackName
          Stages:
            - Name: Source
              Actions:
                - Name: App
                  ActionTypeId:
                    Category: Source
                    Owner: AWS
                    Version: 1
                    Provider: CodeCommit
                  Configuration:
                    RepositoryName: !Ref ProjectName
                    BranchName: master
                  OutputArtifacts:
                    - Name: SCCheckoutArtifact
                  RunOrder: 1
            - Name: Build
              Actions:
                - Name: Build
                  ActionTypeId:
                    Category: Build
                    Owner: AWS
                    Version: 1
                    Provider: CodeBuild
                  Configuration:
                    ProjectName: !Ref BuildProject
                  RunOrder: 1
                  InputArtifacts:
                    - Name: SCCheckoutArtifact
                  OutputArtifacts:
                    - Name: BuildOutput
            - Name: DeployToTest
              Actions:
                ##### DYNAMO START #####
                - Name: CreateChangeSetTestDynamo
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: databases-test
                    ActionMode: CHANGE_SET_REPLACE
                    StackName: databases-test
                    Capabilities: CAPABILITY_NAMED_IAM
                    TemplatePath: BuildOutput::databases.yaml
                    TemplateConfiguration: "BuildOutput::preProd.json"
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 1
                - Name: DeployChangeSetTestDynamo
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: databases-test
                    ActionMode: CHANGE_SET_EXECUTE
                    StackName: databases-test
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 2
                ##### DYNAMO END #####
                ##### LAMBDA START #####
                - Name: CreateChangeSetTestLambda
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: lambda-test
                    ActionMode: CHANGE_SET_REPLACE
                    StackName: lambda-test
                    Capabilities: CAPABILITY_NAMED_IAM
                    TemplatePath: BuildOutput::functions.yaml
                    TemplateConfiguration: "BuildOutput::preProd.json"
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 3
                - Name: DeployChangeSetTestLambda
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: lambda-test
                    ActionMode: CHANGE_SET_EXECUTE
                    StackName: lambda-test
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 4
                ##### LAMBDA END #####
                ##### API START #####
                - Name: CreateChangeSetTestAPI
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: api-test
                    ActionMode: CHANGE_SET_REPLACE
                    StackName: api-test
                    Capabilities: CAPABILITY_NAMED_IAM
                    TemplatePath: BuildOutput::api.yaml
                    TemplateConfiguration: "BuildOutput::preProd.json"
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 5
                - Name: DeployChangeSetTestAPI
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: api-test
                    ActionMode: CHANGE_SET_EXECUTE
                    StackName: api-test
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 6
          ArtifactStore:
            Type: S3
            Location: !Ref ArtifactBucket

Lambda Code

import boto3
import os


def lambda_handler(event, context):
    print('Starting a new build ...')
    cb = boto3.client('codebuild')
    build = {
        'projectName': event['Records'][0]['customData'],
        'sourceVersion': event['Records'][0]['codecommit']['references'][0]['commit'],
        'artifactsOverride': {
                                'type': 'S3',
                                'location': os.environ['ARTIFACT_BUCKET_NAME'],
                                'path': 'builds/',
                                'name': event['Records'][0]['codecommit']['references'][0]['commit'],
                                'packaging': 'ZIP',
                                'overrideArtifactName': True
                            },
        'sourceTypeOverride': 'CODECOMMIT',
        'sourceLocationOverride': os.environ['SOURCE_LOCATION'],
    }
    print('Starting build for project {0} from commit ID {1}'.format(build['projectName'], build['sourceVersion']))
    response = cb.start_build(**build)
    print('Successfully launched a new CodeBuild project build!')
    return