I recently came across a Medium post that claimed "Even though we really like AWS’ services we admit that CodeCommit (Git) and CodePipeline (CI/CD) are not ready for production use, yet. The problem of CodePipeline is that you need to create a pipeline for every branch. You cannot create it easily automatically without running CloudFormation." Which is half true. The part that is true is that CodePipeline is terrible at doing Code Builds that aren't meant to be deployed. If only there was some service in the Code Suite that allowed you to Build Code. If they ever make such a service, I hope it would be name CodeBuilder or BuildCoder, I would hope that this service is capable of running unit/integration tests for any commit on any branch.
Fast forward -835 days and AWS Announces the launch of AWS CodeBuild.
In this post, I will show you how to build a CI/CD pipeline with AWS CodeCommit, CodeBuild, CodePipeline, Lambda, and SNS that will ensure your build process runs for every commit. Here is the high-level overview of what we're going to build.
CFN Template
AWSTemplateFormatVersion: '2010-09-09' Transform: AWS::Serverless-2016-10-31 Description: CICD Pipeline that builds every commit on every branch and only merges passing builds to mainline Parameters: ProjectName: Description: Name for this project Type: String Default: "myProject" Resources: CodeCommitRepo: Type: AWS::CodeCommit::Repository Properties: RepositoryDescription: my super awesome repo RepositoryName: !Ref ProjectName Triggers: - DestinationArn: !GetAtt [HelloWorldFunction, Arn] CustomData: !Sub ${ProjectName} Events: - all Name: SendToBuild CloudWatchEventRule: Type: AWS::Events::Rule Properties: Description: "Rule to Fire SNS Message" EventPattern: !Sub | { "source": [ "aws.codebuild" ], "detail-type": [ "CodeBuild Build State Change" ], "detail": { "build-status": [ "IN_PROGRESS", "SUCCEEDED", "FAILED", "STOPPED" ], "project-name": [ "${ProjectName}" ] } } State: ENABLED Targets: - Arn: !Ref DeveloperNotification Id: "SNSTopic" DeveloperNotification: Type: AWS::SNS::Topic Properties: DisplayName: "Build Notifier" SNSPolicy: Type: AWS::SNS::TopicPolicy Properties: PolicyDocument: Statement: Sid: "MYSID001" Effect: "Allow" Principal: Service: events.amazonaws.com Action: sns:Publish Resource: !Ref DeveloperNotification Topics: - !Ref DeveloperNotification HelloWorldFunction: Type: AWS::Serverless::Function Properties: CodeUri: ../commit_builder/ Handler: app.lambda_handler Runtime: python3.7 Timeout: 30 Policies: - AWSLambdaBasicExecutionRole - Version: '2012-10-17' Statement: - Effect: Allow Action: "*" Resource: "*" Environment: Variables: BUILD_PROJECT_NAME: !Ref ProjectName ARTIFACT_BUCKET_NAME: !Ref ArtifactBucket SOURCE_LOCATION: !Sub https://git-codecommit.${AWS::Region}.amazonaws.com/v1/repos/${ProjectName} InvokePermission: Type: AWS::Lambda::Permission Properties: FunctionName: !Ref HelloWorldFunction Action: 'lambda:InvokeFunction' Principal: apigateway.amazonaws.com SourceArn: !GetAtt [CodeCommitRepo, Arn] ArtifactBucket: Type: AWS::S3::Bucket DeletionPolicy: Retain ToolsCodeCommitRole: Type: AWS::IAM::Role Properties: RoleName: !Sub ${ProjectName}-ToolsAcctCodePipelineCodeCommitRole AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: AWS: - !Ref AWS::AccountId Action: - sts:AssumeRole Path: / Policy: Type: AWS::IAM::Policy Properties: PolicyName: !Sub ${ProjectName}-ToolsAcctCodePipelineCodeCommitPolicy PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - codecommit:BatchGetRepositories - codecommit:Get* - codecommit:GitPull - codecommit:List* - codecommit:CancelUploadArchive - codecommit:UploadArchive - s3:* Resource: "*" Roles: - !Ref ToolsCodeCommitRole BuildProjectRole: Type: AWS::IAM::Role Properties: RoleName: !Sub ${ProjectName}-CodeBuildRole AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - codebuild.amazonaws.com Action: - sts:AssumeRole Path: / BuildProjectPolicy: Type: AWS::IAM::Policy Properties: PolicyName: !Sub ${ProjectName}-CodeBuildPolicy PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - codecommit:BatchGetRepositories - codecommit:Get* - codecommit:GitPull - codecommit:List* - codecommit:CancelUploadArchive - codecommit:UploadArchive - s3:PutObject - s3:GetBucketPolicy - s3:GetObject - s3:ListBucket Resource: - !Join ['',['arn:aws:s3:::',!Ref ArtifactBucket, '/*']] - !Join ['',['arn:aws:s3:::',!Ref ArtifactBucket]] - !GetAtt [CodeCommitRepo, Arn] - Effect: Allow Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents Resource: arn:aws:logs:*:*:* - Effect: Allow Action: - "*" Resource: "*" Roles: - !Ref BuildProjectRole BuildProject: Type: AWS::CodeBuild::Project Properties: Name: !Ref ProjectName Description: !Ref ProjectName ServiceRole: !GetAtt BuildProjectRole.Arn Artifacts: Type: CODEPIPELINE Environment: Type: linuxContainer ComputeType: BUILD_GENERAL1_SMALL Image: aws/codebuild/python:3.7.1-1.7.0 EnvironmentVariables: - Name: S3Bucket Value: !Ref ArtifactBucket Source: Type: CODEPIPELINE BuildSpec: | version: 0.1 phases: install: commands: - printenv - ls -R - pip install -r requirements.txt -t "$PWD" build: commands: - aws cloudformation package --template-file templates/databases.yaml --s3-bucket $S3Bucket --s3-prefix databases/codebuild --output-template-file databases.yaml - aws cloudformation package --template-file templates/functions.yaml --s3-bucket $S3Bucket --s3-prefix functions/codebuild --output-template-file functions.yaml - aws cloudformation package --template-file templates/api.yaml --s3-bucket $S3Bucket --s3-prefix api/codebuild --output-template-file api.yaml artifacts: files: - templateConfigurations/* - databases.yaml - functions.yaml - api.yaml discard-paths: yes TimeoutInMinutes: 10 Tags: - Key: Name Value: !Ref ProjectName PipeLineRole: Type: AWS::IAM::Role Properties: RoleName: !Sub ${ProjectName}-codepipeline-role AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - codepipeline.amazonaws.com Action: - sts:AssumeRole Path: / PipelinePolicy: Type: AWS::IAM::Policy Properties: PolicyName: !Sub ${ProjectName}-codepipeline-policy PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - codepipeline:* - iam:ListRoles - cloudformation:Describe* - cloudFormation:List* - codecommit:List* - codecommit:Get* - codecommit:GitPull - codecommit:UploadArchive - codecommit:CancelUploadArchive - codebuild:BatchGetBuilds - codebuild:StartBuild - cloudformation:CreateStack - cloudformation:DeleteStack - cloudformation:DescribeStacks - cloudformation:UpdateStack - cloudformation:CreateChangeSet - cloudformation:DeleteChangeSet - cloudformation:DescribeChangeSet - cloudformation:ExecuteChangeSet - cloudformation:SetStackPolicy - cloudformation:ValidateTemplate - iam:PassRole - lambda:* - s3:ListAllMyBuckets - s3:GetBucketLocation Resource: - "*" - Effect: Allow Action: - s3:PutObject - s3:GetBucketPolicy - s3:GetObject - s3:ListBucket Resource: - !Join ['',['arn:aws:s3:::',!Ref ArtifactBucket, '/*']] - !Join ['',['arn:aws:s3:::',!Ref ArtifactBucket]] Roles: - !Ref PipeLineRole Pipeline: Type: AWS::CodePipeline::Pipeline Properties: RoleArn: !GetAtt PipeLineRole.Arn Name: !Ref AWS::StackName Stages: - Name: Source Actions: - Name: App ActionTypeId: Category: Source Owner: AWS Version: 1 Provider: CodeCommit Configuration: RepositoryName: !Ref ProjectName BranchName: master OutputArtifacts: - Name: SCCheckoutArtifact RunOrder: 1 - Name: Build Actions: - Name: Build ActionTypeId: Category: Build Owner: AWS Version: 1 Provider: CodeBuild Configuration: ProjectName: !Ref BuildProject RunOrder: 1 InputArtifacts: - Name: SCCheckoutArtifact OutputArtifacts: - Name: BuildOutput - Name: DeployToTest Actions: ##### DYNAMO START ##### - Name: CreateChangeSetTestDynamo ActionTypeId: Category: Deploy Owner: AWS Version: 1 Provider: CloudFormation Configuration: ChangeSetName: databases-test ActionMode: CHANGE_SET_REPLACE StackName: databases-test Capabilities: CAPABILITY_NAMED_IAM TemplatePath: BuildOutput::databases.yaml TemplateConfiguration: "BuildOutput::preProd.json" InputArtifacts: - Name: BuildOutput RunOrder: 1 - Name: DeployChangeSetTestDynamo ActionTypeId: Category: Deploy Owner: AWS Version: 1 Provider: CloudFormation Configuration: ChangeSetName: databases-test ActionMode: CHANGE_SET_EXECUTE StackName: databases-test InputArtifacts: - Name: BuildOutput RunOrder: 2 ##### DYNAMO END ##### ##### LAMBDA START ##### - Name: CreateChangeSetTestLambda ActionTypeId: Category: Deploy Owner: AWS Version: 1 Provider: CloudFormation Configuration: ChangeSetName: lambda-test ActionMode: CHANGE_SET_REPLACE StackName: lambda-test Capabilities: CAPABILITY_NAMED_IAM TemplatePath: BuildOutput::functions.yaml TemplateConfiguration: "BuildOutput::preProd.json" InputArtifacts: - Name: BuildOutput RunOrder: 3 - Name: DeployChangeSetTestLambda ActionTypeId: Category: Deploy Owner: AWS Version: 1 Provider: CloudFormation Configuration: ChangeSetName: lambda-test ActionMode: CHANGE_SET_EXECUTE StackName: lambda-test InputArtifacts: - Name: BuildOutput RunOrder: 4 ##### LAMBDA END ##### ##### API START ##### - Name: CreateChangeSetTestAPI ActionTypeId: Category: Deploy Owner: AWS Version: 1 Provider: CloudFormation Configuration: ChangeSetName: api-test ActionMode: CHANGE_SET_REPLACE StackName: api-test Capabilities: CAPABILITY_NAMED_IAM TemplatePath: BuildOutput::api.yaml TemplateConfiguration: "BuildOutput::preProd.json" InputArtifacts: - Name: BuildOutput RunOrder: 5 - Name: DeployChangeSetTestAPI ActionTypeId: Category: Deploy Owner: AWS Version: 1 Provider: CloudFormation Configuration: ChangeSetName: api-test ActionMode: CHANGE_SET_EXECUTE StackName: api-test InputArtifacts: - Name: BuildOutput RunOrder: 6 ArtifactStore: Type: S3 Location: !Ref ArtifactBucket
Lambda Code
import boto3 import os def lambda_handler(event, context): print('Starting a new build ...') cb = boto3.client('codebuild') build = { 'projectName': event['Records'][0]['customData'], 'sourceVersion': event['Records'][0]['codecommit']['references'][0]['commit'], 'artifactsOverride': { 'type': 'S3', 'location': os.environ['ARTIFACT_BUCKET_NAME'], 'path': 'builds/', 'name': event['Records'][0]['codecommit']['references'][0]['commit'], 'packaging': 'ZIP', 'overrideArtifactName': True }, 'sourceTypeOverride': 'CODECOMMIT', 'sourceLocationOverride': os.environ['SOURCE_LOCATION'], } print('Starting build for project {0} from commit ID {1}'.format(build['projectName'], build['sourceVersion'])) response = cb.start_build(**build) print('Successfully launched a new CodeBuild project build!') return