CDK

Today, the AWS Cloud Development Kit team released experimental support for Python in the CDK. While it's still an experimental release and there are some rough edges, it's a big step forward and I wanted to give it a test drive. The biggest pain-point so far is that "pip install aws-cdk.cdk" only installs the shims that connect the CDK (which is written in TypeScript) to Python language bindings. Users will have to make sure that they have NodeJS (>= 8.11.x) installed and perform "npm install cdk" to install the actual cdk goodies. Once that's out fo the way, we're all set to start declaring our infrastructure. We're going to create a Lambda function that just echoes "Hello, world!".

I've created a new python project in PyCharm.

Now, let's define our requirements in a requirements.txt file.

aws-cdk.aws-lambda
aws-cdk.cdk

Next we create our lambda code. We're going to name this file lambda-handler.py

def main(event, context):
    print("Hello, world!")

So far, this has all been the same development process we follow for developing any lambda function, but here is where we start to leverage CDK. Instead of the SAM approach where we would have a cloudformation template that we either have to craft or copy from a previous project, CDK let's us declare our infrastructure with all the goodness that modern IDEs provide (type-checking, tab completion, syntax highlighting, etc...). Let's create a file call app.py that will contain the logic for building our application.

from aws_cdk import cdk


class MyLambdaStack(cdk.Stack):
    def __init__(self, app: cdk.App, id: str) -> None:
        super().__init__(app, id)


app = cdk.App()
MyLambdaStack(app, "MySimpleLambda")
app.run()

Next we need a file called cdk.json, this file tells CDK how to run the application that builds our infrastructure, the CDK application captures the applications we define here and uses them to generate the cloudformation needed to build this application in AWS.

{
    "app": "python3 app.py"
}

Now we're ready to hop over to the console and build our app. From the root of your package run "cdk synthesize" to see the synthesized cloudformation YAML.

cdk synthesize

Well, that was pretty anti-climactic. there's nothing there but some metadata. Let's go back to our app and add our lambda function code.

First, we're going to read in the contents of lambda-handler.py so that we can pass the raw string to our Lambda Function. This is the same way that you define inline code in a raw cloudformation lambda function.

        with open("lambda-handler.py", encoding="utf8") as fp:
            handler_code = fp.read()

Now that we have our lambda code, let's use it to build a Function.

        lambdaFn = lambda_.Function(
            self,
            "Singleton",
            code=lambda_.InlineCode(handler_code),
            handler="index.main",
            timeout=300,
            runtime=lambda_.Runtime.PYTHON37,
        )

Now we can jump back to our console and synthesize this again

cdk synthesize

Great!!! I have YAML in the console, exactly where I need it ;) . but how do I do anything with this? CDK has solved that for us too.

cdk deploy --profile workshop

"--profile workshop" is just an aws config profile I use for demos, if you have previously run aws configure You won't need to supply that argument.

Let's hop over to the AWS Console and see what's good.

Awesome. Let's go see this Lambda in action.

WOOT!! WOOT!! Now we're building infrastructure in a language that's more fun to work with than YAML/JSON and it handles much of the boilerplate Cloudformation that I don't want to think about.

Comments

Reduce you AWS bill by 10% with this one easy trick, Cloud Economists Hate it

On Thursday, the UK did what UKs are want to do lately. They used Brexit to draw fresh criticisms over seemingly simple tasks. This week was the Parliament Website Edition. The surge of traffic of people wanting up to the minute signature counts crippled their site. My friend James Beswick created a cool application to demonstrate how to properly build a web application that scales dynamically with the request load. While I consider James a friend, his treachery cannot go unmentioned. James committed the most unforgivable of serverlessing crimes; he wrote more code than he needed to and now owns more stuff than he needs to provide the same amount of value.

A quick overview of James' architecture looks like this:

(1) A request comes in to API Gateway, and is proxied to AWS Lambda, (2) AWS Lambda reads the appropriate values from a DynamoDB Table, (3) then the Lambda Function returns the information to the client.

Now that API Gateway supports direct integration with AWS Services, we can eliminate the Lambda completely. and create an architecture that looks more like this:

Without a Lambda Function getting in the way, we see improved performance for our clients​

Latency with Lambda: 
  min: 168.1
  max: 3314.7
  median 242.6
  p95: 336.1
  p99 544.8
Latency without Lambda:
  min: 33.7
  max: 4432
  median: 45
  p95: 86.2
  p99: 886.1

But a far more compelling argument for the simpler architecture is cost. Given the following pricing guides from AWS and our hypothetical scenario, we see a non-trivial cost savings by eliminating Lambda.

API Gateway:
  $3.50/Million requests
  $0.020/hr for 0.5GB Cache on Deployment Stage
Lambda:
  $2.08*10^-7 per 100ms of request time on a 128MB Function
DynamoDB:
  $1.25/Million Writes
  $0.25/Million Reads

If we assume our application will receive 10M requests in the first 24 hours (0.5 Million votes cast and 9.5 Million page refreshes) and our API caches read requests for 5 seconds before the TTL expires. The Lambda-based architecture costs about $41.13 for the first 24 hours, the Lambdaless architecture costs about $36.16 over the same period.

This is where a less scrupulous evangelist would offer to help you eliminate all of your Lambda Functions for 'moderate' consulting fee. But, if we take a closer look we see that by simply enabling the API Gateway Cache on the Deployment Stage, the Dynamo Table RCU cost is almost completely eliminated and offsets the price of the Lambda. No huge refactoring required, no unfamiliar integration changes, just checking one box reduces you AWS Bill by 10%

Comments

[DRAFT] CI/CD With CodeCommit

I recently came across a Medium post that claimed "Even though we really like AWS’ services we admit that CodeCommit (Git) and CodePipeline (CI/CD) are not ready for production use, yet. The problem of CodePipeline is that you need to create a pipeline for every branch. You cannot create it easily automatically without running CloudFormation." Which is half true. The part that is true is that CodePipeline is terrible at doing Code Builds that aren't meant to be deployed. If only there was some service in the Code Suite that allowed you to Build Code. If they ever make such a service, I hope it would be name CodeBuilder or BuildCoder, I would hope that this service is capable of running unit/integration tests for any commit on any branch.

Fast forward -835 days and AWS Announces the launch of AWS CodeBuild.

In this post, I will show you how to build a CI/CD pipeline with AWS CodeCommit, CodeBuild, CodePipeline, Lambda, and SNS that will ensure your build process runs for every commit. Here is the high-level overview of what we're going to build.

CFN Template

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: CICD Pipeline that builds every commit on every branch and only merges passing builds to mainline

Parameters:
  ProjectName:
    Description: Name for this project
    Type: String
    Default: "myProject"
Resources:

    CodeCommitRepo:
      Type: AWS::CodeCommit::Repository
      Properties:
        RepositoryDescription: my super awesome repo
        RepositoryName: !Ref ProjectName
        Triggers:
          - DestinationArn: !GetAtt [HelloWorldFunction, Arn]
            CustomData: !Sub ${ProjectName}
            Events:
              - all
            Name: SendToBuild
    CloudWatchEventRule:
      Type: AWS::Events::Rule
      Properties:
        Description: "Rule to Fire SNS Message"
        EventPattern: !Sub |
            {
              "source": [
                "aws.codebuild"
              ],
              "detail-type": [
                "CodeBuild Build State Change"
              ],
              "detail": {
                "build-status": [
                  "IN_PROGRESS",
                  "SUCCEEDED",
                  "FAILED",
                  "STOPPED"
                ],
                "project-name": [
                  "${ProjectName}"
                ]
              }
            }
        State: ENABLED
        Targets:
         - Arn: !Ref DeveloperNotification
           Id: "SNSTopic"
    DeveloperNotification:
      Type: AWS::SNS::Topic
      Properties:
        DisplayName: "Build Notifier"

    SNSPolicy:
      Type: AWS::SNS::TopicPolicy
      Properties:
        PolicyDocument:
          Statement:
            Sid: "MYSID001"
            Effect: "Allow"
            Principal:
              Service: events.amazonaws.com
            Action: sns:Publish
            Resource: !Ref DeveloperNotification
        Topics:
          - !Ref DeveloperNotification

    HelloWorldFunction:
        Type: AWS::Serverless::Function
        Properties:
            CodeUri: ../commit_builder/
            Handler: app.lambda_handler
            Runtime: python3.7
            Timeout: 30
            Policies:
              - AWSLambdaBasicExecutionRole
              - Version: '2012-10-17'
                Statement:
                  - Effect: Allow
                    Action: "*"
                    Resource: "*"
            Environment:
              Variables:
                BUILD_PROJECT_NAME: !Ref ProjectName
                ARTIFACT_BUCKET_NAME: !Ref ArtifactBucket
                SOURCE_LOCATION: !Sub https://git-codecommit.${AWS::Region}.amazonaws.com/v1/repos/${ProjectName}

    InvokePermission:
      Type: AWS::Lambda::Permission
      Properties:
        FunctionName: !Ref HelloWorldFunction
        Action: 'lambda:InvokeFunction'
        Principal: apigateway.amazonaws.com
        SourceArn: !GetAtt [CodeCommitRepo, Arn]

    ArtifactBucket:
      Type: AWS::S3::Bucket
      DeletionPolicy: Retain

    ToolsCodeCommitRole:
      Type: AWS::IAM::Role
      Properties:
        RoleName: !Sub ${ProjectName}-ToolsAcctCodePipelineCodeCommitRole
        AssumeRolePolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Principal:
                AWS:
                  - !Ref AWS::AccountId
              Action:
                - sts:AssumeRole
        Path: /

    Policy:
        Type: AWS::IAM::Policy
        Properties:
          PolicyName: !Sub ${ProjectName}-ToolsAcctCodePipelineCodeCommitPolicy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - codecommit:BatchGetRepositories
                  - codecommit:Get*
                  - codecommit:GitPull
                  - codecommit:List*
                  - codecommit:CancelUploadArchive
                  - codecommit:UploadArchive
                  - s3:*
                Resource: "*"
          Roles:
            - !Ref ToolsCodeCommitRole
    BuildProjectRole:
      Type: AWS::IAM::Role
      Properties:
        RoleName: !Sub ${ProjectName}-CodeBuildRole
        AssumeRolePolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - codebuild.amazonaws.com
              Action:
                - sts:AssumeRole
        Path: /

    BuildProjectPolicy:
      Type: AWS::IAM::Policy
      Properties:
        PolicyName: !Sub ${ProjectName}-CodeBuildPolicy
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action:
                - codecommit:BatchGetRepositories
                - codecommit:Get*
                - codecommit:GitPull
                - codecommit:List*
                - codecommit:CancelUploadArchive
                - codecommit:UploadArchive
                - s3:PutObject
                - s3:GetBucketPolicy
                - s3:GetObject
                - s3:ListBucket
              Resource:
                - !Join ['',['arn:aws:s3:::',!Ref ArtifactBucket, '/*']]
                - !Join ['',['arn:aws:s3:::',!Ref ArtifactBucket]]
                - !GetAtt [CodeCommitRepo, Arn]
            - Effect: Allow
              Action:
                - logs:CreateLogGroup
                - logs:CreateLogStream
                - logs:PutLogEvents
              Resource: arn:aws:logs:*:*:*
            - Effect: Allow
              Action:
                - "*"
              Resource: "*"
        Roles:
          - !Ref BuildProjectRole
    BuildProject:
      Type: AWS::CodeBuild::Project
      Properties:
        Name: !Ref ProjectName
        Description: !Ref ProjectName
        ServiceRole: !GetAtt BuildProjectRole.Arn
        Artifacts:
          Type: CODEPIPELINE
        Environment:
          Type: linuxContainer
          ComputeType: BUILD_GENERAL1_SMALL
          Image: aws/codebuild/python:3.7.1-1.7.0
          EnvironmentVariables:
            - Name: S3Bucket
              Value: !Ref ArtifactBucket
        Source:
          Type: CODEPIPELINE
          BuildSpec: |
            version: 0.1
            phases:
              install:
                commands:
                  - printenv
                  - ls -R
                  - pip install -r requirements.txt -t "$PWD"
              build:
                commands:
                  - aws cloudformation package --template-file templates/databases.yaml --s3-bucket $S3Bucket --s3-prefix databases/codebuild --output-template-file databases.yaml
                  - aws cloudformation package --template-file templates/functions.yaml --s3-bucket $S3Bucket --s3-prefix functions/codebuild --output-template-file functions.yaml
                  - aws cloudformation package --template-file templates/api.yaml --s3-bucket $S3Bucket --s3-prefix api/codebuild --output-template-file api.yaml
            artifacts:
              files:
                - templateConfigurations/*
                - databases.yaml
                - functions.yaml
                - api.yaml
              discard-paths: yes

        TimeoutInMinutes: 10
        Tags:
          - Key: Name
            Value: !Ref ProjectName
    PipeLineRole:
      Type: AWS::IAM::Role
      Properties:
        RoleName: !Sub ${ProjectName}-codepipeline-role
        AssumeRolePolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - codepipeline.amazonaws.com
              Action:
                - sts:AssumeRole
        Path: /

    PipelinePolicy:
      Type: AWS::IAM::Policy
      Properties:
        PolicyName: !Sub ${ProjectName}-codepipeline-policy
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action:
                - codepipeline:*
                - iam:ListRoles
                - cloudformation:Describe*
                - cloudFormation:List*
                - codecommit:List*
                - codecommit:Get*
                - codecommit:GitPull
                - codecommit:UploadArchive
                - codecommit:CancelUploadArchive
                - codebuild:BatchGetBuilds
                - codebuild:StartBuild
                - cloudformation:CreateStack
                - cloudformation:DeleteStack
                - cloudformation:DescribeStacks
                - cloudformation:UpdateStack
                - cloudformation:CreateChangeSet
                - cloudformation:DeleteChangeSet
                - cloudformation:DescribeChangeSet
                - cloudformation:ExecuteChangeSet
                - cloudformation:SetStackPolicy
                - cloudformation:ValidateTemplate
                - iam:PassRole
                - lambda:*
                - s3:ListAllMyBuckets
                - s3:GetBucketLocation
              Resource:
                - "*"
            - Effect: Allow
              Action:
                - s3:PutObject
                - s3:GetBucketPolicy
                - s3:GetObject
                - s3:ListBucket
              Resource:
                - !Join ['',['arn:aws:s3:::',!Ref ArtifactBucket, '/*']]
                - !Join ['',['arn:aws:s3:::',!Ref ArtifactBucket]]

        Roles:
          - !Ref PipeLineRole
    Pipeline:
        Type: AWS::CodePipeline::Pipeline
        Properties:
          RoleArn: !GetAtt PipeLineRole.Arn
          Name: !Ref AWS::StackName
          Stages:
            - Name: Source
              Actions:
                - Name: App
                  ActionTypeId:
                    Category: Source
                    Owner: AWS
                    Version: 1
                    Provider: CodeCommit
                  Configuration:
                    RepositoryName: !Ref ProjectName
                    BranchName: master
                  OutputArtifacts:
                    - Name: SCCheckoutArtifact
                  RunOrder: 1
            - Name: Build
              Actions:
                - Name: Build
                  ActionTypeId:
                    Category: Build
                    Owner: AWS
                    Version: 1
                    Provider: CodeBuild
                  Configuration:
                    ProjectName: !Ref BuildProject
                  RunOrder: 1
                  InputArtifacts:
                    - Name: SCCheckoutArtifact
                  OutputArtifacts:
                    - Name: BuildOutput
            - Name: DeployToTest
              Actions:
                ##### DYNAMO START #####
                - Name: CreateChangeSetTestDynamo
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: databases-test
                    ActionMode: CHANGE_SET_REPLACE
                    StackName: databases-test
                    Capabilities: CAPABILITY_NAMED_IAM
                    TemplatePath: BuildOutput::databases.yaml
                    TemplateConfiguration: "BuildOutput::preProd.json"
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 1
                - Name: DeployChangeSetTestDynamo
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: databases-test
                    ActionMode: CHANGE_SET_EXECUTE
                    StackName: databases-test
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 2
                ##### DYNAMO END #####
                ##### LAMBDA START #####
                - Name: CreateChangeSetTestLambda
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: lambda-test
                    ActionMode: CHANGE_SET_REPLACE
                    StackName: lambda-test
                    Capabilities: CAPABILITY_NAMED_IAM
                    TemplatePath: BuildOutput::functions.yaml
                    TemplateConfiguration: "BuildOutput::preProd.json"
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 3
                - Name: DeployChangeSetTestLambda
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: lambda-test
                    ActionMode: CHANGE_SET_EXECUTE
                    StackName: lambda-test
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 4
                ##### LAMBDA END #####
                ##### API START #####
                - Name: CreateChangeSetTestAPI
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: api-test
                    ActionMode: CHANGE_SET_REPLACE
                    StackName: api-test
                    Capabilities: CAPABILITY_NAMED_IAM
                    TemplatePath: BuildOutput::api.yaml
                    TemplateConfiguration: "BuildOutput::preProd.json"
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 5
                - Name: DeployChangeSetTestAPI
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: api-test
                    ActionMode: CHANGE_SET_EXECUTE
                    StackName: api-test
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 6
          ArtifactStore:
            Type: S3
            Location: !Ref ArtifactBucket

Lambda Code

import boto3
import os


def lambda_handler(event, context):
    print('Starting a new build ...')
    cb = boto3.client('codebuild')
    build = {
        'projectName': event['Records'][0]['customData'],
        'sourceVersion': event['Records'][0]['codecommit']['references'][0]['commit'],
        'artifactsOverride': {
                                'type': 'S3',
                                'location': os.environ['ARTIFACT_BUCKET_NAME'],
                                'path': 'builds/',
                                'name': event['Records'][0]['codecommit']['references'][0]['commit'],
                                'packaging': 'ZIP',
                                'overrideArtifactName': True
                            },
        'sourceTypeOverride': 'CODECOMMIT',
        'sourceLocationOverride': os.environ['SOURCE_LOCATION'],
    }
    print('Starting build for project {0} from commit ID {1}'.format(build['projectName'], build['sourceVersion']))
    response = cb.start_build(**build)
    print('Successfully launched a new CodeBuild project build!')
    return
Comments