Reduce you AWS bill by 10% with this one easy trick, Cloud Economists Hate it

On Thursday, the UK did what UKs are want to do lately. They used Brexit to draw fresh criticisms over seemingly simple tasks. This week was the Parliament Website Edition. The surge of traffic of people wanting up to the minute signature counts crippled their site. My friend James Beswick created a cool application to demonstrate how to properly build a web application that scales dynamically with the request load. While I consider James a friend, his treachery cannot go unmentioned. James committed the most unforgivable of serverlessing crimes; he wrote more code than he needed to and now owns more stuff than he needs to provide the same amount of value.

A quick overview of James' architecture looks like this:

(1) A request comes in to API Gateway, and is proxied to AWS Lambda, (2) AWS Lambda reads the appropriate values from a DynamoDB Table, (3) then the Lambda Function returns the information to the client.

Now that API Gateway supports direct integration with AWS Services, we can eliminate the Lambda completely. and create an architecture that looks more like this:

Without a Lambda Function getting in the way, we see improved performance for our clients​

Latency with Lambda: 
  min: 168.1
  max: 3314.7
  median 242.6
  p95: 336.1
  p99 544.8
Latency without Lambda:
  min: 33.7
  max: 4432
  median: 45
  p95: 86.2
  p99: 886.1

But a far more compelling argument for the simpler architecture is cost. Given the following pricing guides from AWS and our hypothetical scenario, we see a non-trivial cost savings by eliminating Lambda.

API Gateway:
  $3.50/Million requests
  $0.020/hr for 0.5GB Cache on Deployment Stage
Lambda:
  $2.08*10^-7 per 100ms of request time on a 128MB Function
DynamoDB:
  $1.25/Million Writes
  $0.25/Million Reads

If we assume our application will receive 10M requests in the first 24 hours (0.5 Million votes cast and 9.5 Million page refreshes) and our API caches read requests for 5 seconds before the TTL expires. The Lambda-based architecture costs about $41.13 for the first 24 hours, the Lambdaless architecture costs about $36.16 over the same period.

This is where a less scrupulous evangelist would offer to help you eliminate all of your Lambda Functions for 'moderate' consulting fee. But, if we take a closer look we see that by simply enabling the API Gateway Cache on the Deployment Stage, the Dynamo Table RCU cost is almost completely eliminated and offsets the price of the Lambda. No huge refactoring required, no unfamiliar integration changes, just checking one box reduces you AWS Bill by 10%

Comments

[DRAFT] CI/CD With CodeCommit

I recently came across a Medium post that claimed "Even though we really like AWS’ services we admit that CodeCommit (Git) and CodePipeline (CI/CD) are not ready for production use, yet. The problem of CodePipeline is that you need to create a pipeline for every branch. You cannot create it easily automatically without running CloudFormation." Which is half true. The part that is true is that CodePipeline is terrible at doing Code Builds that aren't meant to be deployed. If only there was some service in the Code Suite that allowed you to Build Code. If they ever make such a service, I hope it would be name CodeBuilder or BuildCoder, I would hope that this service is capable of running unit/integration tests for any commit on any branch.

Fast forward -835 days and AWS Announces the launch of AWS CodeBuild.

In this post, I will show you how to build a CI/CD pipeline with AWS CodeCommit, CodeBuild, CodePipeline, Lambda, and SNS that will ensure your build process runs for every commit. Here is the high-level overview of what we're going to build.

CFN Template

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: CICD Pipeline that builds every commit on every branch and only merges passing builds to mainline

Parameters:
  ProjectName:
    Description: Name for this project
    Type: String
    Default: "myProject"
Resources:

    CodeCommitRepo:
      Type: AWS::CodeCommit::Repository
      Properties:
        RepositoryDescription: my super awesome repo
        RepositoryName: !Ref ProjectName
        Triggers:
          - DestinationArn: !GetAtt [HelloWorldFunction, Arn]
            CustomData: !Sub ${ProjectName}
            Events:
              - all
            Name: SendToBuild
    CloudWatchEventRule:
      Type: AWS::Events::Rule
      Properties:
        Description: "Rule to Fire SNS Message"
        EventPattern: !Sub |
            {
              "source": [
                "aws.codebuild"
              ],
              "detail-type": [
                "CodeBuild Build State Change"
              ],
              "detail": {
                "build-status": [
                  "IN_PROGRESS",
                  "SUCCEEDED",
                  "FAILED",
                  "STOPPED"
                ],
                "project-name": [
                  "${ProjectName}"
                ]
              }
            }
        State: ENABLED
        Targets:
         - Arn: !Ref DeveloperNotification
           Id: "SNSTopic"
    DeveloperNotification:
      Type: AWS::SNS::Topic
      Properties:
        DisplayName: "Build Notifier"

    SNSPolicy:
      Type: AWS::SNS::TopicPolicy
      Properties:
        PolicyDocument:
          Statement:
            Sid: "MYSID001"
            Effect: "Allow"
            Principal:
              Service: events.amazonaws.com
            Action: sns:Publish
            Resource: !Ref DeveloperNotification
        Topics:
          - !Ref DeveloperNotification

    HelloWorldFunction:
        Type: AWS::Serverless::Function
        Properties:
            CodeUri: ../commit_builder/
            Handler: app.lambda_handler
            Runtime: python3.7
            Timeout: 30
            Policies:
              - AWSLambdaBasicExecutionRole
              - Version: '2012-10-17'
                Statement:
                  - Effect: Allow
                    Action: "*"
                    Resource: "*"
            Environment:
              Variables:
                BUILD_PROJECT_NAME: !Ref ProjectName
                ARTIFACT_BUCKET_NAME: !Ref ArtifactBucket
                SOURCE_LOCATION: !Sub https://git-codecommit.${AWS::Region}.amazonaws.com/v1/repos/${ProjectName}

    InvokePermission:
      Type: AWS::Lambda::Permission
      Properties:
        FunctionName: !Ref HelloWorldFunction
        Action: 'lambda:InvokeFunction'
        Principal: apigateway.amazonaws.com
        SourceArn: !GetAtt [CodeCommitRepo, Arn]

    ArtifactBucket:
      Type: AWS::S3::Bucket
      DeletionPolicy: Retain

    ToolsCodeCommitRole:
      Type: AWS::IAM::Role
      Properties:
        RoleName: !Sub ${ProjectName}-ToolsAcctCodePipelineCodeCommitRole
        AssumeRolePolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Principal:
                AWS:
                  - !Ref AWS::AccountId
              Action:
                - sts:AssumeRole
        Path: /

    Policy:
        Type: AWS::IAM::Policy
        Properties:
          PolicyName: !Sub ${ProjectName}-ToolsAcctCodePipelineCodeCommitPolicy
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - codecommit:BatchGetRepositories
                  - codecommit:Get*
                  - codecommit:GitPull
                  - codecommit:List*
                  - codecommit:CancelUploadArchive
                  - codecommit:UploadArchive
                  - s3:*
                Resource: "*"
          Roles:
            - !Ref ToolsCodeCommitRole
    BuildProjectRole:
      Type: AWS::IAM::Role
      Properties:
        RoleName: !Sub ${ProjectName}-CodeBuildRole
        AssumeRolePolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - codebuild.amazonaws.com
              Action:
                - sts:AssumeRole
        Path: /

    BuildProjectPolicy:
      Type: AWS::IAM::Policy
      Properties:
        PolicyName: !Sub ${ProjectName}-CodeBuildPolicy
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action:
                - codecommit:BatchGetRepositories
                - codecommit:Get*
                - codecommit:GitPull
                - codecommit:List*
                - codecommit:CancelUploadArchive
                - codecommit:UploadArchive
                - s3:PutObject
                - s3:GetBucketPolicy
                - s3:GetObject
                - s3:ListBucket
              Resource:
                - !Join ['',['arn:aws:s3:::',!Ref ArtifactBucket, '/*']]
                - !Join ['',['arn:aws:s3:::',!Ref ArtifactBucket]]
                - !GetAtt [CodeCommitRepo, Arn]
            - Effect: Allow
              Action:
                - logs:CreateLogGroup
                - logs:CreateLogStream
                - logs:PutLogEvents
              Resource: arn:aws:logs:*:*:*
            - Effect: Allow
              Action:
                - "*"
              Resource: "*"
        Roles:
          - !Ref BuildProjectRole
    BuildProject:
      Type: AWS::CodeBuild::Project
      Properties:
        Name: !Ref ProjectName
        Description: !Ref ProjectName
        ServiceRole: !GetAtt BuildProjectRole.Arn
        Artifacts:
          Type: CODEPIPELINE
        Environment:
          Type: linuxContainer
          ComputeType: BUILD_GENERAL1_SMALL
          Image: aws/codebuild/python:3.7.1-1.7.0
          EnvironmentVariables:
            - Name: S3Bucket
              Value: !Ref ArtifactBucket
        Source:
          Type: CODEPIPELINE
          BuildSpec: |
            version: 0.1
            phases:
              install:
                commands:
                  - printenv
                  - ls -R
                  - pip install -r requirements.txt -t "$PWD"
              build:
                commands:
                  - aws cloudformation package --template-file templates/databases.yaml --s3-bucket $S3Bucket --s3-prefix databases/codebuild --output-template-file databases.yaml
                  - aws cloudformation package --template-file templates/functions.yaml --s3-bucket $S3Bucket --s3-prefix functions/codebuild --output-template-file functions.yaml
                  - aws cloudformation package --template-file templates/api.yaml --s3-bucket $S3Bucket --s3-prefix api/codebuild --output-template-file api.yaml
            artifacts:
              files:
                - templateConfigurations/*
                - databases.yaml
                - functions.yaml
                - api.yaml
              discard-paths: yes

        TimeoutInMinutes: 10
        Tags:
          - Key: Name
            Value: !Ref ProjectName
    PipeLineRole:
      Type: AWS::IAM::Role
      Properties:
        RoleName: !Sub ${ProjectName}-codepipeline-role
        AssumeRolePolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Principal:
                Service:
                  - codepipeline.amazonaws.com
              Action:
                - sts:AssumeRole
        Path: /

    PipelinePolicy:
      Type: AWS::IAM::Policy
      Properties:
        PolicyName: !Sub ${ProjectName}-codepipeline-policy
        PolicyDocument:
          Version: 2012-10-17
          Statement:
            - Effect: Allow
              Action:
                - codepipeline:*
                - iam:ListRoles
                - cloudformation:Describe*
                - cloudFormation:List*
                - codecommit:List*
                - codecommit:Get*
                - codecommit:GitPull
                - codecommit:UploadArchive
                - codecommit:CancelUploadArchive
                - codebuild:BatchGetBuilds
                - codebuild:StartBuild
                - cloudformation:CreateStack
                - cloudformation:DeleteStack
                - cloudformation:DescribeStacks
                - cloudformation:UpdateStack
                - cloudformation:CreateChangeSet
                - cloudformation:DeleteChangeSet
                - cloudformation:DescribeChangeSet
                - cloudformation:ExecuteChangeSet
                - cloudformation:SetStackPolicy
                - cloudformation:ValidateTemplate
                - iam:PassRole
                - lambda:*
                - s3:ListAllMyBuckets
                - s3:GetBucketLocation
              Resource:
                - "*"
            - Effect: Allow
              Action:
                - s3:PutObject
                - s3:GetBucketPolicy
                - s3:GetObject
                - s3:ListBucket
              Resource:
                - !Join ['',['arn:aws:s3:::',!Ref ArtifactBucket, '/*']]
                - !Join ['',['arn:aws:s3:::',!Ref ArtifactBucket]]

        Roles:
          - !Ref PipeLineRole
    Pipeline:
        Type: AWS::CodePipeline::Pipeline
        Properties:
          RoleArn: !GetAtt PipeLineRole.Arn
          Name: !Ref AWS::StackName
          Stages:
            - Name: Source
              Actions:
                - Name: App
                  ActionTypeId:
                    Category: Source
                    Owner: AWS
                    Version: 1
                    Provider: CodeCommit
                  Configuration:
                    RepositoryName: !Ref ProjectName
                    BranchName: master
                  OutputArtifacts:
                    - Name: SCCheckoutArtifact
                  RunOrder: 1
            - Name: Build
              Actions:
                - Name: Build
                  ActionTypeId:
                    Category: Build
                    Owner: AWS
                    Version: 1
                    Provider: CodeBuild
                  Configuration:
                    ProjectName: !Ref BuildProject
                  RunOrder: 1
                  InputArtifacts:
                    - Name: SCCheckoutArtifact
                  OutputArtifacts:
                    - Name: BuildOutput
            - Name: DeployToTest
              Actions:
                ##### DYNAMO START #####
                - Name: CreateChangeSetTestDynamo
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: databases-test
                    ActionMode: CHANGE_SET_REPLACE
                    StackName: databases-test
                    Capabilities: CAPABILITY_NAMED_IAM
                    TemplatePath: BuildOutput::databases.yaml
                    TemplateConfiguration: "BuildOutput::preProd.json"
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 1
                - Name: DeployChangeSetTestDynamo
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: databases-test
                    ActionMode: CHANGE_SET_EXECUTE
                    StackName: databases-test
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 2
                ##### DYNAMO END #####
                ##### LAMBDA START #####
                - Name: CreateChangeSetTestLambda
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: lambda-test
                    ActionMode: CHANGE_SET_REPLACE
                    StackName: lambda-test
                    Capabilities: CAPABILITY_NAMED_IAM
                    TemplatePath: BuildOutput::functions.yaml
                    TemplateConfiguration: "BuildOutput::preProd.json"
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 3
                - Name: DeployChangeSetTestLambda
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: lambda-test
                    ActionMode: CHANGE_SET_EXECUTE
                    StackName: lambda-test
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 4
                ##### LAMBDA END #####
                ##### API START #####
                - Name: CreateChangeSetTestAPI
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: api-test
                    ActionMode: CHANGE_SET_REPLACE
                    StackName: api-test
                    Capabilities: CAPABILITY_NAMED_IAM
                    TemplatePath: BuildOutput::api.yaml
                    TemplateConfiguration: "BuildOutput::preProd.json"
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 5
                - Name: DeployChangeSetTestAPI
                  ActionTypeId:
                    Category: Deploy
                    Owner: AWS
                    Version: 1
                    Provider: CloudFormation
                  Configuration:
                    ChangeSetName: api-test
                    ActionMode: CHANGE_SET_EXECUTE
                    StackName: api-test
                  InputArtifacts:
                    - Name: BuildOutput
                  RunOrder: 6
          ArtifactStore:
            Type: S3
            Location: !Ref ArtifactBucket

Lambda Code

import boto3
import os


def lambda_handler(event, context):
    print('Starting a new build ...')
    cb = boto3.client('codebuild')
    build = {
        'projectName': event['Records'][0]['customData'],
        'sourceVersion': event['Records'][0]['codecommit']['references'][0]['commit'],
        'artifactsOverride': {
                                'type': 'S3',
                                'location': os.environ['ARTIFACT_BUCKET_NAME'],
                                'path': 'builds/',
                                'name': event['Records'][0]['codecommit']['references'][0]['commit'],
                                'packaging': 'ZIP',
                                'overrideArtifactName': True
                            },
        'sourceTypeOverride': 'CODECOMMIT',
        'sourceLocationOverride': os.environ['SOURCE_LOCATION'],
    }
    print('Starting build for project {0} from commit ID {1}'.format(build['projectName'], build['sourceVersion']))
    response = cb.start_build(**build)
    print('Successfully launched a new CodeBuild project build!')
    return
Comments

Mastering API Gateway With One Weird Trick

So, I set out on a journey to create an API Gateway to Service integration for all 105 services listed in the API Gateway Console. After doing about 8 services I cracked the rosetta stone of API Gateway and here I'll show you how you can think about the way the services are called via their RESTful API to give a more consistent experience across services. One of the key insights from my previous post was that the aws cli somehow "knew" to add the x-amz-json-1.1 Content-Type header to the request. After doing some digging, with a lot of help from Ben K., we discovered how the cli "knew" this. This link in the botocore data directory shows where this data is loaded from.

This got me thinking about the differences between DynamoDB and Athena integration so I fired up the cli again to see how it interacts with DynamoDB

This is really interesting because the API Gateway integration uses the pattern of applying the QueryString parameter "?Action=ListTables" and doesn't include the headers. From the looks of this, we might be able to apply the same technique header-based technique to every service. This way we don't have to worry about which services support ?Actions=[....] query strings and which services don't. Let's walk through a concrete example to make this a bit easier to digest. I'm going to pick a service at random and try to duplicate this with a cloudformation template.

import glob
import random
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
sh = logging.StreamHandler()
logger.addHandler(sh)


def main():
    services = glob.glob('./data/*/*/service-2.json')
    logger.debug("found {} service files".format(len(services)))
    print(random.choice(services))


if __name__ == "__main__":
    main()

Running this gives us ...... *drumroll* .........

Cool. Let's see what we can do with Route53.

Crap. it's not using json at all. No worries though, because we're about to learn the trick. There's only about 4 protocols I've seen so far ['json', 'query', and 'rest-xml'] but I'm sure there's more hiding in there somewhere. This metadata tells us almost everything we need to know. Since there is no field called "targetPrefix" we know that we won't have to add the x-amz-target or application/x-amz-json-1.x headers. Let's try a list command with no parameters to see if we can get the connectivity working, then we'll add some parameters. Let's take a closer look at the ListHostedZones operation.

This tells us the path we want to use '/2014-04-01/hostedzone' and what HTTP method to use in the Integration Request (Note that this can be and often is different from the Method Request in API Gateway). The object also tells us what the input should look like. In this case, we're sending a get request so we don't have to worry about a request body and none of the parameters are required so we can ignore them for now. Here's the template we are going to deploy. I've bolded the parts where we deviate from a standard API Gateway service integration template.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
    Route53 Integration

Resources:
    Route53API:
      Type: AWS::ApiGateway::RestApi
      Properties:
        Name: "Route53 API"
    RootResource:
      Type: AWS::ApiGateway::Resource
      Properties:
        ParentId: !GetAtt Route53API.RootResourceId
        PathPart: "zones"
        RestApiId: !Ref Route53API
    ListHostedZonesMethod:
      Type: AWS::ApiGateway::Method
      DependsOn:
        - RootResource
      Properties:
        HttpMethod: GET
        ResourceId: !Ref RootResource
        RestApiId: !Ref Route53API
        AuthorizationType: AWS_IAM
        MethodResponses:
          - ResponseParameters:
              method.response.header.Access-Control-Allow-Origin: true
            StatusCode: 200
        Integration:
          Type: AWS
          IntegrationHttpMethod: GET
          Credentials:
            Fn::GetAtt:
              - AdminRole
              - Arn
          Uri: !Sub "arn:aws:apigateway:${AWS::Region}:route53:path/2013-04-01/hostedzone"
          IntegrationResponses:
            - StatusCode: 200
    AdminRole:
      Type: "AWS::IAM::Role"
      Properties:
        AssumeRolePolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: "Allow"
              Principal:
                Service:
                  - "apigateway.amazonaws.com"
              Action:
                - "sts:AssumeRole"
        Path: "/"
        ManagedPolicyArns:
          - arn:aws:iam::aws:policy/AdministratorAccess

Let's deploy this and see what happens.

Cool, but I might be lying to you or API Gateway might be lying to you, so we should verify this by implementing some kind of mutating request so we can verify it. Since we have a ListHostedZones, let's look at the CreateHostedZone operation.

Repeating what we did last time, we need to make note of the method, requestUri, and now the input shape.

This tells us that "Name" and "CallerReference" are the only required fields (note: some services don't include a "required" field and instead add the word "Required" to the beginning of the documentation). The CallerReference is an interesting field. Its purpose is so that you can make the same request many times and be sure that the creation is idempotent, which is great if you encounter errors and want to ensure that requesting an item twice doesn't create two identical items. Here's the new API gateway Cloudformation template, with the important parts bolded. Notice that we map the application/json content-type that we expect from our clients to the text/xml that route53 expects based on the metadata["protocol"] entry in the service-2.json file linked above.

    CreateHostedZonesMethod:
      Type: AWS::ApiGateway::Method
      DependsOn:
        - RootResource
      Properties:
        HttpMethod: POST
        ResourceId: !Ref RootResource
        RestApiId: !Ref Route53API
        AuthorizationType: AWS_IAM
        MethodResponses:
          - ResponseParameters:
              method.response.header.Access-Control-Allow-Origin: true
            StatusCode: 200
        Integration:
          Type: AWS
          IntegrationHttpMethod: POST
          RequestParameters:
            integration.request.header.Content-Type: "'text/xml'"
          Credentials:
            Fn::GetAtt:
              - AdminRole
              - Arn
          Uri: !Sub "arn:aws:apigateway:${AWS::Region}:route53:path/2013-04-01/hostedzone"
          RequestTemplates:
            application/json: |
              <CreateHostedZoneRequest xmlns="https://route53.amazonaws.com/doc/2013-04-01/">
                <Name>$input.params('zoneName').</Name>
                <CallerReference>$input.params('user-provided-request-id')</CallerReference>
              </CreateHostedZoneRequest>
          IntegrationResponses:
            - StatusCode: 200

We see that this takes the "zoneName" and "user-provided-request-id" querystring parameters and maps them into the xml of the request. Sending this request with the querystring "zoneName=RichardBoydIsCool.com&user-provided-request-id=19640112" looks like a success

Awesome. Now let's try the "ListHostedZones" that we created earlier and make sure our new HostedZone shows up.

Great!!!! Now we know how to integrate with any service that API Gateway advertises. Stay tuned next week for another post that will make this even easier ;)

Comments