Reduce you AWS bill by 10% with this one easy trick, Cloud Economists Hate it

On Thursday, the UK did what UKs are want to do lately. They used Brexit to draw fresh criticisms over seemingly simple tasks. This week was the Parliament Website Edition. The surge of traffic of people wanting up to the minute signature counts crippled their site. My friend James Beswick created a cool application to demonstrate how to properly build a web application that scales dynamically with the request load. While I consider James a friend, his treachery cannot go unmentioned. James committed the most unforgivable of serverlessing crimes; he wrote more code than he needed to and now owns more stuff than he needs to provide the same amount of value.

A quick overview of James' architecture looks like this:

(1) A request comes in to API Gateway, and is proxied to AWS Lambda, (2) AWS Lambda reads the appropriate values from a DynamoDB Table, (3) then the Lambda Function returns the information to the client.

Now that API Gateway supports direct integration with AWS Services, we can eliminate the Lambda completely. and create an architecture that looks more like this:

Without a Lambda Function getting in the way, we see improved performance for our clients​

Latency with Lambda: 
  min: 168.1
  max: 3314.7
  median 242.6
  p95: 336.1
  p99 544.8
Latency without Lambda:
  min: 33.7
  max: 4432
  median: 45
  p95: 86.2
  p99: 886.1

But a far more compelling argument for the simpler architecture is cost. Given the following pricing guides from AWS and our hypothetical scenario, we see a non-trivial cost savings by eliminating Lambda.

API Gateway:
  $3.50/Million requests
  $0.020/hr for 0.5GB Cache on Deployment Stage
  $2.08*10^-7 per 100ms of request time on a 128MB Function
  $1.25/Million Writes
  $0.25/Million Reads

If we assume our application will receive 10M requests in the first 24 hours (0.5 Million votes cast and 9.5 Million page refreshes) and our API caches read requests for 5 seconds before the TTL expires. The Lambda-based architecture costs about $41.13 for the first 24 hours, the Lambdaless architecture costs about $36.16 over the same period.

This is where a less scrupulous evangelist would offer to help you eliminate all of your Lambda Functions for 'moderate' consulting fee. But, if we take a closer look we see that by simply enabling the API Gateway Cache on the Deployment Stage, the Dynamo Table RCU cost is almost completely eliminated and offsets the price of the Lambda. No huge refactoring required, no unfamiliar integration changes, just checking one box reduces you AWS Bill by 10%


[DRAFT] CI/CD With CodeCommit

I recently came across a Medium post that claimed "Even though we really like AWS’ services we admit that CodeCommit (Git) and CodePipeline (CI/CD) are not ready for production use, yet. The problem of CodePipeline is that you need to create a pipeline for every branch. You cannot create it easily automatically without running CloudFormation." Which is half true. The part that is true is that CodePipeline is terrible at doing Code Builds that aren't meant to be deployed. If only there was some service in the Code Suite that allowed you to Build Code. If they ever make such a service, I hope it would be name CodeBuilder or BuildCoder, I would hope that this service is capable of running unit/integration tests for any commit on any branch.

Fast forward -835 days and AWS Announces the launch of AWS CodeBuild.

In this post, I will show you how to build a CI/CD pipeline with AWS CodeCommit, CodeBuild, CodePipeline, Lambda, and SNS that will ensure your build process runs for every commit. Here is the high-level overview of what we're going to build.

CFN Template

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: CICD Pipeline that builds every commit on every branch and only merges passing builds to mainline

    Description: Name for this project
    Type: String
    Default: "myProject"

      Type: AWS::CodeCommit::Repository
        RepositoryDescription: my super awesome repo
        RepositoryName: !Ref ProjectName
          - DestinationArn: !GetAtt [HelloWorldFunction, Arn]
            CustomData: !Sub ${ProjectName}
              - all
            Name: SendToBuild
      Type: AWS::Events::Rule
        Description: "Rule to Fire SNS Message"
        EventPattern: !Sub |
              "source": [
              "detail-type": [
                "CodeBuild Build State Change"
              "detail": {
                "build-status": [
                "project-name": [
        State: ENABLED
         - Arn: !Ref DeveloperNotification
           Id: "SNSTopic"
      Type: AWS::SNS::Topic
        DisplayName: "Build Notifier"

      Type: AWS::SNS::TopicPolicy
            Sid: "MYSID001"
            Effect: "Allow"
            Action: sns:Publish
            Resource: !Ref DeveloperNotification
          - !Ref DeveloperNotification

        Type: AWS::Serverless::Function
            CodeUri: ../commit_builder/
            Handler: app.lambda_handler
            Runtime: python3.7
            Timeout: 30
              - AWSLambdaBasicExecutionRole
              - Version: '2012-10-17'
                  - Effect: Allow
                    Action: "*"
                    Resource: "*"
                BUILD_PROJECT_NAME: !Ref ProjectName
                ARTIFACT_BUCKET_NAME: !Ref ArtifactBucket
                SOURCE_LOCATION: !Sub https://git-codecommit.${AWS::Region}${ProjectName}

      Type: AWS::Lambda::Permission
        FunctionName: !Ref HelloWorldFunction
        Action: 'lambda:InvokeFunction'
        SourceArn: !GetAtt [CodeCommitRepo, Arn]


Mastering API Gateway With One Weird Trick

So, I set out on a journey to create an API Gateway to Service integration for all 105 services listed in the API Gateway Console. After doing about 8 services I cracked the rosetta stone of API Gateway and here I'll show you how you can think about the way the services are called via their RESTful API to give a more consistent experience across services. One of the key insights from my previous post was that the aws cli somehow "knew" to add the x-amz-json-1.1 Content-Type header to the request. After doing some digging, with a lot of help from Ben K., we discovered how the cli "knew" this. This link in the botocore data directory shows where this data is loaded from.

This got me thinking about the differences between DynamoDB and Athena integration so I fired up the cli again to see how it interacts with DynamoDB

This is really interesting because the API Gateway integration uses the pattern of applying the QueryString parameter "?Action=ListTables" and doesn't include the headers. From the looks of this, we might be able to apply the same technique header-based technique to every service. This way we don't have to worry about which services support ?Actions=[....] query strings and which services don't. Let's walk through a concrete example to make this a bit easier to digest. I'm going to pick a service at random and try to duplicate this with a cloudformation template.

import glob
import random
import logging
logger = logging.getLogger()
sh = logging.StreamHandler()

def main():
    services = glob.glob('./data/*/*/service-2.json')
    logger.debug("found {} service files".format(len(services)))

if __name__ == "__main__":

Running this gives us ...... *drumroll* .........

Cool. Let's see what we can do with Route53.

Crap. it's not using json at all. No worries though, because we're about to learn the trick. There's only about 4 protocols I've seen so far ['json', 'query', and 'rest-xml'] but I'm sure there's more hiding in there somewhere. This metadata tells us almost everything we need to know. Since there is no field called "targetPrefix" we know that we won't have to add the x-amz-target or application/x-amz-json-1.x headers. Let's try a list command with no parameters to see if we can get the connectivity working, then we'll add some parameters. Let's take a closer look at the ListHostedZones operation.

This tells us the path we want to use '/2014-04-01/hostedzone' and what HTTP method to use in the Integration Request (Note that this can be and often is different from the Method Request in API Gateway). The object also tells us what the input should look like. In this case, we're sending a get request so we don't have to worry about a request body and none of the parameters are required so we can ignore them for now. Here's the template we are going to deploy. I've bolded the parts where we deviate from a standard API Gateway service integration template.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
    Route53 Integration

      Type: AWS::ApiGateway::RestApi
        Name: "Route53 API"
      Type: AWS::ApiGateway::Resource
        ParentId: !GetAtt Route53API.RootResourceId
        PathPart: "zones"
        RestApiId: !Ref Route53API
      Type: AWS::ApiGateway::Method
        - RootResource
        HttpMethod: GET
        ResourceId: !Ref RootResource
        RestApiId: !Ref Route53API
        AuthorizationType: AWS_IAM
          - ResponseParameters:
              method.response.header.Access-Control-Allow-Origin: true
            StatusCode: 200
          Type: AWS
          IntegrationHttpMethod: GET
              - AdminRole
              - Arn
          Uri: !Sub "arn:aws:apigateway:${AWS::Region}:route53:path/2013-04-01/hostedzone"
            - StatusCode: 200
      Type: "AWS::IAM::Role"
          Version: "2012-10-17"
            - Effect: "Allow"
                  - ""
                - "sts:AssumeRole"
        Path: "/"
          - arn:aws:iam::aws:policy/AdministratorAccess

Let's deploy this and see what happens.

Cool, but I might be lying to you or API Gateway might be lying to you, so we should verify this by implementing some kind of mutating request so we can verify it. Since we have a ListHostedZones, let's look at the CreateHostedZone operation.

Repeating what we did last time, we need to make note of the method, requestUri, and now the input shape.

This tells us that "Name" and "CallerReference" are the only required fields (note: some services don't include a "required" field and instead add the word "Required" to the beginning of the documentation). The CallerReference is an interesting field. Its purpose is so that you can make the same request many times and be sure that the creation is idempotent, which is great if you encounter errors and want to ensure that requesting an item twice doesn't create two identical items. Here's the new API gateway Cloudformation template, with the important parts bolded. Notice that we map the application/json content-type that we expect from our clients to the text/xml that route53 expects based on the metadata["protocol"] entry in the service-2.json file linked above.

      Type: AWS::ApiGateway::Method
        - RootResource
        HttpMethod: POST
        ResourceId: !Ref RootResource
        RestApiId: !Ref Route53API
        AuthorizationType: AWS_IAM
          - ResponseParameters:
              method.response.header.Access-Control-Allow-Origin: true
            StatusCode: 200
          Type: AWS
          IntegrationHttpMethod: POST
            integration.request.header.Content-Type: "'text/xml'"
              - AdminRole
              - Arn
          Uri: !Sub "arn:aws:apigateway:${AWS::Region}:route53:path/2013-04-01/hostedzone"
            application/json: |
              <CreateHostedZoneRequest xmlns="">
            - StatusCode: 200

We see that this takes the "zoneName" and "user-provided-request-id" querystring parameters and maps them into the xml of the request. Sending this request with the querystring "" looks like a success

Awesome. Now let's try the "ListHostedZones" that we created earlier and make sure our new HostedZone shows up.

Great!!!! Now we know how to integrate with any service that API Gateway advertises. Stay tuned next week for another post that will make this even easier ;)