I recently came across this article on Medium. The author highlights a use-case that gets much less attention than it deserves, connecting API Gateway to services that aren't offered within AWS. In this specific example Brian needs to connect his APIGW endpoint to his BigCommerce endpoint without revealing his BigCommerce credentials to his clients. In the article he uses a Lambda function to add the credentials to the request, modify the path based on a query string, then forward the request to his store's BigCommerce endpoint. If you have followed my blog for more than a few minutes you know how I feel about this pattern. In today's installment, I am going to show you how to accomplish the same integration without requiring a Lambda Function.

First we need to set up a BigCommerce sandbox store. This can be done by signing up for a free 15-day trial here. Once your store is created, we need to set up some credentials. Click on Advanced Settings in the main console.

Then select API Accounts. I have one called "Channel Advisor' because I was playing with it earlier, your list should be blank.

Name your Application and give it the appropriate permissions. I just went down the list clicking all the "read-only" tabs.

Now it will show you your credentials. Don't be like me and show them to the world. I'm a professional. Make sure you save these. Like an AWS Access Key ID and Secret Key, this is the only time you can see them.

Neat. Now you have an application that can talk to your store.

We'll add a product so we can be sure wqe're hitting our store.

And here we see our product in our store's catalog.

Now for the fun part. We are going to create an API that looks just like Brian's from his Medium post.

We start with the basics for our cfn template

AWSTemplateFormatVersion: '2010-09-09'
Description: >
  BigCommerce Proxy API

Now we need to add a REST API

    Type: AWS::ApiGateway::RestApi
      Name: "BigCommerce"
          - REGIONAL

Now let's add a resource to the API. A resource is the path part of the API.

    Type: AWS::ApiGateway::Resource
      ParentId: !GetAtt MyRestAPIForTesting.RootResourceId
      PathPart: "get-variants"
      RestApiId: !Ref MyRestAPIForTesting

Now it get's really exciting. We're going to add the GET method so our clients can call our API

    Type: AWS::ApiGateway::Method
      HttpMethod: GET
      ResourceId: !Ref VariantsResource
      RestApiId: !Ref MyRestAPIForTesting
      AuthorizationType: NONE
        method.request.querystring.product_id: false
        - ResponseParameters:
            method.response.header.Access-Control-Allow-Origin: true
          StatusCode: 200
        IntegrationHttpMethod: GET
        Type: HTTP
          integration.request.header.X-Auth-Token: "'hzkw74s0h0be7mbme5znarql606cd84'"
          integration.request.header.X-Auth-Client: "'r135f84ojwx53si1ty0hsvytczzdj6y'"
          # client Secret: 6cm3y6i63umbc781b5a2pysu4h29hy8 Not sure why they gave me this but whatevs
        PassthroughBehavior: NEVER
          - StatusCode: 200

Here we are taking the product_id query string and using it to replace the {id} variable in the path, as well as adding the auth headers that BigCommerce expects.

Let's deploy this and see what happens.

Crap. I forgot to record the product_id from the server we created. Going back to the storefront and mousing over the product link reveals the product id at the bottom of the page

Now let's go back to APIGW and search for product 113

That looks promising, but it could be any $1,000,000 product. Let's go change the SKU in the product to 1776 and try again and see what happens.

Awesome. Now you can use APIGW to front your BigCommerce store and you have the cloudformation template @ryan_marsh doesn't @ me over this


genetic decipher

I've written fairly extensively on AWS API Gateway (APIGW) and many of the lesser-known features. Today we're going to dive deep on the model validation feature of APIGW and specifically how to handle unexpected input.

The most common use for APIGW is to leverage the "AWS_PROXY" integration that directly connects APIGW to AWS Lambda (this is called "Lambda Function" in the APIGW console). There are several benefits to this approach, including reduced latency, simpler implementation, and more control over the shape of the API response object. Today, I am going to show you how to achieve that same level of control natively with APIGW and use those same techniques to enforce the shape of the request object. We'll wrap up with a discussion of some of the limits of this feature (AWS Tech Evangelists take note!)

Let's start with a brief introduction to the various parts of APIGW. Each API consists of Resources, Methods, Method Request, Integration Request, Integration Response, and Method Response.

* Resources: These are the paths of your API, and API will start with a single resource called the "root" resource and is denoted by "/".

Adding a "child" named "robot" to this resource would look like "/robot".

The robot resource could have "siblings" such as "/mop" or "/vacuum". Likewise, the robot resource could have children of its own, such as "/robot/biped" or "/robot/tracked"

Here is the finished set of resources.

* Methods: methods are the HTTP verbs that describe an interaction with a resource. These are [GET, POST, OPTIONS, DELETE, HEAD, PATCH, PUT, ANY]. Each resource can have any, many, or none of these methods attached to it. One thing to note is that a particular verb can only be attached to a resource once. You cannot have two GETs defined for /robot or three PUTs defined for /mop because when a client makes an HTTP GET request to /robot, there would be no way to know which of the two methods they meant to invoke.

Here we see that the option to add a second GET request is not available. The console is great at preventing this, but if you are using CloudFormation, you could end up trying to deploy a template with conflicting methods and CFN is generally good about telling you which methods are conflicting on which resource within the API.

I've created a sample mock integration to demonstrate some of the next components.

* Method Request: I like to describe the method request as the client-facing incoming side of the APIGW interaction; this is the place where you (the API developer) will define authorizations needed to interact with the resource/method and define expected query strings, headers, and body.

* Integration Request: This (with the Integration Response we will look at shortly) is the meat of an APIGW API. This maps a client's request to an AWS service request by performing transformations on the request body, modifying headers and query strings, and defining what behavior should be performed when an unexpected input is received.

* Integration Response: This is where the response from the AWS service, on-prem server, or other HTTP proxy integration will be transformed into something that the client wants. Here is where you will define what HTTP status codes you expect in response from the integration endpoint and how you want to change your transformation based on those status codes.

* Method Response: Much like the method request, this is the client-facing outgoing side of the APIGW interaction. Here is where you can define specific response "shapes" ("Types" if you come from an typed-language background)

Now that we've covered some of the basics, let's dive into some request/response modeling. Suppose you had an API that was used as a simple CRUD interface for your robot collection. You'd like to be able to add robots to your collection, update their price as they become more rare, query for robots with specific attributes, and delete them when you sell or trade them with other roboticists. When we add a new robot to our collection, we want to make sure it has a serial number and generally looks like how we'd expect a robot's description (hasLaser: True, isCool: True, mass: >=0kg, etc....). If we were using the AWS_PROXY integration we would have to write custom lambda code to parse the request body and validate these fields, co-mingling the request validation logic with our business logic of CRUDing the robots.

Here I've created a simple model for a new robot and mark only the serialNumber field as required. If a mass is supplied, it must be non-negative. Let's add it to an API and poke it a few times to see what happens.

  "$schema" : "",
  "title" : "CreateRobotRequest Schema",
  "type" : "object",
  "required": ["serialNumber"],
  "properties" : {
    "serialNumber" : {
        "type" : "string",
        "minLength": 10
    "mass" : {
        "type" : "number",
        "minimum": 0

I've created a new mock POST method on my robot Resource and added the request model to the Method Request.

An empty request will fail because it is missing the required serialNumber. We see in the Logs dialog box that the required field is missing.

Likewise, adding a serialNumber that is too short will result in a validation failure.

Let's make a request that we expect to pass validation to make sure it's not just rejecting everything.

Great. Now let's get into the fun stuff. within the Integration Request, we can specify how we want the API to perform if the content type provided by the client doesn't match the Content-Types we've specified in our mapping templates. It's interesting to note that the default option is not the recommended option.

Before I can go much further, I have to stop using the mock integrations and the APIGW console for the demo. While the mock integration is great for experimenting or returning hard-coded responses that you're certain will never change, it leaves a bit to be desired that Ben Kehoe describe better than I could several years ago so a fix should be just around the corner. There is also some peculiarities in the way that the console differs from regular cUrl or Postman requests made to the deployed API (maybe a topic for a later blog post) so we will be working on deployed APIs from now on.  For the rest of this post, I'll be using cloudformation to generate the models and APIs so we can build a bunch of them programmatically to test various configurations. think Genetic Algorithms for deciphering API Gateway behavior.

The request body passthrough has 3 options that Alex Debrie describes really well here and I want to experiment with each in a few different scenarios. Below I outline the experiment.


model 1:
application/json (when we require an application/json model in an experiment, this is the model we will use for the definition)
{"props1": "String"}

model 2:
{"props3": "String"}

model 3:
text/plain (when we require an text/plain model in an experiment, this is the model we will use for the definition)

model 4:
{"statusCode": 200}

As far as I can tell, the "content-type" that is required when creating a model in API Gateway is completely un-used. I haven't been able to find anywhere that the the type is enforced and I imagine it is just for making the automatically generated SDKs easier to consume.

Next we will construct 6 experiments:
In these experiments I use "application/jsonx" to refer to content-types that the API author didn't specify ahead of time. I could have just as easily used "appFOOBARlication/somestring" and the results would be unchanged.

Trial 1:
* no models defined in RequestModels
* no templates defined in integration request
* passthrough behavior: NEVER

Content-Type/ Model model 1 model 2 model 3 model 4
application/json 415 415 415 415
application/jsonx 415 415 415 415
text/plain 415 415 415 415

This makes sense, we didn't define any models and we told APIGW to reject any content type we didn't define.

Trial 2:
* no models defined in RequestModels
* no templates defined in integration request
* passthrough behavior: WHEN_NO_MATCH

Content-Type/ Model model 1 model 2 model 3 model 4
application/json 200 200 200 200
application/jsonx 200 200 200 200
text/plain 200 200 200 200

Again, this makes sense. We said "pass the request along if there's no matching content-types" and we didn't identify any content types.

Trial 3:

* no models defined in RequestModels
* no templates defined in integration request
* passthrough behavior: WHEN_NO_TEMPLATES

Content-Type/ Model model 1 model 2 model 3 model 4
application/json 200 200 200 200
application/jsonx 200 200 200 200
text/plain 200 200 200 200

Once again, this makes sense. We said "pass the request through if there are no templates, and all the requests were passed through"

Trial 4:
* content-type application/json and text/plain defined in integration request
* passthrough behavior: NEVER

Content-Type/ Model model 1 model 2 model 3 model 4
application/json 200 400 400 400
application/jsonx 415 415 415 415
text/plain 400 400 200 400

This is where it starts getting fun. The 400 messages mean that validation failed. We told API Gateway that if the content-type is "application/json" then enforce model 1 or if "text/plain" is supplied, then enforce model 3. We see all of the application/jsonx requests being rejecte


Reduce you AWS bill by 10% with this one easy trick, Cloud Economists Hate it

On Thursday, the UK did what UKs are want to do lately. They used Brexit to draw fresh criticisms over seemingly simple tasks. This week was the Parliament Website Edition. The surge of traffic of people wanting up to the minute signature counts crippled their site. My friend James Beswick created a cool application to demonstrate how to properly build a web application that scales dynamically with the request load. While I consider James a friend, his treachery cannot go unmentioned. James committed the most unforgivable of serverlessing crimes; he wrote more code than he needed to and now owns more stuff than he needs to provide the same amount of value.

A quick overview of James' architecture looks like this:

(1) A request comes in to API Gateway, and is proxied to AWS Lambda, (2) AWS Lambda reads the appropriate values from a DynamoDB Table, (3) then the Lambda Function returns the information to the client.

Now that API Gateway supports direct integration with AWS Services, we can eliminate the Lambda completely. and create an architecture that looks more like this:

Without a Lambda Function getting in the way, we see improved performance for our clients​

Latency with Lambda: 
  min: 168.1
  max: 3314.7
  median 242.6
  p95: 336.1
  p99 544.8
Latency without Lambda:
  min: 33.7
  max: 4432
  median: 45
  p95: 86.2
  p99: 886.1

But a far more compelling argument for the simpler architecture is cost. Given the following pricing guides from AWS and our hypothetical scenario, we see a non-trivial cost savings by eliminating Lambda.

API Gateway:
  $3.50/Million requests
  $0.020/hr for 0.5GB Cache on Deployment Stage
  $2.08*10^-7 per 100ms of request time on a 128MB Function
  $1.25/Million Writes
  $0.25/Million Reads

If we assume our application will receive 10M requests in the first 24 hours (0.5 Million votes cast and 9.5 Million page refreshes) and our API caches read requests for 5 seconds before the TTL expires. The Lambda-based architecture costs about $41.13 for the first 24 hours, the Lambdaless architecture costs about $36.16 over the same period.

This is where a less scrupulous evangelist would offer to help you eliminate all of your Lambda Functions for 'moderate' consulting fee. But, if we take a closer look we see that by simply enabling the API Gateway Cache on the Deployment Stage, the Dynamo Table RCU cost is almost completely eliminated and offsets the price of the Lambda. No huge refactoring required, no unfamiliar integration changes, just checking one box reduces you AWS Bill by 10%