BigCommerce

I recently came across this article on Medium. The author highlights a use-case that gets much less attention than it deserves, connecting API Gateway to services that aren't offered within AWS. In this specific example Brian needs to connect his APIGW endpoint to his BigCommerce endpoint without revealing his BigCommerce credentials to his clients. In the article he uses a Lambda function to add the credentials to the request, modify the path based on a query string, then forward the request to his store's BigCommerce endpoint. If you have followed my blog for more than a few minutes you know how I feel about this pattern. In today's installment, I am going to show you how to accomplish the same integration without requiring a Lambda Function.

First we need to set up a BigCommerce sandbox store. This can be done by signing up for a free 15-day trial here. Once your store is created, we need to set up some credentials. Click on Advanced Settings in the main console.

Then select API Accounts. I have one called "Channel Advisor' because I was playing with it earlier, your list should be blank.

Name your Application and give it the appropriate permissions. I just went down the list clicking all the "read-only" tabs.

Now it will show you your credentials. Don't be like me and show them to the world. I'm a professional. Make sure you save these. Like an AWS Access Key ID and Secret Key, this is the only time you can see them.

Neat. Now you have an application that can talk to your store.

We'll add a product so we can be sure wqe're hitting our store.

And here we see our product in our store's catalog.

Now for the fun part. We are going to create an API that looks just like Brian's from his Medium post.

We start with the basics for our cfn template

AWSTemplateFormatVersion: '2010-09-09'
Description: >
  BigCommerce Proxy API

Now we need to add a REST API

Resources:
  MyRestAPIForTesting:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: "BigCommerce"
      EndpointConfiguration:
        Types:
          - REGIONAL

Now let's add a resource to the API. A resource is the path part of the API.

  VariantsResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      ParentId: !GetAtt MyRestAPIForTesting.RootResourceId
      PathPart: "get-variants"
      RestApiId: !Ref MyRestAPIForTesting

Now it get's really exciting. We're going to add the GET method so our clients can call our API

  MatchingMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      HttpMethod: GET
      ResourceId: !Ref VariantsResource
      RestApiId: !Ref MyRestAPIForTesting
      AuthorizationType: NONE
      RequestParameters:
        method.request.querystring.product_id: false
      MethodResponses:
        - ResponseParameters:
            method.response.header.Access-Control-Allow-Origin: true
          StatusCode: 200
      Integration:
        IntegrationHttpMethod: GET
        Type: HTTP
        RequestParameters:
          integration.request.path.id: 'method.request.querystring.product_id'
          integration.request.header.X-Auth-Token: "'hzkw74s0h0be7mbme5znarql606cd84'"
          integration.request.header.X-Auth-Client: "'r135f84ojwx53si1ty0hsvytczzdj6y'"
          # client Secret: 6cm3y6i63umbc781b5a2pysu4h29hy8 Not sure why they gave me this but whatevs
        Uri: https://api.bigcommerce.com/stores/zq0ehvsen5/v3/catalog/products/{id}/variants?include_fields=calculated_price,inventory_level,sku,option_values,image_url
        PassthroughBehavior: NEVER
        IntegrationResponses:
          - StatusCode: 200

Here we are taking the product_id query string and using it to replace the {id} variable in the path, as well as adding the auth headers that BigCommerce expects.

Let's deploy this and see what happens.

Crap. I forgot to record the product_id from the server we created. Going back to the storefront and mousing over the product link reveals the product id at the bottom of the page

Now let's go back to APIGW and search for product 113

That looks promising, but it could be any $1,000,000 product. Let's go change the SKU in the product to 1776 and try again and see what happens.

Awesome. Now you can use APIGW to front your BigCommerce store and you have the cloudformation template @ryan_marsh doesn't @ me over this

Comments

genetic decipher

I've written fairly extensively on AWS API Gateway (APIGW) and many of the lesser-known features. Today we're going to dive deep on the model validation feature of APIGW and specifically how to handle unexpected input.

The most common use for APIGW is to leverage the "AWS_PROXY" integration that directly connects APIGW to AWS Lambda (this is called "Lambda Function" in the APIGW console). There are several benefits to this approach, including reduced latency, simpler implementation, and more control over the shape of the API response object. Today, I am going to show you how to achieve that same level of control natively with APIGW and use those same techniques to enforce the shape of the request object. We'll wrap up with a discussion of some of the limits of this feature (AWS Tech Evangelists take note!)

Let's start with a brief introduction to the various parts of APIGW. Each API consists of Resources, Methods, Method Request, Integration Request, Integration Response, and Method Response.

* Resources: These are the paths of your API, and API will start with a single resource called the "root" resource and is denoted by "/".

Adding a "child" named "robot" to this resource would look like "/robot".

The robot resource could have "siblings" such as "/mop" or "/vacuum". Likewise, the robot resource could have children of its own, such as "/robot/biped" or "/robot/tracked"

Here is the finished set of resources.

* Methods: methods are the HTTP verbs that describe an interaction with a resource. These are [GET, POST, OPTIONS, DELETE, HEAD, PATCH, PUT, ANY]. Each resource can have any, many, or none of these methods attached to it. One thing to note is that a particular verb can only be attached to a resource once. You cannot have two GETs defined for /robot or three PUTs defined for /mop because when a client makes an HTTP GET request to /robot, there would be no way to know which of the two methods they meant to invoke.

Here we see that the option to add a second GET request is not available. The console is great at preventing this, but if you are using CloudFormation, you could end up trying to deploy a template with conflicting methods and CFN is generally good about telling you which methods are conflicting on which resource within the API.

I've created a sample mock integration to demonstrate some of the next components.

* Method Request: I like to describe the method request as the client-facing incoming side of the APIGW interaction; this is the place where you (the API developer) will define authorizations needed to interact with the resource/method and define expected query strings, headers, and body.

* Integration Request: This (with the Integration Response we will look at shortly) is the meat of an APIGW API. This maps a client's request to an AWS service request by performing transformations on the request body, modifying headers and query strings, and defining what behavior should be performed when an unexpected input is received.

* Integration Response: This is where the response from the AWS service, on-prem server, or other HTTP proxy integration will be transformed into something that the client wants. Here is where you will define what HTTP status codes you expect in response from the integration endpoint and how you want to change your transformation based on those status codes.

* Method Response: Much like the method request, this is the client-facing outgoing side of the APIGW interaction. Here is where you can define specific response "shapes" ("Types" if you come from an typed-language background)

Now that we've covered some of the basics, let's dive into some request/response modeling. Suppose you had an API that was used as a simple CRUD interface for your robot collection. You'd like to be able to add robots to your collection, update their price as they become more rare, query for robots with specific attributes, and delete them when you sell or trade them with other roboticists. When we add a new robot to our collection, we want to make sure it has a serial number and generally looks like how we'd expect a robot's description (hasLaser: True, isCool: True, mass: >=0kg, etc....). If we were using the AWS_PROXY integration we would have to write custom lambda code to parse the request body and validate these fields, co-mingling the request validation logic with our business logic of CRUDing the robots.

Here I've created a simple model for a new robot and mark only the serialNumber field as required. If a mass is supplied, it must be non-negative. Let's add it to an API and poke it a few times to see what happens.

{
  "$schema" : "http://json-schema.org/draft-04/schema#",
  "title" : "CreateRobotRequest Schema",
  "type" : "object",
  "required": ["serialNumber"],
  "properties" : {
    "serialNumber" : {
        "type" : "string",
        "minLength": 10
    },
    "mass" : {
        "type" : "number",
        "minimum": 0
    }
  }
}

I've created a new mock POST method on my robot Resource and added the request model to the Method Request.

An empty request will fail because it is missing the required serialNumber. We see in the Logs dialog box that the required field is missing.

Likewise, adding a serialNumber that is too short will result in a validation failure.

Let's make a request that we expect to pass validation to make sure it's not just rejecting everything.

Great. Now let's get into the fun stuff. within the Integration Request, we can specify how we want the API to perform if the content type provided by the client doesn't match the Content-Types we've specified in our mapping templates. It's interesting to note that the default option is not the recommended option.

Before I can go much further, I have to stop using the mock integrations and the APIGW console for the demo. While the mock integration is great for experimenting or returning hard-coded responses that you're certain will never change, it leaves a bit to be desired that Ben Kehoe describe better than I could several years ago so a fix should be just around the corner. There is also some peculiarities in the way that the console differs from regular cUrl or Postman requests made to the deployed API (maybe a topic for a later blog post) so we will be working on deployed APIs from now on.  For the rest of this post, I'll be using cloudformation to generate the models and APIs so we can build a bunch of them programmatically to test various configurations. think Genetic Algorithms for deciphering API Gateway behavior.

The request body passthrough has 3 options that Alex Debrie describes really well here and I want to experiment with each in a few different scenarios. Below I outline the experiment.

Models:

model 1:
application/json (when we require an application/json model in an experiment, this is the model we will use for the definition)
{"props1": "String"}

model 2:
{"props3": "String"}

model 3:
text/plain (when we require an text/plain model in an experiment, this is the model we will use for the definition)
"String"

model 4:
{"statusCode": 200}

As far as I can tell, the "content-type" that is required when creating a model in API Gateway is completely un-used. I haven't been able to find anywhere that the the type is enforced and I imagine it is just for making the automatically generated SDKs easier to consume.

Next we will construct 6 experiments:
In these experiments I use "application/jsonx" to refer to content-types that the API author didn't specify ahead of time. I could have just as easily used "appFOOBARlication/somestring" and the results would be unchanged.

Trial 1:
* no models defined in RequestModels
* no templates defined in integration request
* passthrough behavior: NEVER

Content-Type/ Model model 1 model 2 model 3 model 4
application/json 415 415 415 415
application/jsonx 415 415 415 415
text/plain 415 415 415 415

This makes sense, we didn't define any models and we told APIGW to reject any content type we didn't define.

Trial 2:
* no models defined in RequestModels
* no templates defined in integration request
* passthrough behavior: WHEN_NO_MATCH

Content-Type/ Model model 1 model 2 model 3 model 4
application/json 200 200 200 200
application/jsonx 200 200 200 200
text/plain 200 200 200 200

Again, this makes sense. We said "pass the request along if there's no matching content-types" and we didn't identify any content types.

Trial 3:

* no models defined in RequestModels
* no templates defined in integration request
* passthrough behavior: WHEN_NO_TEMPLATES

Content-Type/ Model model 1 model 2 model 3 model 4
application/json 200 200 200 200
application/jsonx 200 200 200 200
text/plain 200 200 200 200

Once again, this makes sense. We said "pass the request through if there are no templates, and all the requests were passed through"

Trial 4:
* content-type application/json and text/plain defined in integration request
* passthrough behavior: NEVER

Content-Type/ Model model 1 model 2 model 3 model 4
application/json 200 400 400 400
application/jsonx 415 415 415 415
text/plain 400 400 200 400

This is where it starts getting fun. The 400 messages mean that validation failed. We told API Gateway that if the content-type is "application/json" then enforce model 1 or if "text/plain" is supplied, then enforce model 3. We see all of the application/jsonx requests being rejected as the wrong content type (415) and all of the poorly formatted requests of the correct content-type (400)

Trial 5:
* content-type application/json and text/plain defined in integration request
* passthrough behavior: WHEN_NO_MATCH

Content-Type/ Model model 1 model 2 model 3 model 4
application/json 200 400 400 400
application/jsonx 200 200 200 200
text/plain 400 400 200 400

Here is another interesting example. We see the model enforcement completely bypassed by supplying an unexpected content-type (application/jsonx). Poorly formed requests with application/json or text/plain are still bound to the expected model, but application/jsonx is free have the whole request passed directly to the backend.

Trial 6:
* content-type application/json and text/plain defined in integration request
* passthrough behavior: WHEN_NO_TEMPLATES

Content-Type/ Model model 1 model 2 model 3 model 4
application/json 200 400 400 400
application/jsonx 415 415 415 415
text/plain 400 400 200 400

This makes sense, given what we've seen so far. We told APIGW to only pass the requests on if there were no templates specified. Since we specified content-types, the unexpected "application/jsonx" is rejected and the poorly formatted requests are rejected.

Now let's repeat this with the Lambda "AWS_PROXY" integration and see what nonsense falls out.

Trial 1:
* no models defined in RequestModels
* no templates defined in integration request
* passthrough behavior: NEVER

Content-Type/ Model model 1 model 2 model 3 model 4
application/json 415 415 415 415
application/jsonx 415 415 415 415
text/plain 415 415 415 415

This makes sense, we didn't define any models and we told APIGW to reject any content type we didn't define.

Trial 2:
* no models defined in RequestModels
* no templates defined in integration request
* passthrough behavior: WHEN_NO_MATCH

Content-Type/ Model model 1 model 2 model 3 model 4
application/json 200 200 200 200
application/jsonx 200 200 200 200
text/plain 200 200 200 200

Again, this makes sense. We said "pass the request along if there's no matching content-types" and we didn't identify any content types.

Trial 3:

* no models defined in RequestModels
* no templates defined in integration request
* passthrough behavior: WHEN_NO_TEMPLATES

Content-Type/ Model model 1 model 2 model 3 model 4
application/json 200 200 200 200
application/jsonx 200 200 200 200
text/plain 200 200 200 200

Once again, this makes sense. We said "pass the request through if there are no templates, and all the requests were passed through"

Trial 4:
* content-type application/json and text/plain defined in integration request
* passthrough behavior: NEVER

Content-Type/ Model model 1 model 2 model 3 model 4
application/json 200 400 400 400
application/jsonx 200 200 200 200
text/plain 400 400 200 400

This is where the Lambda Proxy integration can get dangerous. If you were to use a regular AWS service integration, unexpected content-types would be rejected with a 415 error, if you use the AWS_PROXY integration for Lambda, unexpected content-types are passed straight through with no model validation.

Trial 5:
* content-type application/json and text/plain defined in integration request
* passthrough behavior: WHEN_NO_MATCH

Content-Type/ Model model 1 model 2 model 3 model 4
application/json 200 400 400 400
application/jsonx 200 200 200 200
text/plain 400 400 200 400

No difference between this and the regular AWS service integration

Trial 6:
* content-type application/json and text/plain defined in integration request
* passthrough behavior: WHEN_NO_TEMPLATES

Content-Type/ Model model 1 model 2 model 3 model 4
application/json 200 400 400 400
application/jsonx 200 200 200 200
text/plain 400 400 200 400

Just like experiment 4, you would expect this configuration to reject application/jsonx, but you would be mistaken.

My assumption is that APIGW is doing some kind of "short-cut" behind the scenes to connect APIGW to Lambda and the Method Request stage is being completely skipped in the path through the system. This creates lower latency (which we saw in the post comparing APIGW service integration to APIGW->Lambda->DDB with James Beswick). It seems odd that the method request stage of the API is completely skipped, considering the price tag that comes with APIGW and that AWS SAM uses the AWS_PROXY integration be default. I would recommend that if you are using APIGW to integrate with a service that isn't Lambda, take the time to specify the "NEVER" request passthrough behavior instead of relying on the default "WHEN_NO_TEMPLATES" behavior.

Comments

CDK

Today, the AWS Cloud Development Kit team released experimental support for Python in the CDK. While it's still an experimental release and there are some rough edges, it's a big step forward and I wanted to give it a test drive. The biggest pain-point so far is that "pip install aws-cdk.cdk" only installs the shims that connect the CDK (which is written in TypeScript) to Python language bindings. Users will have to make sure that they have NodeJS (>= 8.11.x) installed and perform "npm install cdk" to install the actual cdk goodies. Once that's out fo the way, we're all set to start declaring our infrastructure. We're going to create a Lambda function that just echoes "Hello, world!".

I've created a new python project in PyCharm.

Now, let's define our requirements in a requirements.txt file.

aws-cdk.aws-lambda
aws-cdk.cdk

Next we create our lambda code. We're going to name this file lambda-handler.py

def main(event, context):
    print("Hello, world!")

So far, this has all been the same development process we follow for developing any lambda function, but here is where we start to leverage CDK. Instead of the SAM approach where we would have a cloudformation template that we either have to craft or copy from a previous project, CDK let's us declare our infrastructure with all the goodness that modern IDEs provide (type-checking, tab completion, syntax highlighting, etc...). Let's create a file call app.py that will contain the logic for building our application.

from aws_cdk import cdk


class MyLambdaStack(cdk.Stack):
    def __init__(self, app: cdk.App, id: str) -> None:
        super().__init__(app, id)


app = cdk.App()
MyLambdaStack(app, "MySimpleLambda")
app.run()

Next we need a file called cdk.json, this file tells CDK how to run the application that builds our infrastructure, the CDK application captures the applications we define here and uses them to generate the cloudformation needed to build this application in AWS.

{
    "app": "python3 app.py"
}

Now we're ready to hop over to the console and build our app. From the root of your package run "cdk synthesize" to see the synthesized cloudformation YAML.

cdk synthesize

Well, that was pretty anti-climactic. there's nothing there but some metadata. Let's go back to our app and add our lambda function code.

First, we're going to read in the contents of lambda-handler.py so that we can pass the raw string to our Lambda Function. This is the same way that you define inline code in a raw cloudformation lambda function.

        with open("lambda-handler.py", encoding="utf8") as fp:
            handler_code = fp.read()

Now that we have our lambda code, let's use it to build a Function.

        lambdaFn = lambda_.Function(
            self,
            "Singleton",
            code=lambda_.InlineCode(handler_code),
            handler="index.main",
            timeout=300,
            runtime=lambda_.Runtime.PYTHON37,
        )

Now we can jump back to our console and synthesize this again

cdk synthesize

Great!!! I have YAML in the console, exactly where I need it ;) . but how do I do anything with this? CDK has solved that for us too.

cdk deploy --profile workshop

"--profile workshop" is just an aws config profile I use for demos, if you have previously run aws configure You won't need to supply that argument.

Let's hop over to the AWS Console and see what's good.

Awesome. Let's go see this Lambda in action.

WOOT!! WOOT!! Now we're building infrastructure in a language that's more fun to work with than YAML/JSON and it handles much of the boilerplate Cloudformation that I don't want to think about.

Comments