Mastering API Gateway With One Weird Trick

So, I set out on a journey to create an API Gateway to Service integration for all 105 services listed in the API Gateway Console. After doing about 8 services I cracked the rosetta stone of API Gateway and here I'll show you how you can think about the way the services are called via their RESTful API to give a more consistent experience across services. One of the key insights from my previous post was that the aws cli somehow "knew" to add the x-amz-json-1.1 Content-Type header to the request. After doing some digging, with a lot of help from Ben K., we discovered how the cli "knew" this. This link in the botocore data directory shows where this data is loaded from.

This got me thinking about the differences between DynamoDB and Athena integration so I fired up the cli again to see how it interacts with DynamoDB

This is really interesting because the API Gateway integration uses the pattern of applying the QueryString parameter "?Action=ListTables" and doesn't include the headers. From the looks of this, we might be able to apply the same technique header-based technique to every service. This way we don't have to worry about which services support ?Actions=[....] query strings and which services don't. Let's walk through a concrete example to make this a bit easier to digest. I'm going to pick a service at random and try to duplicate this with a cloudformation template.

import glob
import random
import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)
sh = logging.StreamHandler()
logger.addHandler(sh)


def main():
    services = glob.glob('./data/*/*/service-2.json')
    logger.debug("found {} service files".format(len(services)))
    print(random.choice(services))


if __name__ == "__main__":
    main()

Running this gives us ...... *drumroll* .........

Cool. Let's see what we can do with Route53.

Crap. it's not using json at all. No worries though, because we're about to learn the trick. There's only about 4 protocols I've seen so far ['json', 'query', and 'rest-xml'] but I'm sure there's more hiding in there somewhere. This metadata tells us almost everything we need to know. Since there is no field called "targetPrefix" we know that we won't have to add the x-amz-target or application/x-amz-json-1.x headers. Let's try a list command with no parameters to see if we can get the connectivity working, then we'll add some parameters. Let's take a closer look at the ListHostedZones operation.

This tells us the path we want to use '/2014-04-01/hostedzone' and what HTTP method to use in the Integration Request (Note that this can be and often is different from the Method Request in API Gateway). The object also tells us what the input should look like. In this case, we're sending a get request so we don't have to worry about a request body and none of the parameters are required so we can ignore them for now. Here's the template we are going to deploy. I've bolded the parts where we deviate from a standard API Gateway service integration template.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
    Route53 Integration

Resources:
    Route53API:
      Type: AWS::ApiGateway::RestApi
      Properties:
        Name: "Route53 API"
    RootResource:
      Type: AWS::ApiGateway::Resource
      Properties:
        ParentId: !GetAtt Route53API.RootResourceId
        PathPart: "zones"
        RestApiId: !Ref Route53API
    ListHostedZonesMethod:
      Type: AWS::ApiGateway::Method
      DependsOn:
        - RootResource
      Properties:
        HttpMethod: GET
        ResourceId: !Ref RootResource
        RestApiId: !Ref Route53API
        AuthorizationType: AWS_IAM
        MethodResponses:
          - ResponseParameters:
              method.response.header.Access-Control-Allow-Origin: true
            StatusCode: 200
        Integration:
          Type: AWS
          IntegrationHttpMethod: GET
          Credentials:
            Fn::GetAtt:
              - AdminRole
              - Arn
          Uri: !Sub "arn:aws:apigateway:${AWS::Region}:route53:path/2013-04-01/hostedzone"
          IntegrationResponses:
            - StatusCode: 200
    AdminRole:
      Type: "AWS::IAM::Role"
      Properties:
        AssumeRolePolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: "Allow"
              Principal:
                Service:
                  - "apigateway.amazonaws.com"
              Action:
                - "sts:AssumeRole"
        Path: "/"
        ManagedPolicyArns:
          - arn:aws:iam::aws:policy/AdministratorAccess

Let's deploy this and see what happens.

Cool, but I might be lying to you or API Gateway might be lying to you, so we should verify this by implementing some kind of mutating request so we can verify it. Since we have a ListHostedZones, let's look at the CreateHostedZone operation.

Repeating what we did last time, we need to make note of the method, requestUri, and now the input shape.

This tells us that "Name" and "CallerReference" are the only required fields (note: some services don't include a "required" field and instead add the word "Required" to the beginning of the documentation). The CallerReference is an interesting field. Its purpose is so that you can make the same request many times and be sure that the creation is idempotent, which is great if you encounter errors and want to ensure that requesting an item twice doesn't create two identical items. Here's the new API gateway Cloudformation template, with the important parts bolded. Notice that we map the application/json content-type that we expect from our clients to the text/xml that route53 expects based on the metadata["protocol"] entry in the service-2.json file linked above.

    CreateHostedZonesMethod:
      Type: AWS::ApiGateway::Method
      DependsOn:
        - RootResource
      Properties:
        HttpMethod: POST
        ResourceId: !Ref RootResource
        RestApiId: !Ref Route53API
        AuthorizationType: AWS_IAM
        MethodResponses:
          - ResponseParameters:
              method.response.header.Access-Control-Allow-Origin: true
            StatusCode: 200
        Integration:
          Type: AWS
          IntegrationHttpMethod: POST
          RequestParameters:
            integration.request.header.Content-Type: "'text/xml'"
          Credentials:
            Fn::GetAtt:
              - AdminRole
              - Arn
          Uri: !Sub "arn:aws:apigateway:${AWS::Region}:route53:path/2013-04-01/hostedzone"
          RequestTemplates:
            application/json: |
              <CreateHostedZoneRequest xmlns="https://route53.amazonaws.com/doc/2013-04-01/">
                <Name>$input.params('zoneName').</Name>
                <CallerReference>$input.params('user-provided-request-id')</CallerReference>
              </CreateHostedZoneRequest>
          IntegrationResponses:
            - StatusCode: 200

We see that this takes the "zoneName" and "user-provided-request-id" querystring parameters and maps them into the xml of the request. Sending this request with the querystring "zoneName=RichardBoydIsCool.com&user-provided-request-id=19640112" looks like a success

Awesome. Now let's try the "ListHostedZones" that we created earlier and make sure our new HostedZone shows up.

Great!!!! Now we know how to integrate with any service that API Gateway advertises. Stay tuned next week for another post that will make this even easier ;)

Comments

Mastering API Gateway in 105 easy steps

Mastering API Gateway in 105 easy steps

The Serverless 🥑 over at AWS love to tell us how great API Gateway is because it can integrate directly with so many services. I mean, look at this list.

What's not to love?!? I'm going to hop right in and wire up one of these services, selected completely at random to show how easy this is. Let's go with DynamoDB because it's a great example and not at all because it already has about 1/2 million examples on blogs all over the world.

I'll just fill in this handy form

Great, now I update the mapping templates and I'm off to the races.

Awesome. Now let's test this.

You hear that rumbling in the distance? That's the train to Flavor Town with Guy Fieri at the helm because this is some sweet API magic. I have just enough time to check Dynamo to make sure the item was put correctly before the train leaves the station.

The entire train is cheering by now, you're well on your way to not needing any lambdas to proxy requests between API Gateway and any one of those 105 services listed in the console.

This is where the Serverless 🥑 makes the statement that's the DevRel equivalent of "This can work for any service, the proof of which is left to the reader as an exercise" and you're left thinking "Why am I exercising on a train?" but here you are none the less. So you pick another service at random and try to strike out on your own. I hear AWS Athena sounds nice this time of year, let's see if we can run a query through API Gateway.

Well, Shit!!!! What's the name of the Action for starting a query in Athena?!? Let's go ask that one person who knows all these random bits of trivia.

This looks promising

"StartQueryExecution" isn't what I would expect it to be called, but it's the closest thing I can find in the list of available Actions.

On second thought, I really don't want to have to think about all these variable while I'm trying to get this to work (I should crawl before I run). Let's find an Action that doesn't need any inputs and get that to work first, then add some complexity later. The Closest I can find is ListNamedQueries, it has some parameters but none of them are required.

Back to API Gateway. I add "ListNamedQueries" to the Action

Let's give it a whirl.

HAHA, You really thought it would be that easy? You must be new to these parts. That's just great. I don't even know where to start with this. Maybe I'll check the AWS CLI and see how it looks in there.

It works with no parameters from my mac, maybe the debug mode will tell me what's going on.

AH HAH!! There it is. Do you see the difference? If you're like me, you don't see it. If you're like me after about 12 hours, you still don't see the difference. If you're like me after 24 hours, 2 support tickets, and a late night email to our Technical Account Manager you still don't see the difference but you know that other people don't see the difference either. I'm going to save you all of that trouble and show you the difference.

It looks like API Gateway is appending ?Action=[actionName] to the url but Athena actually expects the action specified in a Header named "X-Amz-Target". I'm arrogant enough to think I can fix this.

I've replaced the "Action" field with and empty "Path override" and added the missing header. I'm basically done so I'll just go watch cartoons instead of actually finishing this. The next morning I come into work and test it out.

OOF!!! Maybe it needs an 'empty' Path?!?

Yeah, that'll definitely fix it.

(╯°□°)╯︵ ┻━┻

It's a good thing this is serverless and I don't need to care about the internals of the managed platforms. a week goes by and I try to crack this again. I wonder what happens if I explicitly add an empty body to the request?

This will certainly work!!!

me > (╯°□°)╯︵ ┻━┻

┬─┬ノ( º _ ºノ) < Ben K.

me > (╯°□°)╯︵ ┻━┻

┬─┬ノ( º _ ºノ) < Ben K.

me > (╯°□°)╯︵ ┻━┻

I find a doc that outlines making HTTP requests. 
https://docs.aws.amazon.com/apigateway/api-reference/making-http-requests/
This part looks promising

Specifies JSON and the version, for example, Content-Type: application/x-amz-json-1.0.

Let's see if that helps

I mean, it's in the docs. IT HAS TO WORK!!!!!

(┛ಠ_ಠ)┛彡┻━┻

Okay. I think I've made the point. What you need to do is use application/x-amz-json-1.1, obviously.

FINALLY!!!! 2 down 103 more to go.

Comments

Async APIs

Often we find ourselves wanting to offer a REST API, but the processing we do in a request takes longer than the timeouts provided by API Gateway or Lambda or both. In this guide, We'll walk through a common approach to solving this problem by exposing an async API. At a very high level, this architecture allows a client to submit a request, then they immediately receive a token back that can be used to make a subsequent request to query the API for the status of the initial request.

If typing really isn't your thing, you can just download the whole template from [here]

To accomplish this in AWS, we will use API Gateway, Step Functions, and DynamoDB.

So let's just jump right into it. First, we'll create a simple DynamoDB Table that will be used to hold the statuses of submitted requests.

  RequestStatusTable:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
        - AttributeName: "RequestId"
          AttributeType: "S"
      KeySchema:
        - AttributeName: "RequestId"
          KeyType: "HASH"
      ProvisionedThroughput:
        ReadCapacityUnits: "1"
        WriteCapacityUnits: "1"

Tables are awesome, but if we can't interact with them they are less awesome. Let's go ahead and create some permissions to spread this joy throughout our account. First we start with API Gateway's ability to Query the Table.

  APIGReadDDBRole:
      Type: "AWS::IAM::Role"
      Properties:
        AssumeRolePolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: "Allow"
              Principal:
                Service:
                  - "apigateway.amazonaws.com"
              Action:
                - "sts:AssumeRole"
        Path: "/"
        Policies:
          - PolicyName: "APIGQueryDDB"
            PolicyDocument:
              Version: "2012-10-17"
              Statement:
                - Effect: "Allow"
                  Action:
                   - dynamodb:Query
                  Resource: !GetAtt [ RequestStatusTable, Arn ]

Now we'll give Step function the ability to Put Items in the Table

  StatesExecutionRole:
      Type: "AWS::IAM::Role"
      Properties:
        AssumeRolePolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: "Allow"
              Principal:
                Service:
                  - "states.amazonaws.com"
              Action:
                - "sts:AssumeRole"
        Path: "/"
        Policies:
          - PolicyName: "putDDBitem"
            PolicyDocument:
              Version: "2012-10-17"
              Statement:
                - Effect: "Allow"
                  Action:
                   - dynamodb:PutItem
                  Resource: !GetAtt [ RequestStatusTable, Arn ]

These two Role are basically identical, but it's easier to start with two different Roles and later decide they should be the same Role then to try to split a large Role into 2 smaller ones as their responsibilities diverge.

We're chugging right along, so let's make a simple Step Function state machine that will do some work and then update

  MyStateMachine:
    Type: "AWS::StepFunctions::StateMachine"
    Properties:
      DefinitionString:
        !Sub
        - |-
          {
            "Comment": "Simple State Machine",
            "StartAt": "Submission",
            "TimeoutSeconds": 300,
            "States": {
             "Submission": {
                  "Type": "Task",
                  "Resource": "arn:aws:states:::dynamodb:putItem",
                  "Parameters": {
                      "TableName": "${DynamoTableName}",
                      "Item": {
                          "RequestId": {"S.$": "$.request-id"},
                          "RequestStatus": {"S": "SUBMITTED"},
                          "UserId": {"S.$": "$.user-id"}
                      }
                  },
                  "ResultPath": "$.Result",
                  "Next": "YourPreProcessingCodeHere"
              },
              "YourPreProcessingCodeHere": {
                    "Type": "Wait",
                    "Seconds": 60,
                    "Next": "StartProcessing"
              },
              "StartProcessing": {
                  "Type": "Task",
                  "Resource": "arn:aws:states:::dynamodb:putItem",
                  "Parameters": {
                      "TableName": "${DynamoTableName}",
                      "Item": {
                          "RequestId": {"S.$": "$.request-id"},
                          "RequestStatus": {"S": "PROCESSING"},
                          "UserId": {"S.$": "$.user-id"}
                      }
                  },
                  "ResultPath": "$.Result",
                  "Next": "YourCodeHere"
              },
              "YourCodeHere": {
                    "Type": "Wait",
                    "Seconds": 60,
                    "Next": "TellDynamoWeDone"
              },
              "TellDynamoWeDone": {
                  "Type": "Task",
                  "Resource": "arn:aws:states:::dynamodb:putItem",
                  "Parameters": {
                      "TableName": "${DynamoTableName}",
                      "Item": {
                          "RequestId": {"S.$": "$.request-id"},
                          "RequestStatus": {"S": "COMPLETE"},
                          "UserId": {"S.$": "$.user-id"}
                      }
                  },
                  "ResultPath": "$.Result",
                  "Next": "Succeed"
              },
              "Succeed": {
                  "Type": "Succeed"
              }
            }
          }
        - {DynamoTableName: !Ref RequestStatusTable}
      RoleArn: !GetAtt [ StatesExecutionRole, Arn ]

Now we get into the fun stuff. There's a few ways to create APIs via CloudFormation, but I prefer defining the relationships as CloudFormation Types (as opposed to one large Swagger definition). Let's start with the API resource itself

  AsyncProcessorAPI:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: "Async Processor API"

Now we'll give it the two base resources.

  StatusReportBaseResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      ParentId: !GetAtt AsyncProcessorAPI.RootResourceId
      PathPart: "status"
      RestApiId: !Ref AsyncProcessorAPI
  CreateUserResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      ParentId: !GetAtt AsyncProcessorAPI.RootResourceId
      PathPart: "user"
      RestApiId: !Ref AsyncProcessorAPI

Choo Choo!!! We're on a late night train to API-ville and we're not slowing down. Let's throw some HTTP Methods on here. First we add the ability to query the status of a record.

  GetStatusMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      HttpMethod: GET
      ResourceId: !Ref StatusReportBaseResource
      RestApiId: !Ref AsyncProcessorAPI
      AuthorizationType: AWS_IAM
      RequestParameters:
        method.request.querystring.request-id: true
      MethodResponses:
        - ResponseParameters:
            method.response.header.Access-Control-Allow-Origin: true
          StatusCode: 200
      Integration:
        Type: AWS
        IntegrationHttpMethod: POST
        RequestParameters:
          integration.request.querystring.request-id: 'method.request.querystring.request-id'
        Credentials:
          Fn::GetAtt:
            - APIGReadDDBRole
            - Arn
        Uri: !Sub "arn:aws:apigateway:${AWS::Region}:dynamodb:action/Query"
        RequestTemplates:
          application/json:
            Fn::Sub:
              - |
                #if($input.params('request-id'))
                {
                  "TableName": "${tableName}",
                  "KeyConditionExpression": "PartitionKey = :v1",
                  "ExpressionAttributeValues": {":v1": {"S": "$input.params('request-id')"}}
                }
                #end
              - tableName: !Ref RequestStatusTable
        IntegrationResponses:
          - StatusCode: 200
            ResponseTemplates:
              application/json:
                Fn::Sub: |
                  #set($inputRoot = $input.path('$'))
                  {[
                  #foreach($elem in $inputRoot.Items)
                    {"status": $elem.SortKey.S}#if($foreach.hasNext),#end

                  #end
                  ]}
          - StatusCode: 500

Awesome. Now that we can get data out of the system, let's add something that lets us put some data into it. We'll add the API Gateway to Step Functions integration

  SubmitNewUserMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      HttpMethod: POST
      ResourceId: !Ref CreateUserResource
      RestApiId: !Ref AsyncProcessorAPI
      AuthorizationType: AWS_IAM
      MethodResponses:
        - ResponseParameters:
            method.response.header.Access-Control-Allow-Origin: true
          StatusCode: 200
      Integration:
        Type: AWS
        IntegrationHttpMethod: POST
        Credentials:
          Fn::GetAtt:
            - ExecuteStateMachineRole
            - Arn
        Uri: !Sub "arn:aws:apigateway:${AWS::Region}:states:action/StartExecution"
        RequestTemplates:
          application/json:
            Fn::Sub:
              - |
                {
                  "input": "{\"user-id\": \"$input.params('user-id')\", \"request-id\": \"$context.requestId\", \"name\": \"$context.requestId\"}",
                  "name": "$context.requestId",
                  "stateMachineArn": "${stepFunctionArn}"
                }
              - stepFunctionArn: !Ref MyStateMachine
        IntegrationResponses:
          - StatusCode: 200
            ResponseTemplates:
              application/json:
                Fn::Sub: |
                  {$context.requestId}
          - StatusCode: 500

Whoa, we almost missed something. Can you see it? "ExecuteStateMachineRole" hasn't been defined yet. This Role gives API Gateway the ability to invoke Step Function state machines.. So let's go ahead and add that now.

  ExecuteStateMachineRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: "Allow"
            Principal:
              Service:
                - "apigateway.amazonaws.com"
            Action:
              - "sts:AssumeRole"
      Path: "/"
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AWSStepFunctionsFullAccess

After deploying it, let's run through an example. First we submit a new user named "Richard"

Let's see what happens? We get a request-id back (yes, I should clean up that response object, but then it wouldn't leave anything for you to do when you deploy this template). Now let's see what happens when we check the status of our request.

Let's wait about 60 seconds and check again

That looks promising, let's give it another minute to work through the "wait task" we added for the "YourCodeHere" Step.

Now we're done.

Comments