Mastering API Gateway in 105 easy steps

Mastering API Gateway in 105 easy steps

The Serverless 🥑 over at AWS love to tell us how great API Gateway is because it can integrate directly with so many services. I mean, look at this list.

What's not to love?!? I'm going to hop right in and wire up one of these services, selected completely at random to show how easy this is. Let's go with DynamoDB because it's a great example and not at all because it already has about 1/2 million examples on blogs all over the world.

I'll just fill in this handy form

Great, now I update the mapping templates and I'm off to the races.

Awesome. Now let's test this.

You hear that rumbling in the distance? That's the train to Flavor Town with Guy Fieri at the helm because this is some sweet API magic. I have just enough time to check Dynamo to make sure the item was put correctly before the train leaves the station.

The entire train is cheering by now, you're well on your way to not needing any lambdas to proxy requests between API Gateway and any one of those 105 services listed in the console.

This is where the Serverless 🥑 makes the statement that's the DevRel equivalent of "This can work for any service, the proof of which is left to the reader as an exercise" and you're left thinking "Why am I exercising on a train?" but here you are none the less. So you pick another service at random and try to strike out on your own. I hear AWS Athena sounds nice this time of year, let's see if we can run a query through API Gateway.

Well, Shit!!!! What's the name of the Action for starting a query in Athena?!? Let's go ask that one person who knows all these random bits of trivia.

This looks promising

"StartQueryExecution" isn't what I would expect it to be called, but it's the closest thing I can find in the list of available Actions.

On second thought, I really don't want to have to think about all these variable while I'm trying to get this to work (I should crawl before I run). Let's find an Action that doesn't need any inputs and get that to work first, then add some complexity later. The Closest I can find is ListNamedQueries, it has some parameters but none of them are required.

Back to API Gateway. I add "ListNamedQueries" to the Action

Let's give it a whirl.

HAHA, You really thought it would be that easy? You must be new to these parts. That's just great. I don't even know where to start with this. Maybe I'll check the AWS CLI and see how it looks in there.

It works with no parameters from my mac, maybe the debug mode will tell me what's going on.

AH HAH!! There it is. Do you see the difference? If you're like me, you don't see it. If you're like me after about 12 hours, you still don't see the difference. If you're like me after 24 hours, 2 support tickets, and a late night email to our Technical Account Manager you still don't see the difference but you know that other people don't see the difference either. I'm going to save you all of that trouble and show you the difference.

It looks like API Gateway is appending ?Action=[actionName] to the url but Athena actually expects the action specified in a Header named "X-Amz-Target". I'm arrogant enough to think I can fix this.

I've replaced the "Action" field with and empty "Path override" and added the missing header. I'm basically done so I'll just go watch cartoons instead of actually finishing this. The next morning I come into work and test it out.

OOF!!! Maybe it needs an 'empty' Path?!?

Yeah, that'll definitely fix it.

(╯°□°)╯︵ ┻━┻

It's a good thing this is serverless and I don't need to care about the internals of the managed platforms. a week goes by and I try to crack this again. I wonder what happens if I explicitly add an empty body to the request?

This will certainly work!!!

me > (╯°□°)╯︵ ┻━┻

┬─┬ノ( º _ ºノ) < Ben K.

me > (╯°□°)╯︵ ┻━┻

┬─┬ノ( º _ ºノ) < Ben K.

me > (╯°□°)╯︵ ┻━┻

I find a doc that outlines making HTTP requests. 
https://docs.aws.amazon.com/apigateway/api-reference/making-http-requests/
This part looks promising

Specifies JSON and the version, for example, Content-Type: application/x-amz-json-1.0.

Let's see if that helps

I mean, it's in the docs. IT HAS TO WORK!!!!!

(┛ಠ_ಠ)┛彡┻━┻

Okay. I think I've made the point. What you need to do is use application/x-amz-json-1.1, obviously.

FINALLY!!!! 2 down 103 more to go.

Comments

Async APIs

Often we find ourselves wanting to offer a REST API, but the processing we do in a request takes longer than the timeouts provided by API Gateway or Lambda or both. In this guide, We'll walk through a common approach to solving this problem by exposing an async API. At a very high level, this architecture allows a client to submit a request, then they immediately receive a token back that can be used to make a subsequent request to query the API for the status of the initial request.

If typing really isn't your thing, you can just download the whole template from [here]

To accomplish this in AWS, we will use API Gateway, Step Functions, and DynamoDB.

So let's just jump right into it. First, we'll create a simple DynamoDB Table that will be used to hold the statuses of submitted requests.

  RequestStatusTable:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
        - AttributeName: "RequestId"
          AttributeType: "S"
      KeySchema:
        - AttributeName: "RequestId"
          KeyType: "HASH"
      ProvisionedThroughput:
        ReadCapacityUnits: "1"
        WriteCapacityUnits: "1"

Tables are awesome, but if we can't interact with them they are less awesome. Let's go ahead and create some permissions to spread this joy throughout our account. First we start with API Gateway's ability to Query the Table.

  APIGReadDDBRole:
      Type: "AWS::IAM::Role"
      Properties:
        AssumeRolePolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: "Allow"
              Principal:
                Service:
                  - "apigateway.amazonaws.com"
              Action:
                - "sts:AssumeRole"
        Path: "/"
        Policies:
          - PolicyName: "APIGQueryDDB"
            PolicyDocument:
              Version: "2012-10-17"
              Statement:
                - Effect: "Allow"
                  Action:
                   - dynamodb:Query
                  Resource: !GetAtt [ RequestStatusTable, Arn ]

Now we'll give Step function the ability to Put Items in the Table

  StatesExecutionRole:
      Type: "AWS::IAM::Role"
      Properties:
        AssumeRolePolicyDocument:
          Version: "2012-10-17"
          Statement:
            - Effect: "Allow"
              Principal:
                Service:
                  - "states.amazonaws.com"
              Action:
                - "sts:AssumeRole"
        Path: "/"
        Policies:
          - PolicyName: "putDDBitem"
            PolicyDocument:
              Version: "2012-10-17"
              Statement:
                - Effect: "Allow"
                  Action:
                   - dynamodb:PutItem
                  Resource: !GetAtt [ RequestStatusTable, Arn ]

These two Role are basically identical, but it's easier to start with two different Roles and later decide they should be the same Role then to try to split a large Role into 2 smaller ones as their responsibilities diverge.

We're chugging right along, so let's make a simple Step Function state machine that will do some work and then update

  MyStateMachine:
    Type: "AWS::StepFunctions::StateMachine"
    Properties:
      DefinitionString:
        !Sub
        - |-
          {
            "Comment": "Simple State Machine",
            "StartAt": "Submission",
            "TimeoutSeconds": 300,
            "States": {
             "Submission": {
                  "Type": "Task",
                  "Resource": "arn:aws:states:::dynamodb:putItem",
                  "Parameters": {
                      "TableName": "${DynamoTableName}",
                      "Item": {
                          "RequestId": {"S.$": "$.request-id"},
                          "RequestStatus": {"S": "SUBMITTED"},
                          "UserId": {"S.$": "$.user-id"}
                      }
                  },
                  "ResultPath": "$.Result",
                  "Next": "YourPreProcessingCodeHere"
              },
              "YourPreProcessingCodeHere": {
                    "Type": "Wait",
                    "Seconds": 60,
                    "Next": "StartProcessing"
              },
              "StartProcessing": {
                  "Type": "Task",
                  "Resource": "arn:aws:states:::dynamodb:putItem",
                  "Parameters": {
                      "TableName": "${DynamoTableName}",
                      "Item": {
                          "RequestId": {"S.$": "$.request-id"},
                          "RequestStatus": {"S": "PROCESSING"},
                          "UserId": {"S.$": "$.user-id"}
                      }
                  },
                  "ResultPath": "$.Result",
                  "Next": "YourCodeHere"
              },
              "YourCodeHere": {
                    "Type": "Wait",
                    "Seconds": 60,
                    "Next": "TellDynamoWeDone"
              },
              "TellDynamoWeDone": {
                  "Type": "Task",
                  "Resource": "arn:aws:states:::dynamodb:putItem",
                  "Parameters": {
                      "TableName": "${DynamoTableName}",
                      "Item": {
                          "RequestId": {"S.$": "$.request-id"},
                          "RequestStatus": {"S": "COMPLETE"},
                          "UserId": {"S.$": "$.user-id"}
                      }
                  },
                  "ResultPath": "$.Result",
                  "Next": "Succeed"
              },
              "Succeed": {
                  "Type": "Succeed"
              }
            }
          }
        - {DynamoTableName: !Ref RequestStatusTable}
      RoleArn: !GetAtt [ StatesExecutionRole, Arn ]

Now we get into the fun stuff. There's a few ways to create APIs via CloudFormation, but I prefer defining the relationships as CloudFormation Types (as opposed to one large Swagger definition). Let's start with the API resource itself

  AsyncProcessorAPI:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: "Async Processor API"

Now we'll give it the two base resources.

  StatusReportBaseResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      ParentId: !GetAtt AsyncProcessorAPI.RootResourceId
      PathPart: "status"
      RestApiId: !Ref AsyncProcessorAPI
  CreateUserResource:
    Type: AWS::ApiGateway::Resource
    Properties:
      ParentId: !GetAtt AsyncProcessorAPI.RootResourceId
      PathPart: "user"
      RestApiId: !Ref AsyncProcessorAPI

Choo Choo!!! We're on a late night train to API-ville and we're not slowing down. Let's throw some HTTP Methods on here. First we add the ability to query the status of a record.

  GetStatusMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      HttpMethod: GET
      ResourceId: !Ref StatusReportBaseResource
      RestApiId: !Ref AsyncProcessorAPI
      AuthorizationType: AWS_IAM
      RequestParameters:
        method.request.querystring.request-id: true
      MethodResponses:
        - ResponseParameters:
            method.response.header.Access-Control-Allow-Origin: true
          StatusCode: 200
      Integration:
        Type: AWS
        IntegrationHttpMethod: POST
        RequestParameters:
          integration.request.querystring.request-id: 'method.request.querystring.request-id'
        Credentials:
          Fn::GetAtt:
            - APIGReadDDBRole
            - Arn
        Uri: !Sub "arn:aws:apigateway:${AWS::Region}:dynamodb:action/Query"
        RequestTemplates:
          application/json:
            Fn::Sub:
              - |
                #if($input.params('request-id'))
                {
                  "TableName": "${tableName}",
                  "KeyConditionExpression": "PartitionKey = :v1",
                  "ExpressionAttributeValues": {":v1": {"S": "$input.params('request-id')"}}
                }
                #end
              - tableName: !Ref RequestStatusTable
        IntegrationResponses:
          - StatusCode: 200
            ResponseTemplates:
              application/json:
                Fn::Sub: |
                  #set($inputRoot = $input.path('$'))
                  {[
                  #foreach($elem in $inputRoot.Items)
                    {"status": $elem.SortKey.S}#if($foreach.hasNext),#end

                  #end
                  ]}
          - StatusCode: 500

Awesome. Now that we can get data out of the system, let's add something that lets us put some data into it. We'll add the API Gateway to Step Functions integration

  SubmitNewUserMethod:
    Type: AWS::ApiGateway::Method
    Properties:
      HttpMethod: POST
      ResourceId: !Ref CreateUserResource
      RestApiId: !Ref AsyncProcessorAPI
      AuthorizationType: AWS_IAM
      MethodResponses:
        - ResponseParameters:
            method.response.header.Access-Control-Allow-Origin: true
          StatusCode: 200
      Integration:
        Type: AWS
        IntegrationHttpMethod: POST
        Credentials:
          Fn::GetAtt:
            - ExecuteStateMachineRole
            - Arn
        Uri: !Sub "arn:aws:apigateway:${AWS::Region}:states:action/
Comments