Tuesday, October 3, 2017

A reliable public API for (close to) free!

Sorry for the lack of posts lately, but I've been busy churning out projects (both personal and business related).

I recently went through an exercise that will definitely benefit some of the communities that I interact with (HAD and CoffeeOps) and felt obligated to share because I'm super excited about it.

In the world of IOT and scalable applications, I don't need to spend much time hyping the benefits of a simple API. Along those lines I've been playing around with the Serverless Framework to spin up services (in my case AWS Lambda) but then asked myself 3 distinct questions:

1. How small is "too small" for an API?
2. How cheap could I make a small API?
3. How reliable (or performant) could I make an API?

This seems very similar to the argument about hiring a contractor ("You can have cheap, good or fast....but you can only pick 2 of the 3 options), but it turns out that when you go small you can get pretty close to all three.

With Serverless Framework, in AWS it is pretty easy to setup an API Gateway that passes input to a Lambda function. It's really small and granular and AWS can host it on their infrastructure really easily. That covers the small stuff.

And here is the plus for us - it's really cheap! You pay only for the number of hits to your API Gateway and most AWS accounts give you a minimum number of Lambda functions for free (usually in the thousands or millions). But my problem was the last part - where do you store your data? You could spin up an RDS like PostgreSQL or MySQL, a Redis database or even a DynamoDB. But can you go cheaper than that? Yes....yes you can. 

AWS offers a simple key value store that is most often used for configuration data called SSM Parameter store. In that KV store, you can store strings of up to 4096 characters (I don't think it is bytes according to what I've found online) which is more than enough for the purpose I have. Using parameter store should be a free built in service, so that covers the whole cheap argument.

So finally we get to the reliable and fast part. Depending on how you write your function, you can have an API that completes well under 100ms. As for reliability, AWS has a long history of reliable services, and that isn't even counting that this could run in multiple regions (as long as you could have a way to sync your data). Reliable....check.

So how do you do it? Here is my very basic working example (no authentication in this example BTW...only a proof of concept):

Step 1: 
Learn about and install Serverless Framework here  https://serverless.com/ . Assumed is that you already have an AWS account to work with, but if not, you'll need to set one up.

Step 2:  
You can create a serverless project using the example templates, but to start out with you only need 3 things....a directory to work in, a serverless.yml file and a handler (I'm using python for mine)

Step 3: 
Create a Parameter in the SSM Parameter Store. Its in the AWS console under the EC2 section. Not hard to find and you only need to set an un-encrypted string. I'm using the name 'myParam'.

Step 4: A bit of code (edit your account & region as needed):

serverless.yml 


       
# Welcome to Serverless!
#

service: param-api # NOTE: update this with your service name

# frameworkVersion: "=X.X.X"

provider:
  name: aws
  runtime: python2.7
  stage: dev
  region: us-west-2
  iamRoleStatements:
    - Effect: "Allow"
      Action: 
        - "ssm:GetParameters"
        - "ssm:PutParameter"
      Resource:
        - "arn:aws:ssm:${self:provider.region}:${accountId}:parameter/myParam"

functions:
  get_myParam:
    handler: handler.get_myParam
    events:
      - http:
          path: get
          method: get

  put_myParam:
    handler: handler.put_myParam
    events:
      - http:
          path: put
          method: put
      
 

and my python functions in handler.py

       
import json
import boto3

def get_myParam(event, context):
    ssmResponse = boto3.client('ssm').get_parameters(
        Names=['myParam'],
        WithDecryption=False
    )
    
    body = {
        "message": ssmResponse['Parameters'],
        "input": event['body']
    }

    response = {
        "statusCode": 200,
        "body": json.dumps(body)
    }

    return response


def put_myParam(event, context):
    

    ssmResponse = boto3.client('ssm').put_parameter(
        Name='myParam',
        Description='this is a test',
        Value=event['body'],
        Type='String',
        Overwrite=True
    )

    response = {
        "statusCode": 200,
        "body": json.dumps(ssmResponse)
    }

    return response



      
 Step 5: 
Deploy using serverless. At CLI "serverless deploy -v"

Step 6: 
Test. Curl against the URL that you were given after your deploy completed. This API now supports get and put, but you could add whatever you want as far as HTTP method and authentication.

Step 7: 
Profit!

What could you do with this? Any number of things. Need to share data across IOT devices? Check. Need to build a global service and you don't have datacenters across the globe? Check. The advantage with this is that the investment is very low and this scales to extremely high (just on the micro level, like owning 50 million insects that you control all over the world).

Many thanks to my employer (https://www.flowroute.com/ ) for letting me work on this and my coworker Reed for helping me through the Python bits and bobs. 
Also, thanks to the Seattle CoffeeOps meetup  https://www.meetup.com/Seattle-CoffeeOps/ for listening to me rant about this stuff even though they are asking "What is he talking about ?!?"