Getting Past CORS with an AWS Lambda

Posted by Niky Morgan on October 11, 2018

In my previous post I discussed the process of building an app which shows a visual catalog of the Bitmoji comics available for use in Slack. My Minimum Viable Product was a React frontend which made a request to a Sinatra backend which then requested a list of Bitmojis commands from the Bitmoji API. Why did we even need a backend for this application? Couldn’t our frontend just request this same data via JavaScript?

Unfortunately life is not that simple. Trying to request this data from the browser using JavaScript results in the below error.

No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin is therefore not allowed access. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

This is commonly called a CORS error (which stands for Cross-Origin Resource Sharing). The ‘cross-origin’ portion refers to the fact that the origin from which the request is being made has a different protocol or host than where the request is going to. In our case the request is coming from https://slackmoji-app.herokuapp.com and going to https://api.bitmoji.com/content/templates?app_name=bitmoji&platform=slack. Both websites have an https protocol, but the host api.bitmoji.com is different from slackmoji-app.herokuapp.com.

Why is this an issue? Isn’t the internet built upon different websites talking to each other? The answer is yes, but a website or server does want to ensure that it only exchanges information with trusted sources. When requests are made from a browser in JavaScript (using XMLHttpRequest or Fetch), they use the same-origin policy. This means that unless the server receiving the request specifically allows requests from that origin or any origin, requests from a different origin will be rejected. If the server is not set up to accept cross-origin requests, there is nothing we can do (and no headers we can use) to force it to accept the request.

These stringent limitations exist in the frontend (browser-initiated) requests because JavaScript running in the browser has access to a lot of our information. The same-origin policy is trying to protect us from internet evildoers who might write malicious JavaScript code which retrieves our bank account information from another tab and tries to make a request to our banks. It’s in our best interest that websites are picky about what information they will accept!

Requests made from code running outside the browser (running on a server, for example) don’t have this limitation because they don’t have access to all the secrets hidden in the browser. That is why in the previous version of this app, we used a Sinatra backend. The Ruby code the Sinatra app served could make this request to the Bitmoji API unimpeded. We could have just as easily made a backend in Rails or another framework, but since our backend had only one single task (simply placing a request for data and then returning it), it made sense to use a simple and lightweight framework. Rails is a very powerful tool, but it was a bit overkill for this.

Using any backend here feels like a lot of additional overhead. It means we have to host this backend as well as the frontend. If we deploy this to Heroku, both the backend and frontend would fall asleep when unused. This means that after a period of disuse, anyone visiting the page would have to ‘wake’ the frontend which would take several seconds to restart our React app. Once it was awake, the frontend would wake the backend by making a request to it. Only once the backend was also awake could it make that request to the Bitmoji API. This leads to a very slooooow page load.

I want there to be a simpler way to provide an endpoint to run this request for data. Since I’ve done some previous work setting up a contact form using AWS Lambdas, let’s revisit that solution. AWS Lambdas are a way of running code without setting up a server for it. Instead of paying to keep a server running perpetually, you only have to pay for the time your code actually runs. We don’t have to keep a server running 24/7 just for the hour a week we actually need it to compute.

A Lambda can run our code, but we still need to tell it when to run and be able to receive the data from it. In other words, we need this code to be attached to an endpoint so our React application can trigger it and take the response data from it. AWS also provides API Gateway, a way of managing API calls. Like Lambdas, you are only billed for the amount of use (in this case the number of API calls and the amount of data transferred). Users aren’t billed to keep the service up and running. This seems like the ideal solution. We can write a small amount of code which gets triggered in the same fashion as our previous Sinatra backend. (In fact the only front-end change required is editing the request URL to the new AWS API endpoint.)

To create a Lambda, navigate to the Lambda page in our AWS console and click ‘Create Function’. Give it a name, select a runtime environment (language) and give it a role. The role we select determines what access the Lambda has to other areas of AWS. Since we don’t currently have a role for this, we can select ‘Create a new role from one or more templates’ and name the role after our Lambda function for clarity. We don’t need to select anything from the ‘Policy templates’ menu because the Lambda will automatically be assigned the basic permissions it needs to run.

Once we save, we will be redirected to the page for the Lambda. Under the ‘Function code’ section, there is a nice text editor with syntax highlighting. The starter code in the text editor is the event handler that gets called whenever the endpoint is accessed, so any code we add needs to be kicked off from that function. If we want to name our entrypoint file or function something other than the default, we can edit this in the ‘Handler’ field.

The next step is to establish a way to trigger the code. Under the ‘Designer’ heading, there is a sidebar to the left with several trigger options. Clicking on ‘API Gateway’ will add more fields at the bottom of the page to configure an endpoint. We have the option of using an existing endpoint, but we need to create a new one. Since we aren’t concerned about security for this GET request, we can select ‘Open’ from the ‘Security’ heading. The page will indicate we have unsaved changes, so we can save by clicking the orange button on the top right. ‘The API Gateway’ field will now show the URL for our new API in the lower half of the page. This is the endpoint that the frontend will request data from. If we visit that URL in our browsers, we can see the response returned by the Lambda starter code.

We can select Python 3.6 as our runtime environment and write our code locally before moving it into the Lambda function. In the code below, the lambda_handler function calls a function which requests data from the Bitmoji endpoint. Then we assemble a response from the data we receive back. The Bitmoji API returns a lot of data, but all we want is a list of solo Bitmoji comics and a list of Bitmoji comics featuring two Bitmoji characters. We take those two arrays from the response and filter any duplicate comics from them. We also trim off any keys from the Bitmoji dictionaries that we don’t need and remove duplicate tags. Our goal here is to assemble the cleanest and most minimized data to send to the frontend.

import requests
import json

def lambda_handler(event, context):
    resp = make_request()
    return {
        "statusCode": 200,
        "body": resp
    }
        
def make_request():
    url = 'https://api.bitmoji.com/content/templates?app_name=bitmoji&platform=slack'
    resp = requests.get(url).json()
    return build_response(resp)
    
def build_response(resp):
    solo = build_list(resp['imoji'])
    friends = build_list(resp['friends'])
    return {'solo': solo, 'friends': friends}
    
def build_list(bitmojis):
    filtered_list = {}
    for bitmoji in bitmojis:
      comic_id = bitmoji['comic_id']
      if 'comic_id' not in filtered_list:
        filtered_list[comic_id] = serialize_bitmoji(bitmoji)
    return list(filtered_list.values())
    
def serialize_bitmoji(bitmoji):
    unwanted = set(['comic_id', 'src', 'tags']) - set(bitmoji)
    for unwanted_key in unwanted: del bitmoji[unwanted_key]
    bitmoji['tags'] = filter_tags(bitmoji['tags'])
    return bitmoji
        
def filter_tags(tags):
    tags = map(lambda tag: tag.replace('*', ''), tags)
    s = set(tags)
    return list(s)

While this code prints the data as expected locally, triggering it via the API returns: message: "Internal server error". We can design tests for our Lambda by clicking on the ‘Test’ button at the top of the page. Using the default settings, we can create a test which will log error information at the top of the page. The error that logs is: Unable to import module 'lambda_function': No module named requests. That makes a lot of sense! When developing locally, we install Python packages like ‘request’ package which makes HTTP requests. How does our remote Lambda get access to this magical code?

AWS provides documentation for deploying packages to Lambdas. Using PIP Python package manager, we can install any libraries we need to the local directory containing the Lambda code. In console we run pip install requests -t /slackmoji-api. When we run ls inside the slackmoji-api directory, we see the ‘requests’ package dependencies listed there. If we open the directory in Finder, highlight all the individual files (not the directory containing them), Ctrl + click and select ‘Compress’, we can create a zip file containing all this code. When we unzip these files on the AWS, they will overwrite any code we have in the Lambda, so it is easier to zip our code with the packages.

Back on AWS we need to create a new S3 Bucket to hold these zipped files. After entering the bucket name, we can click through the rest of the setup without changing any settings. Once the bucket is created, we can upload the zip file to it, again making no changes to the default options. Once the upload is complete, the ‘Overview’ tab will show an information about the file. Copy the link to the file at the bottom of the tab for later. There is no need to make this bucket public, as the contents will only be accessed from within our AWS account.

Now if we return to the Lambda Function code view, the dropdown on the left has thus far been set to ‘Edit code inline’. If we select ‘Upload a file from S3’, we can paste the link to our newly uploaded zip file. Once we save, all the files will be unzipped (including the lambda function). Now that our packages are uploaded, we can delete the S3 bucket.

From here the final step is to configure our API to accept requests from any origin. If we click on the API name under the API Gateway section, it will redirect us to the settings page for it. Right now our API is configured to have ‘Any’ and ‘Options’ resource types, but we need to add a ‘GET’ option to configure for CORS. In the ‘Action’ dropdown in the ‘Resource’ column, selecting ‘Create method’ will show a ‘method’ dropdown beneath the existing methods. Select ‘GET’ from the menu and the third column will display the integration information. There is a field for ‘Lambda function’ where we will enter the name of our Lambda function in order to connect it to the GET request. Next return to the ‘Action’ dropdown and select ‘Enable CORS.’ Make sure the ‘GET’ checkbox is checked and click the blue ‘Enable CORS and replace existing CORS headers’ button. Again go to the ‘Action’ menu and select ‘Deploy API.’ Select the stage you want to deploy to. (By default there is only one stage, so we can select that.) Click ‘deploy’ and the new settings for our API should be applied. To test if the CORS settings allow requests from other sites, we can open a new page in the browser (to a website other than AWS), open up the console and try making a fetch request to the new endpoint.

With very little setup, we were able to create an endpoint which serves as a backend. If we want to change the code for it at any point in the future, we can edit it via the Lambda text editor. We won’t have to redeploy or manage this endpoint the way we would a fully-hosted backend, and we get past the CORS issues of the 3rd-party API.