Integrating HAQM Location Service with AWS AI/ML services

HAQM Location, announced in re:Invent 2020, is a service that lets you add location data to your applications including maps, points of interests, geocoding, and tracking. It provides cost-effective location-based service using high quality data from trusted providers – Esri and HERE. The service anonymizes all queries sent to data providers by removing customer metadata and account information.

HAQM Location is fully integrated with AWS CloudTrail, HAQM CloudWatch, HAQM EventBridge, and AWS Identity and Access Management (IAM). It can simplify your development workflow with data integration, and fast tracks apps to production with built-in monitoring, security and compliance features.

In this blog post, we will demonstrate how you can integrate AWS AI/ML services with HAQM Location to add location-based data in your applications. To get started with HAQM Location, we recommend completing the steps given here.

Overview of solution

We’ll consider two use-cases for this blog:

Using HAQM Rekognition with HAQM Location to detect the object in an image and search for corresponding places in the given location
Using HAQM Comprehend with HAQM Location to find addresses in a text and display them on a map

Prerequisites

We’ll use HAQM Cognito authentication as an alternative to using IAM directly for both the services.

For instance, if you are planning to embed the maps publicly (e.g. on your website), make sure to follow these steps for allowing unauthenticated guest access to your application using HAQM Cognito. You can find steps to configure here.

Walkthrough use-case 1: Using HAQM Rekognition with HAQM Location

In this section, we’ll use HAQM Rekognition to identify the object in the image, and then search for nearby places for that object.

The process will involve following steps:

Step 1: Identify the object in the image uploaded in S3 using HAQM Rekognition.
Step 2: Search for nearby places on the map using HAQM Location.
Step 3: Orchestrate the above steps using Step Functions.
Step 4: Display results on a map

Step 1: Identify the object in the image uploaded in S3

In this step, let’s create a Lambda function to identify the object in the image using HAQM Rekognition. Here, we have used node.js as language to write code in Lambda function. You can use any coding language supported by Lambda.

// Import sdk and libraries
const AWS = require('aws-sdk');
const Rekognition=require("aws-sdk/clients/rekognition");
// Create constants to store bucket name, picture name and credentials
const bucket = '<S3 bucket name>'; // the bucketname without s3://
const photo = '<Image name>';
const identityPoolId = '<Cognito Identity Pool ARN>';
const credentials = new AWS.CognitoIdentityCredentials({
    IdentityPoolId: identityPoolId,
});

// Store parameters that will be used to access HAQM Rekognition service
const rekognitionParams = {
    Image: {
        S3Object: {
            Bucket: bucket,
            Name: photo
        },
    },
    MaxLabels: 1
};


exports.handler = async(event) => {
    await credentials.getPromise();
    var client = new Rekognition({
        credentials: credentials,
        region: '<AWS Region>',
    });
    let res;
    await client.detectLabels(rekognitionParams, function(err, response) {
        if (err) {
            console.log(err, err.stack); // an error occurred
        }
        else {
            if (response.Labels !== undefined) {
                res = {
                    statusCode: 200,
                    body: response.Labels[0].Name,
                };
            }
        }
    }).promise();
    return res;
};

This function will return the label of the object detected in the image.

Next, we’ll use this label in another Lambda function to call HAQM Location API and search for nearby places.

Step 2: Search for nearby places on the map using HAQM Location

To make a geocoding or reverse geocoding request to HAQM Location, you first need to create a place index in your AWS account. HAQM Location place index resource allows you to select a data source to support search queries. Follow the steps here to create a place index.

Next, create a Lambda function to use HAQM Location service, and call searchPlaceIndexForText method to get the result.

// Import AWS sdk and libraries
const AWS = require('aws-sdk');
const Location = require('aws-sdk/clients/location');
// Get credentials and store place index name
const identityPoolId = "<HAQM Cognito identity pool ARN>";
const indexName = '<HAQM Location place index';
const credentials = new AWS.CognitoIdentityCredentials({
    IdentityPoolId: identityPoolId,
});
//Parameters to be used when calling HAQM Location searchPlaceIndexForText API
var params = {
    IndexName: indexName,// Place index name
    Text: "",
    MaxResults: 5, //Maximum number of results
    BiasPosition: [-123.1187, 49.2819] // Default longitude and latitude(serve as a base location)
};


exports.handler = async(event) => {
    const data = event;
    const dataInput = data.Input;
    // Get the input from HAQM Rekognitionfunction created in step 1 through Step function 
    if (dataInput !== undefined) {
        const inputPayload = dataInput.Payload;
        params.Text=inputPayload.Payload.body;
    }

    await credentials.getPromise();
    const location = new Location({
        credentials: credentials,
        region: '<AWS region>',
    });
    
    
    let arrayLongLat = [];// Array to store resultant latitude and longitude
    let response = {};// Response of the Lambda function
    
    // Call HAQM Location API to search place index for text
    await location.searchPlaceIndexForText(params, function(err, data) {
        if (err) {
            console.log(err, err.stack);
        } // an error occurred
        else {
            data.Results.map(function(v) {
                arrayLongLat.push({
                    longitude: v.Place.Geometry.Point ? v.Place.Geometry.Point[0] : 0,
                    latitude: v.Place.Geometry.Point ? v.Place.Geometry.Point[1] : 0
                });
            });
            response = {
                statusCode: 200,
                body: JSON.stringify(arrayLongLat),
            };
       
        } // successful response
    }).promise();
    return response;
};

This will return an array of longitudes and latitudes near the default location. Next, let’s create a step function to orchestrate the above Lambda functions.

Step 3: Orchestrate with Step Function

AWS Step Functions is a serverless function orchestrator that makes it easy to sequence multiple AWS Lambda functions and other services. It also provides visual interface that helps you run workflows and maintain application state. The output of one step is input to the next that helps implement business logic.

Here is the visual representation and definition of the step function to orchestrate above Lambda functions:

Start, then invoke rekognition lambda function, then invoke location lambda function, then end.

Step function definition:

{
  "Comment": "HAQM Location Example",
  "StartAt": "Invoke Rekognition Lambda function",
  "States": {
    "Invoke Rekognition Lambda function": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:<region:AccountID>:function:TestRekognitionService:$LATEST",
        "Payload.$": "$"
      },
      "ResultPath": "$.Payload",
      "Next": "Invoke Location Lambda function"
    },
    "Invoke Location Lambda function": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:<region:AccountID>:function:TestLocationService:$LATEST",
        "Payload": {
          "Input.$": "$"
        }
      },
      "End": true
    }
  }
}

Step 4: Display result on map

The last step is to display the results (latitude and longitudes) on a map. HAQM Location lets you create and add map resources to display data. To add a map, you first need to create a map resource and select map providers and styles. You can use popular open-source map SDKs, such as Mapbox GL and Tangram, to display the map in your application. Please read more about maps in HAQM Location here. You can also find samples here to create applications and display map on Android, iOS and web applications.

Walkthrough use-case 2: Using HAQM Comprehend with HAQM Location Service

In this section, we will implement a pipeline that leverage HAQM Comprehend and HAQM Location Service.

The goal of this pipeline is generating a map that contains markers on the addresses that have been detected into a text.

A real example of this use-case could relate to travel blogs that want to generate a map starting from the content mentioning all the cities/country they went through.

The process will involve following steps-

Step 1: Detect the language of the text in input
Step 2: Evaluate if the text is English
Step 3: Detect all the addresses contained in a text
Step 4: Get the coordinates for each of the address
Step 5: Generate an html map with markers

This pipeline will be implemented using AWS Step Functions, below you can find a visual of the Step Function definition.

Link to GitHub repository: http://github.com/aws-samples/amazon-location-detected-addresses-to-map

You can deploy inside you AWS account the latest CloudFormation template by following the link below for your preferred AWS region:

Region	Launch Template
US East (N. Virginia) (us-east-1)
US West (Oregon) (us-west-2)
EU (Ireland) (eu-west-1)
EU (London) (eu-west-2)
EU (Frankfurt) (eu-central-1)
AP (Sydney) (ap-southeast-2)

The CloudFormation template that will deploy:

4x AWS Lambda functions
1x HAQM Cognito Identity Pool, for allowing unauthenticated guest access to your application
1x AWS Step Functions
All the IAM Roles and Lambda Layer needed

Once deployed you can execute the Step Function to execute pipeline described above.

You can leverage the ExampleCLI value provided in the CloudFormation outputs to test the Step Function via CLI.

To invoke the Step Function, you have to provide as input:

The text that you want the pipeline to parse
The S3 bucket name and folder name where the html file has to be stored
The Cognito Identity Pool Id that you can find it in the CloudFormation stack  Outputs  IdentityPoolId. This will allow unauthenticated guest access to the map
The HAQM Location map name you have created

{
    "text": " I live in England, my office is in EC1A 2FD, London.",
    "s3_bucket": "location-to-map-XXXXXX",
    "s3_folder": "map-folder/",
    "identity_pool_id": "us-east-1:205aa252-be23-4616-8849-xxxxxxxxxxxx", 
    "map_name": "ExampleMap01"
}

As output, you will get:

An S3 pre-signed link to securely get your html map (NOTE: every pre-signed link is set to expire in 1 hour)

Once deployed the CloudFormation stack above, you can execute the Step Function with your parameters in input. You can execute the Step function via the ExampleCLI command in the CloudFormation outputs, or starting a new execution directly in Step Functions → State machines.

You can access the generated map via the S3 bucket and folder you specified, or via the S3 pre-signed link in Step Function → State machine → Execution output.

Below an example of generated map. The two makers represent the 2 addresses identified in the input text above: England and EC1A 2FD, London.

HTML map generated by AWS Location Service with markers on the addresses detected in the input text.

Let’s go more in details about what each of the Lambda function is doing.

Step 1: Detect the language of the text in input

The first Lambda function is detecting what is the language of the text in input using the DetectDominantLanguage API in Comprehend. Please find here more information on this API and the languages that Comprehend is able to detect.

Step 2: Evaluate if the text is English

In order to detect the addresses in the text we are going to use the DetectPiiEntities API in Comprehend.

This API inspects the input text for entities that contain personally identifiable information (PII) and returns information about them, in our case we process only the Addresses in the response.

Since HAQM Comprehend supports PII detection in only English text, we have to assert that the text is in English.

Note: We are leveraging the DetectPiiEntities and not the DetectEntities API (which is able to detect Location entities) because the first is better to identify addresses.

An example below starting from the text “60 Holborn Viaduct, Holborn, London EC1A 2FD, United Kingdom”:

Step 3: Detect all the addresses contained in a text

As already mentioned in the step above, in this Lambda we are going to detect the addresses in the text with the DetectPiiEntities API in Comprehend. More info on Detect PII, here.

Step 4: Get the coordinates for each of the address

Given a list of address detected in the text, we now want to get the corresponding coordinates for each address.

In order to do that we are going to create a Place Index resource in HAQM Location Service, this is necessary to then invoke the SearchPlaceIndexForText API in Location Service which will perform the Geocoding to get for each address its coordinates. More info on this API, here.

Note: The SearchPlaceIndexForText API is returning multiple result for an Address, in our case we choose the one with the Label that best match our address.

Step 5: Generate an html map with markers

Now that we have a list of addresses and their respective coordinates, the lambda function is injecting each Marker (composed by: coordinates + address label) in a Mapbox GL template.

If you are looking for HAQM Location Service samples, check out this GitHub repo.

Cleaning up

To avoid incurring future charges, delete the CloudFormation stack.

Please, note that you will have to manually delete the S3 bucket created by the CloudFormation stack containing the generated maps, in order to avoid all potential charges.

Common troubleshooting

The HAQM Location Service was announced in re:Invent 2020, so it’s not present in older version of the AWS SDK. Make sure you are using an updated version of the AWS SDK in your Lambdas.
If you are planning to expose the map publicly (e.g. on your website), make sure you are allowing unauthenticated guest access to your application using HAQM Cognito.

Conclusion

In this blog we went through some examples of how it is possible to integrate AI Services like Rekognition and Comprehend with Location Service to solve some of the business use-cases the customers are facing when implementing Location Based Services (LBS) in their applications.

Front-End Web & Mobile