Mar 5, 2020
Build Your Own Dystopian Future on AWS
Become the surveillance state leveraging facial recognition software from AWS Rekognition.
Facial recognition is a controversial topic. While the technology itself is impressive, it raises a number of privacy concerns, and it’s ripe for potential abuse from bad actors. In my effort to better understand the current capabilities of facial recognition, I came across AWS Rekognition. I was amazed (and somewhat terrified) at how easily I was able to put together a proof-of-concept API leveraging this service to find pictures matching a new reference image. I’m going to walk you through that process so that you too can better understand how this can be used, for good or for evil.
In this example, I’m going to be using Python 3.7 and its AWS related packages. AWS provides libraries for their services in a number of languages on top of their HTTPS endpoints, so it should be possible to recreate much of this in your language of choice. To follow this example on your own, you’ll need an AWS account. We are going to create an S3 bucket to store images. As images are added to this bucket, we’ll connect a Lambda trigger to index any faces detected in the image in a Rekognition collection. With this collection of images, we’ll expose a single API endpoint via API Gateway and Lambda, which takes a new image and looks to see if there is a matching face in our collection. While this should fit within the free-tier usage, you may see charges on your account related if you leave this running or if it sees heavy usage. I’ll touch on how to clean this up before the end of the article.
To get things started we’ll create a new Chalice project. Chalice is a Python framework for building and deploying serverless applications written and maintained by AWS.
$ pyenv virtualenv 3.7.6 rekognition-demo
$ pyenv activate rekognition-demo
(rekognition-demo) $ pip install chalice boto3 awscli
(rekognition-demo) $ chalice new-project rekognition-demo
(rekognition-demo) $ cd rekognition-demo
(rekognition-demo) $ echo "boto3" >> requirements.txtThis assumes some familiarity working with Python and virtual environments to isolate your dependencies. If you aren’t, I recommend checking out my post on how I manage my local Python environment. This won’t be an exhaustive tutorial on the Chalice framework, and what you’ll learn here is focused on the primary goal of creating our API. That being said, this can serve as a nice introduction or sample project if you are interested in exploring Chalice.
With this initial project scaffolding out of the way, we’re ready to start building out this API.
Target Acquired
We’re going to avoid going down a particularly dark route that surveillance technology might take us. Instead we are going to use facial recognition to turn the surveillance state around. I went through the photos on my local police department Facebook page and downloaded images which depicted only officers in uniform, making an effort to avoid any which had clear images of non-police faces. It’s fine for them to contain multiple faces just as long as they are all officers. This was a manual and somewhat tedious process, but it got the job done. I stored them all locally in a data directory inside my Chalice project directory. We’ll sync these to S3 once we have the logic in place to index them.
S3 fires events when items are added or removed from a bucket, and you can connect those to Lambda functions, SNS notifications, and other actions. Chalice allows us to bind our app to functions which trigger off of these events. When deployed it will make use of these event triggers. Let’s replace the app.py which was generated by Chalice and update it with the following:
# app.py
import loggingimport boto3
from chalice import Chaliceapp = Chalice(app_name="rekognition-demo")
app.log.setLevel(logging.DEBUG)BUCKET = "rekognition-apex-police"COLLECTION = "apex-police"rekognition = boto3.client("rekognition")@app.on_s3_event(bucket=BUCKET, events=["s3:ObjectCreated:*"])
def handle_new_image(event):
response = rekognition.index_faces(
CollectionId=COLLECTION,
Image={"S3Object": {"Bucket": event.bucket, "Name": event.key}},
ExternalImageId=event.key,
MaxFaces=10,
QualityFilter="AUTO",
DetectionAttributes=["ALL"],
)
app.log.debug(f"Indexed {len(response['FaceRecords'])} faces from {event.key}")
This binds the handle_new_imagefunction to S3 events from our bucket named rekognition-apex-police. Note that since bucket names must be globally unique, if you want to recreate this yourself you will need to update with a new bucket name. As new images are uploaded, it puts them into our Rekognition collection named apex-police by using IndexFaces. This can take either the image bytes or a reference to an S3 object (as we’ve done here) to add it to the index. Later when we try to match new images, the existence of a match in the index will let us know that this person is part of the police department.
Chalice has a cool feature where it discovers the AWS resources you are using through your boto3 calls and will automatically add permissions for those resources on the deploy. Sadly it’s not perfect and doesn’t quite work here. While it will add the necessary permissions for Rekognition, in order for the call to work, the Lambda function will also need some S3 permissions that won’t get detected here. Thankfully there is a way to manually set the permissions so we’ll do that in .chalice/policy-dev.json.
# .chalice/policy-dev.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"rekognition:CreateCollection",
"rekognition:DescribeCollection",
"rekognition:IndexFaces",
"rekognition:SearchFacesByImage"
],
"Resource": ["*"]
},
{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:ListBucket"],
"Resource": "arn:aws:s3:::rekognition-apex-police/*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
}
]
}Also we need to change autogen_policy to false in the pre-generated .chalice/config.json.
# .chalice/config.json
{
"version": "2.0",
"app_name": "rekognition-demo",
"stages": {
"dev": {
"api_gateway_stage": "api",
"autogen_policy": false
}
}
}Check out the Chalice docs for more information about setting your project permissions: http://chalice.readthedocs.io/en/latest/
With this ready we can do our first deployment and sync our locally saved images to the S3 bucket.
# Configure your AWS API credentials
(rekognition-demo) $ aws configure
# Create the S3 bucket
(rekognition-demo) $ aws s3api create-bucket --bucket rekognition-apex-police
# Create the Rekognition collection
(rekognition-demo) $ aws rekognition create-collection --collection-id apex-police
# Deploy our app
(rekognition-demo) $ chalice deploy
...
# Sync the local images to S3
(rekognition-demo) $ aws s3 sync data s3://rekognition-apex-police
upload: data/71349130_2620493271306848_3732197409082572800_n.jpg to s3://rekognition-apex-police/71349130_2620493271306848_3732197409082572800_n.jpg
...
upload: data/81421938_2819939351362238_4027235321283346432_o.jpg to s3://rekognition-apex-police/81421938_2819939351362238_4027235321283346432_o.jpgChecking the logs we can see that everything is working as expected.
(rekognition-demo) $ chalice logs --name handle_new_image
Now we are ready for for the next phase which is exposing an API to let us check if a new image contains a face of a local police officer.
Recognizing Users
Now that we have our database of faces, we can start to create the API to check image against our database for matches. Rekognition provides a SeachFacesByImage endpoint for just this purpose. Like IndexFaces, it can take either a reference to an S3 image or the bytes for the image we want to search. In this case we are going to get the image bytes out of the current request. Unfortunately, the API gateway and Chalice is structured to work with JSON request bodies rather than multipart/form encoded data. To work around this, rather than posting the image directly we are going to base64 encode the image data into a JSON body.
# app.py continued
# Replacing top level imports
import base64
import loggingimport boto3
from botocore.exceptions import ClientError
from chalice import BadRequestError, Chalice# Previous section would be here ...@app.route("/", methods=["POST"])
def check_image():
request = app.current_request
if "image" not in request.json_body:
raise BadRequestError("No image given")
try:
image = base64.b64decode(request.json_body["image"])
except binascii.Error:
raise BadRequestError("Invalid image given")
try:
response = rekognition.search_faces_by_image(
CollectionId=COLLECTION, Image={"Bytes": image},
)
result = {
"matches": [
match["Face"]["ExternalImageId"] for match in response["FaceMatches"]
],
}
except ClientError:
result = {
"matches": [],
}
return result
After some initial validation that we have what looks like a valid image, the image bytes are passed to Rekognition to search for any matching faces in the collection. If there are matches they are returned including the original file name in the response. The Rekognition docs note that this search will work best if there is only a single face in the image
To get this running for us to test, we’ll need to do another deploy. Chalice will see that we have created a route and wire that to a new API Gateway endpoint.
(rekognition-demo) $ chalice deploy
Creating deployment package.
Updating policy for IAM role: rekognition-demo-dev-handle_new_image
Updating lambda function: rekognition-demo-dev-handle_new_image
Configuring S3 events in bucket rekognition-apex-police to function rekognition-demo-dev-handle_new_image
Creating IAM role: rekognition-demo-dev-api_handler
Creating lambda function: rekognition-demo-dev
Creating Rest API
Resources deployed:
- Lambda ARN: arn:aws:lambda:us-east-1:759954240342:function:rekognition-demo-dev-handle_new_image
- Lambda ARN: arn:aws:lambda:us-east-1:759954240342:function:rekognition-demo-dev
- Rest API URL: https://xrviuv0vtf.execute-api.us-east-1.amazonaws.com/api/With the REST API up, we can test against a couple images: one that we expect should match and another which shouldn’t match. For the matching image, I pulled the chief of police’s head shot from the Town of Apex website.
To send this to the API, it needs to be base64 encoded inside the JSON body under the image key.
$ (echo -n '{"image": "'; base64 Police\ Chief\ John\ Letteney.jpg; echo '"}') |
curl -H "Content-Type: application/json" -d @- https://xrviuv0vtf.execute-api.us-east-1.amazonaws.com/api/
{"matches":["72434429_2645492565473585_3429136798620581888_o.jpg","82393140_2845334582156048_6102571155162398720_n.jpg","82647288_2851484764874363_5294281172166115328_n.jpg","83177039_2881712408518265_7297622762774855680_n.jpg","85026759_2893680170654822_8181472211329089536_o.jpg","71349130_2620493271306848_3732197409082572800_n.jpg"]}✨ It worked! ✨ Not surprisingly, our collection of Facebook pictures had several which included the chief. Now we can test it against an image which shouldn’t match to check for potential false positives. For this we’ll use this picture of me dressed as a Charizard
Running this through a similar command to base64 encode and send to the API yields no matches as expected.
$ (echo -n '{"image": "'; base64 pokemark.jpg; echo '"}') |
curl -H "Content-Type: application/json" -d @ https://xrviuv0vtf.execute-api.us-east-1.amazonaws.com/api/
{"matches":[]}Clean Up
We can clean this up to avoid any overuse of this API and unexpected Amazon bills using chalice delete.
(rekognition-demo) $ chalice delete
Deleting Rest API: xrviuv0vtf
Deleting function: arn:aws:lambda:us-east-1:759954240342:function:rekognition-demo-dev
Deleting IAM role: rekognition-demo-dev-api_handler
Deleting function: arn:aws:lambda:us-east-1:759954240342:function:rekognition-demo-dev-handle_new_image
Deleting IAM role: rekognition-demo-dev-handle_new_imageThis will remove the Lambda functions, API Gateway, and IAM roles that the framework created. It won’t automatically remove the S3 bucket or the Rekognition collection which we created. Those can be removed using the AWS CLI tools similar to how we created them. Note that the S3 bucket needs to be empty first.
# Remove the S3 bucket
(rekognition-demo) $aws s3 rm s3://rekognition-apex-police --recursive
(rekognition-demo) $ aws s3api delete-bucket --bucket rekognition-apex-police
# Remove the Rekognition collection
(rekognition-demo) $ aws rekognition delete-collection --collection-id apex-policeThat takes care of all the AWS resources we created, so let’s wrap it up.
Build the Future You Want to See
The next steps with this API would be to integrate it into a system to process images or video of people which we want to determine if they are Apex police officers or not. Maybe this could be incorporated into a mobile app that would let me determine if a someone is an off-duty or non-uniformed officer. We’ll need to periodically update our images to account for new officers. Right now that’s a fairly manual process, but it could be automated as well.
While Rekognition is certainly impressive in its capabilities and ease of use (at least in this simple case), I’m not sure I’ll need it. Maybe a project will come up where this feels like a good solution. I expect that developers will face difficult ethical choices in the future about how to make use of this technology. It’s not hard to imagine how such a system could be used or abused for other purposes. You could integrate it into a security system to identify returning vs new customers, highlight VIPs, or flag potential shoplifters. It’s easy to be amazed by the possibilities of a new technology without thinking about its impact on the people it will be used on or for. There are certainly interesting ways this technology can be used to enhance customer experience and an equal number of ways this can be used to exclude or repress people. Which will you choose?
