AWS S3 uploads using pre-signed URLs

How can I allow users to access objects in S3?

By default, all objects are private — meaning only the bucket account owner initially has access to the object. If you want a user to have access to a specific bucket or objects without making them public, you can provide the user with the appropriate permissions using an IAM policy. In addition to allowing access using an IAM policy, you can also create a presigned URL — meaning users can interact with objects without the need for AWS credentials or IAM permissions.

So what are presigned URLs anyway?

A presigned URL is a URL that you can provide to your users to grant temporary access to a specific S3 object. Using the URL, a user can either READ the object or WRITE an Object (or update an existing object). The URL contains specific parameters that are set by your application. A pre-signed URL uses three parameters to limit access to the user;

1. Bucket: The bucket that the object is in (or will be in)
2. Key: The name of the object
3. Expires: The amount of time that the URL is valid

As expected, once the expiry time has lapsed the user is unable to interact with the specified object. AWS gives access to the object through the pre-signed URL as the URL can only be correctly signed by the S3 Bucket owner.

Anyone with a valid pre-signed URL can interact with the objects as specified during creation. For example, if a GET (Read) pre-signed URL is provided, a user could not use this as a PUT (Write).

The URL itself is constructed using various parameters, which are created automatically through the AWS JS SDK. These include;

1. X-AMZ-Algorithm
2. X-AMZ-Credential
3. X-AMZ-Date
4. X-AMZ-Expires
5. X-AMZ-Signature
6. X-AMZ-SignedHeaders

https://presignedurldemo.s3.eu-west-2.amazonaws.com/image.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJJWZ7B6WCRGMKFGQ%2F20180210%2Feu-west-2%2Fs3%2Faws4_request&X-Amz-Date=20180210T171315Z&X-Amz-Expires=1800&X-Amz-Signature=12b74b0788aa036bc7c3d03b3f20c61f1f91cc9ad8873e3314255dc479a25351&X-Amz-SignedHeaders=host

Above is an example of a presigned URL that can be used to GET Objects. The link will now be invalid given that the maximum amount of time before a presigned URL expires is 7 days.

How do I create a presigned URL then?

The first thing we need to do is create an IAM user which has access to both reading and writing objects to S3. An API key will then be created for the IAM user, which will be stored as an environment variable in the server.

1. Navigate to S3 and create a bucket. The bucket name must be unique.
2. Navigate to IAM.
3. Create a User with Programmatic access.
4. Click Next: Permissions.
5. Click the Attach existing policies directly box and Create policy.
6. Use the visual editor to select the S3 Service. We only need a couple of access requirements; so expand out the access level groups.
7. Ensure that GetObject under the READ section and PutObject under the write section are both ticked.
8. Set the resources you want to grant access to; specify the bucket name you created earlier and click Any for the object name.
9. We’re not specifying any Request conditions.
10. Click Review Policy and enter a name for the policy. Save the policy.

11. Apply the new policy to the new user you have created and taken note of the aws access credentials.

Generating the presigned URLs using the AWS JS SDK

Below shows the two methods for generating a GET URL and PUT URL using the AWS S3 class.

Create a file “generatePresignedGETURL.js”

Complete file for generating the presigned GET URL

require('dotenv').load();
require('dotenv').config();
var AWS = require('aws-sdk');
var credentials = {
accessKeyId: process.env.S3_ACCESS_KEY,
secretAccessKey : process.env.S3_SECRET_KEY
};
AWS.config.update({credentials: credentials, region: 'eu-west-2'});
var s3 = new AWS.S3();
var presignedGETURL = s3.getSignedUrl('getObject', {
Bucket: 'presignedurldemo',
Key: 'image.jpg', //filename
Expires: 100 //time to expire in seconds
});

Create a file “presignedPUTURL.js” snippet for generating the presigned PUT URL

var presignedPUTURL = s3.getSignedUrl('putObject', {
Bucket: 'presignedurldemo',
Key: 'user12/image.jpg', //filename
Expires: 100 //time to expire in seconds
});

Using the presigned URLs

Using the GET URL, you can simply use it in any web browser. To use the PUT URL, you can use POSTMAN in the configuration as per below. You can attach a file in the body of the PUT request in a binary format.

A successfully uploaded image file

So are there any drawbacks?

At the time of writing, the pre-signed URLs (PUT & GET) do not support limiting the file size. Given that a PUT HTTP request using the presigned URL is a ‘single’-part upload, the object size is limited to 5GB. Using a post presigned URL, however, does give you more flexibility when implementing file upload in your apps. An object, for example, can be uploaded using the multipart upload API as well as limited in size and be a max size of 5TB.

Presigned POST URLs

The POST presigned, like PUT, allows you to add content to an S3 bucket. The GET method only allows you to GET from an S3 bucket. The POST presigned URL takes a lot more parameters than the PUT Presigned URL and is slightly more complex to incorporate into your application. It allows you to upload to S3 directly using an HTML form.

POST URL Parameters

A high-level overview of the required parameters in this article can be found below, however, a thorough description for all parameters for this can be found in AWS Documentation; https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-HTTPPOSTConstructPolicy.html

1. Bucket: process.env.S3_BUCKET (The bucket name)
2. Expires: 1800 (Time to expire in seconds (30m))
3. key: ‘image.jpg’ (Filename)
4. { acl: ‘private’ } (It defines which AWS accounts or groups are granted access and the type of access.)
5. { success_action_status: “201” } (HTTP status code returned if successful)
6. [‘starts-with’, ‘$key’, ‘’] (The value must start with the specified value (e.g. ‘user1/’. In our case image has no additional prefix ‘’)
7. [‘content-length-range’, 0, 100000] (Specify the range of the content you are uploading in Bytes)
8. {‘x-amz-algorithm’: ‘AWS4-HMAC-SHA256’} (Specify the signing algorithm used during signature calculation)

How can I secure this further?

CORS!

Using CORS, you can specify where the S3 can be initiated (AllowedOrigin). The star (*) notation means any (e.g. any origin allowed). You can edit the CORS configuration by selecting the CORS configuration button permissions tab when in a bucket.

The example below allows access from any URL and multiple HTTP methods.

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedMethod>PUT</AllowedMethod>
<AllowedMethod>POST</AllowedMethod>
<MaxAgeSeconds>3000</MaxAgeSeconds>
<AllowedHeader>Authorization</AllowedHeader>
</CORSRule>
</CORSConfiguration>

A full working example of the presigned POST URL can be found below on Github.

git clone https://github.com/abhibvp003/s3_presigned_url.git

Remember you need to add a .env file containing the environment variables below and specify your values.

S3_ACCESS_KEY=anaccesskeyishere
S3_SECRET_KEY=asecretkeyishere
S3_BUCKET=presignedurldemo
S3_REGION=eu-west-2

References

Leonid does a great job at outlining the post presigned URL section, although they wrote the blog post prior to AWS releasing it in their JavaScript SDK. The client-side JS script was taken from his example. Definitely worth a read; https://leonid.shevtsov.me/post/demystifying-s3-browser-upload/

AWS SDK S3 Documentation: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html