How To Upload File To S3 Behind Corporate Proxy

In web and mobile applications, it's common to provide users with the power to upload information. Your application may let users to upload PDFs and documents, or media such as photos or videos. Every modernistic web server engineering science has mechanisms to permit this functionality. Typically, in the server-based surround, the process follows this period:

Application server upload process

The user uploads the file to the application server.
The application server saves the upload to a temporary space for processing.
The application transfers the file to a database, file server, or object store for persistent storage.

While the process is unproblematic, information technology can have significant side-effects on the performance of the web-server in busier applications. Media uploads are typically large, then transferring these tin represent a large share of network I/O and server CPU time. You lot must besides manage the state of the transfer to ensure that the entire object is successfully uploaded, and manage retries and errors.

This is challenging for applications with spiky traffic patterns. For instance, in a web application that specializes in sending vacation greetings, it may experience well-nigh traffic only around holidays. If thousands of users attempt to upload media around the same time, this requires you to scale out the awarding server and ensure that at that place is sufficient network bandwidth available.

Past direct uploading these files to Amazon S3, you can avoid proxying these requests through your application server. This can significantly reduce network traffic and server CPU usage, and enable your application server to handle other requests during decorated periods. S3 likewise is highly available and durable, making it an platonic persistent store for user uploads.

In this blog mail, I walk through how to implement serverless uploads and show the benefits of this arroyo. This blueprint is used in the Happy Path web application. You can download the code from this blog post in this GitHub repo.

Overview of serverless uploading to S3

When you upload directly to an S3 bucket, you must first request a signed URL from the Amazon S3 service. You tin then upload directly using the signed URL. This is two-step process for your application forepart:

Serverless uploading to S3

Call an Amazon API Gateway endpoint, which invokes the getSignedURL Lambda part. This gets a signed URL from the S3 saucepan.
Direct upload the file from the application to the S3 bucket.

To deploy the S3 uploader example in your AWS account:

Navigate to the S3 uploader repo and install the prerequisites listed in the README.md.
In a concluding window, run:
git clone https://github.com/aws-samples/amazon-s3-presigned-urls-aws-sam
cd amazon-s3-presigned-urls-aws-sam
sam deploy --guided
At the prompts, enter s3uploader for Stack Name and select your preferred Region. One time the deployment is complete, annotation the APIendpoint output.The API endpoint value is the base URL. The upload URL is the API endpoint with /uploads appended. For example: https://ab123345677.execute-api.us-west-two.amazonaws.com/uploads.

CloudFormation stack outputs

Testing the application

I prove two ways to test this application. The first is with Postman, which allows you to directly call the API and upload a binary file with the signed URL. The 2nd is with a basic frontend awarding that demonstrates how to integrate the API.

To test using Postman:

First, copy the API endpoint from the output of the deployment.
In the Postman interface, paste the API endpoint into the box labeled Enter asking URL.
Choose Send.
After the asking is complete, the Trunk section shows a JSON response. The uploadURL aspect contains the signed URL. Copy this attribute to the clipboard.
Select the + icon side by side to the tabs to create a new request.
Using the dropdown, alter the method from GET to PUT. Paste the URL into the Enter request URL box.
Choose the Body tab, then the binary radio button.
Cull Select file and choose a JPG file to upload.
Choose Send. You lot see a 200 OK response after the file is uploaded.
Navigate to the S3 console, and open the S3 saucepan created by the deployment. In the bucket, you see the JPG file uploaded via Postman.

To test with the sample frontend application:

Re-create alphabetize.html from the case's repo to an S3 bucket.
Update the object's permissions to make information technology publicly readable.
In a browser, navigate to the public URL of index.html file.
Select Cull file then select a JPG file to upload in the file picker. Choose Upload image. When the upload completes, a confirmation message is displayed.
Navigate to the S3 console, and open the S3 bucket created by the deployment. In the saucepan, you see the 2d JPG file you lot uploaded from the browser.

Understanding the S3 uploading process

When uploading objects to S3 from a web awarding, you must configure S3 for Cross-Origin Resource Sharing (CORS). CORS rules are defined equally an XML certificate on the bucket. Using AWS SAM, you tin can configure CORS as part of the resource definition in the AWS SAM template:

                      S3UploadBucket:     Type: AWS::S3::Bucket     Properties:       CorsConfiguration:         CorsRules:         - AllowedHeaders:             - "*"           AllowedMethods:             - GET             - PUT             - HEAD           AllowedOrigins:             - "*"

The preceding policy allows all headers and origins – it's recommended that yous utilise a more restrictive policy for production workloads.

In the first step of the procedure, the API endpoint invokes the Lambda function to brand the signed URL request. The Lambda function contains the following code:

          const AWS = require('aws-sdk') AWS.config.update({ region: process.env.AWS_REGION }) const s3 = new AWS.S3() const URL_EXPIRATION_SECONDS = 300  // Chief Lambda entry indicate exports.handler = async (event) => {   return await getUploadURL(event) }  const getUploadURL = async function(event) {   const randomID = parseInt(Math.random() * 10000000)   const Key = `${randomID}.jpg`    // Become signed URL from S3   const s3Params = {     Bucket: process.env.UploadBucket,     Key,     Expires: URL_EXPIRATION_SECONDS,     ContentType: 'prototype/jpeg'   }   const uploadURL = await s3.getSignedUrlPromise('putObject', s3Params)   return JSON.stringify({     uploadURL: uploadURL,     Key   }) }

This function determines the name, or key, of the uploaded object, using a random number. The s3Params object defines the accepted content type and too specifies the expiration of the cardinal. In this instance, the key is valid for 300 seconds. The signed URL is returned every bit part of a JSON object including the fundamental for the calling application.

The signed URL contains a security token with permissions to upload this unmarried object to this bucket. To successfully generate this token, the code calling getSignedUrlPromise must take s3:putObject permissions for the bucket. This Lambda function is granted the S3WritePolicy policy to the bucket by the AWS SAM template.

The uploaded object must match the same file name and content blazon as divers in the parameters. An object matching the parameters may be uploaded multiple times, providing that the upload procedure starts before the token expires. The default expiration is xv minutes but you may want to specify shorter expirations depending upon your use instance.

Once the frontend application receives the API endpoint response, information technology has the signed URL. The frontend application and then uses the PUT method to upload binary information directly to the signed URL:

          allow blobData = new Blob([new Uint8Array(array)], {type: 'image/jpeg'}) const result = wait fetch(signedURL, {   method: 'PUT',   body: blobData })

At this bespeak, the caller application is interacting directly with the S3 service and not with your API endpoint or Lambda function. S3 returns a 200 HTML condition code once the upload is consummate.

For applications expecting a large number of user uploads, this provides a simple way to offload a large amount of network traffic to S3, away from your backend infrastructure.

Adding authentication to the upload process

The current API endpoint is open, bachelor to any service on the net. This means that anyone can upload a JPG file once they receive the signed URL. In most production systems, developers desire to use hallmark to control who has admission to the API, and who can upload files to your S3 buckets.

You lot tin can restrict access to this API by using an authorizer. This sample uses HTTP APIs, which support JWT authorizers. This allows yous to control admission to the API via an identity provider, which could be a service such as Amazon Cognito or Auth0.

The Happy Path application merely allows signed-in users to upload files, using Auth0 equally the identity provider. The sample repo contains a second AWS SAM template, templateWithAuth.yaml, which shows how y'all can add an authorizer to the API:

                      MyApi:     Blazon: AWS::Serverless::HttpApi     Properties:       Auth:         Authorizers:           MyAuthorizer:             JwtConfiguration:               issuer: !Ref Auth0issuer               audition:                 - https://auth0-jwt-authorizer             IdentitySource: "$request.header.Authorization"         DefaultAuthorizer: MyAuthorizer

Both the issuer and audience attributes are provided past the Auth0 configuration. By specifying this authorizer as the default authorizer, it is used automatically for all routes using this API. Read part 1 of the Enquire Around Me series to learn more about configuring Auth0 and authorizers with HTTP APIs.

After authentication is added, the calling web application provides a JWT token in the headers of the request:

          const response = look axios.get(API_ENDPOINT_URL, {   headers: {     Potency: `Bearer ${token}`         } })

API Gateway evaluates this token before invoking the getUploadURL Lambda office. This ensures that but authenticated users can upload objects to the S3 bucket.

Modifying ACLs and creating publicly readable objects

In the current implementation, the uploaded object is not publicly accessible. To make an uploaded object publicly readable, yous must set up its access control list (ACL). In that location are preconfigured ACLs available in S3, including a public-read option, which makes an object readable by anyone on the internet. Set the appropriate ACL in the params object before calling s3.getSignedUrl:

          const s3Params = {   Bucket: process.env.UploadBucket,   Cardinal,   Expires: URL_EXPIRATION_SECONDS,   ContentType: 'image/jpeg',   ACL: 'public-read' }

Since the Lambda function must have the appropriate bucket permissions to sign the request, yous must likewise ensure that the part has PutObjectAcl permission. In AWS SAM, you can add together the permission to the Lambda function with this policy:

                      - Statement:           - Effect: Allow             Resources: !Sub 'arn:aws:s3:::${S3UploadBucket}/'             Activeness:               - s3:putObjectAcl

Conclusion

Many web and mobile applications allow users to upload data, including large media files like images and videos. In a traditional server-based application, this tin create heavy load on the awarding server, and also utilize a considerable amount of network bandwidth.

By enabling users to upload files to Amazon S3, this serverless blueprint moves the network load abroad from your service. This tin brand your application much more than scalable, and capable of handling spiky traffic.

This blog mail service walks through a sample application repo and explains the procedure for retrieving a signed URL from S3. It explains how to the test the URLs in both Postman and in a web awarding. Finally, I explain how to add authentication and make uploaded objects publicly accessible.

To learn more than, see this video walkthrough that shows how to upload directly to S3 from a frontend spider web awarding. For more than serverless learning resources, visit https://serverlessland.com.