How to Send a File to S3 Image Upload

• Language Back up
• Node.js
• Direct to S3 File Uploads in Node.js

This commodity was contributed by Will Webberley

Will is a computer scientist and is enthused by well-nigh all aspects of the technology domain. He is specifically interested in mobile and social computing and is currently a researcher in this area at Cardiff University.

Direct to S3 File Uploads in Node.js

English — 日本語に切り替える

Last updated March 09, 2022

Tabular array of Contents

• Uploading direct to S3
• Overview
• Prerequisites
• Initial setup
• Direct uploading
• Running the app
• Summary

Web applications ofttimes require the ability to let users to upload files such every bit images, movies and archives. Amazon S3 is a popular and reliable storage option for these files.

This article demonstrates how to create a Node.js application that uploads files directly to S3 instead of via a spider web application, utilising S3's Cantankerous-Origin Resource Sharing (CORS) back up. The Express spider web framework is used to facilitate request-handling in the examples below, but the procedure should be nigh identical in any Node.js application.

Uploading directly to S3

A complete example of the code discussed in this article is bachelor for direct use in this GitHub repository.

The main advantage of straight uploading is that the load on your awarding'southward dynos would exist considerably reduced. Using app-side processes for receiving files and transferring to S3 can needlessly tie upwardly your dynos and volition hateful that they volition not be able to respond to simultaneous spider web requests as efficiently.

The awarding uses client-side and app-side JavaScript for signing the requests. The actual upload is carried out asynchronously then that you can decide how to handle your awarding'southward menses afterward the upload has completed (for example, you may wish to redirect users to another page upon successful upload rather than a full page refresh).

An example simple account-editing scenario is used as a guide for completing the various steps required to accomplish the directly upload and to relate the awarding of this to a wider range of apply-cases. More than data on this scenario is provided later.

Overview

S3 is comprised of a set of buckets, each with a globally unique name, in which individual files (known as objects) and directories, can be stored.

For uploading files to S3, y'all will need an Access Key ID and a Secret Admission Fundamental, which act as a username and countersign. The access key business relationship volition need to have sufficient access privileges to the target bucket in order for the upload to exist successful.

Please see the S3 Article for more information on this, creating buckets and handling your authentication keys.

In general, the method described in this article follows these unproblematic steps:

• A file is selected for upload past the user in their web browser;
• The user'due south browser makes a request to your web awarding on Heroku, which produces a temporary signature with which to sign the upload request;
• The temporary signed request is returned to the browser in JSON format;
• The browser then uploads the file directly to Amazon S3 using the signed request supplied by your Node.js application.

This guide includes information on how to implement the client-side and app-side lawmaking to form the complete system. After following the guide, you should take a working barebones organization, allowing your users to upload files to S3. Nevertheless, it is unremarkably worth adding actress functionality to aid improve the security of the arrangement and to tailor it for your own item uses. Pointers for this are mentioned in the appropriate parts of the guide.

The signature generation on the server uses AWS'southward official SDK, as explained later. Please meet their documentation for data on the features of this SDK.

Prerequisites

• The Heroku CLI has been installed;
• Node.js has been installed;
• A Heroku application has been created for the current project;
• An AWS S3 bucket has been created;
• You lot have AWS authentication keys that have write access to the bucket.

Initial setup

S3 setup

You will now need to edit some of the permissions properties of the target S3 bucket so that the terminal request has sufficient privileges to write to the bucket. In a spider web-browser, sign in to the AWS console and select the S3 department. Select the appropriate bucket and click the Permissions tab. A few options are at present provided on this page (including Block public access, Admission Command List, Bucket Policy, and CORS configuration).

Firstly, ensure that "Block all public access" is turned off, and in item plough off "Block public admission to buckets and objects granted through new access control lists" and "Cake public access to buckets and objects granted through whatsoever access control lists" for the purposes of this projection. Setting up the bucket in this manner allows united states to read its contents without signed URLs, but this may not exist suitable for services running in production.

Side by side, you volition need to configure the bucket'southward CORS (Cantankerous-Origin Resource Sharing) settings, which volition allow your awarding to access content in the S3 bucket. Each dominion should specify a set of domains from which access to the bucket is granted and also the methods and headers permitted from those domains.

For this to work in your application, click 'Edit' and enter the post-obit JSON for the saucepan's CORS settings:

            [     {         "AllowedHeaders": [             "*"         ],         "AllowedMethods": [             "GET",             "Caput",             "Mail",             "PUT"         ],         "AllowedOrigins": [             "*"         ],         "ExposeHeaders": []     } ]                      

Click 'Salve changes' and close the editor.

This tells S3 to permit whatever domain access to the saucepan and that requests can contain any headers, which is mostly fine for testing. When deploying, you lot should change the 'AllowedOrigin' to just accept requests from your domain.

If yous wish to use S3 credentials specifically for this application, and then more keys tin be generated in the AWS account pages. This provides further security, since you lot tin designate a very specific prepare of requests that this prepare of keys are able to perform. If this is preferable to you lot, and then you will need to configure your IAM users in the Edit saucepan policy selection in your S3 bucket. There are various guides on AWS'due south web pages detailing how this can exist accomplished.

App setup

If your app hasn't however been setup, then it is useful to do and then at this stage. To go started, create a directory somewhere on your local machine:

            $ mkdir NodeDirectUploader                      

Now create two further subdirectories of NodeDirectUploader/ to respectively contain your HTML pages and support files:

            $ cd NodeDirectUploader $ mkdir views $ mkdir public                      

Node's parcel manager, npm, should have been installed by default along with Node and can exist used to handle the installation and updates of the required packages for your app. To begin this, run Node'due south interactive parcel setup tool in the root of your app directory:

            $ npm init                      

The tool will enquire some questions nearly your app, including its proper name, description, licensing, and version-control, and create a file chosen package.json in the app'southward root. This file uses your responses to maintain information almost your app, which you tin edit freehand every bit you develop farther.

The same file tin exist used to easily declare your app's dependencies, which volition facilitate the deployment and share-power of your app. To do so, edit package.json and add a "dependencies" JSON object to incorporate the following parcel dependencies:

            {   "name": "NodeDirectUploader",   "version": "0.0.1",    ...    "dependencies": {     "aws-sdk": "2.x",     "ejs": "2.ten",     "express": "four.x"   } }                      

These dependencies can then be installed using npm:

            $ npm install                      

Use of these packages volition go articulate afterward, and installation of them in this style allows for greater command of your per-app dependencies every bit your apps grow.

Heroku setup

In order for your application to access the AWS credentials for signing upload requests, they will demand to be added as configuration variables in Heroku:

            $ heroku config:ready AWS_ACCESS_KEY_ID=xxx AWS_SECRET_ACCESS_KEY=yyy Adding config vars and restarting app... done, v21     AWS_ACCESS_KEY_ID => xxx     AWS_SECRET_ACCESS_KEY => yyy                      

In addition to the AWS admission credentials, set your target S3 bucket'southward name:

            $ heroku config:set S3_BUCKET=zzz Adding config vars and restarting app... done, v21     S3_BUCKET     => zzz                      

Using config vars is preferable over configuration files for security reasons. Avert placing passwords and admission keys directly in your application's code or in configuration files. Delight encounter the article Configuration and Config Vars for more information.

Setting up local surroundings variables for your app is useful for running and testing your app locally. For more information, come across the Prepare your local environment variables section of the Heroku Local article. Information on launching your app locally is provided afterwards in this article.

Call back to add the .env file to your .gitignore, since this file should only exist used for local testing.

Direct uploading

The processes and steps required to accomplish a straight upload to S3 will be demonstrated through the use of a unproblematic profile-editing scenario for the purposes of this article. This example will involve the user being permitted to select an avatar paradigm to upload and enter some basic information to exist stored equally part of their account.

In this scenario, the following procedure volition take place:

• The user is presented with a web page, containing elements encouraging the user to choose an epitome to upload as their avatar and to enter a username and their own proper name.
• An chemical element is responsible for maintaining a preview of the chosen image by the user. By default, and if no image is chosen for upload, a default avatar image is used instead (making the image-upload effectively optional to the user in this scenario).
• When a user selects an image to be uploaded, the upload to S3 is handled automatically and asynchronously with the process described earlier in this article. The epitome preview is so updated with the selected image one time the upload is consummate and successful.
• The user is then free to move on to filling in the remainder of the information.
• The user then clicks the "submit" button, which posts the username, name and the URL of the uploaded image to the Node awarding to be checked and/or stored. If no image was uploaded by the user earlier the default avatar epitome URL is posted instead.

Setting up the customer-side code

No third-party lawmaking is required to complete the implementation on the client-side.

The HTML and JavaScript tin now be created to handle the file choice, obtain the request and signature from your Node application, and then finally make the upload request.

Firstly, create a file called account.html in your awarding's views/ directory and populate the head and other necessary HTML tags appropriately for your awarding. In the torso of this HTML file, include a file input and an element that will comprise status updates on the upload progress. In addition to this, create a course to allow the user to enter their username and full name and a hidden input element to hold the URL of the called avatar image:

To see the completed HTML file, please run into the advisable code in the companion repository.

            <input type="file" id="file-input"> <p id="status">Please select a file</p> <img id="preview" src="/images/default.png">  <form method="POST" activeness="/salvage-details">   <input type="subconscious" id="avatar-url" name="avatar-url" value="/images/default.png">   <input type="text" name="username" placeholder="Username"><br>   <input type="text" name="full-name" placeholder="Total name"><br><br>   <input type="submit" value="Update profile"> </class>                      

The #preview element initially holds a default avatar prototype (which would go the user's avatar if a new paradigm is not chosen), and the #avatar-url input maintains the electric current URL of the user's chosen avatar image. Both of these are updated past the JavaScript, discussed below, when the user selects a new avatar.

Thus when the user finally clicks the submit button, the URL of the avatar is submitted, along with the username and total name of the user, to your desired endpoint for server-side handling.

The customer-side code is responsible for achieving two things:

1. Recollect a signed request from the app with which the epitome tin be PUT to S3
2. Actually PUT the paradigm to S3 using the signed request

JavaScript'southward XMLHttpRequest objects can exist created and used for making asynchronous HTTP requests.

To achieve this, first create a <script> block and write some code that listens for changes in the file input, in one case the certificate has loaded, and starts the upload process.

            (() => {   document.getElementById("file-input").onchange = () => {     const files = document.getElementById('file-input').files;     const file = files[0];     if(file == null){       return alert('No file selected.');     }     getSignedRequest(file);   }; })();                      

The code likewise determines the file object itself to be uploaded. If one has been selected properly, information technology gain to telephone call a function to obtain a signed PUT asking for the file. Side by side, therefore, write a function that accepts the file object and retrieves an advisable signed request for it from the app.

            role getSignedRequest(file){   const xhr = new XMLHttpRequest();   xhr.open('Become', `/sign-s3?file-proper noun=${file.name}&file-type=${file.blazon}`);   xhr.onreadystatechange = () => {     if(xhr.readyState === 4){       if(xhr.status === 200){         const response = JSON.parse(xhr.responseText);         uploadFile(file, response.signedRequest, response.url);       }       else{         alarm('Could non go signed URL.');       }     }   };   xhr.transport(); }                      

If the name (file.name) and/or mime type (file.type) of the file you upload contains special characters (such as spaces), then they should be encoded first (e.grand. encodeURIComponent(file.proper name)).

The higher up function passes the file's proper name and mime blazon every bit parameters to the GET request since these are needed in the construction of the signed request, equally volition be covered later in this commodity. If the retrieval of the signed request was successful, the function continues by calling a function to upload the bodily file:

            role uploadFile(file, signedRequest, url){   const xhr = new XMLHttpRequest();   xhr.open('PUT', signedRequest);   xhr.onreadystatechange = () => {     if(xhr.readyState === 4){       if(xhr.status === 200){         document.getElementById('preview').src = url;         certificate.getElementById('avatar-url').value = url;       }       else{         alert('Could non upload file.');       }     }   };   xhr.transport(file); }                      

This function accepts the file to exist uploaded, the signed asking, and generated URL representing the eventual retrieval URL of the avatar image. The latter two arguments will be returned as part of the response from the app. The function, if the request to S3 is successful, and so updates the preview chemical element to the new avatar image and stores the URL in the hidden input and then that information technology can be submitted for storage in the app.

At present, once the user has completed the rest of the form and clicked submit, the name, username, and avatar image can all be posted to the same endpoint.

If you find that the page isn't working as you intend after implementing the system, and so consider using console.log() to tape whatsoever errors that are revealed inside the onreadystatechange function and apply your browser'due south error console to assist diagnose the problem.

It is good practice to inform the user of whatever prolonged activity in whatever form of awarding (spider web- or device-based) and to display updates on changes. Therefore a loading indicator could be displayed betwixt selecting a file and the upload existence completed. Without this sort of information, users may suspect that the page has crashed, and could try to refresh the page or otherwise disrupt the upload process.

Setting up the app-side Node lawmaking

This department discusses the employ of Node.js for generating a temporary signature with which the upload request tin can be signed. This temporary signature uses AWS hallmark credentials (the access key and secret key) as a basis for the signature, just users volition non accept straight access to this information. Afterwards the signature has expired, and so upload requests with the aforementioned signature will not exist successful.

To see the completed Node file, please see the appropriate lawmaking in the companion repository.

Start by creating your main application file, app.js, in the root of your application directory and prepare your skeleton application appropriately:

            const limited = require('limited'); const aws = require('aws-sdk');  const app = limited(); app.set up('views', './views'); app.employ(limited.static('./public')); app.engine('html', crave('ejs').renderFile); app.listen(process.env.PORT || 3000);  const S3_BUCKET = process.env.S3_BUCKET;                      

In some scenarios, it may exist necessary to bank check that the environment's PORT var is a number by using Number(process.env.PORT).

The packages installed with npm are imported at the top of the application. The Limited app is then set-up and finally the saucepan name is loaded from the environment.

You should now configure your AWS region. To do so, update the imported aws object. For example:

            aws.config.region = 'eu-west-1';                      

Recall to utilize the region that your target saucepan resides in. If you lot need it, use this page to discover your region.

Adjacent, in the aforementioned file, yous will need to create the views responsible for returning the correct information back to the user's browser when requests are made to various URLs. Inside the app.js file, define the view for requests to /business relationship to return the page account.html, which contains the form for the user to complete:

            app.get('/account', (req, res) => res.render('account.html'));                      

Now create the view, in the same JavaScript file, that is responsible for generating and returning the signature with which the customer-side JavaScript tin upload the image. This is the first request made by the customer earlier attempting an upload to S3. This view responds with requests to /sign-s3:

            app.get('/sign-s3', (req, res) => {   const s3 = new aws.S3();   const fileName = req.query['file-name'];   const fileType = req.query['file-type'];   const s3Params = {     Bucket: S3_BUCKET,     Key: fileName,     Expires: 60,     ContentType: fileType,     ACL: 'public-read'   };    s3.getSignedUrl('putObject', s3Params, (err, data) => {     if(err){       console.log(err);       return res.terminate();     }     const returnData = {       signedRequest: data,       url: `https://${S3_BUCKET}.s3.amazonaws.com/${fileName}`     };     res.write(JSON.stringify(returnData));     res.stop();   }); });                      

This code uses the aws-sdk module to create a signed URL that the browser can use to make a PUT asking to S3. In add-on, the prospective URL of the object to be uploaded is produced equally a combination of the S3 bucket proper noun and the object proper name. This URL and the signed request are then returned to the browser in JSON format.

The Expires parameter describes the number of seconds for which the signed URL will be valid for. In some circumstances, such as when uploading big files, a larger value may be more advisable in order to extend the validity of the signed URL.

Initialising the s3 object automatically loads the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY variables that were fix into the surround earlier.

You may wish to assign some other, customised name to the object instead of using the one that the file is already named with, which is useful for preventing accidental overwrites in the S3 saucepan. This name could be related to the ID of the user's business relationship, for example. If not, you should provide some method for properly quoting the name in case in that location are spaces or other bad-mannered characters present. In addition, this is the phase at which you lot could provide checks on the uploaded file in order to restrict access to certain file types. For example, a simple check could exist implemented to allow but .png files to proceed beyond this point.

Finally, in app.js, create the view responsible for receiving the business relationship information subsequently the user has uploaded an avatar, filled in the form, and clicked submit:

            app.postal service('/save-details', (req, res) => {   // TODO: Read POSTed form data and practise something useful });                      

This function is currently just a stub that y'all'll need to complete in order to allow the app to read and store the submitted contour information and to correctly associate it with the rest of the user'south account details.

Running the app

Everything should now be in place to perform the direct uploads to S3. To test the upload, save any changes and use heroku local to start the application:

Yous will demand a Procfile for this to be successful. Run into Getting Started with Node.js on Heroku for more information. Also recollect to correctly set your environment variables on your own machine before running the application locally.

            $ heroku local 15:44:36 web.1  | started with pid 12417                      

Press Ctrl+C to return to the prompt. If your application is returning 500 errors (or other server-based issues), then start your server in debug way and view the output in the Last emulator to help gear up your problem:

            $ DEBUG=express:* node app.js                      

Summary

This article covers uploading to Amazon S3 directly from the browser using Node.js to temporarily sign the upload asking. Although the guide and companion lawmaking focuses on the Limited framework, the idea should easily carry over to other Node applications.

denhamhosseed.blogspot.com

Source: https://devcenter.heroku.com/articles/s3-upload-node

Share this post