How To Upload Csv File On S3 Bucket Using Nodejs

Uploading files to AWS S3 using Nodejs

By 06th Mar, 2018

Category: Blog
457

Uploading files to AWS S3 using Nodejs

AWS S3. A place where yous tin store files. That'south what most of y'all already know about it. S3 is i of the older service provided by Amazon, earlier the days of revolutionary Lambda functions and game changing Alexa Skills.You can store almost any type of files from doc to pdf, and of size ranging from 0B to 5TB.

According to their official docs, AWS S3 is,

"S3 provides comprehensive security and compliance capabilities that meet fifty-fifty the most stringent regulatory requirements. Information technology gives customers flexibility in the way they manage data for cost optimization, admission control, and compliance. " - AWS Docs

If I endeavour to put it in uncomplicated terms, AWS S3, is an object based storage system where every file your store is saved as object non file. There are many big tech-wigs, which uses S3, and Dropbox is one of them. Recently, Dropbox starts saving metadata of their file in their own service but they are still saving primary data in S3. Why? Well, S3 is not that expensive and it's 99.9% bachelor. Plus, you get the modify to use services similar Glacier which tin save data and charge almost $0.01 per GB.

Then far, if I have gotten your attending, and you're thinking how to use S3 in my nodejs application. Well, you don't take to wait for long.

AWS has official packet which exposes S3 apis for node js apps and makes information technology also easy for developers to access S3 from their apps.

Source: https://youtu.be/FLolHgKRTKg

In next few steps, I will guide you to build a nodejs based app, which can write any file to AWS S3.

ane. Fix node app

A basic node app ordinarily have two file, bundle.json (for dependencies) and a starter file (like app.js, alphabetize.js, server.js).

You tin use your Os'south file managing director or your favourite IDE to create project but I commonly prefer CLI. So, let'due south become into our crush and follow these commands:

mkdir s3-contacts-upload-demo cd s3-contacts-upload-demo affect index.js .gitignore npm init

If you didn't get any fault in above commands, you will have a folder by the name of s3-contacts-upload-demo, 3 files in it, package.json, .gitignore andindex.json. You can use this file to add file in your .gitignore file, so that these won't get committed to github or some other version control.

npm init creates a package.json, which have project's details, you can merely hit enter in your shell for default values if you wish.

Set up node app

2. Install dependencies

Let's beginning by installing the NPM package:

npm install --relieve aws-sdk

After successful installation of this bundle, y'all can too check your package.json file, it volition have aws-sdk listed in "dependencies" field.

This npm parcel tin can be used to access any AWS service from your nodejs app, and hither nosotros will use information technology for S3.

3. Import packages

Once installed, import the bundle in your code:

const fs = require('fs'); const AWS = require('aws-sdk'); const s3 = new AWS.S3({ accessKeyId: process.env.AWS_ACCESS_KEY, secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY });

Every bit yous tin can find, we have also imported fs package which volition be used to write file data in this app. We are also using environment variables, to set up AWS access and hugger-mugger access key, as it is a bad practice to put them on version control like Github or SVN.

At present, yous take your S3 instance, which can admission all the buckets in your AWS account.

4. Pass bucket information and write business logic

Beneath is a simple prototype of how to upload file to S3. Here, Bucket is name of your saucepan and central is name of subfolder. So, if your bucket proper noun is "examination-saucepan" and y'all want to save file in "test-bucket/folder/subfolder/file.csv", and then value of Key should be "older/subfolder/file.csv".

Notation: S3 is object based storage and not file based. So, even in AWS console you can see nested folders, behind the scene they never get saved like that. Every object has two fields: Cardinal and Value. Key hither is just name of file and Value is data which is getting stored.

Then, if a saucepan "bucket1" has key "key1/key2/file.mp3", you tin can visualize information technology like this:

{ "bucket1": { "key1/key2/file.mp3": "<mp3-data>" } }

Below is simple snippe to upload a file,using Key and BucketName.

const params = { Bucket: 'saucepan', Cardinal: 'primal', Body: stream }; s3.upload(params, function(err, data) { console.log(err, information); });

v. File to upload to S3

First, create a file, let'southward say contacts.csv and write some data in it.

File to upload to S3

To a higher place is some dummy data for you to become started. At present you take a contacts.csv file, permit's read information technology using fs module and save it to S3.

S3 upload method returns mistake and information in callback, where data field contains location, saucepan and cardinal of uploaded file. For complete API reference, refer their official docs.

Now run this app past following command:

AWS_ACCESS_KEY=<your_access_key> AWS_SECRET_ACCESS_KEY=<your_secret_key> node index.js

We have passed our AWS keys as environment variables.

If you don't become whatever error in above snippet, your bucket should accept the file.

And that'due south it, under 30 lines of code you lot uploaded a file to AWS S3.

You can find the full projection hither.

There are lot many things you lot can practise with this package like

List out buckets and objects
Set permissions on bucket
Create, get or delete bucket and much more

I would highly recommend you to through this doc for APIs, they are very well explained with parameters you can pass to each API and response format they return.

I hope you found this mail useful and please exercise let us know in instance of whatever question or query.