Backup Files And Folders To Amazon S3 From Linux Terminal

Amazon S3 (Simple Storage Service) offers a flexible option to store files/folders and backups of websites in the cloud. It is a part of various web services Amazon offers like EC2, CloudFront and so on. These are collectively known as AWS (Amazon Web Services).

Creating an AWS account is free. New users get 5 GB of Amazon S3 standard storage, 20,000 Get Requests, 2,000 Put Requests, and 15GB of data transfer out for free for each month of the first year.

The way S3 works is to first make a “bucket” and then storing/uploading data in that bucket.

While uploading files directly in S3 through it’s web console is simple, it can be a bit complex when trying to access S3 directly from Linux Terminal.

Let’s take a look at how to backup an entire website (consisting of lots of files and folders) to Amazon S3 through Linux Terminal. (This is also applicable to backing up specific files/folders from local Linux systems as well).

This example assumes an already existing AWS account, a Linux server access using SSH (root access isn’t needed as this will work even on shared hosting for backing up websites to Amazon S3 as long as SSH is enabled).

Prerequisites :

Access to Linux server from where website content needs to be backed up. This system should have Python version 2.6.3 installed or greater (which most web hosts already have).

This can be checked by the following Linux command :

python --version

Amazon S3 bucket should be created. This can be done by signing into AWS S3 console, choosing Services > S3 and clicking “Create Bucket”.

Once the S3 bucket is created, right click on the bucket name and choose “Properties”.

From the right side of AWS pane, note down the information displayed especially “Region”. Also, from the “Permissions” tab, make sure that your username has all the permissions to the created S3 bucket.

Access to AWS private and public keys. These can be obtained by clicking on your username in AWS and choosing “Security and Credentials”.

Then clicking “Create New Access Key”.

Note down both the keys and download the key file if needed. Keep this information somewhere secure.

So to sum up, the following components need to be available before trying to backup content from Linux Terminal to Amazon S3 :

Python version 2.6.3 or greater installed on the host system.

A created Amazon S3 bucket.

Details of S3 bucket like Region and user id who has access to it.

Access Key ID and Secret Access Key details.

Now, to access S3 bucket from Linux Terminal, AWS command line interface needs to be installed and configured (AWS CLI). This is a tool for managing AWS products directly using commands.

Important : Amazon S3 is a storage service which bills only for as much as you use it. Use the S3 pricing calculator to get an idea of costs as it calculates this based on number of GET and PUT requests. The pricing details can be found here.

So to avoid a lot of individual GET and PUT requests for large number of files, it can be more economical to backup a compressed archive and restore it back when needed. This is simple to do using the tar command in Linux.