How to synchronize your Google Drive with an AWS S3 Bucket?

How to synchronize your Google Drive with an AWS S3 Bucket

One of our clients had tens of terabytes of Google Drive files and folders which we needed to be accessible from a proper cloud storage provider. We also wanted those files to be accessible from their Flaneer machines, so we chose the alternative option of syncing those Google Drives in an S3 Bucket.

Synchronize you Gdrive to AWS with Rclone

If you've ever used Google Drive to store your files, and now want to move them to AWS S3, it's not very straightforward.

We are going to use rclone to synchronize everything. Rclone can be installed on any computer, and it lets you set up a connection between more than 40 cloud storage providers.

What is Rclone? 

Rclone mounts any local, cloud, or virtual filesystem as a disk on Windows, macOS, Linux, and FreeBSD, and also serves these over SFTP, HTTP, WebDAV, FTP, and DLNA

First of all, let's install Rclone on our computer.

If you're on Linux or Mac OS X, open a terminal window and type:

On Windows 10 or Windows 7, download the latest version of Rclone for Windows at https://rclone.org/downloads/ and install it by double-clicking on the downloaded file.

You should now have a program called rclone in your start menu or desktop shortcut bar.

Use your AWS account

Now we need to create an Amazon Web Services account if we don't already have one - if you don't have one here is a link where you can sign up for free: https://portal.aws.amazon.com/billing/signup/iam

You'll need to connect rclone with two services: Google Drive and Amazon S3. This will happen through what rclone calls a remote. We'll create two remotes, one for each service.

Actions to do in your Google Drive first

Launch a terminal to begin using rclone.

To set up an account, enter the following in your terminal:

‍

Input remote-g-drive as the name of the remote, and then drive to select the Google Drive storage. Now follow all of the default settings, and when you're asked to connect to your Google Account, do so.

How to connect to AWS

You have two options for connecting to AWS: you can either enter your credentials directly or create a dedicated rclone user. I recommend the second option, as it's more secure! If you want to know how to set up an AWS access key pair and use it with rclone, visit this page.

Open a terminal to launch rclone. You can now type rclone config. Input remote-aws as the name of the remote, and then s3 followed by AWS. If you created your keys, type 1 and then paste them. Otherwise, you can type 2 and leave the 2 keys blank. You can now follow all of the default settings.

Copy your Google Drive files into your AWS S3 Account

We are going to use the following command:

Some notes:

• to be able to see files/folders share with you, can you add the --drive-sharedwithme option after the name of the google drive folder:

• We to copy all of the files/folders from the google drive account, you can leave it empty after the name of the remote

‍