One of our clients had tens of terabytes of Google Drive files and folders which we needed to be accessible from a proper cloud storage provider. We also wanted those files to be accessible from their Flaneer machines, so we chose the alternative option of syncing those Google Drives in an S3 Bucket.
If you've ever used Google Drive to store your files, and now want to move them to AWS S3, it's not very straightforward.
We are going to use rclone to synchronize everything. Rclone can be installed on any computer, and it lets you set up a connection between more than 40 cloud storage providers.
Rclone mounts any local, cloud, or virtual filesystem as a disk on Windows, macOS, Linux, and FreeBSD, and also serves these over SFTP, HTTP, WebDAV, FTP, and DLNA
First of all, let's install Rclone on our computer.
If you're on Linux or Mac OS X, open a terminal window and type:
On Windows 10 or Windows 7, download the latest version of Rclone for Windows at https://rclone.org/downloads/ and install it by double-clicking on the downloaded file.
You should now have a program called rclone in your start menu or desktop shortcut bar.
Now we need to create an Amazon Web Services account if we don't already have one - if you don't have one here is a link where you can sign up for free: https://portal.aws.amazon.com/billing/signup/iam
You'll need to connect rclone with two services: Google Drive and Amazon S3. This will happen through what rclone calls a remote. We'll create two remotes, one for each service.
Launch a terminal to begin using rclone.
To set up an account, enter the following in your terminal:
Input remote-g-drive as the name of the remote, and then drive to select the Google Drive storage. Now follow all of the default settings, and when you're asked to connect to your Google Account, do so.
You have two options for connecting to AWS: you can either enter your credentials directly or create a dedicated rclone user. I recommend the second option, as it's more secure! If you want to know how to set up an AWS access key pair and use it with rclone, visit this page.
Open a terminal to launch rclone. You can now type rclone config. Input remote-aws as the name of the remote, and then s3 followed by AWS. If you created your keys, type 1 and then paste them. Otherwise, you can type 2 and leave the 2 keys blank. You can now follow all of the default settings.
We are going to use the following command:
Some notes: