Compute Questions

Software Questions

Data Questions

Archive Data to Google Drive

Archiving your data to Google Drive (GDrive) using RClone is a free, fast, and secure way to free up on-premises storage. Google Drive can be used with high-risk data following Stanford’s Minimum Security Standards. Our ZFS storage solution is finite, measuring roughly one-half petabyte of space, which means that space is limited to researchers in perpetuity.

Configuring RClone for Google Drive

You must take the following steps to configure Rclone for your Stanford Google Drive account. These configuration steps only need to be taken once, and then you can use your new “remote” as a target for copying data.

  • Log in using your Stanford credentials. Login instructions can be found here.
  • Type module load rclone && rclone config The following menu will appear.

Config Setup

  • Select n to create a new remote connection to google drive. When prompted, give it a name; in this example, I’ll use smancusoGoogleDrive
  • The next screen will display all the possible connections with the version of RClone installed on the machine. This menu changes between versions, so be sure to select the number corresponding to the “Google Drive.”

Prompt Map

  • When prompted for the next two options, leave them blank and hit return/enter.
    • Google Application Client Id - leave blank normally.
    • Google Application Client Secret - leave blank normally.
  • Choose “N” on this prompt “Say N if you are working on a remote or headless machine or Y didn’t work.” Since the Yen cluster is headless, we must be given a unique URL to validate the connection.
  • On the next prompt, RClone provides you with a unique URL that you need to paste into a browser. After logging in with your Stanford credentials, Google Drive will give a code to paste back into the terminal window.

Prompt Map

  • Finally, to finish the configuration in the terminal window setup, click “Y” and “enter” to complete the process if you believe everything is properly setup. In the last prompt, hit “q” to quit. You have setup RClone successfully on your Google Drive.

RClone Common Operations

Initiaitaing a current version of RClone

ml rclone

** OR **

moudle load rclone

Note: this needs to be run once per server per terminal session. After you have initiated module load rclone, you will only need to call rclone as normally from the command line.

To list connection points

rclone listremotes

Create a remote folder on Google Drive using Rclone. Note this will make the folder within your Google Drive base folder.

rclone mkdir smancusoGoogleDrive:GoogleDriveFolderName

To upload contents of a directory to Google Drive using copy and Rclone.

rclone copy /Path/To/Folder/ smancusoGoogleDrive:GoogleDriveFolderName/

List contents of remote folder on Google Drive

rclone ls smancusoGoogleDrive:GoogleDriveFolderName

Download from remote Google Drive.

rclone copy smancusoGoogleDrive:GoogleDriveFolderName /Path/To/Local/Download/Folder