How do I transfer files to and from Yen Servers to Google Drive using rclone?

For versatile data transfers, you can employ rclone, which supports various destinations, including Google Drive, Amazon S3, Dropbox and others.

Using rclone to Transfer Files to Google Drive

Using rclone Locally

On Windows: download from here.

On Mac: install rclone with:

$ curl https://rclone.org/install.sh | sudo bash

Using rclone on Yen Servers

To utilize rclone the yens, load the rclone module:

$ ml rclone

Verify the loaded module with:

$ ml

Currently Loaded Modules:
  1) rclone/1.63.1

Setting up rclone

Before we can push data from the yens to Google Drive, we need to configure rclone once.

$ rclone config

The configuration menu will be presented:

No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> nrapstinGoogleDrive

Select n to make a new remote and give it a name when prompted. For example, $USERGoogleDrive where $USER is your SUNet ID.

Next, select the number corresponding to Google Drive (the menu changes with rclone version so be careful to select the right remote).

18 / Google Drive
   \ (drive)
Storage> 18

When prompted for the next two options, leave them blank and press Enter.

Then the next menu asks to select permissions you want to give rclone. Choose 1 for full read-write access.

scope> 1

Then leave the next prompt blank and press Enter.

Choose n to not edit advanced config:

Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n

Choose n again since we are working on the remote Yen server:

Remote config
Use auto config?
 * Say Y if not sure
 * Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n> n

Next, we need to finish configuring Google drive using the local machine. For that, you will need to install rclone locally and in the local terminal, run

$ rclone authorize "drive" "xxxxxxxxxxxxxxxxxxxxxxx"

where “xxxxxxxxxxxxxxxxxxxxxxx” is the config token that you see in the Yen terminal from the previous step.

This will open up a local web browser in which you can authenticate into your Google Drive using your Stanford account.

Once you authorize rclone for access, Google Drive will give a code to paste back into the Yen terminal. Copy the code and paste it back into the Yen terminal after config_token>.

Next, you will be asked if you want to configure this as a team drive. Press y if you are connecting to a shared Google Drive or press n if you are connecting to your Google Drive.

Configure this as a team drive?
y) Yes
n) No (default)

Finally, press Enter to complete the config.

y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d>

In the last prompt, hit q to quit. Now, rclone should be set up to push files from the yens to your Google Drive.

Using rclone

Here are common rclone commands for transferring files between Yen servers and Google Drive:

  • list remote connections:
$ rclone listremotes
  • Create a remote folder on Google Drive (within your Google Drive base folder):
$ rclone mkdir $USERGoogleDrive:GoogleDriveFolderName

Alternatively, you can specify the path to the new folder on Google Drive:

$ rclone mkdir $USERGoogleDrive:myFolder/subfolder/data

where I already had myFolder directory on my Google drive and within myFolder I have already created subfolder. This rclone command will make a new folder – data.

  • List contents of a remote folder on Google Drive:
$ rclone ls $USERGoogleDrive:GoogleDriveFolderName
  • Upload a directory to Google Drive from Yen Servers:
$ rclone copy /Yen_Path/To/Folder/ $USERGoogleDrive:GoogleDriveFolderName/

where /Path/To/Folder/ is the path on the yens to the directory you want to upload.

  • Download from Google Drive to Yen Servers:
$ rclone copy $USERGoogleDrive:GoogleDriveFolderName /Yen_Path/To/Folder/

where /Yen_Path/To/Folder/ is the path on the yens where you want to copy files to.

Explore more details in the official rclone documentation.