Prerequisites

Topics

Extras

7. Transferring Files to and from Yen Servers

In the course of your data processing pipeline, you will often need to transfer data to Yen servers for analysis and subsequently move the resulting files back to your local machine.

Using scp for File Transfers

Transferring Files to Yens Servers

To transfer one or more files, you can use the scp command. The scp command takes two arguments, the source path (what files or directories you are copying) and the destination path (where the files/directories will be copied to).

$ scp <source_path> <destination_path>

When transferring data to the Yens from your local machine, the <source_path> is the path to the file(s) on your local computer while the <destination_path> is the path on the Yens where the files from your local machine will be transferred to.

For instance, to transfer a file named mydata.txt to your project space on Yen servers, execute:

$ scp mydata.txt <SUNetID>@yen.stanford.edu:/zfs/projects/students/<my_project_dir>

The scp command uses ssh for file transfer, so you’ll be prompted for your password and Duo authentication.

If you want to transfer all CSV files from a particular directory, use the following:

$ scp *.csv <SUNetID>@yen.stanford.edu:/zfs/projects/students/<my_project_dir>

To use the Yen Data Transfer Node, we can modify the scp command as follows:

$ scp *.csv <SUNetID>@yen-transfer.stanford.edu:/zfs/projects/students/<my_project_dir>

Example

Let’s transfer an R script from your local machine to the Yen servers. On your local machine, in a terminal, run:

$ cd ~/Desktop/intro-to-yens
$ scp investment-npv-parallel.R <SUNetID>@yen.stanford.edu:~

where ~ is your Yen home directory shortcut. Enter your SUNet ID password and Duo authenticate for the file transfer to complete.

Transferring Folders to Yen Servers

On your local machine, open a new terminal and navigate to the parent directory of the folder that you want to transfer to the Yens with the cd command.

Once you are in the parent directory of the folder you want to transfer, run the following to copy the folder to the Yens:

$ scp -r my_folder/ <SUNetID>@yen.stanford.edu:/zfs/projects/students/<my_project_dir>

The -r flag is used to copy folders (recursively copy files). Replace <SUNetID> with your SUNet ID and <my_project_dir> with the destination path on the Yen’s file system, ZFS.

Let’s illustrate this with an example. We’ll create an empty folder called test_from_local and transfer it to the home directory on Yen servers:

$ mkdir test_from_local
$ scp -r test_from_local/ <SUNetID>@yen.stanford.edu:~

Transferring Files from Yen Servers

When transferring data from the Yens to your local machine, the <source_path> is the path to the file(s) on the Yens while the <destination_path> is the path on your local machine where the files from the Yens will be transferred to.

To copy files from Yen servers to your local machine, open a new terminal without connecting to the Yens. Use the cd command to navigate to the local directory where you want to copy the files to. Then run:

$ cd my_local_folder
$ scp -r <SUNetID>@yen.stanford.edu:/zfs/projects/students/<my_project_dir>/results .

In this example, we’re copying the results folder from the Yen’s ZFS file system to your local directory (. signifies the current directory). If you’re copying files (not directories), omit the -r flag. To transfer multiple files, use the wildcard * to match several files.

Using Other Transfer Tools

See rclone and rsync pages to learn two additional transfer tools.