Topics

Yen File Storage

File Storage

You have several options for where to store your research files (data sets, programs, output files, and so forth). This guide will help you decide which storage location is best for your situation.

ZFS Directories

The GSB has 612 TB of high-performance storage available from the yen servers under the path /zfs. Currently all yen user directories (~/) reside on ZFS. In addition to user home directories, Stanford GSB faculty (and in some cases students, see below), may request additional project space on ZFS.

Tip: ZFS is mounted only from the yen servers. You cannot access it from Sherlock, FarmShare or any other system

GSB Faculty

All GSB faculty may request project space on ZFS and DARC will set up a corresponding Stanford workgroup that you may use to add and remove collaborators for your project. Quotas are set for faculty ZFS directories based on expected space requirements. However, we understand that initial space estimates may be insufficient and thus we can grant storage quota increases upon request later. But kindly do your part and be a good steward of the commons by

Archiving projects or infrequently accessed files when you can (see here)
Deleting intermediate files that you no longer need to use

GSB Students

While ZFS is primarily a resource for Stanford GSB faculty, under certain conditions Stanford GSB graduate students may also be granted project space in ZFS. If you feel you are in need of student project space on ZFS please contact us at gsb_darcresearch@stanford.edu.

All student project spaces are hard limited to 200 GB and are marked as a priority for purging should disk space on ZFS ever become an issue. Disk purging will always be preceded by adequate notification!

Backups

Files on ZFS are backed up as “snapshots” and can be restored manually by any user. Please see the page How Do I Recover ZFS Files for instructions on recovering files. We also maintain an off-site disaster recovery solution for both ZFS and home directories.

Yen Home Directories

Home directories on the Yen servers are also stored on our storage system. Your home directory has a quota of 25GB.

Tip: To check your home directory quota, use the gsbquota command

When you log into the Yen servers you will automatically land in your home directory, which is located at /home/users/{SUNet-id} or with the shortcut ~. You are able to access your former AFS home directory by following the afs-home symlink inside your home directory.

Important: Please note that if you continue to work and save files within your older afs-home directory you will still be limited to the old AFS disk quota (5GB)

Local Disk

On each Yen machine, there is local scratch space mounted at /scratch. All yen users are free to make use of this space. Much like a hard drive on your laptop, this can be accessed only from that single Yen machine.

Warning: Note that scratch space on all yens is cleared during system reboots, and is subject to intermittent purging as needed by the admins. Therefore local scratch space is usually best only for temporary files

AFS Volumes

You have a personal AFS volume that is named according to your SUNetID. For example If your SUNetID is johndoe13, then then the path to your AFS directory is: /afs/ir/users/j/o/johndoe13. The two individual letters are the first two letters of the SUNetID.

You may have access to other AFS volumes set up for specific projects, or other people may give you access to a specific directory in their AFS volume. To access other AFS volumes, you need to know what the path is. For example, the path might be something like /afs/ir/data/gsb/nameofyourdirectory.

How to access an AFS volume

There are two options for transferring files to and from AFS:

From your desktop using OpenAFS, a free download available from Stanford. This software will mount your AFS directory so that you can access it using an Explorer (Windows) or Finder (Mac) window as you do with other files.
Through a web interface: https://afs.stanford.edu/. When you go to this url, it will take you to your home directory. To go to a different directory, click the Change button at the top of the page under Current AFS Directory Path.

Important: Please note that AFS is currently being phased out by the university, and will eventually be retired completely. Adequate notice will be provided by University IT prior to any portions of AFS being brought offline

How to create an AFS volume

If you are working with a faculty member on a project that uses AFS, chances are that person already has an AFS directory created for that project. Just ask the faculty member what the path to the directory is, and to grant you permissions to use it.

Size Limitations

As of this writing, AFS volumes at Stanford can be as large as 256 GB. However, it is possible to chain multiple volumes together in one Linux directory using symbolic links.