How Do I Check My Resource Usage on the Yens?

Monitoring your resource usage is important to be a good citizen of the GSB Research Computing community, but also to be able to effectively estimate your needs on scheduled systems.

These tools can help you characterize your usage:

• htop allows you to get an overview of the activity on the whole node that you are logged into.
• userload is a custom tool that allows you to monitor CPU and RAM usage in a simplified view.
• topbyuser is another custom tool that lists out the active users on your current host and the amount of resources they are currently using.

Stay Within User Limits

We have User Limits to illustrate responsible use of our shared resources that our users are expected to follow. These guidelines ensure that the Yen cluster can remain accessible and performant.

Monitoring Compute with htop

One easy method of getting a quick snapshot of your CPU and memory usage is via the htop command line tool. Running htop shows usage graphs and a process list that is sortable by user, top CPU, top RAM, and other metrics. Please use this tool liberally to monitor your resource usage, especially if you are running multiprocessing code on shared systems for the first time.

You can toggle to a threaded view of processes by pressing T. Below is a sample of htop output in threaded mode.

Jobs May be Automatically Killed

Note that jobs that exceed user limits may be terminated automatically to preserve the integrity of the system.

User Limits and Multiprocessing

Per our User Limits, CPU usage should always be limited based on system size. When working with implicitly or explicitly multiprocessed code, care must be taken to ensure your code does not use every available processor. Please refer to our Parallel Processing in R and Parallel Processing in Python articles for information about how to limit resource consumption when using parallel packages in those languages.

Be sure to monitor your processes, particularly when using a new package, to verify that you are using the expected number of cores.

Monitoring Disk Usage

Personal home directories have a 50 GB quota, while faculty project directories are much larger.

Do NOT Exceed Your Home Directory Quota!

If you exceed your home directory quota, you cannot access Jupyter or perform many basic system tasks.

Disk storage is a finite resource so to allow us to continue to provide large project spaces, please always be aware of your disk footprint. This includes removing intermediate and/or temp files whenever possible, and avoiding storing multiple copies of the same data set. See the Storage Solutions page for more information about file storage options.

Disk quotas on the Yen servers can be reviewed by using the gsbquota command. It produces output like this:

Terminal Command

gsbquota

Terminal Output

currently using 39% (20G) of 50G available

You can also check the size of your project space by passing in a full path to your project space to gsbquota command:

Terminal Command

gsbquota /zfs/projects/students/<my-project-dir>/

Terminal Output

/zfs/projects/students/<my-project-dir>/: currently using 39% (78G) of 200G available

Unsure what's taking up space? The gsbbrowser command scans your home directory and provides a visual representation of your directories and files and their associated sizes. You can also provide a path to a project directory to scan that directory:

Terminal Command

gsbbrowser /zfs/projects/students/<my-project-dir>/

Example: Monitoring an R Script

Here we present an example in R, to illustrate monitoring code running on multiple cores.

To monitor the resource usage while running a program, we will need three terminal windows that are all connected to the same Yen server.

Start by ssh'ing into any of the interactive yens in the first terminal window:

Terminal 1 Command

ssh yen.stanford.edu

Check what yen node you are connected to in the first terminal:

Terminal 1 Command

hostname

Terminal 1 Output Example

yen3

Then ssh to the same yen in the second and third terminal windows. So if you are on yen3, you would open two new terminals and ssh to yen3 in both so you can monitor your resources when you start running the R program on yen3.

Terminal 2,3 Command

ssh yen3.stanford.edu

Create an investment-npv-parallel.R file below and save it in your desired directory.

We Explicitly Limit the Cores in the Code!

Note that we set 8 cores for use with the registerDoParallel function.

# In the context of economics and finance, Net Present Value (NPV) is used to assess
# the profitability of investment projects or business decisions.
# This code performs a Monte Carlo simulation of Net Present Value (NPV) with 500,000 trials in parallel,
# utilizing multiple CPU cores. It randomizes input parameters for each trial, calculates the NPV,
# and stores the results for analysis.

# load necessary libraries
library(foreach)
library(doParallel)

options(warn=-1)

# set the number of cores here
ncore <- 8

# register parallel backend to limit threads to the value specified in ncore variable
registerDoParallel(ncore)

# define function for NPV calculation
npv_calculation <- function(cashflows, discount_rate) {
  # inputs: cashflows (a vector of cash flows over time) and discount_rate (the discount rate).
  npv <- sum(cashflows / (1 + discount_rate)^(0:length(cashflows)))
  return(npv)
}

# number of trials
num_trials <- 500000

# measure the execution time of the Monte Carlo simulation
system.time({
  # use the foreach package to loop through the specified number of trials (num_trials) in parallel
  # within each parallel task, random values for input parameters (cash flows and discount rate) are generated for each trial
  # these random input values represent different possible scenarios
  results <- foreach(i = 1:num_trials, .combine = rbind) %dopar% {
    # randomly generate input values for each trial
    cashflows <- runif(10000, min = -100, max = 100)  # random cash flow vector over 10,000 time periods.
    # these cash flows can represent costs (e.g., initial investment) and benefits (e.g., revenue or savings) associated with the project
    discount_rate <- runif(1, min = 0.05, max = 0.15)  # random discount rate at which future cash flows are discounted

    # calculate NPV for the trial
    npv <- npv_calculation(cashflows, discount_rate)
  }
})
cat("Parallel NPV Calculation (using", ncore, "cores):
")
# print summary statistics for NPV and plot a histogram of the results
# positive NPV indicates that the project is expected to generate a profit (the benefits outweigh the costs),
# making it an economically sound decision. If the NPV is negative, it suggests that the project may not be financially viable.
summary(results)
hist(results, main = 'NPV distribution')

Once we have three terminal windows connected to the same yen, we are ready to run the script and monitor its resource consumption. In one of the terminals, load the R module:

Terminal 1 Command

ml R

Install the Necessary Packages

The above script relies on two R packages - foreach and doParallel. If you have not previously installed them for the version of R that you have loaded, you can install them by running the following command in the R console: install.packages(c("foreach", "doParallel")). See this page for details on how to manage R packages on the Yen cluster.

In the same terminal where you loaded the R module, run the investment-npv-parallel.R program:

Terminal 1 Command

Rscript investment-npv-parallel.R

Once the program is running, monitor your usage with userload command in the second window:

Terminal 2 Command

watch -n 1 userload

While the program is running, you should see about 8 CPU cores being utilized in userload output. Note that the watch command allows you to see the real time CPU cores count every second.

Run htop -u $USER in the third window, where $USER is your SUNet ID:

Terminal 3 Command

htop -u $USER

While the program is running, you should see 8 R processes running in the htop output because we specified 8 cores in our R program.

In summary, effective monitoring of your resource usage on the Yen cluster ensures that you stay within system limits, maintain performance, and contribute to a fair and efficient computing environment for all users. By leveraging tools like htop, userload, and gsbquota, you can proactively manage your resource consumption.