Skip to content

Self-Hosting LLMs with NVIDIA NIM on the Yens

NVIDIA NIM (NVIDIA Inference Microservices) lets you deploy optimized, production-ready large language models on your own infrastructure. Instead of sending data to a third-party API, you pull a pre-packaged container, start it on a GPU node, and query it through a standard OpenAI-compatible REST endpoint — all within Stanford's network.

This guide walks through deploying Google's Gemma 4 31B IT model on a single H200 GPU on the Yen cluster using Singularity containers.

H200 GPUs Required

NIM containers require H200 GPUs and do not work on older GPU architectures such as the A30 or A40. On the Yen cluster, the only node with H200 GPUs is yen-gpu4. All examples in this guide use yen-gpu4 as the target node.

Why NIM?

If you've used Ollama to run models on Stanford clusters, NIM fills a similar niche but with a different approach. While Ollama provides a convenient wrapper around open-source models, NIM delivers NVIDIA's own optimized inference stack — the same engine that powers their cloud API — packaged as containers you can run locally.

Key advantages of NIM:

  • Optimized inference: NIM containers use TensorRT-LLM under the hood, providing high-throughput, low-latency inference tuned for NVIDIA GPUs.

  • OpenAI-compatible API: the running container exposes a /v1/chat/completions endpoint, so existing code that talks to OpenAI or other providers works with minimal changes.

  • Data stays local: like Ollama, everything runs on the Yen cluster — no data leaves Stanford's infrastructure, making it suitable for sensitive or licensed datasets.

Check Your Data License

Running a model locally does not automatically mean you can use it on any dataset. Many data licenses carry their own AI and machine learning restrictions. Before using self-hosted LLMs on licensed data, review the terms of your data agreement and Stanford's Usage Policy for Licensed Resources.

  • Reproducible deployments: the container image pins a specific model version, runtime, and optimization profile. Anyone pulling the same image gets identical behavior.

Marlowe Users

This guide is adapted from Marlowe's NGC container example for the Yen cluster environment. If you are running on Marlowe, refer to that documentation instead.

Prerequisites

  1. Create an NVIDIA Developer account — if you don't already have one, sign up at developer.nvidia.com.

  2. Generate an NGC API key — navigate to the Gemma 4 31B IT model card on NVIDIA's Build portal and click on Deploy Tab.

Gemma 4 Model Card on NVIDIA Developer Site

Then select Self-Hosted Deployments and click on Get API Key button in Step 1 Generate API Key. Once you have the key, save it on the Yens in a restricted file so your scripts can reference it without hardcoding credentials:

Generate API Key

Save your NGC API key on the Yens
mkdir -p ~/.secrets
echo "your-ngc-api-key" > ~/.secrets/ngc_api_key
chmod 600 ~/.secrets/ngc_api_key

Some models require accepting license terms

Gemma does not require accepting a license, but other NIM models (such as GPT OSS 120B) may require you to accept terms on the model card before you can generate a key and pull the container.

Do Not Hardcode API Keys

Never paste your NGC API key directly into scripts or commit it to version control. Always read it from ~/.secrets/ngc_api_key as shown above.

Step 1: Set Up the Environment

Before requesting GPU resources, initialize your workspace on any Yen login node. This keeps H200 time dedicated strictly to inference. First, load the Singularity software module to ensure the necessary tools are in your path. Then, define your project root in the /scratch/shared/ directory, which is optimized for the large files associated with LLM containers.

Define paths and load modules
ml singularity

export NIM_ROOT="/scratch/shared/$USER/nim"
export SINGULARITY_CACHEDIR="$NIM_ROOT/singularity_cache"
export SINGULARITY_TMPDIR="$NIM_ROOT/tmp"
export SINGULARITY_CONFIGDIR="$NIM_ROOT/singularity_config"

Why /scratch/shared/?

The NIM container image and model cache can be tens of gigabytes. The /scratch/shared/ filesystem on the Yens is designed for this kind of temporary, large-file storage. Files on scratch are not backed up and may be purged periodically, so don't store anything you can't regenerate.

Next, create the necessary directory structure for the NIM cache, NGINX proxy, and temporary files. We use chmod -R 700 to ensure these files are only accessible by your user account, which is a critical security best practice on shared clusters.

Create secure directories
mkdir -p "$NIM_ROOT"/{cache,work/{nginx,configs},tmp,singularity_cache,singularity_config,triton}
chmod -R 700 "$NIM_ROOT"

Finally, authenticate with the NVIDIA NGC registry using the API key saved in the previous step and pull the container image:

Authenticate and pull the image
export NGC_API_KEY=$(cat ~/.secrets/ngc_api_key)

export SINGULARITY_DOCKER_USERNAME='$oauthtoken'
export SINGULARITY_DOCKER_PASSWORD="$NGC_API_KEY"

singularity pull "$NIM_ROOT/gemma-4-31b-it.sif" docker://nvcr.io/nim/google/gemma-4-31b-it:latest

Note

The .sif file is the Singularity container image. Once pulled, you can reuse it across multiple sessions without re-downloading — just make sure the file persists on scratch between jobs.

Step 2: Request a GPU node

With the image now stored on scratch and your environment configured, you are ready to request the H200 node.

Gemma 4 31B IT fits comfortably on a single H200 GPU. Start a terminal on GPU node with an interactive allocation:

Request an H200 GPU on the Yens
srun -p gpu --mem=100G -c 25 -G 1 -C "GPU_MODEL:H200" -t 5:00:00 --pty /bin/bash

Once Slurm grants the allocation, you will be on yen-gpu4:

You should see yen-gpu4 in your prompt
$USER@yen-gpu4:~$

Step 3: Run the NIM container

NIM ships with multiple optimization profiles tuned for different GPU architectures and precision modes. Before launching, you can list the profiles available for your hardware:

List available model profiles
singularity run --nv --cleanenv \
    --writable-tmpfs \
    --bind "$NIM_ROOT/cache:/opt/nim/.cache" \
    --bind "$NIM_ROOT/work/nginx:/opt/nim/nginx" \
    --bind "$NIM_ROOT/work/configs:/opt/nim/generated_configs" \
    --bind "$NIM_ROOT/triton:/tmp/triton_cache" \
    --env TRITON_CACHE_DIR="/tmp/triton_cache" \
    --env NGC_API_KEY=$NGC_API_KEY \
    --env NIM_TENSOR_PARALLEL_SIZE=1 \
    --env CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES \
    "$NIM_ROOT/gemma-4-31b-it.sif" list-model-profiles

NIM automatically selects the best compatible profile for your GPU at launch. To force a specific profile, add the NIM_MODEL_PROFILE environment variable to the launch command in the next step:

Example: select a specific profile
--env NIM_MODEL_PROFILE="<profile-id-from-list>"

Step 4: Launch the model

Start the NIM container with GPU access. The --nv flag enables NVIDIA GPU passthrough, and the bind mounts give the container access to its required working directories:

Run the NIM container
singularity run --nv --cleanenv \
    --writable-tmpfs \
    --bind "$NIM_ROOT/cache:/opt/nim/.cache" \
    --bind "$NIM_ROOT/work/nginx:/opt/nim/nginx" \
    --bind "$NIM_ROOT/work/configs:/opt/nim/generated_configs" \
    --bind "$NIM_ROOT/triton:/tmp/triton_cache" \
    --env TRITON_CACHE_DIR="/tmp/triton_cache" \
    --env NGC_API_KEY=$NGC_API_KEY \
    --env NIM_TENSOR_PARALLEL_SIZE=1 \
    --env CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES \
    "$NIM_ROOT/gemma-4-31b-it.sif"

Slurm automatically sets CUDA_VISIBLE_DEVICES based on your allocation.

The container will download model weights on first launch (cached for subsequent runs), compile optimized inference kernels, and start serving on port 8000. This startup process can take several minutes — wait until you see log output indicating the server is ready.

Port Access and Choosing a Different Port

NIM serves on port 8000 by default. This port is accessible to any user on the Yen cluster — there is no authentication on the endpoint, so anyone who knows the hostname and port can send requests to your running model.

If port 8000 is already in use by another user on the same node, or if you want to avoid collisions, you can change the port by adding the NIM_HTTP_API_PORT environment variable to the launch command:

--env NIM_HTTP_API_PORT=8001

Remember to update the port in your curl and Python client calls accordingly.

Environment Variables

The launch command above uses the following environment variables:

Variable Description
NGC_API_KEY Your NVIDIA NGC API key. Required to download model weights on first launch.
CUDA_VISIBLE_DEVICES Which GPU device to use. The yen-gpu4 node has two H200 GPUs (devices 0 and 1)
NIM_TENSOR_PARALLEL_SIZE Number of GPUs to shard the model across. Set to 1 when running on a single GPU. Increase this if your Slurm allocation includes multiple GPUs and the model is too large to fit on one.
TRITON_CACHE_DIR Where Triton stores compiled GPU kernels. We redirect this to scratch to avoid filling your home directory (see the warning above).

You can also tune inference behavior with additional variables that are not included in the command above:

Variable Description
NIM_MAX_MODEL_LEN Maximum sequence length (input + output tokens combined). Longer sequences use more GPU memory. Defaults vary by model — set this if you need longer contexts or want to limit memory usage (e.g., 16384).
NIM_GPU_MEMORY_UTILIZATION Fraction of GPU memory NIM is allowed to use, between 0 and 1. A value like 0.85 leaves headroom for the operating system and other processes. Lower it if you see out-of-memory errors.

For a full list of configuration options, see the NVIDIA NIM configuration documentation.

Step 5: Query the Model from a Login Node

Once the server is running, open a second terminal on any interactive Yen node and send a request using curl:

Send a chat completion request
curl -s -X POST http://yen-gpu4:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "google/gemma-4-31b-it",
    "messages": [
      {
        "role": "user",
        "content": "What are three advantages of running LLMs on local infrastructure?"
      }
    ],
    "max_tokens": 256,
    "temperature": 0.7
  }'

The response follows the OpenAI chat completions format, so you can also use the openai Python library by pointing it at http://yen-gpu4:8000:

Python client example
from openai import OpenAI

client = OpenAI(
    base_url="http://yen-gpu4:8000/v1",
    api_key="not-used"  # NIM doesn't require a client-side key
)

response = client.chat.completions.create(
    model="google/gemma-4-31b-it",
    messages=[
        {"role": "user", "content": "Summarize the key ideas behind LoRA fine-tuning."}
    ],
    max_tokens=256
)

print(response.choices[0].message.content)

Clean Up

When you're done, press Ctrl+C in the terminal running the NIM container to stop it, then release your GPU allocation:

Release the Slurm allocation
exit

The cached model weights and container image remain on scratch for future use. If you no longer need them:

Remove NIM files from scratch
rm -rf /scratch/shared/$USER/nim

Summary

Step What happens
Set up environment Load modules, create scratch directories, authenticate with NGC, and pull the container image
Request GPU srun reserves an H200 GPU on yen-gpu4
List profiles Check available optimization profiles for your hardware (optional)
Launch Start the NIM container — model weights are cached after first run
Query Hit the OpenAI-compatible API on port 8000 from any Yen login node

Running as a Batch Job

The interactive workflow above is useful for testing, but for longer-running inference you can submit a Slurm batch script instead. This starts the NIM server on a GPU node in the background so you can query it from any Yen login node.

Create a file called run_nim_server.slurm. This script mirrors the interactive setup from the steps above:

run_nim_server.slurm
#!/bin/bash
#SBATCH -J nim-server
#SBATCH -p gpu
#SBATCH -C "GPU_MODEL:H200"
#SBATCH -G 1
#SBATCH -n 1
#SBATCH -c 25
#SBATCH --mem=100G
#SBATCH -t 5:00:00
#SBATCH -o nim-server-%j.out
#SBATCH --mail-type=ALL
#SBATCH --mail-user=user@stanford.edu

# 1. Define Paths & Credentials
export NGC_API_KEY=$(cat /home/users/$USER/.secrets/ngc_api_key)

export NIM_ROOT="/scratch/shared/$USER/nim"
export SINGULARITY_CACHEDIR="$NIM_ROOT/singularity_cache"
export SINGULARITY_TMPDIR="$NIM_ROOT/tmp"
export SINGULARITY_CONFIGDIR="$NIM_ROOT/singularity_config"
export SINGULARITY_DOCKER_USERNAME='$oauthtoken'
export SINGULARITY_DOCKER_PASSWORD="$NGC_API_KEY"

ml singularity

# 2. Setup directories
mkdir -p "$NIM_ROOT"/{cache,work/{nginx,configs},tmp,singularity_cache,singularity_config,triton}
chmod -R 700 "$NIM_ROOT"

# 3. Download SIF if not found
if [ ! -f "$NIM_ROOT/gemma-4-31b-it.sif" ]; then
    echo "SIF file not found. Downloading..."
    singularity pull "$NIM_ROOT/gemma-4-31b-it.sif" docker://nvcr.io/nim/google/gemma-4-31b-it:latest
else
    echo "SIF file found. Skipping download."
fi

# 4. Launch NIM
singularity run --nv --cleanenv \
    --writable-tmpfs \
    --bind "$NIM_ROOT/cache:/opt/nim/.cache" \
    --bind "$NIM_ROOT/work/nginx:/opt/nim/nginx" \
    --bind "$NIM_ROOT/work/configs:/opt/nim/generated_configs" \
    --bind "$NIM_ROOT/triton:/tmp/triton_cache" \
    --env TRITON_CACHE_DIR="/tmp/triton_cache" \
    --env NGC_API_KEY=$NGC_API_KEY \
    --env NIM_TENSOR_PARALLEL_SIZE=1 \
    --env CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES \
    "$NIM_ROOT/gemma-4-31b-it.sif"

Submit the job:

Submit the batch job
sbatch run_nim_server.slurm

Once the job is running, check the log file to see that the NIM is deployed:

Check the log file
tail -f nim-server-<jobid>.out

Then query the model from any Yen login node:

Query from a login node
curl -s -X POST http://yen-gpu4:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "google/gemma-4-31b-it",
    "messages": [
      {"role": "user", "content": "What is NVIDIA NIM?"}
    ],
    "max_tokens": 256
  }'

To stop the server, cancel the Slurm job:

Cancel the job
scancel <jobid>

NIM gives you a fast, optimized path to self-hosting LLMs on Stanford's GPU infrastructure. Combined with Slurm for resource management and scratch storage for large files, it's a practical way to run inference workloads that need to stay on-premises.