# Self-Hosting LLMs with NVIDIA NIM on the Yens

NVIDIA NIM (NVIDIA Inference Microservices) lets you deploy optimized, production-ready large language models on your own infrastructure. Instead of sending data to a third-party API, you pull a pre-packaged container, start it on a GPU node, and query it through a standard OpenAI-compatible REST endpoint — all within Stanford's network.

This guide walks through deploying Google's [Gemma 4 31B IT](https://build.nvidia.com/google/gemma-4-31b-it) model on a single H200 GPU on the Yen cluster using Singularity containers.

H200 GPUs Required

NIM containers require H200 GPUs and do not work on older GPU architectures such as the A30 or A40. On the Yen cluster, the only node with H200 GPUs is **`yen-gpu4`**. All examples in this guide use `yen-gpu4` as the target node.

## Why NIM?

If you've used [Ollama](/blog/2025/05/12/running-ollama-on-stanford-computing-clusters/) to run models on Stanford clusters, NIM fills a similar niche but with a different approach. While Ollama provides a convenient wrapper around open-source models, NIM delivers NVIDIA's own optimized inference stack — the same engine that powers their cloud API — packaged as containers you can run locally.

Key advantages of NIM:

- **Optimized inference**: NIM containers use TensorRT-LLM under the hood, providing high-throughput, low-latency inference tuned for NVIDIA GPUs.
- **OpenAI-compatible API**: the running container exposes a `/v1/chat/completions` endpoint, so existing code that talks to OpenAI or other providers works with minimal changes.
- **Data stays local**: like Ollama, everything runs on the Yen cluster — no data leaves Stanford's infrastructure, making it suitable for sensitive or licensed datasets.

Check Your Data License

Running a model locally does not automatically mean you can use it on any dataset. Many data licenses carry their own AI and machine learning restrictions. Before using self-hosted LLMs on licensed data, review the terms of your data agreement and Stanford's [Usage Policy for Licensed Resources](https://www.gsb.stanford.edu/library/research-resources/usage-policy).

- **Reproducible deployments**: the container image pins a specific model version, runtime, and optimization profile. Anyone pulling the same image gets identical behavior.

Marlowe Users

This guide is adapted from [Marlowe's NGC container example](https://docs.marlowe.stanford.edu/ngc_example.html) for the Yen cluster environment. If you are running on Marlowe, refer to that documentation instead.

## Prerequisites

1. **Create an NVIDIA Developer account** — if you don't already have one, sign up at [developer.nvidia.com](https://developer.nvidia.com).
1. **Generate an NGC API key** — navigate to the [Gemma 4 31B IT model card](https://build.nvidia.com/google/gemma-4-31b-it) on NVIDIA's Build portal and click on `Deploy` Tab.

Then select `Self-Hosted Deployments` and click on **Get API Key** button in Step 1 Generate API Key. Once you have the key, save it on the Yens in a restricted file so your scripts can reference it without hardcoding credentials:

Save your NGC API key on the Yens

```
mkdir -p ~/.secrets
echo "your-ngc-api-key" > ~/.secrets/ngc_api_key
chmod 600 ~/.secrets/ngc_api_key
```

Some models require accepting license terms

Gemma does not require accepting a license, but other NIM models (such as GPT OSS 120B) may require you to accept terms on the model card before you can generate a key and pull the container.

Do Not Hardcode API Keys

Never paste your NGC API key directly into scripts or commit it to version control. Always read it from `~/.secrets/ngc_api_key` as shown above.

## Step 1: Set Up the Environment

Before requesting GPU resources, initialize your workspace on any Yen login node. This keeps H200 time dedicated strictly to inference. First, load the Singularity software module to ensure the necessary tools are in your path. Then, define your project root in the `/scratch/shared/` directory, which is optimized for the large files associated with LLM containers.

Define paths and load modules

```
ml singularity

export NIM_ROOT="/scratch/shared/$USER/nim"
export SINGULARITY_CACHEDIR="$NIM_ROOT/singularity_cache"
export SINGULARITY_TMPDIR="$NIM_ROOT/tmp"
export SINGULARITY_CONFIGDIR="$NIM_ROOT/singularity_config"
```

Why `/scratch/shared/`?

The NIM container image and model cache can be tens of gigabytes. The `/scratch/shared/` filesystem on the Yens is designed for this kind of temporary, large-file storage. Files on scratch are not backed up and may be purged periodically, so don't store anything you can't regenerate.

Next, create the necessary directory structure for the NIM cache, NGINX proxy, and temporary files. We use `chmod -R 700` to ensure these files are only accessible by your user account, which is a critical security best practice on shared clusters.

Create secure directories

```
mkdir -p "$NIM_ROOT"/{cache,work/{nginx,configs},tmp,singularity_cache,singularity_config,triton}
chmod -R 700 "$NIM_ROOT"
```

Finally, authenticate with the NVIDIA NGC registry using the API key saved in the previous step and pull the container image:

Authenticate and pull the image

```
export NGC_API_KEY=$(cat ~/.secrets/ngc_api_key)

export SINGULARITY_DOCKER_USERNAME='$oauthtoken'
export SINGULARITY_DOCKER_PASSWORD="$NGC_API_KEY"

singularity pull "$NIM_ROOT/gemma-4-31b-it.sif" docker://nvcr.io/nim/google/gemma-4-31b-it:latest
```

Note

The `.sif` file is the Singularity container image. Once pulled, you can reuse it across multiple sessions without re-downloading — just make sure the file persists on scratch between jobs.

## Step 2: Request a GPU node

With the image now stored on scratch and your environment configured, you are ready to request the H200 node.

Gemma 4 31B IT fits comfortably on a single H200 GPU. Start a terminal on GPU node with an interactive allocation:

Request an H200 GPU on the Yens

```
srun -p gpu --mem=100G -c 25 -G 1 -C "GPU_MODEL:H200" -t 5:00:00 --pty /bin/bash
```

Once Slurm grants the allocation, you will be on `yen-gpu4`:

You should see yen-gpu4 in your prompt

```
$USER@yen-gpu4:~$
```

## Step 3: Run the NIM container

NIM ships with multiple optimization profiles tuned for different GPU architectures and precision modes. Before launching, you can list the profiles available for your hardware:

List available model profiles

```
singularity run --nv --cleanenv \
    --writable-tmpfs \
    --bind "$NIM_ROOT/cache:/opt/nim/.cache" \
    --bind "$NIM_ROOT/work/nginx:/opt/nim/nginx" \
    --bind "$NIM_ROOT/work/configs:/opt/nim/generated_configs" \
    --bind "$NIM_ROOT/triton:/tmp/triton_cache" \
    --env TRITON_CACHE_DIR="/tmp/triton_cache" \
    --env NGC_API_KEY=$NGC_API_KEY \
    --env NIM_TENSOR_PARALLEL_SIZE=1 \
    --env CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES \
    "$NIM_ROOT/gemma-4-31b-it.sif" list-model-profiles
```

NIM automatically selects the best compatible profile for your GPU at launch. To force a specific profile, add the `NIM_MODEL_PROFILE` environment variable to the launch command in the next step:

Example: select a specific profile

```
--env NIM_MODEL_PROFILE="<profile-id-from-list>"
```

## Step 4: Launch the model

Start the NIM container with GPU access. The `--nv` flag enables NVIDIA GPU passthrough, and the bind mounts give the container access to its required working directories:

Run the NIM container

```
singularity run --nv --cleanenv \
    --writable-tmpfs \
    --bind "$NIM_ROOT/cache:/opt/nim/.cache" \
    --bind "$NIM_ROOT/work/nginx:/opt/nim/nginx" \
    --bind "$NIM_ROOT/work/configs:/opt/nim/generated_configs" \
    --bind "$NIM_ROOT/triton:/tmp/triton_cache" \
    --env TRITON_CACHE_DIR="/tmp/triton_cache" \
    --env NGC_API_KEY=$NGC_API_KEY \
    --env NIM_TENSOR_PARALLEL_SIZE=1 \
    --env CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES \
    "$NIM_ROOT/gemma-4-31b-it.sif"
```

Slurm automatically sets `CUDA_VISIBLE_DEVICES` based on your allocation.

The container will download model weights on first launch (cached for subsequent runs), compile optimized inference kernels, and start serving on port 8000. This startup process can take several minutes — wait until you see log output indicating the server is ready.

Port Access and Choosing a Different Port

NIM serves on port `8000` by default. This port is accessible to **any user on the Yen cluster** — there is no authentication on the endpoint, so anyone who knows the hostname and port can send requests to your running model.

If port `8000` is already in use by another user on the same node, or if you want to avoid collisions, you can change the port by adding the `NIM_HTTP_API_PORT` environment variable to the launch command:

```
--env NIM_HTTP_API_PORT=8001
```

Remember to update the port in your `curl` and Python client calls accordingly.

### Environment Variables

The launch command above uses the following environment variables:

| Variable                   | Description                                                                                                                                                                                |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `NGC_API_KEY`              | Your NVIDIA NGC API key. Required to download model weights on first launch.                                                                                                               |
| `CUDA_VISIBLE_DEVICES`     | Which GPU device to use. The `yen-gpu4` node has two H200 GPUs (devices `0` and `1`)                                                                                                       |
| `NIM_TENSOR_PARALLEL_SIZE` | Number of GPUs to shard the model across. Set to `1` when running on a single GPU. Increase this if your Slurm allocation includes multiple GPUs and the model is too large to fit on one. |
| `TRITON_CACHE_DIR`         | Where Triton stores compiled GPU kernels. We redirect this to scratch to avoid filling your home directory (see the [warning above](#step-1-set-up-the-environment)).                      |

You can also tune inference behavior with additional variables that are not included in the command above:

| Variable                     | Description                                                                                                                                                                                                  |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `NIM_MAX_MODEL_LEN`          | Maximum sequence length (input + output tokens combined). Longer sequences use more GPU memory. Defaults vary by model — set this if you need longer contexts or want to limit memory usage (e.g., `16384`). |
| `NIM_GPU_MEMORY_UTILIZATION` | Fraction of GPU memory NIM is allowed to use, between `0` and `1`. A value like `0.85` leaves headroom for the operating system and other processes. Lower it if you see out-of-memory errors.               |

For a full list of configuration options, see the [NVIDIA NIM configuration documentation](https://docs.nvidia.com/nim/large-language-models/latest/configuration.html).

## Step 5: Query the Model from a Login Node

Once the server is running, open a **second terminal** on any interactive Yen node and send a request using `curl`:

Send a chat completion request

```
curl -s -X POST http://yen-gpu4:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "google/gemma-4-31b-it",
    "messages": [
      {
        "role": "user",
        "content": "What are three advantages of running LLMs on local infrastructure?"
      }
    ],
    "max_tokens": 256,
    "temperature": 0.7
  }'
```

The response follows the OpenAI chat completions format, so you can also use the `openai` Python library by pointing it at `http://yen-gpu4:8000`:

Python client example

```
from openai import OpenAI

client = OpenAI(
    base_url="http://yen-gpu4:8000/v1",
    api_key="not-used"  # NIM doesn't require a client-side key
)

response = client.chat.completions.create(
    model="google/gemma-4-31b-it",
    messages=[
        {"role": "user", "content": "Summarize the key ideas behind LoRA fine-tuning."}
    ],
    max_tokens=256
)

print(response.choices[0].message.content)
```

## Clean Up

When you're done, press `Ctrl+C` in the terminal running the NIM container to stop it, then release your GPU allocation:

Release the Slurm allocation

```
exit
```

The cached model weights and container image remain on scratch for future use. If you no longer need them:

Remove NIM files from scratch

```
rm -rf /scratch/shared/$USER/nim
```

## Summary

| Step               | What happens                                                                                  |
| ------------------ | --------------------------------------------------------------------------------------------- |
| Set up environment | Load modules, create scratch directories, authenticate with NGC, and pull the container image |
| Request GPU        | `srun` reserves an H200 GPU on `yen-gpu4`                                                     |
| List profiles      | Check available optimization profiles for your hardware (optional)                            |
| Launch             | Start the NIM container — model weights are cached after first run                            |
| Query              | Hit the OpenAI-compatible API on port 8000 from any Yen login node                            |

## Running as a Batch Job

The interactive workflow above is useful for testing, but for longer-running inference you can submit a Slurm batch script instead. This starts the NIM server on a GPU node in the background so you can query it from any Yen login node.

Create a file called `run_nim_server.slurm`. This script mirrors the interactive setup from the steps above:

run_nim_server.slurm

```
#!/bin/bash
#SBATCH -J nim-server
#SBATCH -p gpu
#SBATCH -C "GPU_MODEL:H200"
#SBATCH -G 1
#SBATCH -n 1
#SBATCH -c 25
#SBATCH --mem=100G
#SBATCH -t 5:00:00
#SBATCH -o nim-server-%j.out
#SBATCH --mail-type=ALL
#SBATCH --mail-user=user@stanford.edu

# 1. Define Paths & Credentials
export NGC_API_KEY=$(cat /home/users/$USER/.secrets/ngc_api_key)

export NIM_ROOT="/scratch/shared/$USER/nim"
export SINGULARITY_CACHEDIR="$NIM_ROOT/singularity_cache"
export SINGULARITY_TMPDIR="$NIM_ROOT/tmp"
export SINGULARITY_CONFIGDIR="$NIM_ROOT/singularity_config"
export SINGULARITY_DOCKER_USERNAME='$oauthtoken'
export SINGULARITY_DOCKER_PASSWORD="$NGC_API_KEY"

ml singularity

# 2. Setup directories
mkdir -p "$NIM_ROOT"/{cache,work/{nginx,configs},tmp,singularity_cache,singularity_config,triton}
chmod -R 700 "$NIM_ROOT"

# 3. Download SIF if not found
if [ ! -f "$NIM_ROOT/gemma-4-31b-it.sif" ]; then
    echo "SIF file not found. Downloading..."
    singularity pull "$NIM_ROOT/gemma-4-31b-it.sif" docker://nvcr.io/nim/google/gemma-4-31b-it:latest
else
    echo "SIF file found. Skipping download."
fi

# 4. Launch NIM
singularity run --nv --cleanenv \
    --writable-tmpfs \
    --bind "$NIM_ROOT/cache:/opt/nim/.cache" \
    --bind "$NIM_ROOT/work/nginx:/opt/nim/nginx" \
    --bind "$NIM_ROOT/work/configs:/opt/nim/generated_configs" \
    --bind "$NIM_ROOT/triton:/tmp/triton_cache" \
    --env TRITON_CACHE_DIR="/tmp/triton_cache" \
    --env NGC_API_KEY=$NGC_API_KEY \
    --env NIM_TENSOR_PARALLEL_SIZE=1 \
    --env CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES \
    "$NIM_ROOT/gemma-4-31b-it.sif"
```

Submit the job:

Submit the batch job

```
sbatch run_nim_server.slurm
```

Once the job is running, check the log file to see that the NIM is deployed:

Check the log file

```
tail -f nim-server-<jobid>.out
```

Then query the model from any Yen login node:

Query from a login node

```
curl -s -X POST http://yen-gpu4:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "google/gemma-4-31b-it",
    "messages": [
      {"role": "user", "content": "What is NVIDIA NIM?"}
    ],
    "max_tokens": 256
  }'
```

To stop the server, cancel the Slurm job:

Cancel the job

```
scancel <jobid>
```

NIM gives you a fast, optimized path to self-hosting LLMs on Stanford's GPU infrastructure. Combined with Slurm for resource management and scratch storage for large files, it's a practical way to run inference workloads that need to stay on-premises.