Skip to content

2026

  • in LLM
  • 9 min read

Self-Hosting LLMs with NVIDIA NIM on the Yens

NVIDIA NIM (NVIDIA Inference Microservices) lets you deploy optimized, production-ready large language models on your own infrastructure. Instead of sending data to a third-party API, you pull a pre-packaged container, start it on a GPU node, and query it through a standard OpenAI-compatible REST endpoint — all within Stanford's network.

This guide walks through deploying Google's Gemma 4 31B IT model on a single H200 GPU on the Yen cluster using Singularity containers.

H200 GPUs Required

NIM containers require H200 GPUs and do not work on older GPU architectures such as the A30 or A40. On the Yen cluster, the only node with H200 GPUs is yen-gpu4. All examples in this guide use yen-gpu4 as the target node.

  • in Yen
  • 2 min read

SSH Setup for the Yen Servers

When working on the Yen cluster, you may occasionally open multiple SSH sessions from your local computer or connect from one Yen node to another to check a running process or run a command on a different node.

You can make these workflows smoother by configuring SSH multiplexing on your local computer and setting up SSH key authentication between Yen nodes.