Why Run LLMs on the Yens?

Now for a change of tack. We’ve been looking at how to spread work across the cluster; next comes the work itself.

Since Day 2, every LLM call we’ve made — to the Stanford AI API Gateway, or any third-party model — has gone to an API (application programming interface): your prompt and your data travel to someone else’s server, the model runs there, and the answer comes back. This section asks the opposite question: when should you run the model yourself, on Stanford’s own hardware (the Yens)?

What an LLM API Call Actually Does

When you call an LLM API, three things happen outside your control:

  • Your data leaves. The filing text — and whatever else is in your prompt — is sent over the network to the model provider.
  • The compute is theirs. The model runs on the model provider’s machines, not yours.
  • The model can change. You get the weights the model provider is serving that day.
Calling an LLM API sends your data to a remote server and back Your code on the left, labelled "on your or Stanford's machines", and the model provider's server on the right, labelled "in the cloud", separated by a dashed vertical boundary line. Your code sends a packet labelled "prompt plus data" across the boundary to the provider, which runs the model and sends a "response" packet back. On your or Stanford's machines In the cloud Your code Model provider prompt + data ▶ ◀ response Your prompt and data go to the model provider's servers, and the response comes back.

Why Run It Yourself?

Running the model yourself keeps your prompt and data on your side Your code sits above a local model, both on the left of a dashed vertical boundary — the edge of your, or Stanford's, machines. A labelled bar carries the prompt and data down to the local model and the response back up; neither crosses the boundary. On the right, in the cloud, the model provider's server is greyed out and not contacted. On your or Stanford's machines In the cloud Your code Local model prompt + data ▼ ▲ response Model provider (not contacted) Run the model yourself and your prompt and data never cross into the cloud.

1. Privacy & data requirements — often a hard rule, not a preference. Restricted, confidential, or IRB-governed data may not leave Stanford’s perimeter. Run the model locally and your prompts and documents never leave the cluster — nothing goes to an outside model provider. (Which bucket your data falls in was covered in the data classification section on Day 2.)

2. Cost at scale. No per-token bill. You already have cluster access, so local “inference” (querying an LLM) is effectively free at the margin. A run over 100,000 filings that would rack up a real bill on a metered API costs nothing extra on the Yens.

3. Reproducibility. Versioning proprietary models is complicated — the provider can change the model behind a given name, and old versions eventually get retired. An open model is always reproducible: you control the weights, so the exact model is fixed and a reviewer (or future you) can rerun the identical pipeline years later.

4. No API rate limits. Model providers cap requests and tokens per minute, which throttles a big parallel job array. Locally you’re bounded only by your own compute allocation on the cluster, so a thousand-task array isn’t held back by someone else’s quota.

There’s a catch to “run it yourself”: you can only run models whose weights you can download. The frontier models from OpenAI (GPT-5.6) and Anthropic (Claude Fable 5) are proprietary — the companies never release the weights, so those models exist only behind their cloud APIs. There is no way to run them on the Yens.

Running locally therefore means using open-weight models — ones whose parameters are published for anyone to download and serve, such as Llama (Meta), Mistral, Qwen, or DeepSeek. These are the kind of models you’ll run on the Yens later today.

So the real choice isn’t “any model, local or cloud.” It’s: a proprietary model in the cloud, or an open-weight model you run yourself.

When the API Is Still the Right Call

Running locally isn’t always the answer. The honest tradeoffs:

  • Capability — the proprietary models (GPT-5.6, Claude Fable 5) typically outperform the open-weight ones you can run yourself. For the hardest tasks, the API wins.
  • Convenience — an API key and one line of code: no cluster job to submit, no model server to start, no queue wait.
  • Elastic scale — a hosted API scales on demand; the capacity you can run on the Yens is finite, so very large models or very high volume can exceed it.

The Three Options at a Glance

  Local on the Yens Stanford AI API Gateway Third-party API
Where your data goes Stays on the cluster In the cloud, but regulated and approved by Stanford Leaves Stanford → model provider
Cost Free at the margin Budget-capped Stanford account Per-token billing
Models Open-weight models you can run on the Yens Provider-curated, Stanford-audited Latest, most capable
Best for Restricted data, large batch jobs Everyday research Hardest tasks where data rules allow

For the Stanford AI API Gateway’s tradeoffs in more depth, see the Upsides and Downsides table from Day 2.

Rule of Thumb

Restricted data → run it locally, no exceptions. Otherwise, weigh cost, reproducibility, and throughput (which favor local) against capability and convenience (which favor the API).

What You Learned

  • You know an API call sends your data to a model provider’s servers and returns a response — the computation runs on their machines, not yours
  • You can give the main reasons to run an LLM on the Yens instead: data privacy, cost at scale, reproducibility, and freedom from API rate limits
  • You can name the honest counterweights — capability, convenience, elastic scale — where a hosted API is the better choice
  • You can apply the rule of thumb: restricted data forces local; otherwise weigh the tradeoffs
This site uses Just the Docs, a documentation theme for Jekyll.