Submit

🔴 Red sticky = I need help. Put it up the moment you are stuck — an instructor will come to you.

🟢 Green sticky = I have done 1–4 and am moving on to the bonus work, and I am ready to help my table.

Everything on this page runs from your clone, with the environment active:

cd ~/yens-onboarding-2026
source .venv/bin/activate

Read the Queue First

Before you add a job to the queue, look at the queue.

Task: Look at the live Slurm queue to see what jobs are waiting or running right now.

squeue

Look at the columns:

• JOBID — unique ID for each job
• PARTITION — which partition (queue) the job was submitted to — each partition has different node types, time limits, and resource caps; see the current partitions and their limits
• ST — status: R = running, PD = pending (waiting in queue for resources)
• TIME — how long the job has been running
• NODELIST — which compute node it landed on

There is also a shorthand to filter to just your jobs:

squeue --me

You can also filter by partition — for example, to see only GPU jobs:

squeue -p gpu

Every PD job is waiting for a node with the resources it requested. When Slurm finds a matching node — it runs.

Now run sinfo to see the state of all nodes and the partitions they belong to:

sinfo

Answer each one from your own output, then open it to check.

1How many compute nodes are currently idle?

Count the idle rows in your own output — the number changes minute to minute, so there is no fixed answer here.

idle means the node is running nothing and can take work now; alloc means it is already full; mix means some of its cores are busy and some are still free. PD means a job is waiting for free resources.

2What partitions exist, and which would you use for an ordinary CPU job?

normal — it is the default for CPU work, and the one the job script you are about to write uses.

3What is the maximum time limit for each partition?

Read the TIMELIMIT column in your sinfo output. Each partition sets its own, and a job that asks for longer than its partition allows is rejected rather than queued.

The maintained figures live in current partitions and their limits — worth trusting over any number written on a course page, since they change.

Write and Submit a Job

Task: Build a Slurm job script line by line to run your Form 3 extraction script on a compute node.

Start from a clean shell. If you have a virtual environment active right now (you’ll see (.venv) at the front of your prompt), deactivate it first:

deactivate

sbatch copies your current shell’s environment into the job by default, so if .venv is active when you submit, it rides along — and the job can quietly succeed even if the script forgot to activate it. Deactivate first so the job runs on only what the script sets up — the way it’ll run for a teammate, or for you from a clean login.

Create the file:

Your repo already has a slurm/ folder (with a few prepared scripts). Just make sure a logs/ folder exists for job output:

mkdir -p logs

The logs/ folder must exist before you submit. Slurm opens your --output/--error files the moment the job starts — it does not create missing directories. If you point --output at logs/… but there’s no logs/ folder, the job fails silently: nothing runs and no log file appears to tell you why. Create it once, up front. If instead you point --output at a bare extract.out with no folder, the file lands in whatever directory you ran sbatch from.

Create a new file slurm/extract_form_3_batch.slurm and open it in your editor — you’ll build it up line by line below.

No preferred terminal editor? You can create it right in JupyterHub: in the file browser, open the slurm/ folder, click + New → Text File (or File → New → Text File), edit it in the browser, then rename the file to extract_form_3_batch.slurm and save with Cmd/Ctrl+S.

Copy the next lines into the new file — Steps 1 to 4 build it up in order. None of them are commands to run in your terminal.

Step 1 — The shebang

The first line of every shell script is the shebang:

#!/bin/bash

The #! (the shebang) tells the operating system which interpreter — the program that reads your script and runs it line by line — to use for the rest of the file; here, the Bash shell at /bin/bash. Without it, the system doesn’t know whether your script is Bash, Python, or something else. It has to be the very first line of the file.

Step 2 — SBATCH directives

These are instructions to the Slurm scheduler — add them at the top of the file, right after the shebang:

#SBATCH --job-name=<job-name>
#SBATCH --partition=normal
#SBATCH --output=logs/extract_%j.out
#SBATCH --error=logs/extract_%j.err
#SBATCH --time=<HH:MM:SS>
#SBATCH --mem=<RAM>
#SBATCH --cpus-per-task=<cores>

What each one is:

• --job-name — a short label you pick so you can spot this job in the queue (e.g. form3-extract). It doesn’t affect resources; name it whatever’s memorable.
• --partition — the queue the job runs in. normal is the default for CPU jobs; each partition has its own time limits and resource caps (see the current partitions and their limits).
• --output / --error — files where the job’s normal output and errors get written; %j is auto-filled with the job ID, so each run gets its own log. Leave these as-is.
• --time, --mem, --cpus-per-task — the resources you’re requesting. Fill these in from the time, RAM, and CPU cores you recorded in your Profiling README.

Round up, and you cannot ask for 0. --mem wants a size like 500M or 1G, and the RES you measured will not be a round number. Ask for a bit more than you measured: a job that exceeds its --mem is killed outright, while asking slightly high costs you only a little longer in the queue. If your run barely touched memory, 1G is a sane floor.

The <...> are placeholders — delete the angle brackets too. Replace the whole thing, brackets included: --time=00:30:00, not --time=<00:30:00>. Slurm reads a leftover < as part of the value and rejects the directive.

About the --output and --error files:

• A batch job has no terminal — you’re not watching it run. So Slurm redirects everything your script would normally print: normal output goes to the --output (.out) file, and error messages/tracebacks go to the --error (.err) file. Those files are how you see what the job did and debug it when it fails.
• %j gets replaced with the job ID, so each run writes its own logs/extract_JOBID.out and .err instead of overwriting the last.
• Combine them if you like: omit --error entirely and Slurm sends both normal output and errors to the single --output (.out) file. Keeping them separate just makes errors easier to spot.

Step 3 — Set up the environment

Still writing into the file, not typing in your terminal. These two lines run later, on the compute node, when the job starts:

# Navigate to your project
cd $HOME/yens-onboarding-2026

# Activate your virtual environment
source .venv/bin/activate

What’s already installed. Your .venv was built from requirements.txt on Day 1. Once it’s activated, any job can use these packages:

Package	Used for
anthropic	Calling the Anthropic API (LLM extraction)
python-dotenv	Loading your API key from .env
pydantic	Validating and structuring the LLM output
pandas	Tabular data
numpy	Vectorized numerics (installed with pandas)
requests	Downloading filings over HTTP
ipykernel / jupyter	Notebook and JupyterHub kernels
matplotlib	Plots

Need something else? pip install it into your .venv and add it to requirements.txt so your work stays reproducible.

Step 4 — Add the line that runs your script

The last line of the file is the actual work — the command Slurm will run on the compute node when the job starts. It’s a line you write inside the script, not something you run yourself right now:

python scripts/extract_form_3_batch.py

This runs the 10-filing batch you profiled — scripts/extract_form_3_batch.py loops over NUM_FILINGS (10) SEC Form 3 filings from data/aws_links.csv — so the --time, --mem, and --cpus-per-task you filled in above come straight from your Profiling README.

Save the file. Here’s the whole script, with its four parts labeled:

Every Slurm script has these four parts: the shebang, the #SBATCH directives (requests to the scheduler, not commands that run), the environment setup that runs on the compute node, and the run line(s) that do the actual work.

Slurm starts a fresh shell on the compute node. Your virtual environment is not active. Your working directory is not set. Every setup step must be in the script — cd, source .venv/bin/activate, and any module load commands you need. If it works interactively on the Yens but fails as a job, a missing setup step is usually why.

Submit it

Today only: this class has two dedicated Slurm reservations — class_cpu for the CPU jobs you write today, and class_gpu for the GPU bonus at the end of the day. Add --reservation=class_cpu to every sbatch (and srun) command from here on so your jobs run on the reserved nodes.

sbatch --reservation=class_cpu slurm/extract_form_3_batch.slurm
# Submitted batch job 12345678

Monitor the queue:

squeue --me

Cancel it

scancel JOBID

Replace JOBID with your job’s actual number — the one sbatch printed (Submitted batch job 12345678) and that shows in squeue --me. It’s not the literal word JOBID.

Confirm it is gone:

squeue --me

You may briefly see your job’s status change to CG (completing) before it disappears from the queue — that’s normal, not an error.

Add email notifications

Slurm can email you when the job starts and finishes. That needs two more #SBATCH directives — and rather than typing them, have Claude Code add them.

Start Claude Code on the Yens:

ml claude-code
cd ~/yens-onboarding-2026
claude

Then ask it — the address below is filled in from your sidebar, so read it back before you send it:

> Add --mail-type=ALL and --mail-user=SUNetID@stanford.edu to the #SBATCH directives in slurm/extract_form_3_batch.slurm

Use your own SUNet ID. If an unfilled placeholder ends up in the script instead of your own address, the job still runs and still reports success — the email just goes nowhere, and nothing tells you. Check the line before you resubmit.

Claude will show you the change before writing it. Approve it, then /exit to leave Claude and get your shell back. The two lines it should have added:

#SBATCH --mail-type=ALL
#SBATCH --mail-user=SUNetID@stanford.edu

ALL sends an email when the job starts, ends, and fails — including a utilization summary showing how much CPU and RAM it actually used.

Resubmit:

sbatch --reservation=class_cpu slurm/extract_form_3_batch.slurm

Once your job runs, check your inbox. You should receive two emails: one when the job starts and one when it ends. The start email reports the queued time — how long the job sat waiting before it ran. The end email includes a utilization summary (how much CPU time and memory the job actually used) and the job’s exit status: 0 means success; any other value means it failed.

Low utilization means you asked for too much. If the summary shows the job touched a fraction of the CPU or memory you requested, ask for less next time. A smaller request is easier for Slurm to fit onto a node, so it clears the queue sooner and you get your results sooner — note that this shortens the waiting, not the run itself.

So scale each request down to just above what you measured, rather than padding it for comfort. The floor is your profiling numbers: ask for less than the job actually needs and Slurm kills it for exceeding --mem or --time, which costs you the whole run.

Finish all four pages before you start any bonus. The numbers you write down on one page are the inputs to the next, and Part 2 assumes all four are done.

⭐ Bonus — other ways to run and inspect jobs

Bonus — Go Interactive Instead of Batch

Everything so far has been batch submission — write a script, sbatch it, wait. Slurm also supports an interactive allocation on a dedicated node — handy when you’re debugging and re-running over and over: you hold the allocation, so you don’t re-queue for resources every time a job fails and you fix it:

srun --reservation=class_cpu --pty --cpus-per-task=2 --mem=4G --time=00:30:00 bash

Your interactive session is a Slurm job like any other — run squeue --me and you’ll see it listed (state R) until you release it:

squeue --me

Once the interactive Slurm job starts, you are on a fresh shell on the allocated node — do the same setup your batch script does, then run the script directly:

cd $HOME/yens-onboarding-2026           # into your project
source .venv/bin/activate               # activate your environment
python scripts/extract_form_3_batch.py  # run it and watch the output live

Because you’re interactive, you see the output as it happens and can re-run instantly after a fix — no re-queuing. Type exit to release the allocation when you’re done.

Bonus — Watch a Job Run on Its Node

slurm/mystery.slurm runs the mystery script from Profiling for about 30 seconds across a few cores — long enough to watch it live.

Submit it:

sbatch --reservation=class_cpu slurm/mystery.slurm

While your job is running you can SSH to the node it’s on and watch it work. (Nodes are shared — other users’ jobs run on them too — but your job has its own dedicated cores and RAM.)

First, run squeue --me to find which node it landed on — the NODELIST column (e.g. yen10):

squeue --me

Then SSH to that node and watch your processes live:

ssh SUNetID@yen10  # use your job's actual node
htop -u SUNetID    # or: top -u SUNetID

You’ll see the mystery script’s Python workers pinning the cores you requested. Press q to quit htop, then exit to leave the node.

You can only SSH to a compute node while you have a job running on it — once the job ends (or if you never had one there), SSH to that node is refused. You can’t hop onto arbitrary compute nodes.

Bonus — Chain Two Jobs

Research work usually comes in stages, each starting from what the one before it produced. Scaled up, your Form 3 work is naturally two jobs: (1) extract the structured fields with the API (what your batch script does), then (2) aggregate the per-filing JSON into one dataset and compute summary stats. The second stage reads what the first one wrote, so it cannot start until the extractions land. Rather than babysit them, launching each by hand the moment the last finishes, you queue the whole chain at once: --dependency=afterok tells Slurm to hold each job until the one before it succeeds. Your repo ships a small two-step version of this:

• scripts/chain_step1.py — crunches numbers for ~2 minutes, then writes its result to /scratch/users/SUNetID/chain_demo/step1_result.txt.
• scripts/chain_step2.py — reads that file and does ~30 seconds more math, writing step2_result.txt beside it.

with slurm/chain_step1.slurm and slurm/chain_step2.slurm to run them.

Step 1 — read both job scripts first so you know what you’re submitting. Notice they’re ordinary Slurm scripts, and that step 2 reads the file step 1 wrote:

cat slurm/chain_step1.slurm slurm/chain_step2.slurm

Step 2 — have Claude add the email lines to step 2 before you submit, so you get a note when the chain finishes:

> Add #SBATCH --mail-type=ALL and #SBATCH --mail-user=SUNetID@stanford.edu to slurm/chain_step2.slurm.

Step 3 — submit both back-to-back. Step 1 runs for ~2 minutes, so fire them off one after the other and let it crunch while step 2 queues behind it. Submit step 1 and note the JOBID it prints:

sbatch --reservation=class_cpu slurm/chain_step1.slurm

Then submit step 2 right away, chained to the first — replace JOBID with step 1’s ID:

sbatch --reservation=class_cpu --dependency=afterok:JOBID slurm/chain_step2.slurm

Step 4 — watch the queue. Both jobs are in, but step 2 waits its turn. watch re-runs a command every couple of seconds, so you can see the handoff happen live:

watch squeue --me

Step 1 shows R (running) while step 2 sits PD with reason (Dependency). When step 1 finishes, step 2 flips to R on its own — you do nothing. Press Ctrl+C to stop watching.

💡 If step 1 fails, afterok is never satisfied, so step 2’s reason in squeue changes from (Dependency) to (DependencyNeverSatisfied). That job will never run — but it won’t clear itself either. It sits in the queue until you cancel it with scancel JOBID. Clear it, fix step 1, then requeue the chain.

Step 5 — check the handoff once both are done:

cat /scratch/users/SUNetID/chain_demo/step2_result.txt

Step 2’s number is computed from step 1’s — proof the scratch file passed between them. Had step 1 failed, step 2 would never have started.

Bonus — The dev partition

The Yens have a dedicated dev partition for short, interactive debugging jobs — quick test runs while you’re getting a script working, not production runs. It has tighter time limits but is meant to turn around fast, so you’re not stuck in the main queue while iterating. Learn more: Yen Slurm partitions.

📋 Show steps

Fire a quick throwaway job at dev with -p dev (and --wrap, which runs an inline command as a job). It’s tiny, so it schedules fast, and it emails you when it finishes:

sbatch --reservation=class_cpu -p dev --mail-type=ALL --mail-user=SUNetID@stanford.edu --wrap="hostname; sleep 30"

Watch it — dev usually starts right away:

squeue --me

You’ll get a completion email in a moment. Confirm it says the job completed.

Bonus — Add a longsqueue alias

The default squeue output is sparse. Pass a custom format to see what each job actually requested — CPU cores, memory, and time limit:

squeue -o "%.18i %.9P %.8j %.8u %.8T %.10M %.10l %.4C %.7m %.15R"

The columns are: job ID, partition, job name, user, state, time elapsed, time limit, CPU cores requested, memory requested, and reason/node.

To keep it, append an alias. The quoted heredoc (<<'EOF') means nothing inside needs escaping — paste the whole block at once:

cat >> ~/.bash_profile <<'EOF'
alias longsqueue='squeue -o "%.18i %.9P %.8j %.8u %.8T %.10M %.10l %.4C %.7m %.15R"'
EOF
source ~/.bash_profile

Now run longsqueue. If the alias comes back “not found”, check the tail of the file with tail -3 ~/.bash_profile before appending again.

Bonus — Inspect any job with scontrol

Pick any job from squeue and look up its full details:

scontrol show job JOBID

Three fields to pick out of the output:

Field	What it tells you
NumCPUs	How many CPU cores the job asked for
mem=	How much RAM it asked for
TimeLimit	The wall-clock limit it gets killed at

Works on any job — yours or anyone else’s — as long as it is still queued or running.

Bonus — Compare partitions

Run sinfo -p gpu and sinfo -p normal to compare node counts and time limits.

sinfo does not show the per-user caps — the ceiling on how much any one person can hold at once. Those live in the partition’s QoS (“quality of service”), the policy Slurm attaches to each partition saying how much of it a single user may take. Compare the two with sacctmgr show qos gpu against sacctmgr show qos normal (or read the current partitions and their limits). When would you request one over the other?

⭐ Bonus — turn this into a Claude skill

You just wrote a Slurm script by hand and got it working. That working setup is the raw material for a skill — a set of standing instructions Claude picks up automatically, so it follows your conventions without you re-explaining them.

You are the reviewer throughout: you submit and check the work, not Claude.

Two homes for a skill

Every skill is a directory holding one file named exactly SKILL.md. The directory name is the skill’s name — lowercase letters, digits and hyphens only — and it is also how you invoke it: yen-slurm/ gives you the /yen-slurm command.

Where the directory lives decides its scope:

• Global skill → ~/.claude/skills/<skill-name>/SKILL.md — in your home directory (~/.claude/), so it loads in every project you work on. Best for conventions that follow you across projects (like how the Yens work).
• Project skill → <your-repo>/.claude/skills/<skill-name>/SKILL.md — in the repo’s own .claude/ (no ~/), so it loads only in this repo and, once committed, ships to anyone who clones it. Best for repo-specific conventions — where results land, which script owns which stage, the naming your collaborators expect.

Same .claude/skills/ layout, two different homes: the global skill in ~/.claude/ follows you into every project; the project skill in the repo’s .claude/ is committed and ships to anyone who clones it.

Write a Global Skill

Task: Have Claude distill a global Yen skill from the Slurm job you just ran by hand, then invoke it on a fresh job.

You’ll do this from Claude Code running on the Yens. Load the module and launch it inside your repo:

ml claude-code
cd ~/yens-onboarding-2026
claude

A global skill — distilled from the job you just ran

You already got a batch Slurm script working by hand. Rather than describe the Yen conventions from scratch, point Claude at that script and have it capture the reusable parts:

> Read slurm/extract_form_3_batch.slurm and turn its reusable Yen conventions into a global skill at ~/.claude/skills/yen-slurm/SKILL.md: partition choice, email, %j log naming, always setting --time/--mem/--cpus-per-task, and checking current limits on [RCpedia](https://rcpedia.stanford.edu/_user_guide/slurm/#current-partitions-and-their-limits). Keep it short and repo-agnostic — no project paths.

Check what it wrote. A good skill is short, and its description is what makes Claude reach for it later — so open the file and read it:

cat ~/.claude/skills/yen-slurm/SKILL.md

It should look something like this — a little frontmatter, then a few bullet conventions:

# ~/.claude/skills/yen-slurm/SKILL.md
---
name: yen-slurm
description: Write and check Slurm batch scripts for Stanford's Yen cluster — partitions, email, %j logs, resource requests. Use when writing a .slurm job.
---

When writing a Yen job script:
- Choose a partition: `normal` for production, `dev` for short jobs (limits on RCpedia)
- Always set `--time`, `--mem`, and `--cpus-per-task`
- Email on completion: `--mail-type=ALL`, `--mail-user=SUNetID@stanford.edu`
- Name logs `logs/<job-name>_%j.out` and `.err`

The description is the trigger — Claude reads it to decide when to pull the skill in. Leave it vague (“slurm stuff”) and it won’t fire when you need it; say what it does and when to use it.

Then invoke it on a fresh job. Claude Code turns each skill’s folder name into a /-command, so the yen-slurm/ folder gives you /yen-slurm — type it and add your request:

> /yen-slurm write a Slurm job for a new run and save it as slurm/extract_form_3_batch_claude.slurm

Submit and review: the conventions should come straight from the skill, matching what you hand-wrote.

You’re still the reviewer. A skill makes Claude follow your conventions, but Claude can still invent partition names, time limits, or per-user caps that don’t exist. Check its choices against RCpedia — the current partitions and their limits page and sacctmgr show qos <partition> — and against your own profiling.