Day 3 — Cluster Computing

Today you’ll move from running your code interactively — one piece at a time, watching it go — to handing bigger jobs off to the scheduled Yens and letting them run on their own. You’ll pick up the ideas and habits for that step by step: how to size up what a job needs before you run it, how to send it off and check how it went, how to fix it when something breaks, and how to write down what you did so you (or a labmate) can run it again. By the end you’ll have run the whole thing on a real job.

Here’s what we’ll do today, start to finish:

The Day 3 research pipeline: profile your script to estimate resources, submit the batch job to Slurm, read the logs — debugging and resubmitting if it fails — then document, with scaling up on Day 4. profile submit toSlurm read logs document scale (Day 4) debug 1 2 3 4 5

Day 2 Recap

  • Built a virtual environment on the Yens and installed dependencies from requirements.txt
  • Built scripts/extract_form_3_one_file.py in three stages — calls the Stanford AI API to extract structured fields from one SEC Form 3 filing, with logging and a validated result file
  • Used Pydantic to validate and structure the LLM output

Any questions about Day 1–2 we can discuss before we move on.

Sections

Work through the sections in order — later ones build on earlier ones, and the Day 3 Capstone draws on everything you’ve learned.

Section Format What you master
Compute Environments 🥪💬 Demo + discussion Compute environments — so you pick the right one (laptop, Yens, or cloud) instead of wasting your advisor’s compute budget (or your afternoon)
Profiling Resource Usage 💻 Hands-on Profiling — so your own jobs request the resources they actually need instead of guessing and stalling in the queue
Exploring Cluster Usage Data 💻🤖 Hands-on (Claude-assisted) Data exploration — the same messy-real-data instinct you’ll need on your own datasets
The Slurm Scheduler 💬💻 Discussion + hands-on Interactive vs. scheduled Yen nodes — so your research jobs get dedicated resources instead of competing with everyone else on a shared login node
Writing & Submitting a Slurm Job 💻 Hands-on Batch jobs — the actual mechanism you’ll use to run your own research code on the cluster (plus debugging broken jobs)
Writing a Slurm Job with Claude 💻🤖 Hands-on (Claude-assisted) Reusable skills — distill a global Yen skill from the job you just ran, then make a figure and distill a project plotting skill from it, invoking each — project vs. global scope
Documenting Your Pipeline 💻🤖 Hands-on (Claude-assisted) Reproducibility — what makes collaborators, and your future self, trust and rerun your results
Day 3 Capstone 🔑 Capstone Profiled submission — the exact loop you’ll repeat for every real research pipeline from here on
This site uses Just the Docs, a documentation theme for Jekyll.