Parallelization Basics

Before you scale a job across the cluster, it helps to picture what “in parallel” actually means, when it works, and why it’s usually the single biggest speedup available to you.

When Parallelization Helps

Think back to the kitchen from Day 3: your machine is a kitchen, and every CPU core is a burner.

Say you want four grilled cheeses. The steps within one sandwich mostly don’t split: you can’t grill a side before the cheese is on the bread — nearly every step needs the previous one finished. Put four cooks on a single sandwich and three of them stand around watching. And no matter how many cooks you hire, a sandwich that takes four minutes takes four minutes.

Grilled cheese on one burner One burner cooks four grilled cheeses one after another — a single burner box shows the four steps (get bread, put cheese on bread, grill one side, grill the other side) lighting up in sequence, with a note to repeat four times, one sandwich after another. One burner, four grilled cheeses Burner sandwich 1 t = 1 t = 2 t = 3 t = 4 get bread put cheese on bread grill one side of sandwich grill other side of sandwich t = 5–8 t = 9–12 t = 13–16 sandwich 2 — same four steps sandwich 3 — same four steps sandwich 4 — same four steps × 4 one sandwich after another The four steps of one sandwich are a sequence — grilling can't start before assembly.

But the sandwiches don’t depend on each other. Each is its own pan — get the bread, put on the cheese, grill one side, grill the other — and no sandwich needs anything from the others. Fire up four burners and lunch is ready in a quarter of the time.

Grilled cheese on four burners Four burner boxes side by side each run the same four steps at the same time, cooking all four sandwiches at once. Four burners, four grilled cheeses t = 1 t = 2 t = 3 t = 4 Burner 1 get bread put cheese on bread grill one side of sandwich grill other side of sandwich Burner 2 get bread put cheese on bread grill one side of sandwich grill other side of sandwich Burner 3 get bread put cheese on bread grill one side of sandwich grill other side of sandwich Burner 4 get bread put cheese on bread grill one side of sandwich grill other side of sandwich The sandwiches are independent: four burners finish at t = 4 what one burner finishes at t = 16.

Code is the same. Parallelization pays off when the pieces of work are independent — each can run without waiting on the results of another. When a job splits cleanly into fully independent tasks with no coordination between them, it’s called embarrassingly parallel — the easiest, highest-payoff kind of work to parallelize.

A quick test, in kitchen terms: if you could hand each task to a different cook and never have them talk to each other, it will parallelize. Your extraction job fits perfectly — every filing is its own self-contained task.

Parallelization doesn’t make a single task faster — one grilled cheese still takes its four steps. It makes many tasks finish sooner by running them at the same time. If your bottleneck is one slow step, parallelizing won’t help; you need a faster step.

Let’s run through some examples together and discuss what types of tasks are parallelizable (or useful to parallelize):

Example 1

You’re given an array of numbers, and you’re asked to compute the sum of the numbers in the array:

∑x ∈ array x

Is parallelizing possible and/or worthwhile?

What if we want to compute the sum of some complicated function of each number (see below)?

∑x ∈ array f(x)
Example 2

You have a dataset A where rows are meant to have a unique key. You want to merge dataset B onto A, but without the merge creating new rows. A standard way to avoid that is to check that A’s rows are unique. Can this be parallelized?

Example 3

You’re scraping a website to compile a dataset. After scraping each page, you want to write some variables you found on the page to a common .csv that will store the results from all the webpages you’ve scraped. Can this be parallelized?

🔮 How to cast your progress — click to reveal

Every quest you check reveals a 🔮 Cast to the leaderboard button with a one-line ./cast spell. To record it:

  1. SSH to the Yens — ssh SUNetID@yen.stanford.edu
  2. cd to your repo — cd ~/gsb-research-computing-ai-skills
  3. Paste the ./cast … spell from the quest and run it.

Ways to Parallelize

Let’s make this concrete with the job you ran yesterday: processing SEC filings. Each filing is a sandwich: fetch it, send it to the API, save the fields that come back — those steps run in order, and no filing needs anything from another. So 100 filings are 100 independent sandwiches. There’s more than one way to spread them out: give one job more cores, run more jobs, or do both.

Recall yesterday’s script. It ran as one job on one core — a for loop walking the filings in sequence:

for filing in filings:          # 100 filings in the list
    result = extract(filing)     # one core does this, start to finish
    save(result)
# the loop can't start filing 2 until filing 1 is done
One job, one core A single Slurm job box holds one CPU and eight filings in a row. The CPU moves from filing to filing one at a time. Job filing 1 filing 2 filing 3 filing 4 filing 5 filing 6 filing 7 filing 8 CPU One job, one core — the filings are processed one after another. ≈ 8 × 5s = 40s.

If one filing takes 5 seconds, 100 filings take ~500 seconds — and the whole time your script is using exactly one core. A Yen node has dozens more you could have asked for.

Watch as we run this baseline and each of the three approaches below on the Yens — 20 filings every time, so the only thing that changes is how the work is spread. (The diagrams use 8 filings to stay legible; the live runs do 20.) This first one is the baseline: 1 job, 1 core, 20 filings, 20 API calls.

After each run, watch sacct -X -j JOBID --format=JobID,State,Elapsed,TotalCPU,ReqCPUS,MaxRSS shows what it cost: Elapsed is the wall-clock time, TotalCPU against ReqCPUS shows how busy the reserved cores actually stayed, and MaxRSS is the peak memory — the same field you compared against your estimate on Day 3.

Approach 1: One job, many cores — parallelize within a job. Ask the same job for several cores (on the Yens, set #SBATCH --cpus-per-task in your .slurm script) and split the filings across them in your code. But you’re capped at the cores on a single machine:

One job, many cores A single Slurm job box holds two CPUs and eight filings. The two CPUs work different filings at the same time, sweeping the row in four waves. Job filing 1 filing 2 filing 3 filing 4 filing 5 filing 6 filing 7 filing 8 CPU 1 CPU 2 One job, two cores — they split the filings and finish in waves. ≈ 4 × 5s = 20s.

Two cores clear the eight filings in four waves — ≈ 4 × 5s = 20s of wall-clock, versus ~40s one at a time. Same total work, spread across two cores.

Ask Claude Code for help with parallelizing within a job. Get it to split the filings across the cores for you: describe your loop and say how many cores you asked for. Read what it gives you before you run it — check that the work really is independent.

Let’s run an example like this on the Yens and see how it performs — the same 20 filings as the baseline, now split across the cores of a single job: 1 job, 2 cores, 20 filings, 20 API calls.

Approach 2: Many jobs, one core each — parallelize across jobs. Submit a job array: the scheduler launches many near-identical jobs at once, each an independent task on (possibly) a different node, each working its own slice of the filings. This scales past a single machine, and because every task stands alone, a failure costs you only that task:

Many jobs, one core each Two separate Slurm job boxes stacked vertically, each with its own CPU. The first job's CPU processes filings 1 to 4; the second job's CPU processes filings 5 to 8, both at the same time. Job 1 filing 1 filing 2 filing 3 filing 4 CPU 1 Job 2 filing 5 filing 6 filing 7 filing 8 CPU 2 Two jobs, one core each — each job works its own slice, in parallel. ≈ 4 × 5s = 20s.

We’ll cover job arrays in detail on the next page.

The tasks are identical — so you have to tell them apart. Every task in an array runs the same script, which means nothing decides on its own which filing each one takes. That mapping is yours to write.

Let’s run an example like this on the Yens and see how it performs — the same 20 filings again, this time split across independent jobs: 2 jobs, 1 core each, 10 filings apiece, 20 API calls.

Approach 3: Many jobs, many cores — do both. Nothing stops an array task from itself requesting several cores. Reach for this when one alone isn’t enough: many jobs to spread across nodes, several cores inside each to chew through a big slice:

Many jobs, many cores Two stacked Slurm job boxes, each holding two CPUs and four filings. In each job the two CPUs split the four filings two-and-two, and the two jobs run at the same time. Job 1 filing 1 filing 2 filing 3 filing 4 CPU 1 CPU 2 Job 2 filing 5 filing 6 filing 7 filing 8 CPU 3 CPU 4 Two jobs, two cores each — both at once. ≈ 2 × 5s = 10s.

Let’s run an example like this on the Yens and see how it performs — the same 20 filings once more, now spread across both jobs and cores: 2 jobs, 2 cores each, 10 filings apiece, 20 API calls.

Putting the four possibilities side by side:

  One core per job Many cores per job
One job Day 3’s baseline — a for loop, one filing at a time. No speedup. Approach 1. --cpus-per-task=N, then split the filings across the cores in your code. Capped at one node.
Many jobs Approach 2. A job array. Scales past one node, and one task failing costs you only that task. The number of tasks you can queue is capped. Approach 3. A job array whose tasks each ask for several cores. The most throughput, but the largest request — and the longer the queue wait.

More Filings Than Cores (and Vice Versa)

The illustrations above show a tidy few-filings-per-core picture. In practice the ratios vary, and both directions come up constantly.

More filings than cores — the usual case. Say you have 100 filings but only 8 cores. The cores work in waves: the first 8 filings run at once, and the moment a core finishes it picks up the next filing in line, until all 100 are done. The total time is roughly the number of waves times the per-filing time — about ⌈100 ÷ 8⌉ = 13 waves × 5s ≈ 65s, versus ~500s in serial. You don’t manage the waves yourself: within a job your code hands out the next filing; across array jobs, the scheduler does.

More cores than filings. Say you have 5 filings but the same 8 cores. Only 5 cores can do anything — a single filing can’t be split across cores — so the other 3 sit idle. Cores beyond the number of independent tasks buy you nothing, and requesting more than you’ll use can mean a longer wait in the queue for resources you never touch. Once the job starts, Slurm holds those cores for you whether you use them or not — so they sit idle in your job instead of running someone else’s. The Yens are shared; ask for what you’ll actually use.

Hitting a wall on resources in the future? Ask us. If a job is too slow, or you’re bumping up against what you can reasonably request, the DARC team is happy to look at your code with you.

What You Learned

  • You can explain the difference between running tasks sequentially on one core and in parallel across many
  • You can tell whether a workload is independent enough to parallelize
  • You can explain a few ways to parallelize — more cores within one job, more jobs, or both — and name the limit each one runs into
  • You understand that parallelization speeds up many tasks, not a single slow one
This site uses Just the Docs, a documentation theme for Jekyll.