Yen-Slurm Array Limits
🔴 Red sticky = I need help. Put it up the moment you are stuck — an instructor will come to you.
🟢 Green sticky = I have done 1–4 and am moving on to the bonus work, and I am ready to help my table.
Everything on this page runs from your clone, with the environment active:
cd ~/yens-onboarding-2026
source .venv/bin/activate
Task: Work out how to get all ~992 filings through an array when Slurm will not give you 992 tasks, run it, then write down what the full run cost.
The limit
Every scheduler puts a ceiling on arrays, and on the Yens’ normal partition it is
MaxArraySize = 512. Read it off the cluster rather than taking our word for it:
scontrol show config | grep MaxArraySize
What that number caps is the task index, not the number of tasks. Slurm allows an index
of 0, so the highest index you may use is one less than MaxArraySize — 511. An array
therefore holds at most 512 tasks, numbered 0 to 511, and --array=0-511 is the largest
one you can submit.
Try to go past it — with the array you wrote on the previous page — and the submission is simply refused:
sbatch --reservation=class_cpu --array=0-991 slurm/extract_array.slurm
The important part is that this cap applies to every submission, independently. There is
no second window of higher indices waiting for a second sbatch — index 512 is out of
bounds in a fresh array exactly as it was in the first one.
The other limits worth knowing
MaxArraySize is the one that stops you today, but it is not the only ceiling on an array:
| What it limits | Where to see it | |
|---|---|---|
MaxArraySize |
The highest task index — so also the largest array | scontrol show config \| grep MaxArraySize |
% throttle |
How many tasks run at once, without changing how many exist — --array=0-511%20 still submits all 512, but never more than 20 at a time. Not to be confused with a step, which is : — --array=0-511:20 submits 26 tasks, indices 0, 20, 40 … 500 |
your own directive |
| QoS caps | How much of a partition one user may hold — running jobs, submitted jobs, total cores | sacctmgr show qos normal |
--time |
Per task, not per array. A hundred tasks each get the full allowance | your own directive |
Your problem
data/aws_links.csv lists 992 filings. You have run 100 of them. The ceiling is 512
tasks per submission.
Work out how to process all 992. You have everything you need — page 2’s array, page 3’s rerun-safety, and the limit above. Hints are at the bottom if you want them; use Claude to think it through or work it out on paper, but be able to explain the route you chose.
Size it
Whatever route you pick, the per-task numbers may change — if a task now handles more than
one filing, one --time has to cover all of them. You are not guessing this time:
2. Scale the Loop to an Array left you real
sacct numbers for a single filing to build on.
Run it
mkdir -p logs
watch squeue --me
When it drains, count what landed:
ls results/*.json | wc -l
Short of 992? That is what rerun-safety is for — resubmit. Finished tasks find their output and exit immediately, so only the gaps are redone.
Document it, and push
In README.md, next to your 100-filing numbers, record what the full run actually took:
per-task time and memory from sacct, total wall-clock, how many tasks you split it into,
and which route you chose and why.
sacct -j JOBID --format=JobID,State,Elapsed,MaxRSS
Then commit it. Ask Claude Code to handle it:
> Add and commit my array scripts and README changes with a message like "Day 2: all 992 filings through an array", then push to my fork.
Stuck?
Put up a red sticky and talk it through with your table. Nudges, roughly in the order they help:
💡 What exactly does the ceiling forbid?
Write the two numbers next to each other: 992 filings, and a hard maximum of 512 tasks in any one array.
So one array cannot have one task per filing. That is the only thing the limit rules out — and notice how much it leaves open. Two of the three things in “992 filings, 512 tasks, one array” are assumptions you chose, not rules Slurm imposed.
💡 Does one task have to mean one filing?
On page 2 it did, because 100 filings fitted comfortably inside one array. Nothing in Slurm requires it.
What is stopping a single task from handling several filings? And if it did, what would that
change about the --time you ask for?
💡 What if one submission is not enough?
Suppose you submit twice. Every array starts numbering at 0, so the second submission’s task 0 gets handed exactly the same number as the first submission’s task 0 — and would therefore process exactly the same filing.
What extra piece of information would the second submission need in order to work on a different slice? And how could you get that information to it at submit time, without editing the script in between?
💡 How will you know it worked?
Decide the number you expect before it finishes: how many files should be in results/
when all 992 have been processed?
If you are short, you already have the machinery to fix it cheaply — that was page 3. And
per-task .err files tell you which tasks to look at rather than which array.