Scale the Loop to an Array

🔴 Red sticky = I need help. Put it up the moment you are stuck — an instructor will come to you.

🟢 Green sticky = I have done 1–4 and am moving on to the bonus work, and I am ready to help my table.

Everything on this page runs from your clone, with the environment active:

cd ~/yens-onboarding-2026
source .venv/bin/activate

Task: Turn Part 1’s loop over filings into a job array that processes 100 filings, get it running, then find out what it actually used.

What you already have

scripts/extract_form_3_batch.py from Part 1 — Python that walks a list of filings, one API call at a time, waiting for each before starting the next.
The array from 1. Hello World Array — one submission, many tasks, running the identical script. The only thing that differs between them is SLURM_ARRAY_TASK_ID, and that page also shows how that number gets into Python.
data/aws_links.csv — already in your clone, with a urls column listing the filings.

You will end up writing two files:

scripts/extract_array.py — the Python that one task runs
slurm/extract_array.slurm — the array job that launches it

Make them the same way you made Part 1’s .slurm — a terminal editor, or + New → Text File in JupyterHub’s file browser. Keep those names — the pages either side of this one already use them in commands you will run.

Claude is fair game. Use it to draft, to debug, to explain an error you have not seen before. The one rule: you should be able to say what every line you keep is doing, and why. An array that works but that you cannot explain is worth less today than one that is broken and you can.

If you get stuck, there are hints at the bottom of this page — but try the reasoning first, and ask the person next to you before you scroll.

Estimate, document, submit

The same flow as Part 1, so you already know how each step goes:

Estimate what one task needs. Profile it the way you did this morning.
Document the numbers in your README.md, before you submit.
Submit, with those numbers in the .slurm, and think about what the array directive has to say for 100 filings.

Run it

Submit it, then watch it go. watch re-runs a command every couple of seconds, so you can see the tasks start in parallel and drop off as they finish:

watch squeue --me

The thing to notice is the job IDs: an array shows up as many rows sharing one ID, with a task number after it — 12345678_0, 12345678_1, and so on — each moving through the same PD → R → gone lifecycle you watched in 3. Submit.

What did it actually use?

MaxRSS is peak memory and Elapsed is wall-clock, per task:

sacct -j JOBID --format=JobID,State,Elapsed,MaxRSS

Compare that against what you estimated and wrote down. Over-asking for memory is the normal result, and worth noticing.

Stuck?

Put up a red sticky and ask your table — someone near you has probably just hit the same thing. Nudges, roughly in the order they help:

💡 How do you divide the work?

An array hands you a set of tasks that all run the identical script. The work has to be divided between them somehow, and the only thing that differs from one task to the next is its task ID — not the filing list, not the Python file.

So: 100 filings, and however many tasks you ask for. How are you splitting them, and how does a task work out which part of the work is its own?

💡 Where does the array start?

Did you start the index at 0 or at 1? Whichever you pick has to agree with how you index the filing list.

💡 Can you see what each task did?

Do you have a log per task? %A_%a in --output and --error gives every task its own pair of files. Without it, 100 tasks write over one log and you cannot tell which of them failed, or why.

A job that finishes is not the same as a job that did the work. Check:

the per-task .err files, for the tasks that failed
the number of output files you ended up with, against the number you expected

💡 Run it interactively to test

Run your Python by hand in a terminal first, on one task’s worth of work. Did it do what you expected? You can pass a task number in yourself to check it picks the right thing.

💡 Debug with Claude, and write down what you found

Paste the error and the part of your script it points at into Claude, and work through it until you can say what went wrong rather than just that it now runs.

Then write it down — what broke, and what fixed it. That note is worth more to you next month than the fix is.

⭐ Bonus — combine the results into one CSV

The array leaves you a directory of JSON files, one per filing. For analysis you want a single table instead — one row per filing, one column per field.

Have Claude write you a short script — scripts/merge_results.py — that reads every JSON and writes them out as one CSV, then document it as a step in your README. It is part of your pipeline now, not a one-off you ran once and forgot.

Think before you type: what happens to a task that failed and never wrote a file? And what does the row count tell you afterwards?

Going further. Give the merge a .slurm of its own — slurm/merge_results.slurm — so Slurm runs it only if the array finished cleanly:

sbatch --dependency=afterok:ARRAYJOBID slurm/merge_results.slurm

It sits in the queue as PD until the array succeeds, then runs on its own. If any task fails, it never starts — which is what you want, rather than merging a half-finished directory.