A3. Fine-tune on two nodes
Goal: run the A2 fine-tune across two full Marlowe nodes, 16 H100s, and compare its time per step with one node.
| Where | Marlowe: 2 nodes, 16 H100s |
| Data | train.jsonl, 54,426 filings with Jev’s soft labels |
| Model | Qwen3.8-27B |
| Trainer | Kev’s trainer, the same full fine-tune as A2 |
| Before you start | A2: setup done, and the 8-GPU run finished |
The training is the same as A2’s: 100 optimizer steps of 16 filings each. With 16 GPUs, each step gives one filing to each GPU at once, where 8 GPUs took two passes of 8.
The commands, in order
| Command | Where | What it does |
|---|---|---|
cd /scratch/m000106/gsb_hackathon/$USER/marlowe-hackathon |
login node | Moves to your clone from A2. |
source env/marlowe.sh |
login node | Sets the paths every script uses. |
sbatch slurm/marlowe/advanced/a3_train_2node.slurm |
login node | Starts the fine-tune on two nodes. |
tail -f logs/a3-27b-2node-<jobid>.out |
login node | Follows the job’s log. |
$PY scripts/step_times.py ... |
login node | Compares the time per step of the one-node and two-node runs. |
Step 1: Get ready
Work from your clone from A2:
ssh <sunet>@login.marlowe.stanford.edu
cd /scratch/m000106/gsb_hackathon/$USER/marlowe-hackathon
source env/marlowe.sh
Have Slurm email you when the job starts, ends or fails. Open the job file in an editor:
vi slurm/marlowe/advanced/a3_train_2node.slurm
Find the #SBATCH --mail-user line near the top and put your Stanford email address in place of the placeholder, so
it reads:
#SBATCH --mail-user=<sunet>@stanford.edu
Save the file.
Step 2: Fine-tune on two nodes
sbatch slurm/marlowe/advanced/a3_train_2node.slurm
The job asks for two whole nodes, so it can wait in the queue longer than A2’s. Slurm starts one copy of torchrun
on each node, and each starts 8 trainer processes, one per GPU. The 16 processes find each other through the first
node of the job and share the model 16 ways with FSDP.
Follow the log:
tail -f logs/a3-27b-2node-<jobid>.out
Every line starts with the minutes and seconds since the job began. After the model loads, the trainer prints a line
every 10 optimizer steps; world=16 in its first line is the 16 GPUs. Each node also writes its own GPU log,
logs/gpu-<jobid>-<node>.csv.
Step 3: Compare with one node
When the log prints saved, compare the two runs:
$PY scripts/step_times.py $WORK/runs/a2-8gpu $WORK/runs/a3-2node
It prints, for each run, the GPUs, the seconds per optimizer step, the peak GPU memory, and the hours and GPU-hours one epoch would take.
After you compare: what the numbers mean
Twice the GPUs does not halve the time per step. With FSDP, the GPUs exchange each layer’s weights before using it and the gradients after. Inside one node the 8 H100s are connected by NVLink; between nodes the traffic goes over InfiniBand, which is several times slower. With 16 GPUs, every exchange crosses that link, so part of the time the second node saves on computing goes to waiting for data.
Each GPU also holds half as much of the model as on one node, so the peak memory per GPU drops. GPU-hours are the cost: if 16 GPUs take more than half the time of 8, the same epoch costs more GPU-hours on two nodes.
A3. Reply in the 🧵 A3 thread in the hackathon’s Slack channel: the seconds per step on one node and on two, and the GPU-hours for one epoch at each. Then put up a 🟩 green sticky. 🟥 if you are stuck.
Step 4: Clean up
The two-node run saved another fine-tuned model, about 51 GB. Delete the runs you do not need:
rm -rf $WORK/runs/a3-2node