A2. Fine-tune Qwen 27B

Goal: fine-tune Qwen3.8-27B on Jev’s labels on one H100, then on all 8 H100s of one Marlowe node, and compare what each needs.


Where	Marlowe: 1 H100, then the 8 H100s of one node
Data	`train.jsonl`, 54,426 filings (2004 to 2018) with Jev’s soft labels
Model	Qwen3.8-27B, 27 billion parameters
Trainer	Kev’s trainer (`kev.train`): a full fine-tune, which trains every weight of the model
Time	about 1 hour, plus any wait in the queue

Both jobs run the same training: 100 optimizer steps of 16 filings each, so their times compare. A3 runs it again on two nodes.

The commands, in order

Command	Where	What it does
`ssh <sunet>@login.marlowe.stanford.edu`	your laptop	Logs you in to a Marlowe login node. Setup and job submission run here.
`git clone https://github.com/gsbdarc/marlowe-hackathon.git`	login node	Makes your own copy of this repository in the hackathon’s project space.
`source env/marlowe.sh`	login node	Sets the paths every script uses: `$DATA`, `$WORK`, `$KEV_HOME`, the shared model cache, and `$PY`. Run it again in every new terminal.
`bash scripts/setup_kev.sh`	login node	Installs Kev in `$KEV_HOME` and checks that the models are in the shared cache. About 3 minutes, and safe to rerun.
`cp /scratch/m000106/gsb_hackathon/ceo-turnover/data/...`	login node	Copies the training filings from the shared read-only data folder.
`sbatch slurm/marlowe/advanced/a2_train_1gpu.slurm`	login node	Starts the fine-tune on one H100.
`sbatch slurm/marlowe/advanced/a2_train_8gpu.slurm`	login node	Starts the same fine-tune on all 8 H100s of one node.
`tail -f logs/<job name>-<jobid>.out`	login node	Follows a job’s log. Ctrl-C stops following; the job keeps running.
`$PY scripts/step_times.py ...`	login node	Prints the time per step, and how long and how many GPU-hours a full pass over the data would take.

Where everything goes

Everything you make lives in your folder in the hackathon’s project space, /scratch/m000106/gsb_hackathon/$USER. source env/marlowe.sh sets the variables in the second column.

Path	Variable	What is there
`/scratch/m000106/gsb_hackathon/$USER/marlowe-hackathon`		your clone of this repository: the scripts and Slurm jobs. You run every command from here.
`.../marlowe-hackathon/logs/`		each Slurm job’s log, named after the job and its ID: `a2-27b-1gpu-<jobid>.out` and `a2-27b-8gpu-<jobid>.out` (the `-J` line of each job file), and `gpu-<jobid>.csv`, the job’s GPU memory and utilization
`/scratch/m000106/gsb_hackathon/$USER/ceo-turnover`	`$WORK`	your data and runs for every task
`$WORK/data/train.jsonl`		the training filings with Jev’s labels, copied from the shared folder in Step 1
`$WORK/runs/a2-1gpu/`		the one-GPU run’s folder
`$WORK/runs/a2-8gpu/`		the 8-GPU run’s folder: the fine-tuned model (`model-*.safetensors`, about 51 GB, plus `head.pt` and the tokenizer) and `training_metrics.json`, the time of every step, which `step_times.py` reads
`/scratch/m000106/gsb_hackathon/$USER/kev`	`$KEV_HOME`	Kev’s code and its Python environment, from `setup_kev.sh`

A job replaces its run folder when it starts. To keep a run, give the next one another name with RUN, as the challenge below does.

Two shared folders, read only, are DARC’s: /scratch/m000106/gsb_hackathon/ceo-turnover/data/ (Jev’s data) and /scratch/m000106/gsb_hackathon/hf_cache (the model weights, $HF_HOME). You read from them and never write to them.

Step 1: Set up on Marlowe

Log in to Marlowe and clone this repository into the hackathon’s project space. Your Marlowe home directory holds only 32 GB, so everything in this task lives under /scratch/m000106/gsb_hackathon/.

ssh <sunet>@login.marlowe.stanford.edu
mkdir -p /scratch/m000106/gsb_hackathon/$USER
cd /scratch/m000106/gsb_hackathon/$USER
git clone https://github.com/gsbdarc/marlowe-hackathon.git
cd marlowe-hackathon
source env/marlowe.sh

If you already have the folder, cd into it and run git pull instead of git clone.

Install Kev. The models are already in Marlowe’s shared cache: setup checks they are there and never downloads one.

bash scripts/setup_kev.sh

It takes about 3 minutes and ends with:

model ok: jaredpalmer/kev-4b@v1.0
model ok: Qwen/Qwen3.5-4B-Base@main
model ok: jaredpalmer/kev-27b@af0e6d551bdc2cc724f3e9d7a8bee1cd4fb8f7bf
tokenizer ok: Qwen/Qwen3.8-27B@1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Kev ready: /scratch/m000106/gsb_hackathon/<sunet>/kev/kev at 53cb0b5, Python 3.13.16

Have Slurm email you when each job starts, ends or fails. Open both A2 job files in an editor, for example vi in your terminal:

vi slurm/marlowe/advanced/a2_train_1gpu.slurm
vi slurm/marlowe/advanced/a2_train_8gpu.slurm

In each, find the #SBATCH --mail-user line near the top and put your Stanford email address in place of the placeholder, so it reads:

#SBATCH --mail-user=<sunet>@stanford.edu

Save the file.

Copy the training filings into your work folder:

mkdir -p $WORK/data
cp /scratch/m000106/gsb_hackathon/ceo-turnover/data/{train,valid,test1000}.* $WORK/data/

Step 2: Fine-tune on one H100

mkdir -p logs
sbatch slurm/marlowe/advanced/a2_train_1gpu.slurm

sbatch prints the job ID. squeue --me shows the job: PD while it waits for a GPU, R once it runs. Follow its log:

tail -f logs/a2-27b-1gpu-<jobid>.out

Early in the log the trainer reports dropped 1669 of 54426 records that exceed the training context (2048 state ...). That is expected: the job trains on filings of up to 2,048 tokens (--max_state 2048), and 3% of them are longer, so it trains on the other 52,757.

Every line of the log starts with the minutes and seconds since the job began, so you can see how long each phase takes, and the last line says how long the whole job ran. The job also writes logs/gpu-<jobid>.csv, the GPU’s memory and utilization every 15 seconds. Watch what happens. When the job ends, read the last lines of its log and the memory.used column of the GPU log:

tail -20 logs/a2-27b-1gpu-<jobid>.out
cat logs/gpu-<jobid>.csv

After the job ends: what happened

The GPU log shows the memory: about 52 GB once the weights are loaded, then full when the backward pass starts.

A full fine-tune keeps about 16 bytes for every parameter of the model:

What	Bytes per parameter	For 27B
Weights, in bf16	2	51 GB
Gradients, in bf16	2	51 GB
fp32 master copy of the weights, and AdamW’s two moments	12	310 GB

On one GPU, Kev’s trainer moves the 310 GB of master weights and optimizer state to the node’s host memory (the job asks for 480 GB of it). The weights and gradients still have to be on the GPU, about 102 GB, and an H100 has 80 GB: the backward pass runs out of room as the gradients fill in. On one H200 (141 GB) it would fit.

Step 3: Fine-tune on 8 H100s with FSDP

FSDP, Fully Sharded Data Parallel, is PyTorch’s way of training one model on several GPUs at once:

Data parallel: each GPU trains on different filings at the same time, and the GPUs combine their gradients before every optimizer step. Eight GPUs process eight filings at once.
Fully sharded: no GPU holds the whole model. Each holds one eighth of the weights, one eighth of the gradients and one eighth of the optimizer state. When the model runs a layer, the GPUs pass each other their pieces of that layer, compute, and drop the borrowed pieces again. After the backward pass, each GPU keeps only the gradients for its own piece.

The price is communication: the GPUs exchange data at every layer. Inside one Marlowe node the 8 H100s are connected by NVLink, which is fast enough that the exchange costs little. Kev’s trainer uses FSDP2, the current version in PyTorch. torchrun starts 8 copies of the trainer, one per GPU, and they find each other to share the work.

sbatch slurm/marlowe/advanced/a2_train_8gpu.slurm

The job takes a whole node, so it may wait in the queue longer than the one-GPU job. Follow its log:

tail -f logs/a2-27b-8gpu-<jobid>.out

After a few minutes of loading, the trainer prints how it set up the run, then a line every 10 optimizer steps.

The time to the first step line is the model load; the time between step lines is the training.

world=8 is the number of GPUs sharing the job. Monitoring Training explains the other fields. Compare the memory.used column of this job’s logs/gpu-<jobid>.csv with the one-GPU job’s.

Step 4: Measure it

When the job prints saved, turn its step times into how long a full pass over the training data would take, and how many GPU-hours it would use:

$PY scripts/step_times.py $WORK/runs/a2-8gpu

It prints the seconds per optimizer step, the peak GPU memory, and how many hours and GPU-hours one epoch (every training filing once) would take at this speed.

How busy were the GPUs? The job’s GPU log samples each GPU’s utilization every 15 seconds. Average it:

awk -F', ' 'NR > 1 { u = $4 + 0; s += u; n++; if (u > m) m = u } END { printf "GPU utilization: average %.0f%%, peak %d%%\n", s / n, m }' logs/gpu-<jobid>.csv

The peak is close to 100% on almost any run. The average, which includes the minutes of loading, is the one to watch: a GPU waiting on data or on other GPUs is not training.

A2. Reply in the 🧵 A2 thread in the hackathon’s Slack channel with these numbers:

Peak GPU memory on 1 H100: the highest memory.used in the one-GPU job’s GPU log

Peak GPU memory per GPU on 8 H100s: peak GPU GB from step_times.py

Seconds per step on 8 H100s: s/step from step_times.py

GPU utilization on 8 H100s: the average and peak from the awk command

GPU-hours for one epoch: epoch GPU-hours from step_times.py

Then put up a 🟩 green sticky. 🟥 if you are stuck.

Optional challenge: train it faster

How would you make this run faster? Each attempt is the same 100-step job as your first 8-GPU run, about 15 minutes, so aim for at least three attempts. Change one thing at a time, so you know what made the difference, and keep going: try other settings, combine the ones that helped, and find where each one stops helping or runs out of memory.

Claude Code or Codex is fair game.

Give each run its own name with RUN, so it gets its own folder and does not replace the last one. Settings go in front of sbatch:

RUN=a2-try1 sbatch slurm/marlowe/advanced/a2_train_8gpu.slurm

To watch your runs side by side as they train, add WANDB=1 as well. Each run appears in your Weights & Biases project marlowe-hackathon under its RUN name, with its loss, its speed, and its GPUs’ memory and utilization on the System tab. Log in to W&B on Marlowe once first, as Monitoring Training describes:

$KEV_HOME/kev/.venv/bin/wandb login
WANDB=1 RUN=a2-try1 sbatch slurm/marlowe/advanced/a2_train_8gpu.slurm

Give each attempt its own name (a2-try1, a2-try2, …) and note what you changed. When runs finish, score them all against your first run:

$PY scripts/step_times.py $WORK/runs/a2-8gpu $WORK/runs/a2-try*

It projects, from each run’s 100 steps, how long a full epoch (every training record once) would take. That projected epoch hours is the score: lower is faster. $PY scripts/step_times.py --help lists its options.

Hints

Look at the GPU log of your first 8-GPU run: how much of each H100’s 80 GB did it use, and how busy was each GPU?
slurm/train_27b_args.sh reads BATCH (filings per GPU in each forward and backward pass) and ACCUM (passes per optimizer step) from the environment. Each step trains BATCH × ACCUM × 8 filings, so BATCH=2 ACCUM=1 is the same 16 filings in one pass instead of two.
EXTRA_ARGS adds any kev.train option. Filings in a pass are padded to the longest one; --length_sort 1 groups filings of similar length so less of each pass is padding.
A pass that does not fit runs out of memory, like the one-GPU job did.
If you change the filings per step, pass --per-step to step_times.py. More filings per step can still train well, but it is no longer the same training as the first run.

Challenge. Reply in the 🧵 A2 challenge thread in the hackathon’s Slack channel with:

Projected epoch hours: your lowest epoch hours from step_times.py

What you changed: the settings or code, compared with the first 8-GPU run

Peak GPU memory per GPU: peak GPU GB from step_times.py

Average GPU utilization: from the awk command in Step 4

Step 5: Clean up

Each 8-GPU run saved a fine-tuned model in $WORK/runs/, about 51 GB each. Delete them if you do not need them:

rm -rf $WORK/runs/a2-*