A4. Inference at scale with Qwen 235B

Goal: serve Qwen 235B across all 8 H100s of one Marlowe node, time how long it takes to load, then label filings with it and measure how many it answers per second.


Where	Marlowe: the server on the 8 H100s of one node, the queries from the login node
Data	the 1,000 test filings with Jev’s labels, then training filings (no labels needed)
Model	Qwen3-235B-A22B-Instruct-2507-FP8: 235 billion parameters in FP8, 237 GB of weights
Server	vLLM 0.31.0, in an Apptainer container
Before you start	A2 Step 1: your clone, Kev and the data on Marlowe

The work is split across two machines, as in A1. A Slurm job on a GPU node runs the server and stays up. From the login node, you send it the filings and score its answers.

The commands, in order

Command	Where	What it does
`cd /scratch/m000106/gsb_hackathon/$USER/marlowe-hackathon`	login node	Moves to your clone from A2.
`source env/marlowe.sh`	login node	Sets the paths every script uses.
`sbatch slurm/marlowe/advanced/a4_serve_qwen235b.slurm`	login node	Starts the vLLM server as a Slurm job on 8 H100s.
`tail -f logs/a4-qwen235b-server-<jobid>.out`	login node	Follows the server’s log while the model loads.
`curl http://$(cat $WORK/qwen-235b.server)/v1/models`	login node	Asks the server whether it is ready.
`$PY scripts/label.py --api chat ...`	login node	Sends filings to the server and saves its answers.
`$PY scripts/score.py ...`	login node	Compares the answers with Jev’s.
`scancel <jobid>`	login node	Stops the server and frees the node.

Where everything goes

Path	Variable	What is there
`/scratch/m000106/gsb_hackathon/$USER/marlowe-hackathon`		your clone; you run every command from here
`.../marlowe-hackathon/logs/`		the server’s log, `a4-qwen235b-server-<jobid>.out`, and its GPU log, `gpu-<jobid>.csv`
`$WORK/qwen-235b.server`	`$WORK`	the running server’s address, `node:port`, written by the server job
`$WORK/results/qwen-235b.jsonl`		Qwen 235B’s answers on the test filings, one line per filing
`$WORK/results/qwen-235b-train.jsonl`		its answers on training filings, from Step 5
`/scratch/m000106/gsb_hackathon/hf_cache`	`$HF_HOME`	the shared model weights, read only
`/scratch/m000106/gsb_hackathon/containers/vllm-openai_v0.31.0.sif`		the vLLM container, read only

Step 1: Get ready

Work from your clone from A2:

ssh <sunet>@login.marlowe.stanford.edu
cd /scratch/m000106/gsb_hackathon/$USER/marlowe-hackathon
source env/marlowe.sh

Have Slurm email you when the job starts, ends or fails. Open the job file in an editor:

vi slurm/marlowe/advanced/a4_serve_qwen235b.slurm

Find the #SBATCH --mail-user line near the top and put your Stanford email address in place of the placeholder, so it reads:

#SBATCH --mail-user=<sunet>@stanford.edu

Save the file.

Step 2: Start the server

Qwen 235B’s weights are 237 GB, three times one H100’s 80 GB, so the server splits the model across all 8 GPUs. vLLM does this with tensor parallelism: every layer of the model is cut into 8 pieces, one per GPU, and the GPUs compute each layer together, exchanging partial results over NVLink. A2’s FSDP also shares a model across GPUs, but for training: there, each GPU keeps a slice of the weights and borrows the rest when it needs them.

The model is a mixture of experts: each layer holds many small expert networks, and each word it reads goes through only a few of them, 22 billion of the 235 billion parameters. That makes it much faster to run than its size suggests.

mkdir -p logs
sbatch slurm/marlowe/advanced/a4_serve_qwen235b.slurm

The job takes a whole node, starts vLLM in its container, and serves until you stop it or it reaches its 2-hour limit. It writes its address to $WORK/qwen-235b.server:

cat $WORK/qwen-235b.server

Step 3: Time the load

Follow the log:

tail -f logs/a4-qwen235b-server-<jobid>.out

Every line starts with the minutes and seconds since the job began. vLLM reads the weights from the shared cache onto the 8 GPUs, sets up its memory for requests, and then reports that the application has started. Find those moments:

grep -E "Loading weights took|Model loading took|startup complete" logs/a4-qwen235b-server-<jobid>.out

How long did the weights take to load, and how long until the server was ready? When it is ready, it answers:

curl http://$(cat $WORK/qwen-235b.server)/v1/models

Until then curl reports Connection refused.

The server has no authentication, and anyone on Marlowe can reach its port. Send it only the public filings in this task, and stop it when you are done.

Step 4: Label the test filings and score them

Send the 1,000 test filings from the login node:

$PY scripts/label.py --api chat --model qwen-235b --url http://$(cat $WORK/qwen-235b.server) \
  --data $WORK/data/test1000.jsonl --out $WORK/results/qwen-235b.jsonl

With --api chat, label.py writes the filing and Jev’s three questions as one prompt with numbered options, and asks vLLM for a JSON answer limited to those options. Then score the answers against Jev’s:

$PY scripts/score.py $WORK/results/qwen-235b.jsonl

Kev-27B was trained to answer questions like Jev’s; Qwen 235B was not. Compare its scores with Kev-27B’s from A1.

Step 5: Label at scale

Now label training filings, as you would to make a teacher’s labels for A5. --concurrency is how many filings label.py keeps in flight at once; vLLM batches them on the GPUs:

$PY scripts/label.py --api chat --model qwen-235b --url http://$(cat $WORK/qwen-235b.server) \
  --data $WORK/data/train.jsonl --limit 5000 --concurrency 64 --out $WORK/results/qwen-235b-train.jsonl

It prints the filings per second as it goes, and a summary at the end. Try other values of --concurrency, each into its own --out file: where does the rate stop growing? Watch the GPUs in logs/gpu-<jobid>.csv as you do.

A4. Reply in the 🧵 A4 thread in the hackathon’s Slack channel with:

Weight load time: from Loading weights took in the server’s log

Time until the server was ready: the job’s minutes and seconds at startup complete

Agreement with Jev on CEO changes: status_agree_on_ceo_changes from score.py

Filings per second: your best rate in Step 5, and the --concurrency that gave it

Then put up a 🟩 green sticky. 🟥 if you are stuck.

Step 6: Clean up

Stop the server, so the node is free for the next person:

scancel <jobid>

squeue --me should no longer list the job.