A1. Score Kev-27B before training

Goal: serve Kev-27B on one Yens H200 and measure how often it agrees with Jev. That score is the bar your own fine-tuned teacher has to reach in A2.


Where	Yens: the server on 1 H200, the queries from an interactive node
Data	the 1,000 test filings (2022 to 2026) with Jev’s labels
Model	Kev-27B, 51 GB of weights, built on Qwen3.8-27B
Time	about 15 minutes

The work is split across two machines. A Slurm job on a GPU node runs the Kev server and stays up. From your interactive Yen node, you send it the filings and score its answers.

Kev-27B needs an H200. Its 51 GB of weights do not fit on an A40 (48 GB). The Yens have two H200s, both on yen-gpu4.

The commands, in order

Command	Where	What it does
`ssh <sunet>@yen.stanford.edu`	your laptop	Logs you in to an interactive Yen node (`yen1` to `yen5`). Installs and queries run here.
`git clone https://github.com/gsbdarc/marlowe-hackathon.git`	interactive node	Makes your own copy of this repository in `/scratch/users/$USER`.
`source env/yens.sh`	interactive node	Sets the paths every script uses: `$DATA` (Jev’s filings), `$WORK` (your data, runs and results), `$KEV_HOME` (your Kev install), `$HF_HUB_CACHE` (the shared model weights) and `$PY`. Run it again in every new terminal.
`bash scripts/setup_kev.sh`	interactive node	Installs Kev in `$KEV_HOME` and checks that the models are in the shared cache. About 3 minutes, and safe to rerun.
`cp /scratch/shared/marlowe_hackathon/data/test1000.* $WORK/data/`	interactive node	Copies the 1,000 test filings, and Jev’s answers for them, from the shared read-only data folder.
`sbatch slurm/yens/advanced/a1_serve_kev27b.slurm`	interactive node	Starts the Kev server as a Slurm job on an H200 and prints its job ID.
`tail -f logs/kev27b-server-<jobid>.out`	interactive node	Follows the server’s log while the model loads. Ctrl-C stops following; the server keeps running.
`curl http://$(cat $WORK/kev-27b.server)/v1/models`	interactive node	Asks the server whether it is ready.
`$PY scripts/label.py ...`	interactive node	Sends each test filing to the server and saves Kev’s answers.
`$PY scripts/score.py ...`	interactive node	Compares Kev’s answers with Jev’s and prints the scores.
`scancel <jobid>`	interactive node	Stops the server and frees the GPU.

Where everything goes

Everything you make lives in your own scratch space, /scratch/users/$USER. source env/yens.sh sets the variables in the second column.

Path	Variable	What is there
`/scratch/users/$USER/marlowe-hackathon`		your clone of this repository: the scripts and Slurm jobs. You run every command from here.
`/scratch/users/$USER/marlowe-hackathon/logs/`		each Slurm job’s log, named after the job and its ID: `kev27b-server-<jobid>.out` is the server job’s (`-J kev27b-server` in the job file)
`/scratch/users/$USER/ceo-turnover`	`$WORK`	your data and results for every task
`$WORK/data/test1000.jsonl`		the 1,000 test filings and Jev’s answers, copied from the shared folder in Step 1
`$WORK/kev-27b.server`		the running server’s address, `node:port`, written by the server job
`$WORK/results/kev-27b-zeroshot.jsonl`		Kev-27B’s answers, one line per filing. Zero-shot means Kev-27B as released, before any fine-tuning.
`/scratch/users/$USER/kev`	`$KEV_HOME`	Kev’s code and its Python environment, from `setup_kev.sh`
`/scratch/users/$USER/hf`	`$HF_HOME`	your Hugging Face settings

Two shared folders, read only, are DARC’s: /scratch/shared/marlowe_hackathon/data/ (Jev’s data) and /scratch/shared/marlowe_hackathon/hf/hub (the model weights, $HF_HUB_CACHE). You read from them and never write to them.

Step 1: Set up the environment

ssh yen.stanford.edu puts you on an interactive node, a full-size server where you install Kev and run the client side of this task. Clone this repository into your scratch space and set the paths:

ssh <sunet>@yen.stanford.edu
cd /scratch/users/$USER
git clone https://github.com/gsbdarc/marlowe-hackathon.git
cd marlowe-hackathon
source env/yens.sh

Install Kev. The models are already in the shared cache, /scratch/shared/marlowe_hackathon/hf/hub: setup checks they are there and never downloads one.

bash scripts/setup_kev.sh

It takes about 3 minutes and ends with:

model ok: jaredpalmer/kev-4b@v1.0
model ok: Qwen/Qwen3.5-4B-Base@main
model ok: jaredpalmer/kev-27b@af0e6d551bdc2cc724f3e9d7a8bee1cd4fb8f7bf
tokenizer ok: Qwen/Qwen3.8-27B@1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0
Kev ready: /scratch/users/<sunet>/kev/kev at 53cb0b5, Python 3.13.16

Jev’s data is in /scratch/shared/marlowe_hackathon/data/, readable by everyone on the Yens. Copy the test filings into your own work folder:

mkdir -p $WORK/data
cp /scratch/shared/marlowe_hackathon/data/test1000.* $WORK/data/

Step 2: Start the server

Have Slurm email you when the job starts, ends or fails. Open the job file in an editor, the text editor in JupyterHub or vi in your terminal:

vi slurm/yens/advanced/a1_serve_kev27b.slurm

In JupyterHub, choose File > Open from Path… and paste the file’s full path: /scratch/users/<sunet>/marlowe-hackathon/slurm/yens/advanced/a1_serve_kev27b.slurm

Find the #SBATCH --mail-user line near the top and put your Stanford email address in place of the placeholder, so it reads:

#SBATCH --mail-user=<sunet>@stanford.edu

Save the file.

Then submit it:

mkdir -p logs
sbatch slurm/yens/advanced/a1_serve_kev27b.slurm

sbatch prints the job ID. The job asks for one H200, starts a Kev server with Kev-27B on it, and keeps running until you stop it or it reaches its 2-hour limit. Before the server starts, the job writes its address, the GPU node and a port, to $WORK/kev-27b.server:

cat $WORK/kev-27b.server

yen-gpu4:8973

The port is worked out from the job ID, so two servers on the same node do not collide.

Step 3: Wait until it is ready

If both H200s are busy, the job waits in the queue first; squeue --me shows PD until it starts. Once it runs, loading 51 GB of weights onto the GPU takes about 3 minutes. Follow the log:

tail -f logs/kev27b-server-<jobid>.out

When the model has loaded, the server answers on its address:

curl http://$(cat $WORK/kev-27b.server)/v1/models

Until then curl reports Connection refused. Wait and try again.

The server has no authentication, and its port can be reached by anyone on the Yens. Send it only the public filings in this task, and stop it when you are done.

Step 4: Label and score from the interactive node

With the server ready, send it the 1,000 test filings from your interactive node:

$PY scripts/label.py --url http://$(cat $WORK/kev-27b.server) \
  --data $WORK/data/test1000.jsonl --out $WORK/results/kev-27b-zeroshot.jsonl

label.py asks Kev the three questions about each filing and saves its answers. It reports filings per second as it goes. If it stops, run the same command again: it skips filings already answered. Then score the answers against Jev’s:

$PY scripts/score.py $WORK/results/kev-27b-zeroshot.jsonl

A1. Reply in the 🧵 A1 thread in the hackathon’s Slack channel with Kev-27B’s agreement with Jev on CEO changes and its filings per second. Then put up a 🟩 green sticky. 🟥 if you are stuck.

Step 5: Clean up

Stop the server, so the H200 is free for the next person:

scancel <jobid>

squeue --me should no longer list the job.

Optional: where does Kev-27B get it wrong?

Kev-27B agrees with Jev on most filings, but not all. Which filings does it get wrong, and what do they have in common? Look for patterns: the kind of change, the wording of the filing, its length, the year, how confident Kev was. Then ask what that means for research: which findings would change if you used Kev’s labels in place of Jev’s, and where you would still want a person to check. Make at least one plot that shows something you found. This does not need the server, so do it after Step 5.

These zero-shot labels are your baseline. Keep the notebook or the code: after fine-tuning later in the track, run the same analysis on the new labels and see which mistakes went away and which are still there.

Claude Code or Codex is fair game. Work in a JupyterHub notebook, or ask Claude Code to load the files and make the plots for you.

Everything you need is in three files, joined by the filing’s ID:

File	What is in it
`$WORK/results/kev-27b-zeroshot.jsonl`	Kev’s answers: `id`, and under `answers` each question’s `choice`, `confidence` and `probabilities`
`$WORK/data/test1000.parquet`	Jev’s labels for the same filings: `accession` (the ID), `status_name`, `why_name`, `ceo_change`, `words`, `year`, `company_name`, `edgar_url`
`$WORK/data/test1000.jsonl`	The text Kev read: `state`, with the ID in `_meta`

Hints

$PY scripts/score.py prints agreement by Jev’s status. Start with the statuses Kev gets wrong most often.
A confusion table of Jev’s status against Kev’s shows which answers Kev mixes up.
Compare Kev’s confidence on the filings it gets right and wrong. Is a wrong answer also an unsure one? If so, sending only the unsure filings to a person would catch most mistakes.
Check whether the mistakes are spread evenly or pile up in some years, industries or reasons for leaving. An even spread adds noise to a study; a pile-up biases it.
Read a handful of the wrong filings in full. Sometimes Jev is the one that is wrong.

A1 challenge. Reply in the 🧵 A1 challenge thread in the hackathon’s Slack channel with your plot and one or two sentences on what kind of filing Kev-27B finds hard and what that would mean for a study that used its labels.