Nish Tahir · 2026-10-10 · notable
Build your own decision model — a hands-on guide with Qwen3-1.7B
Nish Tahir's tutorial turns Qwen3-1.7B into a decision model that picks one of five options in a single forward pass. A quick finetune lifts CommonsenseQA accuracy from 59.4% to 62.4%, and temperature scaling fixes its overconfidence.
A step-by-step guide to building a small "System one" decision model from an open LLM, with code.
What is it?
"Build your own decision model" is a tutorial by Nish Tahir that shows how to make a decision model: a model that picks one answer from a fixed set of options instead of writing text token by token. The guide uses Qwen3-1.7B as the base and CommonsenseQA as the test task, and it ships a GitHub repo with scripts to build the dataset, evaluate, finetune and calibrate.
How does it work?
The trick is to limit the model's output to the option tokens A to E and read the probabilities of those five tokens in one forward pass. On a 1,221-question CommonsenseQA holdout, the base Qwen3-1.7B gets 725 right (59.4%); after a quick finetune it gets 762 (62.4%). Tahir then fits one temperature value (about 3.80) so the model's confidence scores better match how often it is actually right.
Why does it matter?
Decision models such as Jev, Clef and Laya are a fast-growing class of small, cheap classifiers for agents. This tutorial shows the core idea is reproducible on an open 1.7B model, and it points out a practical trap: a raw model said it was 98.6% sure on answers it got right only 70% of the time, until calibration fixed it.
Who is it for?
ML engineers building agent classifiers
Try it
git clone https://github.com/nishtahir/build-your-own-jev