Deciding with LLMs
Responding to the decision model hype train
The decision model hype train has reached fever pitch. Fair enough, they’re not a bad idea: we had machine learning classifiers long before autoregressive transformers turned up.
But here’s the thing. Any ordinary autoregressive LLM can already be used as a decision model, with no fine-tuning and no architectural changes. Instead of asking it to generate an answer, we give it a document, a question and a set of permitted answers, usually single tokens such as A (yes) and B (no). The model processes the prompt as normal, but instead of sampling the next token, we look at the raw output logits: the scores it assigns to every token in its vocabulary.
In a Python stack using Hugging Face Transformers, these come from model(**inputs).logits[0, -1, :]. We pick out the logits for our permitted answer tokens, which effectively masks out the rest of the vocabulary, and apply softmax to turn the remaining scores into a probability distribution. For two options with logits (z_A) and (z_B), the probability of A is (P(A)=e^{z_A}/(e^{z_A}+e^{z_B})), or equivalently (\sigma(z_A-z_B)).
If A gets 0.85 and B gets 0.15, the model favours A, and we have a margin we can measure rather than just a Boolean. One caveat: these probabilities are relative to the permitted answers. They aren’t necessarily calibrated estimates of being right.
The appeal is that we get a full distribution over the answers from a single forward pass, with no decoding loop and no parsing of free text. To ask several questions about the same document, we build separate prompts and run them as a batch, pulling the answer logits out of each. Fast.
In a production stack like vLLM, you can do the same thing with its logprob support or a custom logits processor, depending on the API. Put the document first and the question last, and prefix caching means the document only gets processed once per batch. Watch for a couple of gotchas: tokenisers often treat A and A (with a leading space) as different tokens, and logprob APIs usually return only the top-k tokens, so make sure your answer tokens actually make the cut.
We can also extend this to reasoning. Let the model think first, append an answer cue, then read the answer logits at that final decision point and compare them with the no-reasoning probabilities. Given a dataset coded by human researchers, this lets us compare models, model sizes, prompts and reasoning budgets, measuring not just whether they get the right label but how much probability they put on it. With suitable calibration and validation, it could help identify difficult codes, show which tasks actually benefit from bigger models or more reasoning, and let us spend inference budget accordingly.
It just takes a bit of rolling up the sleeves and getting your hands dirty in the inference stack.
Note this paper demonstrates doing Jev-like things with small fine-tuned models. There’s also a stack of decision models on Hugging Face which are essentially based on Qwen 3.6/3.8 27B. Lots of people seem to be getting excited about fine tuning again, focusing on making decisions but you know what, if you do it with regular Qwen as these guys have, they got #8 on decision index without any fine tuning at all.
I think if you’re going to fine-tune ought to consider if you can just train a really small dedicated model, turns out those are competitive with so-called decision models for certain tasks.