Asking LLM for confidence
LLM-based agents can do a lot of tasks. Or, at least, they can appear to do them. For example, they write code, summarize papers, plan trips, and fill in forms. If you ask one for a probability or a confidence score, it will give you one of those too. I have seen many agent designers use LLMs to generate confidence scores in this way. The problem is that there is no good reason to trust that number. This post is about why we cannot rely on an LLM to generate a confidence score.
What confidence means
Before we can understand why an LLM cannot give a trustworthy confidence score, we need to understand what confidence really means.
For example, think of a weather forecaster who says “I am 70% confident that it’s going to rain tomorrow.”
The forecaster is predicting that it will rain. The 70% is how sure they are about that prediction. Tomorrow, it will either rain or it will not, and how sure they are today depends on what they know.
Their confidence comes from the information that they use. Some of it is about tomorrow: air pressure, humidity, wind, and the clouds on the radar. Some of it is from the past: years of weather records, and how often it rained after days that looked like this one. The more information they have, and the more strongly it points one way, the closer their confidence can get to 100% or to 0%.
In theory, turning this information into a confidence is a calculation, not an opinion. Let’s say that this calculation can be represented by a hypothetical rule. You start with what the past tells you, and you update it with the information about tomorrow. If two forecasters have exactly the same information, and both apply the rule correctly, they must arrive at exactly the same confidence. When forecasters disagree, it is because they have different information, or because one of them applied the rule badly.
Now suppose that the forecaster is asked about a day two weeks from now. They have no information about that day yet. All that they have is the past. Suppose that, at this time of year, it rains on about 30% of days. Then the best that they can say is “I am 30% confident that it’s going to rain.” As the day gets closer, they get more information, and their confidence can move up toward 100% or down toward 0%.
How do we measure the accuracy of a confidence score
Now that we know what confidence means, let’s look at how we can tell whether a confidence score is accurate.
A single day cannot show whether the forecaster’s 70% was right, because it either rained or it did not. We can check their confidence only over many days.
Take all of the days on which the forecaster said that they were 70% confident of rain. If it rained on about 70% of them, their 70% is accurate. If it rained on only 40% of them, they were overconfident. If it rained on 90% of them, they were underconfident. We can do the same for each level of confidence. Suppose that this is the forecaster’s record for one year:
| They said | Days | Days it rained | Rate |
|---|---|---|---|
| 10% | 120 | 13 | 11% |
| 30% | 90 | 26 | 29% |
| 70% | 80 | 57 | 71% |
| 90% | 75 | 67 | 89% |
At every level, the rate of rain is close to what they said. When the confidence matches the actual rate like this, it is calibrated. This is also how we find out how well the forecaster follows the rule.
Calibration alone is not enough. A forecaster who ignores tomorrow’s weather and always says 30%, the usual rate for the season, is also calibrated. Over the year, it will rain on about 30% of their 30% days. But they never use the information about tomorrow, so their confidence never tells us anything. The forecaster in the table is better, because most of their days are near 10% or near 90%. They use the information, and they are still calibrated.
There is one more condition, and it is easy to miss. To do any of this, someone has to write down every forecast and what actually happened. Without that record, there is no way to know whether a confidence score is accurate.
So a confidence score that you can rely on needs two things:
- It reflects how much information the prediction is based on.
- It has been checked against what actually happened, on the same kind of question.
How an LLM gives a confidence score
An LLM produces text one token at a time. Each token is chosen based on the tokens that came before it. This is what “autoregressive” means. For example, when you ask “How confident are you?”, the answer “0.85” is not read from a gauge inside the model. The model writes it, token by token, in the same way that it writes any other text. “0.85” is a plausible continuation of the conversation. It is not a measurement of anything.
In the terms of the previous section, the number has neither property. It does not reflect how much information the model had, because it is a continuation of the text, not a summary of the evidence. And it was never calibrated. Pretraining does score the model against outcomes, but the outcome is the next token of text. So the model is calibrated, to some degree, on the question “What text comes next?” It was never scored on whether the confidence that it writes matches how often it is correct. The model learned what confidence scores look like in text. It did not learn to make its scores match its accuracy.
This has some consequences that you can see for yourself:
- Ask again, and the number changes. For example, the same question can get 0.85 one time and 0.7 the next. The information did not change, so if the model were applying a rule, the confidence would not change either.
- Change the wording, and the number changes. For example, if you ask “Are you sure?”, the score usually moves, even when nothing about the problem is different. A different wording is not new information.
- The text before the number decides the number. For example, if the model has already written “This is clearly the right answer,” a high score is the natural continuation. The score follows from the words, not from the evidence.
The model will always give you a number. It will look precise. And, as with the summary of a paper in my last post, an incorrect number looks the same as a correct one. You can find out whether the numbers are trustworthy only by grading many of them against an answer key. Most people who ask an LLM for a confidence score do not do that.
Measuring confidence without an LLM
If we cannot ask the LLM for its confidence, where do we get it? We can do what the forecaster does: build a rule that turns information into a confidence, and keep checking it against what actually happens. The LLM can still do the task, such as writing an answer. But the confidence score comes from the rule, not from the LLM.
For example, think of an agent that answers customer-support questions by searching a help center and writing a reply. We want a confidence score for each reply, so that the uncertain replies go to a person before they are sent.
1. Choose the information. List the things that we can observe about each reply, without asking the LLM how it feels. Did the search find an article on this topic? How many? Is this a type of question that the agent has answered before? If we ask the agent the same question twice, do the two replies agree? This is the agent’s version of air pressure, humidity, and wind.
2. Start with the past. Before we have any other data, the best confidence is the usual rate, as with the forecast for two weeks from now. If a person has checked 200 past replies and 150 of them were correct, every reply starts at 75%.
3. Record every reply and what happened. For each reply, save the information from step 1, the confidence that the rule gave, and whether the reply turned out to be correct. A reply is correct if a reviewer approved it, or if the customer did not have to ask again. This is the record that the previous sections said we need.
4. Update the rule from the record. Group the past replies by their information, and see how often each group was correct. If replies with a matching article and two agreeing answers were correct 95% of the time, then the rule should say 95% for the next reply like that. If replies with no matching article were correct only 40% of the time, the rule should say 40%. At first, the groups are small and the rule is rough. With more data, the groups get bigger, and the rule can use more of the information.
5. Keep checking. Every week or so, make the same table that we made for the forecaster: for each level of confidence, how often was the reply correct? If the 90% replies were correct only 70% of the time, the rule is overconfident, and the new records show how to fix it.
A rule like this has the properties that an LLM’s number does not have. The same information always gives the same confidence. The confidence depends on the information, not on the wording. And it is calibrated on our own kind of question, because it was built from our own records. It also gets better as we use it, because every reply adds to the record.
The rule is only as good as the record, though. If nobody checks whether the replies were correct, there is nothing to learn from, and we are back to a number that nobody has measured.