Calibrating Judgment: The Real Hard Part of FAIR Isn't Math, It's Honest Ranges

You can build the perfect FAIR model. You can define loss events with precision, map threat communities with rigor, and compute annualized loss exposure down to the dollar. And if the person feeding you the inputs is guessing, the whole thing is a confident lie.

FAIR is not a math problem. It's a measurement problem, and the instrument you're measuring with is a human being. The math is the easy part. The hard part is getting a busy CISO or an IT lead to give you a range they actually believe, rather than a number they think you want to hear.

This is the third post in our deep-dive series on FAIR. The first covered the fundamentals of the model. The second walked through the common mistakes that derail an analysis. This one is about the craft that nobody teaches: how to elicit well-calibrated estimates from people who are, by default, overconfident and under-prepared to answer your questions.

The uncomfortable truth: FAIR's output is only as good as its inputs, and the inputs are human judgment

Here's the uncomfortable truth about FAIR: the output is only as good as the inputs, and the inputs are human judgment. You can have a perfect model, perfect data, and perfect calculations, and still produce a result that is worthless because the person who estimated the loss event frequency was guessing.

Most FAIR analyses fail not because the math is wrong, but because the elicitation was sloppy. The analyst asked a question, the expert gave a number, and nobody challenged it. The number went into the model, the model produced a result, and the result went to the board as if it were a fact.

It isn't a fact. It's a guess dressed up in a probability distribution — and the whole art of calibration is making that guess honest.

This is not a reason to abandon FAIR. It's a reason to take the elicitation process seriously. The model is only as good as the people feeding it, and the people feeding it need discipline, structure, and a process that forces them to be honest about what they know and what they don't.

If you're a board member reading this, here's what you need to know: when your security team presents a FAIR analysis, the quality of that analysis depends on how the numbers were gathered. If they were gathered in a hallway conversation, treat the result with suspicion. If they were gathered through a structured elicitation process, you can have more confidence.

Why security experts are overconfident — anchoring, availability bias, and the "I've seen it once" trap

Security experts are not naturally well-calibrated. They are naturally overconfident. This is not a character flaw. It's a cognitive feature that comes from years of experience, and it manifests in three specific ways that will corrupt your FAIR analysis if you don't account for them.

Anchoring

Anchoring is the tendency to latch onto the first number you hear and adjust from there. If you ask a CISO "how many times per year do you think a credential-based attack will succeed?" and then add "I've seen estimates ranging from 1 to 50," you've just anchored them. They will adjust from that range, not from their own experience.

The fix is to ask for the estimate before you provide any context. No ranges, no examples, no hints. Just the question, and then silence. Let them think. Let them struggle. The first number they produce is the most honest one.

Availability bias

Availability bias is the tendency to overestimate the likelihood of events that are easy to recall. If a security professional has dealt with a ransomware incident in the last six months, they will overestimate the frequency of ransomware attacks. If they haven't seen a data breach in three years, they will underestimate it.

This is the "I've seen it once" trap. One incident, one anecdote, one news story, and suddenly the probability doubles in their mind. The fix is to force them to think in terms of base rates, not personal experience. Ask them: "how many times in the last 10 years has this happened across your industry?" rather than "how many times have you personally dealt with this?"

The confidence trap

The third issue is the confidence trap. When you ask a security expert for a range, they will often give you a narrow one. They are confident, and confidence feels like accuracy. But research shows that when people say they are 90% confident, they are right about 70% of the time. When they say they are 99% confident, they are right about 85% of the time.

This is not a criticism of security professionals. It's a universal human trait. The fix is not to tell people to be less confident. The fix is to build a process that forces them to confront their own uncertainty.

The calibration discipline — turning "how likely" into "how many events in 10 years"

The core of calibration is turning vague questions into concrete ones. "How likely is a data breach?" is a question that invites a vague answer. "How many data breaches will we experience in the next 10 years?" is a question that forces a concrete answer.

This is the single most important technique in elicitation. You are not asking for a probability. You are asking for a frequency. And you are asking for it over a specific time horizon.

Here's why this works. When you ask "how likely," people think in terms of gut feeling. When you ask "how many times in 10 years," people are forced to think in terms of concrete events. They have to imagine the event happening, and they have to count the instances. This is a much more reliable cognitive process.

Here's the drill. For each loss event in your FAIR analysis, ask the expert three questions:

• How many times do you think this event will occur in the next 10 years?
• What is the lowest number you would be surprised to see?
• What is the highest number you would be surprised to see?

The first question gives you the expected value. The second and third give you the range. And the word "surprised" is critical. You are not asking for the absolute minimum and maximum. You are asking for the range that contains the true value 90% of the time. This is the 90% confidence interval, and it is the foundation of a defensible FAIR analysis.

If you do this correctly, you will find that most experts give you a range that is wider than they initially wanted to give. That's good. That's honest. That's the range you want to put into the model.

Elicitation technique — how to interview a busy CISO or IT lead to get a defensible range (not a point)

You have 45 minutes with a busy CISO. You need a defensible range for loss event frequency and loss magnitude. Here's how to run that interview.

Step 1: Set the stage

Before you ask a single question, explain what you're doing and why. Tell them: "I'm going to ask you a series of questions about how often you think certain events will occur. There are no right answers. I'm looking for your honest judgment, not a number you think I want to hear. I will not use your answers to evaluate your performance. I will use them to build a model that helps us make better decisions."

This is not a formality. It's a critical step. If the CISO thinks you're testing them, they will give you confident, narrow ranges. If they think you're building a model, they will give you honest, wide ranges.

Step 2: Ask for the 90% confidence interval

For each loss event, ask the three questions from the calibration drill. But here's the technique: ask for the expected value first, then ask for the range. Do not ask for the range first. If you ask for the range first, they will anchor on the midpoint and give you a narrow range. If you ask for the expected value first, they will give you a number, and then the range will be a genuine adjustment.

Here's the exact phrasing: "How many times do you think this event will occur in the next 10 years?" Wait for the answer. Then: "What is the lowest number you would be surprised to see?" Wait. Then: "What is the highest number you would be surprised to see?" Wait.

The silence is important. Do not fill it. Let them think. Let them adjust. The first answer is rarely the best answer.

Step 3: Avoid loaded framing

Do not say "how likely is a ransomware attack?" Do not say "how vulnerable are we to phishing?" These are loaded questions. They imply that the event is likely and that the organization is vulnerable. Instead, ask neutral questions: "How many times in the next 10 years do you expect a ransomware attack to succeed?"

Also avoid providing examples. Do not say "I've seen estimates ranging from 1 to 50." Do not say "some organizations report this happening monthly." Every example you provide is an anchor, and every anchor corrupts the estimate.

Step 4: Separate best case, worst case, and expected

This is a powerful technique for getting a defensible range. Ask for three numbers: the best case, the worst case, and the expected case. Then use the best case and worst case as the boundaries of your range, and the expected case as the midpoint.

Here's the phrasing: "If everything goes right, what's the fewest times this could happen in 10 years? If everything goes wrong, what's the most times? And what's your honest expectation?"

This technique works because it separates the cognitive processes. The best case and worst case are relatively easy to imagine. The expected case requires more thought. And the gap between the best case and worst case is usually wider than the expert initially wanted to admit.

Step 5: Challenge the range

Once you have a range, challenge it. Say: "You said between 2 and 10 times in 10 years. Are you 90% confident that the true number is in this range? If I offered you even odds on a bet, would you take it?"

This is not adversarial. It's a calibration check. If the expert is not 90% confident, widen the range. If they are 90% confident, keep it. The goal is not to get the narrowest range. The goal is to get the most honest range.

Frequency vs. magnitude elicitation — the two different conversations

FAIR separates loss into two components: loss event frequency and loss magnitude. These are two different conversations, and they require two different elicitation techniques.

Loss event frequency

Loss event frequency is about how often an event occurs. This is a counting exercise. You are asking "how many times in 10 years?" The calibration drill works well here because people can count events.

The key is to be specific about what counts as an event. Define the event clearly before you ask. "A successful credential-based attack" is not a clear definition. "An attacker using stolen credentials to gain unauthorized access to a system that contains sensitive data" is clearer. The more specific the definition, the more reliable the estimate.

Loss magnitude

Loss magnitude is about how much damage an event causes. This is a valuation exercise. You are asking "how much money will this cost?" And this is where the conversation gets harder.

People are terrible at estimating costs. They underestimate the cost of downtime, the cost of legal fees, the cost of reputational damage. They overestimate the cost of hardware replacement. The fix is to break the loss into components and elicit each one separately.

Here's the breakdown:

• Productivity loss: how many hours of work will be lost, and what is the hourly cost?
• Response cost: how many hours of incident response, and at what rate?
• Replacement cost: what hardware or software needs to be replaced?
• Legal and regulatory: what fines, legal fees, and notification costs will be incurred?
• Reputation: what is the estimated impact on customer churn and future revenue?

Elicit each component separately, then sum them. Do not ask for a single number. A single number will be a guess. A sum of components is a calculation.

For each component, use the same calibration drill. "How many hours of incident response do you expect? What's the lowest you'd be surprised by? What's the highest?" Then multiply by the hourly rate. This gives you a range for each component, and you can sum the ranges to get a range for total loss magnitude.

A calibration drill — the deck-of-cards test that reveals how miscalibrated people actually are

Before you start eliciting estimates from your team, run a calibration drill. It takes 15 minutes, and it will change how you think about every number you collect.

Here's the drill. Take a deck of cards. Shuffle it. Ask the participant to estimate the probability that the next card is red. They will say 50%. Correct. Now ask them to estimate the probability that the next card is a heart. They will say 25%. Correct.

Now make it harder. Deal 10 cards face down. Ask them to estimate the probability that at least one is a heart. They will probably say something like 75% or 80%. The correct answer is about 94%. They are underconfident.

Now deal 20 cards face down. Ask them to estimate the probability that at least one is a heart. They will probably say 95% or 99%. The correct answer is about 99.7%. They are close.

Now the real test. Ask them to estimate the probability that the next card is a 7. They will say about 7.7%. Correct. Now ask them to estimate the probability that the next card is a 7 or a face card. They will say about 38%. Correct.

Now ask them a question they cannot calculate. "What is the probability that a randomly selected person in this building has a birthday in January?" They will say about 8%. Correct. Now ask: "What is the probability that two randomly selected people in this building have birthdays in the same month?" They will probably say something like 10% or 15%. The correct answer is about 87%.

This is the point. People are systematically miscalibrated. They are overconfident in some situations and underconfident in others. And they have no idea which is which.

After the drill, ask them to rate their confidence on a series of questions where you know the answer. "What is the capital of Australia?" (Canberra, not Sydney.) "What is the population of Canada?" (About 38 million.) "What is the boiling point of water at sea level?" (100°C.)

For each question, ask them to give a range that they are 90% confident contains the true answer. Then score them. If they are well-calibrated, about 90% of their ranges will contain the true answer. If they are overconfident, fewer than 90% will. Most people score between 50% and 70%.

This drill is not a test of intelligence. It's a test of calibration. And it's a powerful way to show your team why the elicitation process matters. When they see that their 90% confidence intervals are actually 60% confidence intervals, they will be more humble about their estimates. And humility is the foundation of a defensible FAIR analysis.

When to use data instead of judgment — where actuarial and trade data should override elicitation

Elicitation is not the only source of inputs for a FAIR analysis. In many cases, you have data that should override human judgment entirely. The rule is simple: when you have data, use it. When you don't, elicit.

Here are the sources of data that should take precedence over elicitation:

Industry breach data

If you are analyzing the frequency of credential-based attacks, and you have industry data showing that the average organization in your sector experiences 2.3 successful attacks per year, use that data. Do not ask the CISO for their opinion. The data is more reliable than the opinion.

Sources include Verizon's Data Breach Investigations Report, the IBM Cost of a Data Breach Report, and industry-specific reports from regulators and trade associations. These are not perfect, but they are better than a single person's judgment.

Actuarial tables

For certain types of loss, actuarial tables exist. If you are analyzing the probability of a fire destroying a data center, there are actuarial tables for that. If you are analyzing the probability of a key employee dying, there are actuarial tables for that. Use them.

The key is to adjust the actuarial data to your specific context. A data center in a flood zone has a different risk than one in a desert. An employee with a heart condition has a different risk than a marathon runner. But the base rate is still the starting point.

Internal historical data

If your organization has been tracking security incidents for the last five years, you have data. Use it. Count the number of incidents per year, calculate the frequency, and use that as your input. Do not ask the CISO to estimate what you already know.

The caveat is that historical data is backward-looking. If your security posture has changed significantly, the historical frequency may not be representative of the future. But it is still a better starting point than a guess.

When to elicit anyway

There are situations where data is not available, or where the data is not relevant to your specific context. In those cases, elicitation is the only option. But even then, you should use the data you have to anchor the elicitation.

For example, if you are analyzing the frequency of a new type of attack that has only been observed in the wild for six months, you have no historical data. But you can use the data from similar attack types as a starting point. "We know that phishing attacks succeed about 1% of the time. How does this new attack compare?"

The key is to be transparent about the source of your inputs. If you used data, say so. If you used elicitation, say so. If you used a combination, say so. The board deserves to know the difference between a measured number and a guessed number.

Build the discipline into your process

Calibration is not a one-time exercise. It's a discipline that has to be built into your FAIR process. Every analysis should include a structured elicitation session, and every elicitation session should follow the same protocol.

If you are doing this internally, assign a dedicated elicitation lead. This person is not the CISO and not the IT lead. They are a neutral facilitator whose job is to extract honest estimates. They should be trained in the techniques described here, and they should be empowered to challenge every number.

If you are working with an external partner, make sure they have a documented elicitation methodology. Ask them how they handle overconfidence, anchoring, and availability bias. If they don't have a good answer, find a different partner.

The payoff is a FAIR analysis that the board can trust. Not because the math is right, but because the inputs are honest. And honest inputs are the only foundation for a defensible risk decision.

The math is easy. The judgment is hard. But with the right process, you can turn confident guesses into honest ranges, and honest ranges into decisions you can defend.