A state reported a 94% Crisis Intervention Team training compliance rate.
At first glance, that sounds like a strong result.
If you were looking at a dashboard, you might see the number, mark the indicator green, and move on.
But then you ask one more question:
“Which counties are not compliant?”
That question changes what the 94% means.
In the HB1222 Crisis Intervention Team case, the statewide figure was 94%.
But five counties had zero trained officers.
The 94% wasn’t necessarily wrong.
The problem was that the aggregate number did not show where the remaining gap existed.
And when you’re making decisions about funding, implementation, accountability, or public safety, knowing where the gap is can matter just as much as knowing the statewide percentage.
Let’s audit the logic.
A Correct Number Can Still Hide a Decision-Critical Gap
This is one of the easiest problems to miss in impact reporting.
We tend to treat accuracy and usefulness as the same thing.
They aren’t.
A number can be calculated correctly and still leave out information a decision-maker needs.
Think about a hospital reporting that:
94% of patients received follow-up care.
That sounds reassuring.
But suppose the remaining 6% are concentrated in one community.
The statewide figure hasn’t become false.
But the decision-making picture has changed.
Leadership now needs to know:
Why are those patients concentrated there?
Is access different?
Is staffing different?
Is the referral process breaking down?
Does that location require additional resources?
The same principle applies to program evaluation.
The aggregate number tells you what happened across the whole system.
The underlying distribution tells you where to look next.
That’s why stopping at the headline percentage can be risky.
What the 94% Number Didn’t Tell Us
In the HB1222 CIT case, the 94% training compliance rate gave a statewide picture.
But the county-level information exposed something the statewide percentage could not communicate on its own:
Five counties had zero trained officers.
That isn’t a small detail.
It changes the questions a leader should ask.
A statewide compliance figure might lead someone to conclude:
Training coverage is strong.
The county-level finding prompts a different set of questions:
Which counties have no trained officers?
Why?
How long have they been without trained officers?
Are those counties serving populations with significant need?
What resources or implementation barriers exist there?
What needs to happen next?
The purpose of breaking down the data isn’t to make a good number look bad.
It’s to make the number more useful for action.
The Dashboard Problem: When the Headline Becomes the Whole Story
Dashboards are designed to simplify information.
That’s useful.
Nobody wants to open a dashboard and be buried under hundreds of rows of data before finding the number that matters.
But simplification creates a risk.
The more a dashboard compresses information into a single headline metric, the easier it becomes to lose the variation underneath it.
Imagine a school district reporting:
94% graduation rate.
That sounds excellent.
But now divide the result by school.
School A: 99%
School B: 98%
School C: 97%
School D: 62%
The district-wide figure hasn’t changed.
But your understanding of the situation has.
You now know where the problem is.
That’s the role of disaggregation in evaluation.
The goal isn’t to create more charts.
It’s to make sure important differences aren’t hidden by the average.
The Questions I Would Ask Before Trusting the 94%
When I see an aggregate compliance or outcome figure, I don’t immediately ask whether the calculation is correct.
I start with the questions underneath it.
1. What is included in the denominator?
What exactly does the 94% represent?
Who or what was counted?
If you can’t clearly define the population behind the percentage, the number is already harder to interpret.
2. What does the percentage look like by location?
A statewide or organization-wide average can hide substantial variation.
Break the figure down by county, site, department, region, or another meaningful unit.
Then look for differences.
3. Where are the zeros?
Zeros deserve attention.
A location with zero trained officers tells you something that an average may never reveal.
Don’t automatically treat zero as a data error.
First ask what it means.
4. Are the gaps concentrated?
A 6% gap spread evenly across 82 counties tells a different story from the same 6% concentrated in five counties.
The percentage is identical.
The intervention may not be.
5. What action would the number trigger?
This is one of the most important questions.
If leadership saw the 94% tomorrow, what would they do?
If the answer is nothing, ask whether the metric is giving them enough information to make a decision.
A useful indicator should do more than describe performance.
It should help someone understand what deserves attention.
This Is Why Indicator Definitions Matter
One way to prevent these problems is to define your indicators before the reporting cycle begins.
Not just the name of the indicator.
The logic behind it.
For example, an indicator definition should make clear:
- What is being measured?
- Who is included?
- What is the denominator?
- What counts as compliant?
- What is the reporting period?
- Where does the data come from?
- How is the calculation performed?
- At what level should the result be reviewed?
Without that structure, two people can look at the same indicator and interpret it differently.
One person sees:
94% compliance.
Another asks:
94% of whom?
Both may be looking at the same dashboard.
Only one is auditing the logic.
Don’t Just Report the Percentage. Test It.
Before your next board, funder, or executive report goes out, take your strongest percentage and break it apart.
Start with the headline.
Then ask:
What sits underneath this number?
Break it down by the units that matter.
For a nonprofit, that might be:
- Program site
- County
- Participant group
- Service type
- Age group
- Reporting period
For a corporate program, it might be:
- Department
- Business unit
- Location
- Employee group
- Quarter
The exact breakdown depends on the decision you’re trying to support.
The principle is the same:
Don’t let an average hide a meaningful difference.
A Simple Dashboard Test
Here’s a practical test you can use with your own reporting system.
Take one of your most important indicators.
Write down the headline number.
Then ask these five questions:
1. What is the overall result?
2. What are the results by meaningful subgroup or location?
3. Where are the lowest results?
4. Where are the zeros or missing values?
5. Would those underlying results change what leadership does next?
If the answer to the last question is yes, that information belongs somewhere in the reporting conversation.
It doesn’t necessarily need to sit on the first screen of your dashboard.
But it shouldn’t be impossible to find.
The Goal Isn’t More Data
This isn’t an argument for adding another 20 charts to your dashboard.
More information can create another problem: decision-makers stop knowing where to look.
The goal is better visibility.
A good dashboard gives leadership the headline.
A strong evaluation system also makes it possible to investigate the headline.
That’s the difference.
The 94% figure can still be useful.
But it becomes much more useful when you can move from:
“We’re at 94%.”
to:
“We’re at 94%, and here’s exactly where the remaining gap sits.”
Now leadership has something they can act on.
What This Means for Boards and Funders
This matters because boards and funders rarely care about a percentage in isolation.
They care about what the percentage tells them about the program.
Is the program working?
Where is it working?
Where isn’t it working?
Who is being reached?
Who isn’t?
Where should resources go?
What needs to change?
A headline metric may start the conversation.
The underlying evidence determines whether the conversation goes anywhere useful.
That’s why a strong report doesn’t simply make the number easy to find.
It makes the logic behind the number easy to examine.
Before You Publish Your Next Big Number
The next time your team prepares a report and someone says:
“The number looks good.”
Pause there.
Ask:
“What does it look like underneath?”
Then check.
Who is included?
Who isn’t?
Where are the gaps?
Where are the zeros?
Where is performance concentrated?
What would we see if we broke this number down?
And most importantly:
Would the answer change a decision?
Because the purpose of evaluation isn’t to produce numbers that look reassuring.
It’s to produce evidence that helps people see what is actually happening.
The 94% compliance rate was a useful starting point.
The five counties with zero trained officers made the picture more complete.
That’s the lesson.
A strong aggregate number should invite a second look, not end the conversation.
Want to strengthen the logic behind your indicators?
Start with an indicator definition that makes the number easier to understand, trace, and defend.
Download the Indicator Dictionary Template and use it to document the definitions, data sources, calculations, and evidence behind your most important indicators.
Download the Indicator Dictionary Template: https://olpadrindicatordictionarytemplate.claritytoimpact.com/?utm_source=linkedin&utm_medium=social&utm_campaign=indicator_dictionary
Evidence-led, impact-driven. The OLPADR way.