Microsoft AI · 2024 - 2026
Commercial Copilot: answers you can act on, not just search results
Enterprise customers were getting help answers no better than a web search or a generic chatbot. I connected Copilot to trusted sources and built the LLM-as-judge evals that decided when an answer was good enough to ship, raising session success 7 points.
- role
- product manager
- customers
- commercial and enterprise Copilot users
- owned
- grounding sources, eval methodology, release thresholds
- partners
- Applied Science
Try this registry edit. A forum thread says it works on most versions.
forum · 2019Set it as a device policy so it covers every managed laptop.1 Pilot it on one group first.2
1 · official docs2 · your IT guidanceAn illustration of grounded versus ungrounded answers and the rubric that scores them. Not real eval data.
The problem
When an IT admin asks Copilot how to configure something across a fleet of company laptops, a wrong answer has a cost: they act on it. The alternatives they had were a web search that returns forum threads and stale docs, or a general-purpose chatbot that sounds equally confident whether or not it’s right.
For commercial customers, “better than search” has to mean two concrete things. The answer comes from a source the customer can trust, and someone has measured that it’s correct often enough to rely on.
What I did
Connected trusted sources. I worked on grounding Copilot’s help answers in trusted sources, so responses come from content we could stand behind instead of whatever ranked well on the web.
Built the evals. I defined the LLM-as-judge evaluation strategy for Copilot’s help experiences: the methodology, the scoring criteria and the success thresholds. Every answer was scored on two things:
- Tool-invocation accuracy. Did Copilot call the right source or tool for this question?
- Response quality. Is the answer correct, complete and safe to act on?
A judge model is only useful if it agrees with the people it’s standing in for, so I calibrated the judge against domain-expert assessments and ran offline evals with Applied Science before trusting it to gate a release.
Set release thresholds. Changes had to clear those thresholds to ship, which turned “it feels better” into a number the team could argue about.
Outcome
Session success rate rose 7 percentage points, and the team had an eval calibrated to expert judgment that every later change to sources or prompts had to pass.
What I learned
Grounding and evals are the same project. Adding trusted sources without measuring the answers just moves the hallucination somewhere harder to see, and measuring answers without fixing the sources only gives you a precise number for a bad experience.