The short answer: 97% can be a warning sign, not a victory

A RAG system that scores 97% in evaluation is not automatically production-ready. If the same evaluation set was reused during prompt tuning, retriever changes, document chunking experiments, reranking adjustments, or answer grading calibration, the score may reflect overfitting rather than real reliability.

In plain language: the system may have become excellent at passing yesterday's exam.

That distinction matters because enterprise RAG is rarely a demo feature. It is often connected to policies, contracts, product documentation, customer histories, HR procedures, legal opinions, financial controls, support playbooks, and other knowledge assets that people use to make decisions. A misleading score can turn into operational risk, compliance exposure, customer dissatisfaction, and wasted budget.

A high RAG score is useful only when it measures generalization. If it measures familiarity with the test set, it is a polished business illusion.

Why RAG overfitting happens so easily

Classic machine learning has a familiar discipline: training set, validation set, and test set. The test set is supposed to remain untouched until the end. Once teams use test results to make design decisions, it stops being a test set and becomes part of development.

RAG makes this harder to see because the work feels less like model training and more like product tuning. Teams adjust prompts. They change chunk sizes. They improve metadata. They remove documents that create noise. They add synonyms. They tune retrieval parameters. They rewrite questions. Each individual step can be reasonable. Together, they can contaminate the evaluation.

This is especially common when teams repeatedly run the same question set:

  • The team sees which questions fail.
  • The prompt is adjusted to handle those failures.
  • The retriever is tuned around known weak cases.
  • Ambiguous questions are rewritten or removed.
  • The benchmark score improves.
  • Leadership sees a clean upward trend.

The problem is that the business does not ask the same 120 questions forever. Employees ask messy questions. Customers phrase problems unpredictably. Legal teams ask cross-document questions. Support agents mix symptoms, history, and policy exceptions. Finance teams need controlled answers, not confident approximations.

A system that performs beautifully on a known benchmark can still fail when reality changes the wording, context, or risk level.

Goodhart's law in the AI program office

Goodhart's law says that when a measure becomes a target, it stops being a good measure. RAG evaluation is a perfect example.

Metrics such as Precision@k, Recall@k, MRR, answer similarity, groundedness, and faithfulness are valuable. They help teams see patterns, compare experiments, and detect regressions. But once the organizational goal becomes 'get the dashboard to 95%+', the metric starts to attract gaming behavior, even when nobody intends to game it.

The team starts optimizing the benchmark instead of the business outcome.

That is when AI governance becomes practical rather than ceremonial. The question is not only whether the model answered correctly. The real questions are sharper:

  • Did it retrieve the right source?
  • Did it ignore irrelevant context?
  • Did it admit uncertainty when the answer was not available?
  • Did it preserve policy nuance?
  • Did it handle conflicting documents responsibly?
  • Did it fail safely when the question was outside scope?
  • Did performance hold on fresh questions nobody tuned against?

These are not purely technical questions. They require domain knowledge, managerial judgment, academic discipline, and real operational experience. AI is not only an engineering layer. It is a multidisciplinary field where the best implementations combine machine learning expertise, process design, governance, finance, legal awareness, and human behavior.

The business cost of a contaminated evaluation

A contaminated evaluation set creates a false sense of maturity. That false maturity affects decisions across the company.

For operations, it can mean automating a workflow before the exception cases are understood. The result is escalation volume, rework, and frustrated teams who lose trust in AI.

For finance, it can mean approving larger budgets based on inflated performance claims. A 97% score can justify licenses, integration work, infrastructure, consulting fees, and change management. If the system later performs at 72% in production, the financial model collapses.

For compliance and legal teams, it can mean relying on answers that sound grounded but miss a clause, jurisdiction, date, or policy exception. In RAG, the most dangerous answer is often not obviously wrong. It is mostly right, fluently written, and missing the one detail that mattered.

For strategy, it can push leadership into the wrong conclusion: either overconfidence that AI is ready for broad autonomy, or disappointment that 'AI does not work' when the real problem was poor evaluation design.

Both reactions are expensive.

What a serious RAG evaluation should include

A credible RAG evaluation program needs separation, version control, and ongoing measurement. It should look less like a one-time demo checklist and more like an operational control system.

A practical structure includes:

  1. Development set: Questions the team can use freely while tuning prompts, retrieval, chunking, routing, and answer format.
  1. Validation set: A controlled set used to compare candidate configurations before release decisions.
  1. Locked test set: A protected set that is not used for routine tuning and is accessed only at defined decision points.
  1. Fresh field set: Real questions collected from users, support teams, sales, operations, finance, or legal workflows after deployment.
  1. Adversarial and ambiguity set: Questions designed to test unclear wording, missing information, conflicting sources, and out-of-scope requests.
  1. Human review protocol: Domain experts evaluate not only correctness, but also source fidelity, risk level, and whether the system should have refused to answer.

This structure is not bureaucracy. It is how an organization protects itself from self-deception.

Human in the loop, without turning humans into bottlenecks

Human review is critical in enterprise AI, especially when the workflow is non-deterministic and judgment-heavy. But there is a common mistake: organizations insert a person into every step and then claim they have implemented responsible AI.

That is not transformation. That is manual work with a more expensive interface.

The correct goal is different. A person who previously executed or supervised one process should be able to supervise dozens or hundreds of AI-assisted processes with better controls, better sampling, and better escalation logic.

For RAG evaluation, this means human experts should focus on high-value review points:

  • Reviewing samples from fresh production questions.
  • Auditing high-risk categories.
  • Investigating disagreement between automated graders and user feedback.
  • Defining what a safe refusal looks like.
  • Updating evaluation criteria when business policies change.
  • Approving release gates for material system changes.

Human-in-the-loop is not a checkbox. It is an operating model.

The hidden governance question: who saw the test set?

Before approving a RAG system for production, leaders should ask uncomfortable questions. These questions often reveal more than the headline score.

  • When was the evaluation set created?
  • Who wrote the questions?
  • Were the questions written before or after the first system outputs were reviewed?
  • How many times was this set used during tuning?
  • Were failed questions removed?
  • Were prompts or retrieval rules changed because of specific test failures?
  • Are evaluation sets versioned?
  • Is there a locked holdout set?
  • Are real user questions being sampled after deployment?
  • Are answer failures categorized by business impact, not only technical cause?

If the team cannot answer these clearly, the score should be treated as directional at best.

Why expertise matters more than AI theater

The market is full of self-appointed AI experts selling shortcuts. Large enterprises are usually better at filtering weak advice, although not immune. Small and mid-sized businesses are more exposed because they often lack internal AI governance, procurement skepticism, or senior technical review.

RAG evaluation is one of the areas where shallow expertise causes real damage. It is easy to show a beautiful chatbot. It is harder to design a statistically honest benchmark, map failure modes to business risk, understand retrieval behavior, and build a feedback loop that improves the system without contaminating the test.

This is why education, research discipline, and practical business experience matter. Academic thinking contributes rigor. Field experience contributes realism. Management experience ensures the system serves an operating model rather than a laboratory metric.

The strongest AI teams are not just prompt writers or tool operators. They understand processes, incentives, data quality, user behavior, risk, and deployment constraints.

RAG, agents, and the next enterprise platform problem

RAG evaluation is also becoming more important because enterprises are moving from isolated AI tools to AI agents. Agents often rely on retrieval to decide what to do, what context to use, and when to take action. If the retrieval layer is overfit or poorly evaluated, the agent inherits that weakness and may amplify it.

Organizations need two parallel tracks.

First, they need AI literacy across the workforce. Employees must learn how to communicate effectively with models, challenge outputs, ask better questions, and understand limitations.

Second, they need internal capability to build, manage, and monitor AI agents. This requires platforms, governance, release management, permissions, observability, and lifecycle ownership. Information systems departments will increasingly behave like human resources departments for AI agents: onboarding them, defining roles, monitoring performance, managing access, and retiring them when they are no longer fit for purpose.

This is why evaluation cannot remain an afterthought. It becomes part of the enterprise AI operating system.

A better executive interpretation of the 97% score

A 97% RAG score should not be celebrated or dismissed automatically. It should trigger a disciplined review.

The right executive interpretation is:

  • If the test set was clean, representative, and protected, 97% may indicate strong readiness.
  • If the test set was reused during development, 97% may indicate overfitting.
  • If the evaluation ignored refusal quality, source fidelity, and ambiguity, 97% is incomplete.
  • If performance has not been tested on fresh field questions, 97% is not yet evidence of operational reliability.
  • If business impact was not measured, 97% is a technical signal rather than a management conclusion.

The number is not the answer. The evaluation design behind the number is the answer.

The standard enterprises should demand

A production-grade RAG system should prove that it can generalize, not merely remember the exam. It should retrieve relevant evidence, answer faithfully, refuse when needed, handle ambiguity, and maintain performance on new questions. It should be monitored after release, not frozen at launch.

Most importantly, it should be governed by people who understand both AI and the business process it is meant to improve.

AI creates major operational efficiency opportunities because it can execute non-deterministic processes that previously required human judgment. But that value appears only when the organization treats implementation as a professional discipline, not a technology trend.

A clean evaluation program will not make every RAG system perfect. It will do something more valuable: it will tell the truth early enough for the business to act on it.