AI contracting: a greenfield planted with wishful thinking

AI principles taking root: Truthful, unbiased, and compliant principles growing from an AI contract foundation amid a lush field.

Procurement can now ask an AI supplier to commit to truthfulness. We still need to agree on how to establish whether it has delivered.

I came across a useful example in the US defense guidance on implementing “Unbiased Artificial Intelligence Principles” in AI procurement. It includes three connected templates: the buyer selects its disclosure requirements, bidders submit the requested evidence, and the contract establishes continuing supplier obligations.

The structure makes sense. My concern is what happens when somebody tries to manage the resulting contract.

From principles to an argument

The guidance requires “Truth-seeking” and “Ideological Neutrality”. Suppliers provide model documentation, acceptable-use policies, user resources, and a mechanism for reporting potentially noncompliant outputs. Buyers can request additional disclosures, including evaluation methods and benchmark results.

After notification of a suspected violation, the supplier must investigate and respond within 15 business days. Its response must include findings, proposed corrective actions, and a proposed timetable. Significant model modifications require updated documentation.

There is enough here to start a procurement process. Whether there is enough to establish acceptable performance is another question.

Suppose the AI assistant produces an incorrect answer. The buyer considers it a breach. The supplier sees it as an occasional error within the technology's limitations.

Both have read the contract. Neither has an agreed basis for settling the matter.

Calling compliance a material contractual requirement gives it significance. It still leaves the parties needing a method for determining compliance.

AI Contracting: what the buyer needs to specify

I have turned this guidance into a contract-review exercise for my IT Procurement Foundations course. Four gaps deserve particular attention.

Potentially missing element	What the parties should agree
Testing methodology	Representative tasks, reference sources, sampling, model settings, repeated trials, and testing responsibilities. Standardized datasets can help, especially when they reflect the intended use.
Objective criteria	Scoring rules and acceptance thresholds, including distinctions between isolated errors, recurring failures, and critical incidents.
Verification and dispute resolution	Buyer verification rights, access to relevant evidence, escalation and independent review where findings are disputed.
Change controls and reacceptance	Changes requiring notification, regression testing and renewed acceptance, with workable arrangements for service continuity or rollback.

There is also a timing problem. The 15-business-day requirement covers investigation and a written response. The supplier proposes a timeline for completing the corrective action.

Buyers need to agree on resolution deadlines appropriate to the problem's severity.

These are adaptable templates, and the guidance expressly allows buyers to supplement them. The criticism therefore concerns the compliance framework these provisions provide alone. Other procurement documents may fill the gaps.

“The supplier shall prevent hallucinations”

This is where wishful thinking can become expensive.

Before negotiating a hallucination SLA, define what counts as a hallucination in the purchased service—a fabricated citation, an unsupported factual assertion, and an answer contradicted by the supplied documents. The measurement also needs a denominator: claims, answers, or completed tasks.

Then establish how you will sample, verify, and classify failures. A plausible answer with a hallucinated contract reference can have far different consequences than a minor factual error in an internal draft.

Supplier-funded retraining may be an appropriate corrective action, but prescribing it for every failure would be another loose assumption.

The cause of the hallucination could require changes to retrieval, source material, configuration, or the model itself. The contract should assign responsibility for diagnosis and correction, including retraining where necessary, followed by testing and buyer reacceptance.

Before accepting a promise of truthful AI, I would want to know who tests it, what evidence it is tested against, and whose decision applies when the results are disputed.