Artificial intelligence is rapidly becoming part of the machinery of American law. Attorneys use it to research cases, summarize discovery, draft motions, analyze contracts, and organize enormous collections of documents. Police agencies are experimenting with AI to transcribe body-camera footage, analyze reports, identify patterns, and reduce administrative workloads. Prosecutors and investigators can employ sophisticated software to sort evidence that would once have required hundreds of hours of human labor.
These technologies offer substantial benefits. They also introduce a peculiar danger to institutions whose legitimacy depends on the accurate determination of facts: AI hallucination.
A hallucination occurs when an artificial-intelligence system generates information that appears authoritative but is false, unsupported, or nonexistent. The machine may invent a judicial opinion, fabricate a quotation, attribute a statement to the wrong person, misstate what a document contains, or confidently supply details absent from the underlying evidence. The problem is not simply that computers make mistakes. Humans make mistakes as well. The deeper problem is that generative AI can manufacture errors with extraordinary fluency.
A fictional case citation can look exactly like a legitimate one. An inaccurate summary can sound perfectly reasonable. An invented factual connection may fit comfortably among genuine facts. The machine does not necessarily announce uncertainty. It can present fiction in the linguistic clothing of certainty.
That characteristic should concern every institution entrusted with administering justice.
The legal profession has already received warnings. Attorneys have submitted court filings containing nonexistent cases produced by generative AI. Judges have consequently had to remind lawyers of something that should never have required technological clarification: an attorney remains responsible for representations made to a court. Delegating research to software does not delegate professional responsibility.
This principle must remain inviolate.
AI can assist an attorney, but it cannot become an invisible junior lawyer whose work escapes review. Every quotation should be checked against the original. Every case should be verified through an authoritative legal database or court record. Every factual assertion should be traceable to evidence. The traditional habits of careful lawyering become more important, not less important, when machines dramatically increase the speed at which information can be produced.
Law enforcement presents an even more consequential problem.
Consider an AI system asked to summarize hours of body-camera footage, witness interviews, dispatch records, and investigative reports. A detective receiving the resulting summary may naturally assume that each statement originated somewhere in those materials. If the system hallucinates a detail—a suspect’s supposed statement, the sequence of events, the presence of an object, or a connection between two individuals—the error could quietly enter the investigative process.
Once introduced, erroneous information can acquire institutional momentum.
A detective may incorporate it into a report. Another investigator may rely upon that report. A prosecutor may read the investigator’s summary. The assertion may influence a search-warrant application, a charging decision, an interrogation strategy, or a plea negotiation. Eventually, determining where the supposedly established “fact” originated could become surprisingly difficult.
That possibility should fundamentally shape how government agencies deploy generative AI.
The government possesses powers that private organizations do not. It can investigate people, search homes, seize property, arrest suspects, initiate prosecutions, and deprive convicted defendants of liberty. Technologies used in exercising those powers therefore require standards considerably higher than ordinary commercial convenience.
No person should find himself confronting the coercive power of the state because software invented something.
There is another danger: automation bias. People frequently assign greater credibility to computerized outputs precisely because computers are presumed to be objective. A police officer who would question an uncertain witness might treat a polished AI-generated summary as neutral analysis. Yet the machine’s apparent neutrality can be deceptive. Generative systems operate probabilistically. They are exceptionally sophisticated tools for producing and analyzing language, but linguistic confidence is not evidentiary certainty.
The appropriate response is not to prohibit artificial intelligence from being used in the legal system. That would sacrifice genuine advantages and would probably prove impractical. AI can help locate relevant documents, organize discovery, transcribe recordings, identify inconsistencies, and reduce the administrative burden consuming the time of attorneys and officers. Properly controlled, it may allow professionals to spend more time exercising judgment rather than performing clerical work.
The wiser approach is governed by adoption built around human accountability.
Any AI-generated material affecting an investigation or judicial proceeding should be independently verifiable. Agencies should maintain records identifying when AI was used and, where appropriate, preserve the source materials from which an output was generated. AI-generated investigative summaries should never quietly become primary evidence. Prosecutors should know when consequential information was generated or materially processed by AI. Defense attorneys must have sufficient access to challenge technologically derived assertions when those assertions bear upon guilt, probable cause, credibility, or sentencing.
Courts, meanwhile, should continue insisting that lawyers remain personally accountable for their submissions. “The AI generated it” cannot become a twenty-first-century version of “the computer made a mistake.”
The constitutional order ultimately rests upon human responsibility. Judges judge. Juries determine facts. Attorneys answer for their advocacy. Police officers are responsible for investigations. Prosecutors are answerable for charging decisions. Technology can inform those responsibilities, but it cannot absorb them.
Artificial intelligence may become one of the most useful tools ever introduced into American law. But the justice system cannot permit efficiency to outrun verification. In ordinary commerce, an AI hallucination may produce inconvenience or embarrassment. In a courtroom or police investigation, the same failure can damage reputations, destroy livelihoods, compromise prosecutions, or contribute to the loss of an innocent person’s liberty.
The standard, therefore, should be simple and demanding: AI may assist in discovering the truth, but it must never be permitted to manufacture it.