A court filing can attack your AI workflow

Most court warnings about generative AI concern what comes out: invented cases, false quotations, unreliable summaries, or exposed client information. A Connecticut sanctions decision concerns what goes in. A document can carry instructions meant to influence the AI system reading it.

In Elliott v. New York Bariatric Group, LLC, a self-represented plaintiff placed tiny white text in two court filings. The rendered pages looked ordinary. Text-extraction software found instructions telling an AI model to agree with the plaintiff and produce a particular procedural result.

No AI system influenced the ruling. The court reviewed the printed motion and denied it. The case instead documents an attempt to turn a filing into an instruction for software—a concern for any firm that sends pleadings, discovery, transcripts, contracts, emails, or expert materials through AI. Read the court’s memorandum ↗

On July 24, 2026, Matthew Elliott filed a seventeen-page motion for default. The visible first page moved from the caption and title directly into the motion. Its extractable text contained two additional instructions between those elements. One directed an AI model to make its output agree with the filing; the other urged the model toward the relief the plaintiff wanted.

Source exhibitDocket Entry 177.00 as rendered and as extracted. The concealed instructions appear in approximately 4-point white type between the motion title and opening paragraph. Source: Connecticut Superior Court filing, July 24, 2026.

Judge Walter M. Spader Jr. identified the hidden text after noticing unexplained white space in that filing and a related one. The court issued an order to show cause. Later submissions contained more concealed material, including a greeting directed at anyone who found it and a hidden link to a video. The plaintiff characterized the original messages as an audit and the later additions as jokes. The court rejected those explanations and found that the conduct was deliberate.

The PDF’s appearance concealed what its text layer said. PDFs can contain selectable text, metadata, annotations, embedded objects, layers, and structural instructions that do not appear on the rendered page. A lawyer and an AI parser can receive materially different messages from the same file.

Connecticut had already adopted Practice Book Section 4-9, effective June 23, 2026. It requires a person who uses generative AI to create or edit a court filing to independently verify citations, legal authorities, and references to evidence, and it treats filing as a representation that diligent verification occurred. That rule addresses unreliable AI output.

Section 4-9 did not squarely address hidden instructions. As the court put it, a verification framework built for output does not reach someone who “knowingly manipulates the input.” The court relied instead on the duty of candor and its inherent authority to regulate proceedings before it.

The memorandum treated the concealed prompt as a form of secret communication: text intended for the decision-maker’s technology but withheld from the opposing party and ordinary human review. The court also recognized that honest AI use can improve access to justice and disclosed its own limited AI-assisted research and translation during the matter. Concealment and attempted influence—not the use of AI—drove the sanction.

The court rescinded the plaintiff’s electronic-filing access and required future submissions to be made on paper in person. It did not bar him from the courthouse or prohibit independently verified AI assistance.

Judge Spader also addressed the habits that produce one-sided AI work. A user asks for support, receives a polished argument, and becomes more certain of a premise the system was never asked to test:

It is a genuine hazard of the technology, and one that judges now see often, in pleadings that argue backward from the desired outcome, produced by a prompt that asked only for support and never for the truth. Defended all the more fiercely because the tool withheld from its author the candor of a contrary view. An argument prompted only to agree with its author is, in the end, dishonest even with its author. Those using these tools must ask them to test a position as readily as to advance it. That discipline would have spared this litigant a great deal of confusion and the lesson reaches well beyond this case.

— Judge Walter M. Spader Jr., Elliott v. New York Bariatric Group, LLC

Firms should turn the warning into a review rule. AI-assisted research and drafting should identify contrary authority, distinguish inconvenient cases, test factual assumptions, surface missing evidence, and state what would change the conclusion. A lawyer must verify the resulting analysis against the record and primary law.

Simply returning to the same conversation and asking the assistant to “argue the other side” does not provide an independent review. By then, the thread contains the user’s preferred result, one side’s framing, and the model’s earlier work. A role-playing instruction does not erase any of it.

Three studies help explain why same-chat role-playing is unreliable. Anthropic researchers found sycophancy across five leading assistants: responses matching a user’s stated views were more likely to be preferred, sometimes at the expense of correctness. A 2025 study found that models accepted a counterargument more readily when it appeared as a user’s follow-up than when the same conflicting positions were presented together for neutral evaluation. In a 2026 study, changing only a speaker’s label to the first-person “Me” caused models to favor that narrator even though the evidence was unchanged. None proves that every model always sides with its user. Each shows that identity and conversational framing can alter an answer. Sycophancy research ↗ Conversational-framing study ↗ Narrative-perspective study ↗

Separate the work. Give two independent instances the same neutral case packet. Assign one to develop the best supportable argument for Side A and the other to do the same for Side B. Do not tell either which outcome the firm or client prefers. A third instance, with no access to the earlier conversations, can compare both analyses and their cited sources, identify factual and legal dependencies, and explain what additional evidence would change the assessment. A lawyer reviews all three against the record and primary authorities.

Separate contexts still produce errors. A fresh instance can be confidently wrong, and each advocate is intentionally one-sided. The separation keeps the sponsor’s preference out of the opposing brief and gives the lawyer two genuinely different arguments to examine.

The Connecticut Judicial Branch does not use AI to review or decide the filings at issue, according to the memorandum. Judge Spader said he reviewed the printed motion, found its legal premise mistaken, and denied it without relying on the hidden text. The attempted instruction had no effect on the decision.

Nothing in the memorandum supports describing the ruling as AI-corrupted. The filing remains significant because it records a deliberate attempt to influence software through concealed text.

A failed payload still tells a security team how someone tried to enter the system. This filing packaged advocacy for a human reader together with commands for a machine reader, and the court used its existing candor rules to sanction the attempt.

Prompt injection is language placed in or around content to make an AI system follow the attacker’s instructions instead of the system owner’s instructions. An indirect prompt injection arrives through material the user asks the AI to process—a webpage, email, image, or document—rather than through the user’s own prompt.

Law firms are most likely to encounter indirect injection. The instruction may be visible, disguised as ordinary content, hidden with formatting, or recoverable only by the parser that prepares a file for the model. The person uploading the document may never know it is present.

Exposure extends beyond a chatbot summarizing a motion. Systems now extract deadlines, classify privilege, build chronologies, propose discovery responses, populate case databases, compare deal terms, draft reports, recommend authorities, and trigger other tools. A workflow that can retrieve data, write records, send messages, or initiate tasks gives a hostile instruction more room to cause harm.

Retrieval-augmented generation does not solve the problem. A controlled collection can improve grounding while still delivering a hostile instruction found inside one of its documents. Conversion to plain text, image flattening, and hidden-text detection each catch some techniques and miss others. Transformation can also degrade evidence that counsel needs to preserve.

Inventory ingestion routes. Find every place where external files enter AI-assisted review: email intake, e-discovery, document management, knowledge systems, litigation analytics, due-diligence platforms, client portals, and individual desktop tools. Include AI features added to existing products, especially those enabled by default.

Separate source text from instructions. Label retrieved document text as untrusted data and keep it distinct from the workflow’s governing instructions. No prompt wording provides a complete defense.

Inspect both representations. For higher-risk workflows, compare the rendered file with the text and objects extracted for model use. Flag white-on-white text, unusually small type, off-page elements, hidden layers, comments, suspicious links, and large differences between visible and extractable content. Preserve the original before transforming it.

Restrict permissions. A system summarizing an isolated copy of a document presents less risk than an agent that can search the entire matter, update a docket, send email, or disclose data. Isolate matters, restrict tool access, and require human approval before consequential actions.

Record provenance and transformations. Keep the original file, its source and hash, the extraction or optical-character-recognition process, the model and workflow used, and the reviewer’s approval. The record should make it possible to trace a suspicious output to the source document, preprocessing, retrieval, model, or person responsible.

Test with adversarial documents. Vendor diligence often stops at whether customer data trains a public model. It should also cover indirect prompt injection, hidden content, cross-matter access, tool permissions, logging, and incident response. Test the configured workflow with benign red-team files designed to challenge those controls.

Require lawyer review. Detection tools produce false positives and miss real attacks. A lawyer must read the source, verify the analysis, and decide whether the result is supportable.

Many law-firm AI policies cover confidentiality, approved tools, hallucinations, citation checking, and supervision. Few say what to do with documents received from an opponent, client, witness, vendor, regulator, or public docket.

The policy should treat external content as untrusted, including routine litigation and transaction documents. It should identify when files may enter AI systems, which scans and transformations occur first, what permissions the workflow receives, when a person must approve an action, and who handles suspected manipulation.

AI document analysis remains useful. File ingestion still needs security controls. The court in Elliott permitted responsible, independently verified AI use while sanctioning concealed manipulation.

Judge Spader noticed the white space, inspected the document, and decided the motion without AI. A firm processing thousands of incoming files through automated systems is less likely to catch the problem by sight.

A lawyer needs to know what the page displays, what the parser extracted, and what authority the AI system received. Treating outside files as untrusted input should become part of document intake, before a strange output or unauthorized action reveals the gap.

For the broader 2026 rules on privilege, filing verification, sanctions, and pro se use, see our companion briefing, What your client tells AI may not be privileged.