FDA is already forming expectations about your AI. Most quality organizations are not ready to answer.

AI governance is not a future problem. It is a present one.

The FDA is already forming expectations about how organizations should govern AI tools used in drug manufacturing and quality workflows. Most quality organizations are not ready to answer the questions the FDA will ask. The agency has published a draft guidance, co-authored a set of international guiding principles, opened a public docket specifically on AI in drug manufacturing, and established an internal council to coordinate its position.

Meanwhile, AI has entered quality organizations through the side door. Deviation triage. Complaint classification. Document drafting. Trend detection. Batch record review support. Vendor platforms that quietly added an AI feature in a release note.

The gap between those two facts is where the regulatory exposure lives.

What FDA has actually said

Precision matters here, because a great deal of what circulates about FDA and AI is inference presented as fact. Here is the verified record as of this writing.

Discussion Paper: Artificial Intelligence in Drug Manufacturing (March 2023). FDA published this through CDER and opened a public docket, FDA-2023-N-0487. The paper set out areas for consideration and policy development identified by CDER scientific and policy experts, and posed a series of questions to industry rather than issuing requirements. FDA followed it with a public workshop on the regulatory framework for AI in pharmaceutical manufacturing in September 2023. This work sits within CDER's Framework for Regulatory Advanced Manufacturing Evaluation, the FRAME initiative.

Considerations for the Use of Artificial Intelligence To Support Regulatory Decision-Making for Drug and Biological Products (January 2025). FDA's first guidance on AI for drug and biological product development. It introduces a risk-based credibility assessment framework for establishing and evaluating the credibility of an AI model for a particular context of use. The comment period closed April 7, 2025.

**This document remains in draft status.** It has not been finalized, and it carries the standard designation that it contains non-binding recommendations and is not for implementation. Anyone telling you FDA requires the credibility framework today is overstating it. What the draft does tell you is what the agency is thinking, which is a different and arguably more useful thing.

Guiding Principles of Good AI Practice in Drug Development (January 2026). CDER and CBER collaborated with the European Medicines Agency on ten guiding principles for industry and product developers using AI in drug and biological product development. The ten principles, as published by FDA, are:

1. Human-centric by design
2. Risk-based approach
3. Adherence to standards
4. Clear context of use
5. Multidisciplinary expertise
6. Data governance and documentation
7. Model design and development practices
8. Risk-based performance assessment
9. Life cycle management
10. Clear, essential information

The CDER AI Council, established in 2024, coordinates CDER's AI activities and is intended to ensure the center speaks with a consistent voice on AI policy and evaluation.

Two figures worth holding onto, both from FDA's own account: CDER received more than 500 submissions containing AI components between 2016 and 2023, and more than 800 external comments on its 2023 discussion paper on AI in drug development. This is not an agency waiting to encounter the issue.

On status. Regulatory positions in this area are moving quickly. The draft guidance may be finalized, revised, or superseded after this article is published. Verify current status directly on FDA's guidance database before relying on any characterization here, including ours.

The three questions you cannot currently answer

Read the ten principles again and notice what they are asking for. Clear context of use. Data governance and documentation. Risk-based performance assessment. Life cycle management. Human-centric by design.

Those resolve into three questions an investigator can ask without any new regulation being issued, because the underlying obligations already exist under 21 CFR Part 11, Part 211, and ICH Q9(R1) and Q10.

What is this tool's intended use? Not what the vendor calls it. What decision does its output inform, and what happens downstream of that output? A model that "helps with deviation triage" either influences a GxP decision or it does not. If it does, it is in scope. If your answer begins with a product name rather than a decision, you do not have an intended use statement.

What is your validation rationale, and where is it written down? Not whether you validated. Why you validated the way you did, at that depth, for that use, and what risk assessment drove that choice.

How is human oversight demonstrated? Not asserted in a policy. Demonstrated in a record that an investigator can examine.

Most quality organizations can gesture at all three. Very few can produce documentation for any of them.

Context of use is the governance unit, not the tool

This is the single most useful concept in FDA's current thinking, and the one most often missed.

Organizations govern AI at the tool level. They ask whether a platform is validated, add it to an inventory, and consider the question closed. FDA's framing is different. Credibility is established for a model within a specific context of use, meaning the particular question the model addresses and how its output will be used.

The practical consequence is significant. The same tool deployed against two different decisions is two governance objects, not one. A language model summarizing internal meeting notes and the same model drafting deviation investigation narratives carry entirely different risk. One touches nothing. The other enters a GxP record and shapes a root cause determination.

A tool inventory that does not record context of use is a procurement list, not a governance artifact. If your AI register has a column for vendor and a column for validation status but no column for what decision the output feeds, it will not survive the first serious question.

Why traditional validation thinking breaks

Quality organizations are fluent in computer system validation. That fluency is useful and it is also the source of the problem, because AI systems violate assumptions that CSV was built on.

Determinism. Traditional validation rests on the premise that identical inputs produce identical outputs. Many AI systems, particularly generative ones, do not behave that way. Test scripts written to confirm exact output matching will either fail or, worse, be quietly rewritten until they pass.

Static behavior. Conventional systems change through change control. Models can change through retraining, through vendor updates you did not initiate, or through drift in the underlying data distribution while the model itself sits untouched. The system that was validated in March may not be the system running in September, and nothing in your change control record will show it.

Verifiable logic. Validation traditionally traces a requirement to a design element to a test. Many AI models cannot be decomposed that way. This is why FDA's framing centers on credibility for a defined purpose rather than a one-time certification of correctness.

None of this means AI cannot be governed. It means the validation rationale has to be built for the technology rather than borrowed from a paradigm that does not fit. That rationale is a written document explaining what the model does, what could go wrong, how likely and how consequential that is, what evidence establishes fitness for the intended use, and what ongoing monitoring detects degradation.

Organizations that skip the rationale and jump to testing produce protocols that look rigorous and prove very little.

Human oversight has to be demonstrable

• "Human in the loop" is the most overused and least evidenced claim in AI governance.
  
  The claim usually means a person reviews the output before it is used. The question an investigator will ask is what that review consisted of, and whether the record shows it happened.
  
  Three failure patterns are worth naming:
  1. Rubber-stamp review. A reviewer approves AI output at a rate and volume that makes meaningful examination impossible. If a system produces forty draft investigations a week and one reviewer approves all of them within a day, the record documents throughput rather than oversight.
  

• 2. Automation bias. Reviewers accept plausible-sounding output more readily than they would accept the same conclusion from a colleague. Fluent text reads as authoritative. This is a documented human factors problem and it does not resolve through training reminders alone.
  

• 3. Undefined competence. Reviewing AI output for a GxP decision requires knowing the failure modes of the tool, not just the subject matter. Most organizations have not defined what qualifies a person to perform that review.
  

Demonstrable oversight means the reviewer, the scope of review, the qualification basis, and the disposition are all in the record, and that the review is designed at a scale a human can actually perform.

The problem underneath all of this

Before any of the above can be addressed, an organization has to know what AI it is running.

In most quality organizations the honest answer is that nobody knows. Tools arrive through vendor updates that add AI features to existing validated systems. Through departmental subscriptions purchased below procurement thresholds. Through individual staff using consumer AI tools on work content without anyone having said they could not.

That last category is the sharpest exposure, because GxP content leaving the organization through an unassessed tool is a data governance failure that exists independent of anything FDA says about AI.

Discovery comes first. Governance frameworks applied to an unknown population govern nothing.

What to do in the next ninety days

• None of this requires waiting for final guidance.
  - Inventory what is actually in use. Survey by workflow rather than by software license, because that surfaces the vendor features and shadow usage a license audit misses.
  - Classify by context of use. For each entry, record the decision the output informs and whether that decision is GxP. Everything else follows from this classification.
  - Write the intended use statement. One paragraph per GxP use, stating what the tool does, what it does not do, and what decision consumes its output.
  - Assess risk before validating. Model influence on the decision, and consequence if the output is wrong. That assessment determines validation depth and produces the rationale you will be asked for.
  - Define the oversight record. For each GxP use, specify who reviews, against what, with what qualification, evidenced how.
  - Establish a change trigger. Define what constitutes a change to an AI system in your environment, including vendor-initiated model updates, and route it through change control.
  - Assign ownership. AI governance without a named owner in Quality drifts to IT, where the regulatory obligation does not sit.
  

Questions for your Quality leadership

1. Can we produce a list of every AI-enabled tool touching a GxP decision, today, from a controlled source?
2. For each one, can we state the context of use in a sentence that names a decision?
3. Where is the written rationale explaining why we validated each tool to the depth we chose?
4. What in our change control system would detect a vendor retraining a model inside a system we already validated?
5. What evidence would we hand an investigator to demonstrate that human oversight is real rather than nominal?
6. What is our position on staff using consumer AI tools with GxP content, and how do we know whether it is being followed?
7. Who in Quality owns AI governance, and what authority do they hold?

If those questions produce discussion rather than documents, that is your finding.

Where Bayou Vantage fits

FDA's expectations are forming in public and in writing. The organizations that will answer well are the ones building intended use documentation, validation rationale, and demonstrable oversight now, while the guidance is still in draft and the cost of getting it right is low.

Bayou Vantage translates emerging regulatory expectation into governance your Quality organization can actually operate and defend.

[Start a conversation](/contact)

Sources

FDA, Considerations for the Use of Artificial Intelligence To Support Regulatory Decision-Making for Drug and Biological Products, Draft Guidance for Industry, January 2025. Draft status as of publication.
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/considerations-use-artificial-intelligence-support-regulatory-decision-making-drug-and-biological

FDA and EMA, Guiding Principles of Good AI Practice in Drug Development, January 2026.
https://www.fda.gov/about-fda/artificial-intelligence-drug-development/guiding-principles-good-ai-practice-drug-development

FDA CDER, Artificial Intelligence for Drug Development.
https://www.fda.gov/about-fda/center-drug-evaluation-and-research-cder/artificial-intelligence-drug-development

FDA CDER, Discussion Paper: Artificial Intelligence in Drug Manufacturing, March 2023, Docket FDA-2023-N-0487.

FDA, Data Integrity and Compliance With Drug CGMP: Questions and Answers, Guidance for Industry, December 2018.
https://www.fda.gov/regulatory-information/search-fda-guidance-documents/data-integrity-and-compliance-drug-cgmp-questions-and-answers-guidance-industry

All regulatory characterizations above were verified against FDA primary sources on the date of writing. The three-question diagnostic, the validation-rationale framework, the oversight failure patterns, the ninety-day sequence, and the Quality leadership questions are Bayou Vantage's own analysis and are not FDA requirements.