Model Risk in the Age of Generative AI

Model risk management is one of the quiet success stories of post-crisis regulation: a discipline, codified in guidance like the Federal Reserve's SR 11-7, that taught banks to inventory, validate, and monitor the models they rely on. Then generative AI arrived, and a framework built for credit-scoring equations met systems that write prose, hold conversations, and behave differently on Tuesday than they did on Monday. The temptation is to declare the old discipline obsolete. It is not. But three of its core assumptions no longer hold, and knowing which three is the difference between governing generative AI and merely filing paperwork about it.

What model risk management is, and why it worked

Model risk is the risk of loss from decisions based on models that are wrong, misused, or misunderstood. The classical framework manages it through five controls: a complete model inventory; risk tiering by materiality; independent validation before deployment; documentation of design, assumptions, and limitations; and ongoing monitoring against defined thresholds, all overseen by governance with teeth. The framework worked because its assumptions matched its subjects. Traditional models were deterministic, built in-house on known data, used by trained specialists, and changed only through controlled releases. Every one of those assumptions is now negotiable.

What generative AI breaks

Determinism. A credit model gives the same answer to the same input every time; a large language model may not. Point-in-time validation, test it once, approve it, loses its meaning when behavior is probabilistic. Assessment has to move from verifying outputs to characterizing behavior: error rates, failure modes, and variance across thousands of representative and adversarial prompts.

Transparency and ownership. Most organizations consume foundation models through vendor APIs. You cannot inspect the training data, the weights, or the fine-tuning of a model you rent, and the vendor can update it under you. Validation therefore shifts to the parts you do control, prompts, retrieval pipelines, guardrails, and output filters, plus contractual and monitoring controls on the parts you do not, including version pinning and change notification.

The user population. Classical model risk assumed a handful of expert users. Generative AI puts a model in front of everyone in the company, most of whom have never heard of model risk. Misuse risk moves from the modeling team to the entire workforce: confidential data pasted into public tools, fabricated citations in client documents, unauthorized use cases invented one prompt at a time.

Add two genuinely new exposure classes and the risk surface looks different enough that ai model risk management has become its own conversation. Hallucination, fluent output that is factually wrong, is not a bug to be patched but a property of how these systems generate text, which means controls must assume it will occur and catch it downstream: grounding responses in verified sources, requiring citations that can be checked, and routing consequential outputs through review. Prompt injection, where adversarial content in an email, a document, or a webpage hijacks the model's instructions, has no complete technical fix today; the mitigations are architectural, limiting what the system can access and do, treating all external content as untrusted, and monitoring for anomalous behavior rather than assuming the perimeter holds.

What stays the same

The spine of the discipline holds, and organizations with mature model risk functions are adapting faster than those starting fresh, because the instincts transfer.

Inventory still comes first. You cannot govern what you have not counted, and generative AI is proliferating through vendor features and individual experimentation far faster than models ever did. The inventory must now include use cases and embedded AI, not just models you built.

Tiering still allocates effort. A brainstorming assistant and a system drafting regulated customer communications do not deserve the same scrutiny. Materiality, what decisions the output touches, what data it sees, who relies on it, remains the right sorting principle.

Independent review, documentation, and monitoring still apply. Someone other than the builder examines the system before it matters; assumptions and limitations get written down; production behavior gets watched against thresholds. The techniques change, the logic does not. The quality of what goes into these systems matters as much as ever, which is why trusted data remains the foundation under any AI program.

Adapting the framework: a practical playbook

1. Inventory generative AI by use case. Catalog every use, sanctioned or discovered, with its purpose, model, data exposure, output destination, and owner. Expect the discovered list to be longer than the sanctioned one.

2. Tier by materiality. Reserve heavyweight governance for use cases that touch customers, regulators, financial decisions, or sensitive data. A three-tier scheme, minimal, standard, enhanced, is enough for most organizations.

3. Build evaluation suites instead of one-time tests. For each material use case, maintain a battery of representative prompts, edge cases, and adversarial attempts, with acceptance thresholds. Rerun it on every model version change, yours or the vendor's.

4. Engineer the control layer you own. System prompts, retrieval grounding, output filters, data-loss prevention, and human review checkpoints are your controllable surface around a rented model. Validate the assembly, not just the engine.

5. Monitor outputs continuously. Sample production interactions, score them for accuracy, leakage, and drift, and wire findings into an incident process with named owners and escalation thresholds.

6. Govern the people, not just the systems. Usage policy, training, and clear escalation for "the tool did something strange" close the gap that no technical control covers.

What tiering looks like in practice

Because tiering is where most programs either become proportionate or collapse into paperwork, it is worth making the three tiers concrete. Tier three, minimal oversight, covers internal productivity uses whose output a knowledgeable person reviews before it matters: drafting, summarization, brainstorming, code suggestions with review. Controls amount to the usage policy, data-loss prevention, and inclusion in the inventory. Tier two, standard oversight, covers systems whose output feeds real processes with a human in the loop: customer-response drafting, document triage, analysis that informs decisions. These get documented reviews, evaluation suites, output sampling, and a named owner. Tier one, enhanced oversight, covers anything customer-facing, regulator-facing, or decision-making with limited human review: automated communications, eligibility or pricing influence, anything touching protected classes or sensitive data. These get independent validation, adversarial testing, continuous monitoring with thresholds, documented fallback procedures, and standing reporting to risk governance.

Two placement rules do most of the work. Tier by the consequence of the worst plausible output, not the average one. And re-tier when usage changes: the internal drafting tool that quietly starts sending unreviewed customer emails has changed tiers, whether or not anyone updated the inventory.

Questions to put to your own team

For executives who want to test their organization's real posture rather than its stated one, five questions are usually sufficient. How many generative AI use cases are in production or daily use, and when was the count last verified? Which of them could plausibly reach a customer, a regulator, or a financial decision, and who validated those? What happens, concretely, when a model vendor ships a new version, who is notified, what gets retested? Where do staff report strange or concerning outputs, and how many such reports arrived last quarter? And if an examiner asked for our ai model risk management documentation tomorrow, how many days would we need? Confident, specific answers to those five are what maturity looks like; anything else locates the gap precisely.

The regulatory direction of travel

Supervisors have been clear that existing model risk expectations extend to AI: the discipline is technology-neutral, and "the vendor built it" transfers no accountability. The NIST AI Risk Management Framework has emerged as the common vocabulary for AI-specific risk, and the EU AI Act adds binding obligations, inventory, documentation, human oversight, monitoring, for higher-risk uses, phasing in through 2026 and beyond. The convergence across all of them is the same short list: know what you run, assess it in proportion to its stakes, control it, watch it, and be able to prove all four. Boards are being asked these questions directly; our board readiness checklist covers that layer of the conversation.

Where to start

If your organization has a model risk function, extend it: add generative use cases to the inventory this quarter, tier them, and stand up evaluation suites for the material few. If it does not, begin with the inventory and one rule, no generative AI in customer-facing or regulated processes without documented review. Either way, resist the two failure modes we see most: governing nothing because the technology feels ungovernable, and governing everything so heavily that the business routes around you. Model risk management earned its keep by being proportionate. Generative AI does not change that; it just raises the price of getting the proportions wrong.

A realistic first quarter looks like this. Month one: build the use-case inventory through a combination of declared registration and discovery, and publish the usage policy. Month two: tier the inventory, write one-page risk profiles for everything in the top tier, and define acceptance thresholds for the two or three most material use cases. Month three: stand up evaluation suites and output sampling for those material cases, run the first adversarial tests, and take the results, including the uncomfortable ones, to risk governance. That cadence produces a defensible posture in ninety days without a single new hire, and it surfaces the real risk decisions, which use cases to constrain, which to invest in, which to shut off, that were being made implicitly anyway.

Frequently asked questions

Does SR 11-7 apply to generative AI?

In substance, yes. Supervisory guidance on model risk is technology-neutral: if a system's output informs business decisions, inventory, validation, documentation, and monitoring expectations apply, adapted to probabilistic behavior.

How do you validate a large language model?

You characterize behavior rather than verify answers: evaluation suites of representative and adversarial prompts, acceptance thresholds, testing of the surrounding controls (prompts, retrieval, filters), and revalidation whenever the model or its configuration changes.

What is the biggest generative AI risk for most firms?

Unmanaged use: staff pasting sensitive data into public tools and relying on unverified output in real work. An inventory, a usage policy, and training address more actual exposure than any single technical control.

Who should own AI model risk?

An independent risk function should set standards and challenge, while use-case owners in the business carry first-line accountability. In smaller organizations, a fractional risk or data executive can anchor the discipline without a full-time hire.