ai-governance-model-risk

Reviews a foundation-model vendor's published evidence pack (system card, model card, evaluation reports, red-team summaries, security and privacy attestations, responsible-use policies, trust-and-safety pages) against firm criteria for the firm's deployment context. Produces a sufficiency view, named gaps with supplemental evidence requested, residual reliance with caveats, recommended owner actions, and re-review triggers. The artefact a model risk lead, AI risk committee, or vendor-diligence officer uses to decide whether to depend on a foundation model in scope. The model-evidence layer that pairs with vendor-diligence in third-party-operational-resilience for the entity-level wrapper. Best for: - A new foundation-model vendor is being onboarded for one or more in-scope use cases and the model-evidence layer is needed for the deployment-context decision. - A foundation-model provider has published a new system card, model variant, or version and the firm needs the delta review. - A periodic re-attestation cycle requires updated evidence-pack review for in-flight vendor models. - An upstream vendor-diligence record has set eval_evidence_status to deferred-to-evidence-review and chained the model-evidence question over. Not the right tool when: - The work is entity-level vendor diligence (financial health, security attestations as entity-level evidence, contract terms, exit posture, sub-contractor footprint). Use vendor-diligence in third-party-operational-resilience. - The vendor model is in-house or open-source with no published vendor evidence pack (the deep work is internal validation under validation-plan). - The work is the deployment-level pre-prod gate; this skill feeds genai-pre-prod-review. - The work is the focused prompt-injection deep-dive on the deployed system (use prompt-injection-risk).

Drafts a model card for a financial-services AI or model use case, with named sections for intended use, training and reference data, performance, limitations and known failure modes, monitoring plan, controls, change management, and sign-off questions. The card is the firm-side governance artifact that supports model risk committee review, pre-prod gates, the model inventory of record, validator handoff, and regulator response files. Best for: - A first-line owner has proposed an AI use case and second-line needs the model card before a tier decision or pre-prod gate. - A model risk team is refreshing model cards as part of an annual model inventory exercise. - A new vendor model is replacing an existing one and the card needs to be updated to reflect the swap. - A regulator response file or examiner request requires the firm's documented view of an in-scope model. Not the right tool when: - The use case has not been intaked yet (use ai-use-case-intake first). - Validation testing has not run and there are no performance results to summarise (use validation-plan; the card consumes its outputs). - The artifact required is the EU AI Act Annex IV technical documentation. That is the provider-side file; the firm-side card sits alongside it. Use ai-act-triage to scope Annex IV deltas.

Reviews the prompt-injection threat surface for a deployed or near-deployed GenAI use case in a regulated financial-services firm. Catalogues the carriers (system prompt, user input, retrieved content, tool output, agent memory, multi-agent message, multimodal input), the trust posture on each, the tested attack classes, the mitigations in place with evidence, the residual risk with likelihood and impact framing, the production monitoring and detection signals, the incident-response classes with regulator-notification triggers, and the recommended owner actions. Output is a second-line-grade memo a CISO function, AI Governance Lead, MRMO, or AI risk committee can act on. Best for: - A GenAI assistant or agent is approaching pre-prod and second-line needs an explicit prompt-injection review before the gate. - An incident or near-miss in a deployed GenAI system has surfaced a prompt-injection vector and the committee needs a refreshed residual-risk view and notification-trigger evaluation. - A pre-exam or pre-cyber-audit motion needs the firm-wide prompt-injection posture documented for a defined population of GenAI use cases. - A foundation-model swap or a tool-inventory change has triggered the re-validation flag in change management. Not the right tool when: - The system has no LLM, no instructions in natural language, and no retrieved or third-party-supplied text in its prompt path; use validation-plan and the standard model-risk skills. - The system uses a foundation model only for embedding generation or classification with no instruction-following surface. - The work is broader red-teaming across hallucination, bias, and abuse; use validation-plan with the GenAI testing block. This skill is the prompt-injection-only deep-dive. - The work is the firm-side governance card for the use case; use model-card-builder. This skill consumes the card and produces the focused memo.

Reviews the retrieval and grounding evaluation for a RAG-based GenAI use case in a regulated financial-services firm. Confirms the corpus is fit for the intended purpose, retrieval quality is measured with named methods on a labelled set, grounding (faithfulness, citation precision, refusal on out-of-scope queries) is tested with documented metric semantics, failure modes are catalogued, mitigations are evidenced, and ongoing monitoring runs against a real ground-truth pipeline. Output is a second-line memo on whether the RAG implementation can be relied on for the use case's intended purpose, with named gaps, residual-risk framing, and owner actions. Best for: - A first-line owner has built a RAG system and second-line needs an evaluation review before pre-prod, expansion, or annual revalidation. - A foundation-model swap or a retrieval-stack change has happened and the grounding evaluation needs to be re-confirmed. - An incident on hallucination, off-corpus answer, or cross-scope leakage has surfaced and the committee needs an updated grounding posture. - An exam-readiness motion needs the firm's RAG-evaluation posture documented for a defined population of GenAI use cases. Not the right tool when: - The system has no retrieval; use validation-plan with the GenAI testing block. - The work is the prompt-injection threat-surface review; use prompt-injection-risk. The two skills sit side by side for a RAG agent and reference each other. - The work is the corpus access-control review only; use the cyber and privacy overlays inside vendor-diligence or genai-pre-prod-review. - The work is the firm-side governance card for the use case; use model-card-builder. This skill consumes the card and produces the focused memo.

Drafts the validator-side scope contract for an AI or model use case before testing starts. Sizes the work to tier, names the conceptual soundness, data review, outcomes analysis, robustness, fairness, and ongoing monitoring scope per pillar, and frames the effective challenge questions the model owner is expected to answer. Output is what the validator and the model owner agree to before validation executes. Best for: - A use case has cleared intake and tiering and validation is the next step before pre-prod or production approval. - An annual revalidation cycle needs a tailored plan rather than a copy-paste of last year's scope. - A vendor or foundation-model swap on an existing use case needs a delta-scoped revalidation plan. - A regulator request lands on a tier-1 or tier-2 model and the firm needs a documented validator scope to point to. Not the right tool when: - Validation has already executed and the work is the validation report write-up. - The use case has not been intaked or tiered (use ai-use-case-intake or ai-risk-tiering first). - The artifact required is the firm-side model card itself (use model-card-builder; this plan consumes the card and scopes the testing against it). - The artifact required is the full GenAI pre-prod gate review across people, process, and technology (use genai-pre-prod-review; this plan is the validator-side scope only).