Breaking the Black Box: A Tabletop Guide to AI Governance and Maturity

Originally delivered at BSides San Diego 2026 by Chris Ward and Dr. Josh Harguess of Fire Mountain Labs, this session walks security and engineering leaders through the AI governance frameworks that actually matter — NIST AI RMF, the EU AI Act, ISO 42001, GDPR, and MITRE's AI Maturity Model

Prefer a PDF? Download the deck

Workshop Agenda

• Intro (10 min)

• Part 1: AI Governance Fundamentals (20 min)

• Part 2: Tabletop Exercise (30 minutes)

• Part 3: Wrap-Up & Q&A (5 minutes)

AI Governance Fundamentals

What is AI Governance?

“Bureaucracy and approval gates”

“Something only regulated industries need”

“A 50-page policy no one reads”

Goverwhat?

• Most organizations lack basic AI oversight

• No inventory, no review process, no accountability

• Failures aren’t edge cases - they’re systemic and recurring

• No governance = no way to control exposure, liability, or impact

Common Organizational Pitfalls

What is AI Governance?

A way to reduce inherent risks of operationalizing AI

Core AI Governance Components

• Develop comprehensive AI policies

• Assess and monitor AI risks

• Ensure accountability and documentation.

• Align with regulatory frameworks.

AI Lifecycle – CRISP-ML(Q)

Governance ≠ MLOps

Governance:

• Determines the what and why for responsible AI.

• Ensures ethical, legal, and compliant AI use.

• Directs MLOps to build trustworthy AI systems.

MLOps:

• MLOps focuses on how to deploy AI efficiently.

• Streamlines AI lifecycle for speed and reliability.

Governance ensures that MLOps is doing the right things, not just efficiently. They're complimentary

Frameworks Overview

• NIST AI Risk Management Framework (RMF)

• EU AI Act

• GDPR

• ISO/IEC 42001 (AI Management Systems)

• US AI Governance Landscape

NIST AI RMF Overview

• Purpose: Voluntary, risk-based framework for trustworthy AI

• Key Functions: GOVERN, MAP, MEASURE, MANAGE AI risks

• Focuses on trustworthiness characteristics for reliable AI

• Addresses unique AI risks, like data and complexity issues

EU AI Act: Key Details

• World's first binding legal framework for AI

• Focuses on AI and data protection, complementing GDPR

• Categorizes AI systems into four risk tiers

• High-risk AI requires strict rules and assessments

• Mandates robust risk management and data governance

GDPR & AI: Key Principles

• Applies to AI processing EU residents' data.

• Reinforces strong data protection principles.

• Mandates lawfulness, fairness, and transparency.

• Requires data minimization and accuracy.

• Emphasizes accountability and human oversight.

ISO 42001: AI Governance Standard

• First international AI management system standard

• Ensures ethical and transparent AI development

• Covers entire AI system lifecycle

• Enhances credibility and mitigates AI risks

• Builds trust and provides competitive advantage

2026 US AI Governance Landscape

• Federal actions formalize NIST AI RMF for trustworthy AI.

• States rapidly adopt AI, establishing governance structures.

• State initiatives focus on impact assessments and responsible AI use.

AI Governance Ownership

• Senior leadership sets AI governance direction and accountability.

• Cross-functional teams ensure enterprise-wide AI policy integration.

• Providers and deployers have distinct compliance responsibilities.

• Organizations are accountable for third-party AI risks.

AI Approval Workflows: Key Steps

• Executive leadership drives AI governance & accountability.

• Cross-functional teams identify & assess AI risks early.

• Rigorous testing verifies technical quality.

• Human oversight & secure design principles are crucial.

• Continuous monitoring ensures ongoing improvement & compliance.

AI Governance: Key Takeaways

• AI governance requires enterprise-wide accountability.

• Adopt a proactive, risk-based AI approach.

• Leverage frameworks for continuous AI adaptation.

Checklist: Is Your AI Governable?

• Develop a comprehensive model inventory

• Include Model Cards, Data Cards, and AI-BOMs

• Conduct thorough risk analysis for AI systems

• Gather forensic evidence for incidents

• Implement robust logging and response mechanisms

Data Provenance

• AI data provenance documents the data lifecycle.

• Key aspects are origin, processing, and training.

• It ensures transparency, accountability, and compliance.

• Provenance manages risks like bias and privacy.

• It is critical for trustworthiness and human oversight.

AI Bill of Materials

• Do any of our models come from near-peer adversaries?

• Is any of our AI built on legacy models that are no longer supported?

• Do we have the legal right to use these models and datasets in our industry?

• Does any of our AI ship with software vulnerabilities?

• Where’s the complete inventory of all our models and datasets?

Governance Board Setup

• Diverse, cross-functional experts should form the internal AI Board.

• Establish a review cadence for AI policies.

• The board needs authority to make and overturn decisions.

• Senior leadership drives accountability for AI systems.

Cultural Enablement

• Governance does not equal bureaucracy.

• Effective communication fosters good AI governance.

Cultural Enablement

• Governance does not equal bureaucracy.

• Effective communication fosters good AI governance.

Culture eats AI governance for breakfast. Make it easy to do the right thing.

Need help running this workshop, or putting these ideas into action?

We build AI assurance programs for enterprise and government - model inventories, AI bills of materials, governance boards, and the controls that turn the frameworks above into something operational.

Fire Mountain Labs is a Service-Disabled Veteran Owned consultancy specializing in AI security, assurance, and operational risk management. We help organizations deploy, test, and defend AI systems by providing frameworks, training, and hands-on expertise for mission assurance in high-stakes environments.

At Fire Mountain Labs, we help business leaders transform AI uncertainty into safe and impactful outcomes.

Explore our services to lean more.

Page updated

Report abuse