Veröffentliche eine Notiz:

AI governance has received a lot more attention as AI regulations are being formulated and passed. But AI Governance is not only about regulation, it is the key discipline to master the complexity induced by the variety of AI frameworks, models and tools.

AI Governance relies on proper Data Governance which has been discussed in some detail in previous blogs articles like Approaches and Considerations becoming a Data-Driven Enterprise and will not be reiterated here.

Motivation

The motivating factors to adopt AI governance can be grouped into three areas.

Regulatory Compliance

is an external motivation and is dependent on where you are operating or making business (e.g. the EU) and the use case you want to implement. E.g., the EU AI act defines various levels of (AI induced) risk that are dependent on the use-case. An example is to use AI with an impact on the availability of critical infrastructure or workforce management. Both are examples which the EU defines as high risk. This does not mean you must not use AI (this would be unacceptable) but it requires you to implement an AI risk management system and adequate Data Governance. If they don´t businesses may be subject to lawsuits and receive fines and judgements for operationalizing bad models.

Reputational Risks

may lead to reputational losses as company perception wanes which could affect various stakeholders. Existing customers brand loyalty may decline or, in general brand popularity may decrease. Analysts may lower their ratings, and finally, stock owners may divest their share, all negatively impacting stock performance.

Operational Risks

are motivated by the organization itself. Primary losses include expenses responding to an incorrect model, revenue losses as customers churn, and replacement costs scrapping your models for new ones. Additional secondary costs will include losing competitive advantage.

Operational risks are directly related to running AI models in a technical manner (e.g., availability) or business wise (quality, accuracy) and have an impact on operational excellence. Operational excellence in AI has a direct impact on the ROI of Machine Learning applications: A system that makes more accurate decisions will gain acceptance from customers (e.g., running in the backbone of a chatbot) or has a direct impact e.g., applied to optimization tasks.

Operational Excellence is a strong motivational factor to implement an AI Governance solution. Similar to implementing a software development lifecycle, AI models need a lifecycle from use case definition to decommissioning. Higher accurate predictions make a model more valuable. Taking a step aside the decision to use ML or AI is driven by the idea to scale a business task that requires intelligent action respective automation. AI Governance can help to track the quality and to quantify (measure) the impact of AI use cases. Better decisions have a direct impact on revenue or savings. On one hand, ML use cases that appeal in the ideation phase may turn out to have inadequate risk or do not deliver enough value – AI Governance will help to collect reliable figures to stop those uses cases and to focus on those with the highest expected value. On the other hand, if a model is considered to have high risk this does not imply it cannot be used. If the expected business value is high enough, mitigation tasks can be established to compensate for the risks.

AI Governance in ML-Operations

In which phase of the AI development lifecycle is AI Governance relevant? In a nutshell, in all. Let’s review the established standard process (CRISP-DM) to check. The reference to CRISP-DM may sound outdated however the phases are still relevant and describe a typical AI Systems lifecycle.

Problem Specification: In case of the EU AI Act the risk level is dependent on the use case. The use case is what is on the core of the problem specification. To ensure proper governance, a structured questionnaire can aid in defining the required procedures and risk metrics to track. It helps define guardrails that keep the model trustworthy and explainable.

Data Understanding: Here, the required datasets are collected and reviewed. This is the point where, e.g. columns that could induce bias can be detected. In general data quality issues can be detected and mitigated accordingly.

Data preparation: Here, two aspects are important: data curation (i.e. remove attributes that introduce bias) and proper lineage. Transparency is a central requirement for AI Governance. Applied to data preparation, it implies that all transformation steps are traceable.

Modelling: Modelling starts with experimenting which model (i.e. type) performs best. Quality figures need to be captured. Governance guardrails defined and refined in previous steps must be respected.

Evaluation: Models will be tested against production or production equivalent data. Besides checking if the required parameters are met, proper testing needs to be documented.

Deploying and Monitoring: From a business process perspective it is crucial to ensure only properly governed (i.e., models that passed evaluation) are deployed. Once models are deployed, they need to be monitored since they can behave differently than during development. Lastly, data drift can occur – this means the characteristic of the data will change over time – and retraining the models may be required.

AI Governance Approach

AI Governance relies on three pillars.

Lifecycle Governance focuses on the documentation and monitoring of AI models facts during its lifecycle.

AI Risk Management puts AI models in context of the business task to be automated. An AI based approach can be perfectly good, e.g., applied to a product recommendation use case and yet be considered high-risk if applied, e.g., to credit risk prediction.

Regulatory Compliance takes the relevant regulations into scope. The EU AI Act is a prominent example, where depending on the use case, companies must take care e.g., not to make biased decisions.

AI Risk Dimensions

AI governance can be complex. Breaking down required tasks, metrics or figures into dimensions, can make governance easier to manage. Also, the dimensions define an order criterium by which our metrics and figures can be aggregated.

People and Culture

AI Governance is a team sport with different roles involved, each with specific requirements.

• Data Scientists or AI Engineers: They need a flexible framework that allows them to stick with their preferred tooling and easily use new (model) libraries. Governance is often perceived as an afterthought. But realizing they can be held responsible e.g., when legal issues arise, an AI Governance framework could shine in a new light.
• Model Operations: Operation Engineers need to have a consistent overview of all models deployed. Before productive deployment, they need to verify if a model is eligible (e.g., has passed verification, is connected to monitoring frameworks, etc.). They are responsible to monitor the deployed models and act accordingly e.g., when model drift occurs.
• Product Owner: They describe the use case and performs a high-level risk assessment that defines metrics to be captured and the ML-Ops workflow.
• Board Executives: They need a consolidated view of the productive models. For example, a proper Key Performance Indicator (KPI) could define the overall risk exposure. The KPI should be traceable along the AI Risk dimensions (like introduced above) and organizational unit.
• All: Team sports work best with collaboration and without media disruption. This works best with an integrated platform that incorporates model development, model monitoring, workflow- and risk management.

AI governance is about ensuring greater transparency across the AI lifecycle and the model itself. IBM recently announced watsonx.governance, a next generation enterprise toolkit which is designed to automate and accelerate workloads across the AI lifecycle while providing risk management and facilitating regulatory compliance.

IBM Solutions

watsonx.governance

watsonx.governance accelerates responsible, transparent and explainable AI workflows. The more AI is embedded into daily workflows, the more you need proactive governance to drive responsible, ethical decisions across the business. watsonx.governance allows you to direct, manage, and monitor your organization’s AI activities. It employs software automation to strengthen your ability to mitigate risk, manage regulatory requirements and address ethical concerns without the excessive costs of switching your data science platform—even for models developed using third-party tools. The AI governance capabilities will include governance of classical data science (i.e., task specific self-trained models) and “Foundation Models” ([6]: generic, pre-trained models which became very popular recently e.g. applied to “Generative AI” tasks). General availability of watsonx.governance is planned for December 2023 (4).

Govern models through complete AI workflow considering policies and regulations

The next generation governance-toolkit provides a range of capabilities to identify, manage, monitor, and report on risk and compliance. It accelerates the creation of models at scale, from use case idea (model candidates) to production deployment, by incorporating approvals in the workflow-based approach. Full transparency of any type of model (e.g., task specific data science artefacts or foundation models) is ensured and made visible in customisable risk monitoring dashboards. Additionally in Open Pages corporate policies and regulations can be assigned to models, e.g., annual bias review (required for EU AI ACT) to ensure that models are fair, transparent, and compliant [4].

Automated collection and documentation of model metadata at all stages, from model idea to production

Model and process metadata is captured in a central meta store. Having all model facts in central place is important both to increase the productivity of the MLOps process (model facts are immediately visible to all parties involved in the lifecycle of an AI model) and to comply with regulatory requirements. Data scientists benefit from assistance and automation of the documentation process. Transparency of model metadata supports audits and brings more clarity to stakeholder or customer requests. Metadata captured in AI factsheets includes model details, training information, metrics, input and output schemas, or details about the models used, such as quality metrics, fairness or drift details.

Monitor, explain, and benchmark your model

Model monitoring is an ongoing task to track models and to drive transparency. This includes the monitoring of the general model performance (e.g., accuracy) and more specifically monitoring of fairness or model and data consistency over time (i.e. drift). Open Pages supports threshold definitions for model performance metrics and combines those with automated detection of threshold violations to trigger model retraining. It implements explainability by supporting explanations how the model arrived at certain predictions. Model benchmarking is supported – it is common practice to compare and benchmark a challenger model with a model in production to ensure that the best model is the one in production.

AI Governance applied to foundation models and generative AI

Foundation models resp. large language models introduced new complexity to AI Governance: They are pre-trained and are customized to specific use cases via either prompting or (fine-) tuning [7]. A risk that arises from pre-trained models is that the data that was used to build the model may not have been properly cleansed. On these base the generative AI may produce hateful or defamatory output. To address this, IBM integrated a “HAP detector” to detect and root out hateful, abusive or profane content (hence “HAP”). This can be used for filtering LLM output but also prevent harmful prompts issued from users.

Other filters are focused around the detection of sensitive (PII) data.

As foundation models are customized e.g. with prompt tuning those prompts need to be governed as well because this influences the generated output. Those prompt (-templates) will be versioned and captured in a similar way as parameters from classical data science.

Resources

1. Quantitative AI Risk Assessments: Opportunities and Challenges https://arxiv.org/pdf/2209.06317.pdf
2. Varshney, Kush R.: Trustworthy Machine Learning https://www.trustworthymachinelearning.com/
3. Fraunhofer: Leitfaden zur Gestaltung vertrauenswürdiger Künstlicher Intelligenz: https://www.iais.fraunhofer.de/content/dam/iais/fb/Kuenstliche_intelligenz/ki-pruefkatalog/202107_KI-Pruefkatalog.pdf
4. IBM watsonx.governance: https://www.ibm.com/products/watsonx-governance
5. Introducing watsonx: The future of AI for business (AI Guardrails): https://www.ibm.com/blog/introducing-watsonx-the-future-of-ai-for-business/
6. https://research.ibm.com/topics/foundation-models
7. https://research.ibm.com/blog/what-is-ai-prompt-tuning