Addressing Data Governance challenges with IBM DataOps

Share this post:

Authors: Elaine Hanley, DataOps Centre of Excellence Worldwide Lead at IBM & Julie Lockner, IBM Data and AI Portfolio Operations, Customer Experience and Offering Management

At the Gartner Data and Analytics Summit in Sydney earlier this year, IBM hosted a design thinking workshop on the emerging topic of DataOps. This is the third blog in a four-part series describing the workshop – read the first entry explaining how we helped participants assess their DataOps maturity, and the second about how they addressed Data Inventory challenges.

IBM provides a path to a DataOps practice with a prescriptive methodology, artificial intelligence (AI)-enabled automation and the IBM DataOps Center of Excellence. Data Governance focuses on ensuring we can provide trust in the data we want to make available for use. It includes asserting ownership of data domains to provide the person who can help address any concerns (data stewardship); quality assessment to measure the reliability of data, including invalid values, missing values and anomalies; and mastering the data for entity resolution and de-duplication.

Let’s look at the case of a bank that wants to understand how to dynamically recognise a fraudulent credit card transaction. Preparing to address Data Governance for this data sprint involves deciding the acceptable quality score thresholds that need to be met before data is acceptable for use, as well as ensuring data stewards have been assigned, providing the Data Governance KPIs used to instrument the sprint.

Our participants discussed the following challenges:

Our attendees’ challenges for Data Governance clearly highlighted a need for agreement and buy-in, with an established, executive-sponsored and consensus-driven data governance program with a supporting organisational structure, Roles and Responsibilities (RACI), and formal processes. A Data Governance program can be perceived as heavy, bureaucratic, and slow – but it doesn’t have to be. When data catalogues are set up properly, they can be the place that formalises many data owners, automates many of the workflows when questions come up on data meaning and its quality, and communicates any changes to data when necessary. Data Governance is required in many highly regulated industries, and can be the competitive differentiation for how enterprises move to data-driven, digital transformations.

When looking at the achievements the attendees have made, themes emerge around setting up governance teams, establishing a strategy and assigning data stewards. 

Data governance success relies on a few key requirements

1. A well-defined goal to form alignment in the organisation. The goal needs to deliver real business value, and so should be tied to a business initiative and communicated to all of the stakeholders in a way that makes it relevant to their day to day activities The goal should be made of intermediate milestones that are reasonable and achievable.
2. Clear roles and responsibilities so teams know what is expected of them to meet the goals.
3. Defined processes to remediate data issues when something goes wrong.
4. Recognition when teams achieve their milestones, like sharing the news to motivate other teams to jump on the bandwagon. Implementing effective KPIs provides a readily available view of progress and achievement.

Some of the questions attendees had (along with our answers) included:

• How do we get teams to care about data quality enough to fix it?

You need to be able to describe the impact the poor data quality has on their output and contribution to the business. At the end of the day, data quality impacts everyone. Make the argument in their language and based on a metric they’re measured on.

• How do you embed data stewards in BAU (Business as Usual)?

Business processes are owned by business leaders, as should be the data that represents the process. Being a data steward doesn’t need to be a full-time job, but it provides accountability. Once people are given responsibility, most live up to the task of ownership.

• How do you make it flexible enough to work for different data?

Processes should not be contingent on the data format. Technology has been designed to solve for that. For those situations where automation is required to keep up with the volumes, IBM’s DataOps offerings support both structured and unstructured data.

• How do we find the right balance for governance for ML/self-service?

Start with a small data analytics team and pilot project. Every organisation’s threshold for self-service and controls are different and based on the company’s tolerance for risk – and a culture based on empowering users with data. Iterate after each project, and the teams and risk managers will strike the right balance.

• Is there a specific issue/tipping point that leads to the creation of data governance committees?

Committees are usually created after a new regulation has been introduced that the company needs to adhere to, and the data pipelines span multiple organisational boundaries. Sometimes failed audits become the tipping point, or a new leader brings in best practices with them.

• Who is responsible? Who pays for data governance initiatives?

Data governance belongs under an operational cost centre – depending on the industry, this could fall under finance or specific lines of business.

• Who has a good framework, so I don’t have to reinvent the wheel?

IBM offers right-sized programs for teams starting from scratch to multi-national, multi-disciplined deployments. There are also several data governance associations that publish frameworks.

• How do you consistently protect data from misuse?

Data privacy regulations challenge organisations to track personal data, recognise it as such during data discovery, and ensure its use respects the policies that need to be adhered to. IBM Watson Catalog automates the process by allowing enforceability of data privacy rules which will automatically mask or deny access to data that can be recognised as sensitive. For example, credit card details can be obfuscated before the data user views them.

Read more about IBM DataOps

Feel free to register your interest in IBM DataOps and to request an IBM DataOps Design Thinking Virtual Workshop for your organisation, by emailing the IBM DataOps team.