Accelerating the delivery of Trusted Data by adopting DataOps

Share this post:

Authors: Elaine Hanley, DataOps Centre of Excellence Worldwide Lead at IBM & Julie Lockner, IBM Data and AI Portfolio Operations, Customer Experience and Offering Management

At the Gartner Data and Analytics Summit in Sydney in February, IBM hosted a design thinking workshop on the emerging topic of DataOps. During the session, we met with representatives from governance, data engineering, data science and business analysts from several organisations. We talked about how every organisation is trying to maximise data to harness its knowledge of current opportunities, past history and predictive indicators. They all want to make the right operational decisions for their business, whether it’s balancing risk in insurance, driving revenue in banking, managing field resources in utilities, or evolving an education that addresses future skill and knowledge gaps in universities. The common factor for all of our participants was the challenge of finding the right data, knowing how and when to trust what it reveals about their use case, and using it effectively to drive results.

DataOps provides a framework to drive acceleration and confidence in data

Our workshop began with a brief introduction on what DataOps is. Gartner defines DataOps as “a collaborative data management practice focused on improving the communication, integration and automation of data flows between data managers and data consumers across an organisation.” IBM’s view is that DataOps helps organisations deliver business-ready data quickly because users know what data they have and can trust its quality and meaning, so they can better use it.

IBM’s DataOps methodology starts by assessing where the organisation currently is now in its maturity level, then guides it to iteratively improve and automate its delivery of data. Our model breaks maturity down into three main areas of capabilities:

1. Data inventory (knowing your data)
2. Data governance (trusting your data)
3. Data pipelines (using your data)

Using this framework, each iteration focuses on the business value the DataOps sprint delivers to the data consumer.

Detailing an organisation’s DataOps requirements

For example, a bank may want to understand how to dynamically recognise a fraudulent credit card transaction. Its team of data scientists can use machine learning techniques to recognise fraudulent patterns based on previous detections, and the data elements it might need include:

• a data inventory, answering where that data can come from, with detailed profiling to understand what it means.
• more information about that data, such as a quality score to guide the user on its reliability, what currency is used to specify the amount, what is the valid set of spending categories to be used, how the technical metadata can be transformed into business-friendly language. These are all considerations governance adds to provide trust in the data.
• an understanding of how to combine multiple sources into one coordinated view and serve that up to the data scientists, agreeing how to convert multiple currency amounts into a common base currency, and removing data noise and duplicates.

Using design thinking to determine maturity

At the workshop, we adopted an IBM Design Thinking approach and asked each participant to share where they are on the maturity curve. We had quite a range, from teams who were completely new to governance to those with advanced capabilities.

What was most interesting was the spread of skills among participants. Those with fewer capabilities learned a lot from conversations with the more advanced teams. To facilitate the engagement further, we asked each participant to offer what challenges they had in each area, what achievements they made, and what questions they still had. In future blogs, we’ll address these questions, and demonstrate how IBM DataOps provides rigour and repeatability that drives fast delivery of business value from an organisation’s data team.

Read more about IBM DataOps

Feel free to register your interest in IBM DataOps and to request an IBM DataOps Design Thinking Virtual Workshop for your organisation, by emailing the IBM DataOps team.