“We understand that audit is going to be disrupted,” says Marisa Ferrara Boston, the Automation and AI Lead Architect at KPMG, one of the world’s top providers of auditing services. Marisa is clear-eyed about the seismic shifts that her industry faces, but she is hopeful that with preparation, these changes will bring opportunities along with the challenges.

At KPMG, Marisa has been involved in implementing AI technologies in various operational areas. Her main focus is in document investigation, where specialists review financial and legal documents to determine risk across a portfolio. These documents can be spreadsheets, presentation decks, emails, scanned documents, and more. The sheer volume of data is difficult for workers to analyze in a cost-effective manner, but because auditing sits within a highly regulated financial industry, highly specialized auditors can’t rely completely on machines to get the job done.

Augmenting, not replacing

A human-machine hybrid solution is required, and that’s where Marisa thinks her industry is headed. KPMG has partnered with IBM to integrate Watson Discovery and Watson Machine Learning into the auditing workflow. KPMG uses Watson as a backbone to a question-answering pipeline for auditors and risk analysts, enabling KPMG audit professionals to better review, classify, and search across documents to extract important attribute values. Under Marisa’s leadership, they are seeing results, and elevating the firm’s operations into a new automated era.

“I was looking to move out of my comfort zone a little bit,” says Marisa, when discussing her start at KPMG, after she’d spent years on teams in cognitive innovation research and natural language processing. When she joined the firm, an AI project was already underway, but it was hitting critical scalability and generalizability issues. She shepherded this project to completion towards a vital proof of concept, and then demonstrated with a revised strategy that the technology could scale.

Marisa had seen technology projects struggle with AI implementation in previous roles. She feels that the key to successful automation is to not cut the human actor out of the equation too quickly. She spent time working in healthcare and customer service, on problems where efficiency gains seem obvious for a small percentage of questions, but a full solution requires top experts to make decisions. This is a classic example of an area where automation can augment the human worker while observing and collecting the data essential for pushing the needle forward on solving difficult problems.

Build AI tools around user behavior

When analyzing the limitations of this “Version One” project, Marisa’s team determined that the team erred in taking technologies designed to analyze small pieces text and applying them to full documents. Instead, the technology needed to be rooted in the search function, so that documents could be broken down to find specific pieces of relevant information.

“Search is still relevant to our users,” says Marisa, “so if we don’t get the extraction at 100%, we can still give them search results.” This way, auditors spend less time sifting through mountains of paperwork to find relevant information, and more time using their highly specialized knowledge to make better decisions.

The team initiated a pivot, made some organizational changes and secured funding. Now KPMG has successfully integrated AI into their auditing process in a way that can scale not only for this use case, but with a backbone that reduces the cost of transition for other document reviews as well.

It’s an important case study in integrating AI solutions. Practitioners cannot easily bring an AI tool into an existing workflow. They need to think about how people do their jobs, and let AI tools work alongside and augment that user behavior. Furthermore, the project taught Marisa that incremental improvements can facilitate the best working relationship between machines and humans.

“Auditors have extensive knowledge, and they do iterative investigation and hypothesis refinement as they’re looking through information,” she says. “They’re often looking across documents to find conflicting information. It would be difficult for us to build out automation across all of the possible use cases and documents that auditors have, but if we provide them with these kinds of enabling technologies, we can give them a better way to find information, without having to define from the beginning what that information is.”

Once the AI tool was in place, it could track the behavior of the specialists and use that behavioral data to improve its own performance in servicing them.

Maintaining continuous buy-in across the organization

Marisa stresses the importance of overcoming the organizational hurdles that may work against a successful AI implementation.

“Now that I’m in the CTO organization,” she says, “it’s interesting to understand all of the pieces that have to move into place in order to further mature an organization to support these types of technologies. It requires a lot of strategy, trust, leadership, and a commitment to change for the better.”

“I feel really lucky to be able to be in a position where I’m still in the fight to be able to help push these things along,” says Marisa. But deployment is only half the battle. When it comes to maintaining innovation in automation over time, “it’s never over,” she says. “These AIs are living. They need to be nurtured in an appropriate environment. They’re not just something that you create and consider the job to be done. If so, you have failed, and probably in a very expensive way.”

Read Marisa’s Women Leaders in AI interview