The Art of Automation: Chapter 4 - Automation and APIs

To assist clients in accelerating their automation projects with integration, IBM has pulled its automation capabilities together into a single platform, based on a single consistent architecture.

Covered in this chapter

Integration and APIs are the key to making enterprises automatable
How AI accelerates API creation with machine learning models for “smart field mapping”
Creating a “closed-loop integration” with AI and automation

The Art of Automation: Table of Contents

Automation, integration and APIs

Most enterprises are driving toward a digital nirvana, where they can deliver an experience to their clients that is both automated and bespoke to the individual. In order to do this, they need to leverage the data from the sum total of every interaction any part of the enterprise has ever had with that user and combine it with any relevant information from outside the enterprise. This data, digested by artificial intelligence (AI) technologies, can then be used deliver a delightful experience to that individual user.

What this means is that the folks working on this client experience need to be able to access every piece of data they have and potentially interact with any and every backend system that the enterprise is using. What makes this tricky is that many enterprises today have a mixture of legacy systems that they have been using for years combined with systems that have been brought together via acquisitions and new applications built to support emerging new lines of business. Furthermore, many organisations are part-way through a migration to cloud and already are using multiple SaaS applications to replace some of their legacy systems.

So, to build a truly engaging, automated experience, the team needs to access and bring data together from all these systems, applications and environments to their AI models and then to the user experience layer. Put another way, integration underpins automation. This is the integration challenge that IBM is working on with legions of clients around the world.

These days, we typically find that the teams building these new applications are small and agile and deadlines are tight, so the integration team needs the technology to do as much of the work as possible — leaving the human brains free to be creative and think about the really hard problems. Put another way, we need to accelerate integration using AI.

Thus, the interaction of automation, AI and integration can be divided into two focus areas:

Integration as an enabler for automation: In order to automate the enterprise, the automation must be able to access to all of the data in the enterprise, and it needs to act on that data and on the processes and applications the enterprise uses.
AI as an enabler for integration: Using AI and automation, integration teams can accelerate the pace at which an enterprise can create the APIs they need.

Making the enterprise automatable with APIs

Let’s look at what happens when you automate. Something that is happening in the enterprise is going to trigger an automation to run. It might be an event that occurs or a piece of data that gets generated. The automation needs to be able to access the data or event so that it can know that it needs to take action. Then, of course, the automation has to be able to take actions wherever it needs to, so it needs to be able to reach out and interact with the existing applications and systems in the enterprise.

If you’re thinking about automation, you’re almost certainly doing it because you want to go faster and/or to free up the human brains in your organisation to do more productive work. So, as you do this, you need the people building the automations to have quick and easy access to the data, to the events and the actions that they can perform. The best way to do this is to adopt a strategy to make application programming interfaces (APIs) ubiquitous across your company and to make those APIs as consistent as you can by adopting a common standard and documenting them in the same way.

When I talk about APIs there, I’m not just talking about REST APIs or even SOAP APIs that you might document using OpenAPI or WSDL; I also mean Streams of Events or ‘Asynchronous APIs’ that you might document using a standard like AsyncAPI as well as GraphQL APIs that we’re starting to see adopted.

API management

In order to make usage of APIs ubiquitous, what is needed is an API management practice based on tools that can help you to Create, Secure, Manage and Socialise APIs, irrespective of whether the API endpoints are in your on-premises data centres, in the public or private cloud(s).

Whichever tool you choose should have powerful tooling to help you create high quality APIs from the endpoints you have. That may involve significant orchestration flows implemented in an application integration tool, or it might simply involve creating an OpenAPI description file and implementing a simple gateway flow.

You will need to secure all these new APIs to control which applications and which developers have access to them and even implement policies like rate limiting to prevent runaway applications and automations from taking down your critical backend applications.

You will need to manage the lifecycle of your APIs so you can keep control of which versions are being used, publish new versions and retire old versions.

Just having the APIs is only part of the story. You will also need is a way to socialise the APIs with your community of automation developers. Allowing them to search and find APIs and to understand and access them in a developer portal.

It is all about speed. Each developer building an automation using an automation tool or framework needs to be able to access APIs of all kinds that the enterprise has all directly from that tool. They need to be able to understand what those APIs do, how they relate to each other, and they want to be able to call the APIs and understand them right there within the tool.

IBM API Connect is built to support all of these needs, providing state-of-the-art capabilities to create, secure and manage APIs of all types:

Accelerating integration using artificial intelligence (AI)

There is no end to the appetite for integration in a modern business. Every central IT team has a backlog of integration projects many times longer than their capacity to deliver. New business imperatives insert high priority projects at the ‘head of the queue,’ leaving a very long tail of valuable projects that will never be realized. In response, frustrated LOB executives are bypassing central IT and spinning up their own ‘shadow IT’ projects.

We think there are three solutions to this problem:

Enable integration developers to develop new integrations at many times the speed they were previously able to do by equipping them with powerful intuitive AI-based tools.
Enable many more individuals to develop high-quality integrations by creating tooling that allows non-specialists to leverage expertise built into the tools.
Integrate these tools into a platform, including specialist tools and API management, to allow central IT to retain control over integrations that are built by shadow Line of Business IT developers.

Powerful integration tools that build in expertise

We discussed earlier that the key to making the enterprise automatable is to expose all of the capabilities as APIs, yet the task of creating large number of high-quality APIs is a potentially daunting one, requiring legions of skilled integration developers. A quick search in any online bookstore will elicit dozens of books running to thousands of pages on the subject of designing high quality REST APIs. The question, then, is how can we enable engineers who don’t have the time to spend weeks studying to create high-quality API integrations?

The approach that we take in App Connect — part of the IBM Automation platform — is to build the expertise into the tool. App Connect adopts an opinionated view about the shape of an API based on established best practices. Rather than starting the journey by asking the user to create an OpenAPI definition, App Connect starts by asking for the definition of the data model, as shown in the figures below:

A data model typically supports four basic functions: Create, Read, Update and Delete (CRUD). In a similar convention, an API model is typically defined by RESTful actions, specifically the corresponding HTTP methods of POST, GET, PUT and DELETE. With App Connect, the user does not need to know REST — they just select the appropriate CRUD operation and App Connect knows how to map this onto the appropriate RESTful HTTP method:

Then, the flow can be built up by selecting the appropriate connectors for the applications and endpoints they need to integrate with. In the past, this would have required the specialist integration engineer to study the documentation for the application to understand its data model and how its APIs were structured in order to figure out how to extract the data they needed.

App Connect skips this time-consuming step by building this information into the connector itself. The IBM development teams did all the work to understand the application and map it onto the consistent CRUD model that App Connect uses and built this into the connectors to be smart connectors. For example, the below figure shows a user searching in Salesforce for contacts with email addresses that match the submitted address:

The simplification that App Connect brings by rendering every application that it connects to in the same way significantly accelerates work, especially for shadow IT professionals who perhaps do not spend every hour of every day creating integrations and are not steeped in all of the intricacies of each and every SaaS and bespoke application that their enterprise uses. Wherever possible, App Connect provides an interface that uses the objects that the users will be familiar with from using the application. For example, when adding a ‘card’ in the productivity application Trello, Trello API requires the ‘board_id’ and ‘list_id’ for the list to add the card to. That means that an integrator needs to understand what those IDs are and how to find the IDs for the particular board and list they want to add the card to. Rather than the integration developer figuring this out, it’s much more useful for the connector developer to build this knowledge into the connector, so that App Connect can allow the user to pick the human readable board name and list name that they recognize from the Trello UI rather than the IDs that the API needs:

Event-driven automations

One of the most common automation scenarios is “When this event happens, do this action or set of actions.” Integration folks will recognize that as an event-driven integration flow. Using the same tooling that we use to create API flows as described above, we can create automations that are event-driven. For example, suppose we want to send an email to all new sales leads that get entered into Salesforce, but we know that sometimes the leads get added without an email address. We might create an automation follow like the one below that is triggered whenever a new Lead object is created in Salesforce. If the email field is not filled in, it sends a Slack message to the team asking them to reach out by phone, otherwise it sends an email. Building a flow like this would be complex without smart connectors that can trigger event-driven flows, but by using the expertise built into App Connect, a flow like this can be built in a few minutes:

AI-based ‘Smart Mapper’

We asked our customers what activity their integration developers spend the most time on. The answer that came back resoundingly was ‘mapping data’ — specifically converting the fields and data representations in one API or application to the fields and data representations in another. Spurred on by that, we developed App Connect Smart Mapper. We think of this as an assistant to the integration user. It does not take over completely, but it can lighten the load considerably.

It uses a range of natural language processing techniques, including machine learning models and rules, to calculate the most likely mappings to a set of target fields from all of the data that’s in context at a given point in an integration flow. Like all good AI assistants, it is driven by confidence levels. The user can choose to have the tool insert all of the highest confidence mappings or just a subset. Even for mappings that the AI is less sure of, it can provide assistance, narrowing down the mapping task from thousands of possible fields to a small number of suggestions with confidence levels, allowing the user to make selections:

Although clients have told us already how much time this saves them, what we have so far is only the start. In the lab, we have online learning models that can adapt the mapping suggestions made based on data from previous mappings. Not simply learning particular mapping that can be repeated again verbatim but building up concept associations that allow the AI model to make suggestions in situations they have never seen before based on the concepts that they have learned from previous mappings. This online learning creates a feedback loop that we call ‘closed-loop integration.’ Using operational and development time data, feeding it back through learning models to accelerate and improve integrations.

Closed-loop integration

This theme of ‘closed-loop integration’ is an important and recurring concept in integration. By taking real-world, company-specific operational data and feeding it through AI/ML models, we can use it to make recommendations and optimizations that make integrations more accurate, faster and more robust.

One example of this is the API testing feature of API Connect. This feature addresses one of the most significant pain-points from our clients — that of developing a comprehensive suite of tests to validate the correct operation of APIs and to ensure that they do not break as they are enhanced and versioned. The API test feature can generate a suite of tests based on the OpenAPI specification for an API. It can even generate tests that establish a particular application state within the API implementation, necessary to execute the full range of the API’s capabilities.

Following the principle of closed loop integration, we have AI models from IBM Research that take Open-Tracing data from production deployments and use this to calculate the behavioral coverage that the current test suite produces. Where coverage gaps are detected, the model can suggest new tests that can be generated to enhance test coverage. Thus, feeding back operational data allows us to improve the quality of the resulting API and accelerate the development of new versions of the API because the API developers can proceed at speed knowing that they have a thorough test suite to guard against regressions.

Learn more

We have seen how integration underpins automation by allowing an enterprise to quickly access and automate all of the data and processes. We have also seen that in order for a modern enterprise to achieve this Herculean task, it must leverage powerful tools that provide a dramatically simplified user experience, leveraging AI to enable many more of its employees to get involved with creating integrations.

In order to assist clients to accelerate their automation projects with integration, IBM has pulled all of its automation capabilities together into a single platform, based on a single consistent architecture that will allow integration and automation capabilities to be effortlessly deployed alongside each other.

In a short chapter like this, it is impossible to fully explore this topic, so the references below go further explaining how IBM Cloud Pak® for Integration delivers the capabilities needed to underpin automation.

Make sure you check out The Art of Automation podcast, especially Episode 11 in which I sit down with Jerry Cuomo to discuss automation and APIs.

Check out the other chapters in the ongoing series, The Art of Automation:

The Art of Automation: Landing Page

Was this article helpful?

YesNo

Rob Nicholson

Distinguished Engineer and Master Inventor