What is an enterprise service bus (ESB)?

An ESB performs transformations of data models, handles connectivity, performs message routing, converts communication protocols and potentially manages the composition of multiple requests. The ESB can make these integrations and transformations available as a service interface for reuse by new applications.

The ESB pattern is typically implemented using a specially designed integration runtime and toolset (i.e., esb product) that ensures the best possible productivity.

An ESB is an essential component of SOA, or service-oriented architecture, a software architecture that emerged in the late 1990s. SOA defines a way to make software components reusable via service interfaces. These services typically use standard interfaces (i.e., web services) in such a way that they can be rapidly incorporated into new applications without having to duplicate the functionality performed by the service in new applications.

Each service in an SOA embodies the code and data required to execute a complete, discrete business function (e.g. checking a customer’s credit, calculating a monthly loan payment, or processing a mortgage application). The service interfaces provide loose coupling, meaning they can be called with little or no knowledge of how the service is implemented underneath, reducing the dependencies between applications. Applications behind the service interface can be written in Java, Microsoft .Net, Cobol or any other programming language, supplied as packaged enterprise applications by a vendor (e.g., SAP), SaaS applications (e.g., Salesforce CRM), or obtained as open source applications.

Service interfaces are frequently defined using Web Service Definition Language (WSDL) which is a standard tag structure based on xml (extensible markup language). The services are exposed using standard network protocols—such as SOAP (simple object access protocol)/HTTP or JSON/HTTP—to send requests to read or change data. Service governance controls the lifecycle for development and at the appropriate stage the services are published in a registry that enables developers to quickly find them and reuse them to assemble new applications or business processes.

Services can be built from scratch but are often created by exposing functions from legacy systems of record. Businesses can choose to provide a standards based service interface in front of the legacy systems, use the ESB to directly connect to the legacy system through an adapter or connector, or the application may provide its own api. In any case the enterprise service bus shields the new application from the legacy interface. An ESB performs the necessary transformation and routing to connect to the legacy system service.

It is possible to implement a SOA without an ESB architecture, but this would be equivalent to just having a bunch of services. Each application owner would need to directly connect to any service it needs and perform the necessary data transformations to meet each of the service interfaces. This is a lot of work (even if the interfaces are reusable) and creates a significant maintenance challenges in the future as each connection is point to point.

In theory, a centralized ESB offers the potential to standardize—and dramatically simplify—communication, messaging, and integration between services across the enterprise. Hardware and software costs can be shared, provisioning the servers as needed for the combined usage, providing a scalable centralized solution. A single team of specialists can be tasked (and, if necessary, trained) to develop and maintain the integrations.

Software applications simply connect (‘talk’) to the ESB and leave it to the ESB to transform the protocols, route the messages, and transform into the data formats as required providing the interoperability to get the transactions executed. The enterprise service bus architectural approach supports scenarios for application integration, data integration, and service orchestration style automation of business processes. This enables developers to spend dramatically less time integrating and much more time focusing on delivering and improving their applications. And the ability to reuse these integrations from one project to the next offered the potential for still greater productivity gains and savings downstream.

But while ESBs were deployed successfully in many organizations, in many other organizations the ESB came to be seen as the bottleneck. Making changes or enhancements to one integration could destabilize others who used that same integration. Updates to the ESB middleware often impacted existing integrations, so there was significant testing required to perform any update. Because the ESB was centrally managed, application teams soon found themselves waiting in line for their integrations. As the volume of integrations grew, implementing high availability and disaster recovery for the ESB servers became more costly. And as a cross-enterprise project, the ESB proved difficult to fund, making these technical challenges that much more difficult to resolve.

Ultimately the challenges of maintaining, updating, and scaling a centralized ESB proved so overwhelming and expensive that the ESB often delayed the very productivity gains that it, and the SOA, were intended to yield, frustrating line of business teams who anticipated a greater pace of innovation.

For a deeper dive into the rise and fall of the ESB, read “The fate of the ESB.”

Microservices architecture enables the internals of a single application to be broken up into small pieces that can be independently changed, scaled, and administered. Microservices emerged and gained steam with the rise of virtualization, cloud computing, Agile development practices, and DevOps. In these contexts, microservices offer the following:

• Improved developer agility and productivity by enabling developers to incorporate new technologies into one part of an application without touching or ‘catching up’ the rest of the application.
• Simpler, more cost-effective scalability by enabling any component to be scaled independently of others, for the fastest possible response to workload demands and the most efficient use of computing resources.
• Greater resilience, because the failure of one component does not impact the others, and each microservice can perform to its own availability requirements without staking the other components to a ‘greatest common availability’ requirement.

The same granularity that microservices bring to application design can be brought to integration, with similar benefits. This is the idea behind agile integration, which breaks up the ESB into fine-grained, decentralized integration components, without interdependencies, that individual application teams can own and manage themselves.