Using the AI Gateway to support APIs for AI applications

API Connect provides a UI wizard to create AI-aware APIs and products, plus integration with AI services to forward requests and manage responses.

AI Gateway functionality allows you to access AI Gateway policies directly on the DataPower API Gateway. You can create REST proxies that use the AI Gateway to control access to AI models, providing a seamless way to manage AI access within your API infrastructure while ensuring enhanced security and governance of your AI models.

The AI Gateway makes it easy for enterprises to manage access to API endpoints used by AI applications. The AI Gateway simplifies the integration of AI into new and existing OpenAPI 3.0 APIs in API Connect to access a set of operations exposed by an AI service.

Note: The AI Gateway is not a separate gateway service and requires no additional installation. AI Gateway is a feature of the DataPower® API Gateway service in API Connect, and provides a single point of control for organizations to access AI services through APIs.
Use cases
There are two cases where you should use the AI Gateway:
  • As a reverse proxy for an existing API

    In this case, the API contract with the AI service is unchanged.

  • As a development tool for new APIs that use an AI service.

    You can create APIs directly in API Connect and define policies in the API workflow to manage access to AI service.

Benefits
With the API Connect AI Gateway, you can centrally manage the use of AI through policy enforcement, data encryption, masking of sensitive data, access control, audit trails and more, in support of your compliance obligations.

Using the AI Gateway to manage your API access to AI service lets you add controls to the API execution through the following features:

  • An interface to each supported AI service

    API Connect provides a policy that enables your API to send requests to a specified AI service.

  • Response caching

    The response to an API call to an AI service is cached, which improves response time for API calls and provides cost optimization for the API provider. When you create an API in API Connect, you can specify the duration for that API's response caching.

  • Rate limiting

    The AI Gateway enforces defined rate limits on APIs and API Plans (which manage access to individual APIs). You can configure rate limits based on the number of requests or generated tokens allowed within a particular time interval.

  • Tokenization

    Tokens are used as a unit of cost for LLM APIs. With the AI Gateway, rate limits can use the number of tokens generated by a request as a means of limiting usage and thus controlling costs. The token limit determines how many tokens are allowed to pass thru the gateway within a specified period of time.

  • Predefined policies and logic structures

    The AI Gateway can leverage all of the existing policies and logic constructs provided by API Connect (for example, invoke, redact, and validate) in the API execution flow.

  • API analytics

    API Connect provides an analytics service that provides insights into AI Gateway usage. You can use the analytics data on the AI usage dashboard to manage costs and track performance. For more information, see Accessing analytics in the API Manager UI.

The AI Gateway supports the following AI services: