Chat
Overview
The watsonx.ai chat API & SDK provides ways to interact with foundation models in a conversational format. It allows you to recognize different types of messages, such as system prompts, user inputs, and responses from the foundation model, including follow-up questions and answers. You can use the chat API to recreate the experience of interacting with a foundation model, just like you would in the Prompt Lab’s chat mode.
Not all the available foundation models can be used with the chat API to create a conversational workflow. For information about the full set of supported IBM and third-party foundation models, see Models.
These models support conversational tasks and can help you integrate foundation models into your applications using the watsonx.ai API & SDK.
Example
You can generate text by sending a structured list of messages by prompting foundation models programmatically with the API or SDKs.
The following example show a chat interaction discussing the distance between Paris and Bangalore. The example uses the meta-llama/llama-3-8b-instruct foundation model, which works well with text generation and also supports tool calling.
Replace {token}, {region}, and {project_id} with your information.
1curl -X POST \ 2-H 'Authorization: Bearer {token}' \ 3-H 'Content-Type: application/json' \ 4-H 'Accept: application/json' \ 5--data-raw '{ 6 "messages": [ 7 { 8 "role": "system", 9 "content": "You are a helpful assistant." 10 }, 11 { 12 "role": "user", 13 "content": [ 14 { 15 "type": "text", 16 "text": "How far is Paris from Bangalore?" 17 } 18 ] 19 }, 20 { 21 "role": "assistant", 22 "content": "The distance between Paris, France, and Bangalore, India, is approximately 7,800 kilometers (4,850 miles)" 23 }, 24 { 25 "role": "user", 26 "content": [ 27 { 28 "type": "text", 29 "text": "What is the flight distance?" 30 } 31 ] 32 } 33 ], 34 "parameters": { 35 "max_new_tokens": 100, 36 "time_limit": 10000 37 }, 38 "model_id": "meta-llama/llama-3-8b-instruct", 39 "project_id": "{project_id}" 40}' \ 41"https://{region}/ml/v1/text/chat?version=2023-10-25" 42
For more information and examples, see the API reference.
The response will include the flight distance between Paris and Bangalore, including any relevant meta data.
Streaming
You can also enable streaming when using the SDK for Node.js or Python.
1try { 2 watsonxAIService.generateTextStream(params).then(async (res) => { 3 for await (const line of res) { 4 console.log(line); 5 } 6 }); 7} catch (err) { 8 console.warn(err); 9} 10