Many factors go into choosing a models that will work well for your project.

The model's license may restrict how it can be used. For example, a model's license may prevent it from being used as part of a commercial application.

The data set used to train the model training has a direct impact how well the model works for a specific application and significantly affects the risk that the model may generate non-sensical, offensive, or simply unwanted responses. Similarly, models trained on copyrighted or private data may open their users to legal liability. IBM provides full training data transparency and indemnification from legal claims arising from its models.

The size of the model, how many parameters it is trained with, and the size of its context window (how long of a passage of text can the model accept) affect model performance, resource requirements, and throughput. While it's tempting to go with a "bigger is better" philosophy and choose a 20 billion parameter model, the resource requirements and improvement (if any) in accuracy may not justify it. Recent studies have shown that smaller models can significantly outperform larger ones for some solutions.

Any fine-tuning applied to a model can affect its suitability for a task. For example, IBM offers two versions of the Granite model: one tuned for general chat applications, and another tuned to follow instructions.

Other considerations when choosing a model include:

• Selection of model parameters, eg. the model temperature, to balance the creation of human-like text and factual responses. Setting the model temperature to a high value will generate consistent but potentially uninteresting or overly terse responses, while setting the temperature to a low value will introduce more variety into the responses but will add unpredictability in the response length and content.

• Selection and implementation of model guardrails to guard against ineffective or offensive results.

The choice of model depends on the application, type of data and language support requirements. Embedding models may have to be extended to accurately encode and search on industry or client specific terms or acronyms.

Vectors databases are only one option for implementing the embedding data store. Watson Discovery provides additional tools and functionality that can improve the performance and accuracy of a RAG solution; and some 'traditional' databases provide vector storage and searching, and/or similarity searching that will support a RAG solution.

There are also numerous options for vector databases. Simple in-memory databases that are embedded directly in GenAI applications provide excellent run-time performance but may not scale well to large data sets, and may introduce significant operational challenges to keep current, or to scale to multi-server configurations. Other databases that use a central server architecture are easier to operate and scale but may not meet the performance needs of the specific solution.

There are a number of methods available to integrate the retrieval and generation model. Retrieving the top K passages and using them to augment the user query is simple and expedient but it can lack the nuance necessary to answer complex questions. Simple searching by keyword may also yield satisfactory results.

More complex solutions may use an LLM to generate multiple queries from the user's original query and use them to retrieve a larger set of passages. Additional logic may added to further sort and select the retrieved passages with the highest relevancy.

Preprocessing the data before feeding it into the RAG system is an important step to ensure that the input data is in a suitable format for the model. Simple methods invole breaking input data into fixed-size chunks with overlaps, eg. the last 10 characters of a chunk is the same first 10 characters of the next one, but this can miss nuances in the input data.

More advanced pre-processing could manipulate the input text to remove common word ending, eg. stopper, stopping, and stopped all become stop; remove un-informative 'stop' words such as the, as, is, and the like; and other techniques. These can substantially improve the relevancy of retrieved information but adds complexity to both the data embedding and user prompting phases.

Even more advanced techniques may operate on full sentences, to keep as much of the meaning as possible in the text.

Evaluating the performance of a RAG system can be challenging due to the complex nature of the task. Common evaluation metrics include perplexity, fluency, relevance, and coherence - as well the BLU and ROUGE metrics. It's important to choose metrics that align with the specific goals of the task and the desired outcomes.

RAG requires plain text, and the choice of conversion methods has a big impact on data quality. For example, when converting PDF files, how are tables, images and other metadata elements handled.

Generating a human-like response from an LLM requires substantial computing resources and can often take several seconds depending on the size of the model, the complexity of the user query, and the amount of augmented information passed to the model. Solutions that need to service large groups of users or require quick response times may need to implement a mechanism to cache model responses to frequently occurring queries.

Embedding proprietary, potentially confidential, and potentially personally identifiable information into LLM prompts is core and necessary to the RAG pattern. Organizations using hosted model platforms must be aware of provider policies such as prompt data retention and usage policies (eg. does the provider capture prompt data and use it for model re-training?), controls to prevent prompt data from 'leaking' to other users, etc; and balance these against their own information security policies and controls.

While transmission of some proprietary information is unavoidable, organizations can limit their exposure by including only document or URL references to the most sensitive information in the processed data. For example, rather than embedding a price discounting table into the RAG data, include only a description of the table and a reference or link to an internal document or website in the content.

Simple transport level security (TLS) on inter-zone communications may be enough to satisfy data security requirements but architects may need to consider providing additional protection by adding components to encrypt and decrypt prompts and responses before passing them across the zone-boundary.

The type of connection between the deployment zones has impacts on several non-functional requirements. Using a virtual private network (VPN) connection over the public Internet is a low-cost option but it may not fully allay security concerns, and may not be able to meet the solution's response time or throughput requirements. A private network connection to the model hosting environment comes at a much higher cost but offers significantly better security and provides architects with the ability to control for network latency and bandwidth.