Summarization is the ability to condense long documents into a concise summary that captures the key points of the larger work. From a technology perspective, summarization is challenging because it requires a broad range of capabilities: comprehending long passages of text, identification of key points and topics, and the generation of new text that captures the intent of the larger work. Fortunately, Large Language Models (LLMs) are well-suited to these tasks. Using LLMs, architects can create solutions that help users minimize the burden of having to read long documents in details; resulting in productivity gains and more positive user experiences.

Architecture

The diagram above shows the two forms of the summarization pattern. The simplest form of the pattern is the Stuff variant. In this pattern:

1. The contents of a document is read and 'stuffed', ie. copied in its entirety, into a LLM prompt.
2. A prompt template is commonly used to 'wrap' the content with directions and keywords to direct the target model to generate a summary.
3. The resulting prompt is submitted to a trained LLM which generates a summary in response.

The Stuff approach is great for small documents but it doesn't work for documents too large for the LLM's context window, or for collections of documents. Fortunately we have the Map-Reduce variant for these situations. In the Map phase of the variant, individual documents and/or subsections of documents are stuffed into LLM prompts using the Stuff approach. The summaries returned for the documents and/or chunks are aggregated by the application and then submitted to an LLM (4) to generate an overall summary of the larger work and/or document set. It's possible to use the same LLM can be used for the Map and Reduce phases but more often the Reduce model will need to be fine-tuned to generate aggregate summaries without losing key details.

Conceptually summarization is similar to a machine translation task: we want the LLM to 'translate' a long document into a shorter summary. Thus encoder-decoder models such as BART and T5 are well-suited to summarization solutions. The majority of LLMs suitable for summarization are trained using one or more publicly available training sets drawn from sources such as news stories, Wikipedia, legislation, and scientific publications but will generally require fine-tuning before they can generate acceptable summaries for targeted business processes and input data.

A complex business process will typically require multiple fine-tuned models to generate summaries for different user groups. For example, an insurance claims process would potentially require LLMs fine-tuned for claims summarization and routing, fraud detection and investigation, and for summarization of reports from service provides such as medical or engineering consultants.

Summarization solutions require architects to make a number of significant decisions to achieve the solution's functional and non-functional requirements.

Choice of Generation Model

As documented above, many LLMs are capable of performing text summarization 'out of the box'. If the capabilities inherent in the model meet the solution requirements than architects must consider factors such the model's size (which drives infrastructure requirements), quality of responses, and inferencing speed. If fine-tuning is required then architects must also consider the amount of tuning data, and the complexity of the tuning process required to tune a selected base model to their specific needs

Evaluation Metrics

Evaluating the performance of generative AI solutions can be challenging due to the qualitative nature of the their task, ie. how one generated summary 'better' than another. Common metrics include perplexity, fluency, relevancy, and coherence; as well as BLU and ROUGE metrics. An architect must select metrics that align with the solution's functional requirements and overall business goals.