Llama 2 is a family of transformer-based autoregressive causal language models. Autoregressive language models take a sequence of words as input and recursively predict—output—the next word(s).

During self-supervised pre-training, LLMs are provided the beginning of sample sentences drawn from a massive corpus of unlabeled data and tasked with predicting the next word. In training the model to minimize divergence between ground truth (the actual next word) and its own predictions, the model learns to replicate linguistic and logical patterns in the training data. Though the research paper notably omits details on specific data sources, it states that Llama 2 was trained with 2 trillion tokens—the numerically-represented words, word parts, phrases and other semantic fragments that transformer-based neural networks use for language processing—from publicly available sources.

On a fundamental level, base foundation models are not pre-trained to actually answer a prompt: they append text to it in a grammatically coherent way. An out-of-the-box foundation model might respond to a prompt of “teach me to bake cookies” with “for a holiday party.” Further fine-tuning, via techniques like supervised learning and reinforcement learning, is required to train a foundation model for a specific application like dialogue, instruction following or creative writing.

Instead, base Llama 2 models are intended to serve as a foundation to build a purpose-specific model upon. To date, Llama 2 (and the original LLaMa) models have served as the base of several prominent open source LLMs, including:

• Alpaca: A version of LLaMa 7B fine-tuned for instruction-following by Stanford University researchers. Notably, it achieved results competitive with GPT-3.5 despite costing only USD 600 in compute resources.⁵
• Vicuna: A chat assistant model from LMSYS Org, trained by fine-tuning Llama 2 13B on user conversations from ShareGPT (link resides outside ibm.com). It outperformed Alpaca in over 90% of cases at a mere USD 300 training cost.⁶
• Orca: A fine-tuned version of Llama 2, trained by Microsoft using a “teacher-student” scheme, in which a larger, more powerful LLM is used to generate examples of useful reasoning behavior for the smaller model to follow.⁷
• WizardLM: Fine-tuned using Evol-Instruct, a method to create large amounts of synthetic instruction data using LLMs, WizardLM achieved over 90% of the performance of ChatGPT in 17 of 29 skills evaluated.⁸

Llama-2-chat models are fine-tuned for dialogue-driven use cases, similar to the specific GPT model versions used in ChatGPT.

Supervised fine tuning (SFT) was used to prime the pre-trained Llama 2 base model to generate responses in the format expected by users in a chatbot or virtual agent setting. In a series of supervised learning tasks, labeled pairs of dialogue-style exchanges, annotated as (prompt, response), are used to train the model to minimize the divergence between its own response for a given prompt and the example response provided by the labeled data. The model thus learns, for example, that the proper response to a prompt of “teach me to bake cookies” is to provide actual instructions to bake cookies, rather than merely complete the sentence.

Rather than using millions of labeled examples, the paper states that results were improved by using “fewer but higher-quality examples,” noting that Meta AI collected 27,540 annotated samples.

Following SFT, Meta used reinforcement learning with human feedback (RLHF) to further align the chat models’ behavior with human preferences and instructions. In RLHF, direct human feedback is used to train a “reward model” to learn patterns of the kind of responses humans prefer. By translating the reward model’s predictions (regarding whether a given response would be preferred by humans) into a scalar reward signal, the reward model is then used to further train Llama-2-chat via reinforcement learning.

There are many different methods and formats in which that human feedback can be collected. Meta AI used a simple method of binary comparison: human annotators were asked to write a prompt, then choose between two model responses—based on criteria provided by Meta—generated by two different variants of Llama 2. To help the reward model properly weight these choices, annotators were also asked to rate the degree to which they preferred their chosen response over the other: “significantly better,” “slightly better” or “negligibly better/unsure.”

Human preferences were used to train two separate reward models: one optimized for helpfulness, the other optimized for safety (i.e. avoiding toxic, hateful responses or responses that might be used to aid in violence or criminal activity). In addition to proximal policy optimization (PPO), the algorithm typically used to update LLM model weights in RLHF, Meta also used rejection sampling (link resides outside ibm.com) to update Llama-2-chat-70B.

Code Llama, built on top of Llama 2, is fine-tuned for generating code (and natural language about code) from both code-based and natural language-based prompts. Introduced shortly after the release of the Llama 2 base and chat models, it’s free for research and commercial use.

Supporting most popular programming languages, including Python, C++, Java, PHP, and Javascript (among others), it’s available in model sizes of 7B, 13B and 34B parameters, and boasts a context length of up to 100,000 tokens. Two additional variations, Code Llama - Python and Code Llama - Instruct, are fine-tuned for Python (and PyTorch) and instruction following, respectively.