To understand vector embeddings requires the explanation of a few key concepts:

• How vector embeddings represent data.
  
  
• How vector embeddings can be compared.
  
  
• How models can be used to generate vector embeddings.
  

In machine learning, the “dimensions” of data do not refer to the familiar and intuitive dimensions of physical space. In the vector space, each dimension corresponds to an individual feature of data, in the same way that length, width and depth are each features of an object in physical space.

Vector embeddings typically deal with high-dimensional data. In practice, most nonnumerical information is high-dimensional. For example, even a small, simple 28x28-pixel black-and-white image of a handwritten digit from the MNIST data set can be represented as a 784-dimensional vector in which each dimension corresponds to an individual pixel whose grayscale value ranges from 0 (for black) to 1 (for white).

However, not all of those dimensions of the data contain useful information. In our MNIST example, the actual digit itself represents only a small fraction of the image. The rest is a blank background or “noise.” It would thus be more accurate to say that we’re “embedding a representation of the image in 784-dimension space” than to say  we’re “representing 784 different features of the image.”

Efficient vector embeddings of high-dimensional data thus often entail some degree of dimensionality reduction: the compression of high-dimensional data down to a lower-dimensional space that omits irrelevant or redundant information.

Dimensionality reduction increases model speed and efficiency, albeit with a potential tradeoff in accuracy or precision, because smaller vectors require less computational power for mathematical operations. It can also help decrease the risk of overfitting the training data. Different dimensionality reduction methods, such as autoencoders, convolutions, principal component analysis and T-distributed stochastic neighbor embedding (t-SNE), are best suited to different data types and tasks.

Whereas the dimensions of image vector data are relatively objective and intuitive, determining the relevant features of some data modalities such as the semantic meanings and contextual relationships of language, is more abstract or subjective. In such cases, the specific features represented by the dimensions of vector embeddings can be established through manual feature engineering or, more commonly in the era of deep learning, determined implicitly through the process of training a model to make accurate predictions.

The core logic of vector embeddings is that n-dimensional embeddings of similar data points should be grouped closely together in n-dimensional space. However, embeddings can have dozens, hundreds or even thousands of dimensions. This goes well beyond the 2- or 3-dimensional spaces that our minds can intuitively visualize.

Instead, multiple measures are used to infer the relative similarity of different vector embeddings. The best measure of similarity for a specific situation depends largely on the nature of the data and what the comparisons are being used for.

• Euclidian distance measures the average straight-line distance between the corresponding points of different vectors. The difference between two n-dimensional vectors a and b is calculated by first adding the squares of the differences between each of their corresponding components—so, (a₁–b₁)² + (a₂–b₂)² + ... (a_n–b_n)²—and then taking the square root of that sum. Because Euclidian distance is sensitive to magnitude, it’s useful for data reflecting things like size or counts. Values range from 0 (for identical vectors) to ∞.
   

• Cosine distance, also called cosine similarity, is a normalized measure of the cosine of the angle between two vectors. Cosine distance ranges from -1 to 1, in which 1 represents identical vectors, 0 represents orthogonal (or unrelated) vectors, and -1 represents fully opposite vectors. Cosine similarity is used widely in NLP tasks because it naturally normalizes vector magnitudes and it’s less sensitive to the relative frequency of words in training data than Euclidian distance.
   

• Dot product is, algebraically speaking, the sum of the product of the corresponding components of each vector. Geometrically speaking, it’s a nonnormalized version of cosine distance that also reflects frequency or magnitude.

Stand-alone embedding models might be pretrained offerings or trained from scratch on specific tasks or training data. Each form of data typically benefits from a specific neural network architecture, but these are best practices rather than explicit rules in most cases. 

Sometimes, the embedding process is an integrated part of a larger neural network. For example, in the encoder-decoder convolutional neural networks (CNNs) used for tasks such as image segmentation, the act of optimizing the entire network to make accurate predictions entails training the encoder layers to output effective vector embeddings of input images.
 

Pretrained models
For many use cases and fields of study, pretrained models can provide useful embeddings that can serve as inputs to custom models or vector databases. Such open source models are typically trained on a massive and broad set of training data to learn embeddings useful to many downstream tasks such as few-shot learning or  zero-shot learning.

For text data, basic open source word embedding models such as Google’s Word2Vec or Stanford University’s Global Vectors (GloVe) can be trained from scratch, but are also offered in variants pretrained on public text data such as Wikipedia and Common Crawl. Likewise, encoder-decoder large language models (LLMs) often used for embeddings, such as BERT and its many variants, are pretrained on a huge amount of text data. 

For computer vision tasks, pretrained image classification models such as ImageNet, ResNet or VGG can be adapted to output embeddings by simply removing their final, fully connected prediction layer.

Custom embedding models
Some use cases, particularly those involving esoteric concepts or novel classes of data, benefit from the fine-tuning of pretrained models or the training of fully custom embedding models. 

The legal and medical domains are prominent examples of fields that often rely on esoteric and highly specialized vocabulary, knowledge bases or imagery unlikely to have been included in the training data of more generalist models. Supplementing the base knowledge of pretrained models through further training on domain-specific examples can help the model output more effective embeddings.

While this can also be achieved through designing a bespoke neural network architecture or training a known architecture from scratch, doing so requires resources and institutional knowledge that might be out of reach to most organizations or hobbyists.

Images with a typical RGB color scheme are numerically represented as a three-dimensional matrix, in which those three matrices correspond to the respective red, green and blue values of each pixel. RGB images are usually 8-bit, meaning each color value for a pixel can range from 0 to 256 (or 2⁸). As described earlier, black-and-white images are numerically represented as a two-dimensional matrix of pixels wherein each pixel has a value between 0 and 1.

Convolutions use 2-dimensional numerical filters, called kernels, to extract features from the image. The weights of the kernels most conducive to extracting relevant features are themselves a learnable parameter during model training. These convolutions yield a feature map of the image.

When necessary, padding is used to maintain the original size of the input by adding extra layers of zeros to the outer rows and columns of the array. Conversely, pooling, which essentially summarizes visual features by taking only their minimum, maximum or average values, can be used for further dimensionality reduction.

Finally, the compressed representation is then flattened into a vector.

One intuitive application of image embedding is image search: a system taking image data as input and returning other images with similar vector embeddings, such as a smartphone app that identifies a plant species from a photograph.

A more complex execution is multimodal image search, taking text as input and returning images related to that text. This cannot be accomplished by taking a text embedding from a language model and using it as input to a separate computer vision model. The two embedding models must be explicitly trained to correlate with one another.

One prominent algorithm used for both image and text embeddings is contrastive language-image pretraining (CLIP), originally developed by OpenAI. CLIP was trained on an enormous unlabeled data set of over 400 million image-caption pairs taken from the internet. These pairings were used to jointly train an image encoder and text encoder from scratch by using contrastive loss to maximize the cosine similarity between image embeddings and the embeddings for their corresponding captions.

Another important application for image embedding is image generation: the creation of new images.

One method to generate new images from image embeddings is variational autoencoders (VAEs). VAEs encode two different vector embeddings of input data: a vector of means and a vector of standard deviations. By randomly sampling from the probability distribution these vector embeddings represent, VAEs can use their decoder network to generate variations of that input data.

A leading embedding-based image generation method, especially in recent years, uses the previously mentioned CLIP algorithm. Image synthesis models such as DALL-E, Midjourney and Stable Diffusion take text prompts as input, using CLIP to embed a vector representation of the text; that same vector embedding, in turn, is used to essentially reconstruct a new image.

Simply representing words in terms of their letters, the way image embeddings represent visuals in terms of their pixel values, would not yield meaningful embeddings.

Whereas most computer vision models are trained using conventional supervised learning, embedding models for NLP require self-supervised learning on a truly massive amount of training data to adequately capture the many potential meanings of language in different contexts. 

The resulting embeddings power many of the tasks commonly associated with generative AI, from language translation to conversational chatbots to document summarization to question-answering services.

The models used to generate vector embeddings for text data are often not the same as those used for generating actual text.

The popular LLMs commonly used for text generation and other generative AI tasks, such as ChatGPT or Llama, are decoder-only autoregressive models, also called causal language models. In training, they’re presented with the beginning of a particular text sample and tasked with continuously predicting the next word until the end of the sequence. While this lends itself well to learning to generate coherent text, it’s not optimal for learning useful stand-alone vector embeddings.   

Instead, text embeddings typically rely on masked language models such as bidirectional encoder representations from transformers (BERT), first released in 2018. In training, these encoder-decoder models are provided text sequences with certain words masked or hidden, and tasked with completing the blanks.

This exercise rewards embeddings that better capture information about a specific word or sentence and how it relates to the context around it. Word2vec pursues a similar training task, albeit with a  simpler 2-layer neural network architecture.

As of June 2024, BERT remains the most popular language model on Hugging Face, having been downloaded over 60 million times in the month prior.³ Several prominent BERT variants have been adapted to specific types of language embeddings and scenarios:

• SBERT: Also known as sentence BERT and sentence transformers, SBERT is a variant of BERT with an adapted Siamese neural network (link resides outside ibm.com) structure, fine-tuned on pairs of sentences to improve its ability to encode sentence embeddings.

• DistilBERT: A lightweight BERT variant, created through knowledge distillation of the BERT base model into a smaller model that runs 60% faster while preserving over 95% of BERT’s performance by some metrics.⁴

• RoBERTa: Short for robustly optimized BERT pretraining approach, RoBERTa refined the BERT training procedure to optimize its performance.

Vector embeddings can be used to represent various natural language data.

Word embeddings
Word embeddings aim to capture not only the semantic meaning of individual words but also their contextual relationship to other words with which they often cooccur. In doing so, word embeddings can generalize well to new contexts and even rare or previously unseen words.

GloVe, a popular word embedding model, was trained on a “global word-word cooccurrence matrix,” inferring semantic meaning and semantic relationships from how often specific words are used close to one another. For example, meaning can be derived from how “ice” and “steam” coincide with “water” at roughly the same frequency, but coincide with “solid” and “gas” at very different rates.⁵

The way the dimensions of a word embedding vector implicitly capture these relationships enables us to mathematically manipulate them in useful and intuitive ways. In a well-configured word embedding scheme, subtracting the vector for “man” from the vector for “king” and adding the vector for “woman” should essentially yield the vector for “queen.”

Sentence embeddings
Sentence embeddings embed the semantic meaning of entire phrases or sentences, rather than individual words. They’re typically generated with SBERT or other variants of sentence transformers.

• Sentence embeddings can embed representations of user queries, for use in search engines or question-answering applications.

• In machine translation, the vector embedding of a sentence in one language can be used to output a sentence in a different language with a similar vector embedding.

• Sentence embeddings are often used in sentiment analysis. Classifiers can be either trained on labeled examples of each category of sentiment or by using supervised learning, then classify new samples by matching their vector embedding to the learned embedding for each class. Sentiment analysis is also possible through zero-shot learning, in which the embedding for a specific sentence is compared to the word embedding of a particular categorization.
   

Document embeddings
Document embeddingsare often used to classify documents or web pages for indexing in search engines or vector databases. Typical models for document embedding include BERT variants, Doc2vec (which is an expansion of the Word2vec model) or other open source embedding models such as Instructor (link resides outside ibm.com).