Artificial intelligence is a rapidly evolving field in which the latest advances in training algorithms to build foundation models are being applied to multimodal research. This discipline saw prior multimodal innovations such as audio-visual speech recognition and multimedia content indexing, which had developed before advances in deep learning and data science paved the way for gen AI.

Today, practitioners use multimodal AI in all kinds of use cases, from analyzing medical images in healthcare to using computer vision alongside other sensory inputs in AI-powered autonomous vehicles.

A 2022 paper out of Carnegie Mellon¹ describes three characteristics of multimodal AI: heterogeneity, connections and interactions. Heterogeneity refers to the diverse qualities, structures and representations of modalities. A text description of an event will be fundamentally different in quality, structure and representation from a photograph of the same event. Connections refers to the complementary information shared between different modalities. These connections may be reflected in statistical similarities or in semantic correspondence. Lastly, interactions refers to how different modalities interact when they are brought together.

The core engineering challenge for multimodal AI lies in effectively integrating and processing diverse types of data to create models that can leverage the strengths of each modality while overcoming their individual limitations. The paper’s authors also put forth several challenges: representation, alignment, reasoning, generation, transference and quantification.

• Representation refers to how to represent and summarize multimodal data to reflect the heterogeneity and interconnections between modalities. Practitioners use specialized neural networks (for example, CNNs for images, transformers for text) to extract features, and employ joint embedding spaces or attention mechanisms for representation learning.

• Alignment aims to identify connections and interactions across elements. For example, engineers use techniques for temporal alignment in video and audio data, spatial alignment for images and text.

• Reasoning aims to compose knowledge from multimodal evidence, usually through multiple inferential steps.

• Generation involves learning a generative process to produce raw modalities that reflect cross-modal interactions, structure and coherence.

• Transference aims to transfer knowledge between modalities. Advanced transfer learning techniques and shared embedding spaces allow knowledge to be transferred across modalities.

• Quantification involves empirical and theoretical studies to understand multimodal learning to better evaluate their performance within multimodal models.

  Multimodal models add a layer of complexity to large language models (LLMs), which are based on transformers, themselves built on an encoder-decoder architecture with an attention mechanism to efficiently process data. Multimodal AI uses data fusion techniques to integrate different modalities. This fusion can be described as early (when modalities are encoded into the model to create a common representation space) mid (when modalities are combined at different preprocessing stages) and late (when multiple models process different modalities and combine the outputs).