Multimodal model

Four architectural families are in use. Dual encoders such as CLIP (Radford et al., 2021) train an image encoder and a text encoder with a symmetric contrastive (InfoNCE) loss on about 400 million image-caption pairs, so that matching pairs have high cosine similarity; they do not generate text but enable zero-shot classification and cross-modal search, and SigLIP later replaced the softmax with a pairwise sigmoid loss. Cross-attention models such as Flamingo (2022) keep a frozen language model and insert gated cross-attention layers that read visual features. The now dominant late-fusion or "adapter" design, popularised by LLaVA (2023), takes a pretrained vision encoder, maps its patch embeddings through a small projection into the language model's token-embedding space and feeds them to the decoder as if they were ordinary tokens. Early-fusion models such as Chameleon (2024) tokenise images into discrete codes and train one transformer on interleaved sequences from the start.

Images become tokens via a Vision Transformer: a 224 × 224 image cut into 14 × 14-pixel patches yields 16 × 16 = 256 patch embeddings. High-resolution inputs are handled by tiling or dynamic resolution, so one screenshot or scanned page can cost hundreds to thousands of tokens, which dominates latency and price for document workloads. Audio is typically encoded with a Whisper-style encoder over log-mel spectrograms, and video is sampled as frames, which multiplies token counts quickly. On the output side, many systems still route image generation to a separate diffusion model, while "any-to-any" models generate discrete image or audio tokens directly.

Characteristic failure modes differ from text-only models. Object hallucination (describing things not in the image), errors in counting, spatial relations, small text and chart values, and overconfident reading of handwriting are well documented, which is why a model's reading of a floor plan or invoice needs an arithmetic or human check. Typographic attacks showed early on that CLIP could be steered by a handwritten label on an object, and OWASP's Top 10 for LLM Applications 2025 lists multimodal injection, instructions hidden in images or audio, under LLM01 Prompt Injection; such text can be invisible to the user (low contrast, tiny fonts, steganography) yet readable to the model.

For governance, images and audio often carry more personal data than the prompt suggests: faces, number plates, screens in the background, voices and EXIF metadata such as GPS coordinates. Under the GDPR, a photo of a person is personal data, and facial images become special-category biometric data under Art. 9 only when processed through specific technical means for unique identification, so the purpose of processing matters for the legal basis.