1. What multimodal AI is (the short version)

• "Reading" and "making" grew up on different mechanisms: vision-language models, which read an image and answer in words, are basically a vision encoder joined to a language model. Generative models for images, video and audio are mostly diffusion models or flow matching. Recently, attempts to handle both in a single model have appearedSourced [Sources 1, 3, 9, 15, 20]
• The common key is tokenization: images become patches or latent representations, audio becomes the codes of an audio codec, and video becomes spacetime patches — all turned into sequences a Transformer can handle. As resolution or duration grows, the sequence gets longer and the burden on computation and memory grows sharplySourced / Our calculation
• Evaluation is not settled by one number: FID measures whether the distribution of generated images is close to that of real images, CLIPScore whether an image and a sentence match, and human evaluation which output people prefer. Each changes with settings and evaluator, and they can disagreeSourced [Sources 28, 31, 30]

2. The big picture — the understanding branch and the generation branch

Multimodal AI is easier to see when sorted by input and output. Taking in images, audio or video and answering in text is "understanding"; taking in text or images and producing images, video or audio is "generation". The former has mostly been a design that joins a vision encoder to a language model, the latter mainly diffusion models and flow matching (the following sections).

Recently, more models straddle this boundary. For example, the GPT-4o system card describes GPT-4o as an "autoregressive omni model" that accepts any combination of text, audio, image and video as input and generates any combination of text, audio and image as output, "trained end-to-end across text, vision, and audio, meaning that all inputs and outputs are processed by the same neural network" (dated 8 August 2024)Sourced [Source 24]. On the research side, Chameleon presented an "early-fusion, token-based" model that understands and generates images and text in any sequence [Source 19], and Transfusion showed how to train one Transformer with both next-token prediction and diffusion [Source 20]Sourced.

However, labels such as "omni" and "natively multimodal" are the developers' own descriptions, and what is handled by a single model, and how far, has to be checked in each company's technical documents. This article does not assert the internal structure of any particular model.

3. Vision-language models — how they read images, and where they are weak

(1) Lining images up like "words": ViT and CLIP

The Vision Transformer (ViT) cuts an image into P x P patches and feeds the resulting sequence to a Transformer. The paper explains that for an image of resolution H x W the number of patches is N = HW/P², which is also the length of the input sequence to the Transformer (the "16x16 words" of the title are 16 x 16-pixel patches)Sourced [Source 1].

CLIP is pretrained on 400 million (image, text) pairs collected from the internet to predict which text goes with which image, which maps images and text into one space. This allows "zero-shot" transfer: classifying images by concepts specified in natural language rather than a predetermined set of categories. The paper reports matching the accuracy of the original ResNet-50 on ImageNet without using any of the 1.28 million ImageNet training imagesSourced [Source 2].

(2) Joining a vision encoder to a language model

Many models that read an image and answer in words are built by joining a pretrained vision encoder and a pretrained language model with a small connecting component. The main designs, as described in the papers, are sorted belowSourced.

All Sourced (abstracts and text of each paper). The way of grouping the designs is this article's own (commentary).

(3) Weak points: fine geometry, and "seeing" what is not there

Vision-language models score highly on many benchmarks, yet they fail unexpectedly on simple visual tasks. Rahmanzadehgervi et al. report that on seven simple tasks (BlindTest) — whether two circles overlap, how many times two lines intersect, which letter of a word is circled, how many circles are in an Olympic-style logo — four state-of-the-art vision-language models at the time of the paper averaged 58.07% accuracy. The best model reached 77.84%, against an expected human accuracy of 100%. They also show that accuracy becomes close to 100% when more space is added between shapes, and that the vision encoders contain the information needed to solve the tasks while the language model fails to turn it into the right answer (linear-probing experiments)Sourced [Source 38]. These are results for the models at the time of the paper (July 2024; final version March 2025), not the performance of today's models.

Li et al. also studied systematically "object hallucination", in which vision-language models describe objects that are not in the image, and reported that objects that appear frequently in the instruction data, or that tend to co-occur with objects in the image, are especially likely to be invented. Because the evaluation method itself is affected by the input instruction and output style, they proposed a method that asks repeated yes/no questions (POPE)Sourced [Source 37].

4. Tokenization — turning images, audio and video into sequences

A language model processes text after turning it into a sequence of integers called tokens. To treat images, audio and video the same way, one first has to turn a continuous signal into a sequence of finite length. The design of this conversion (tokenization, or compression into a latent representation) largely decides quality, computation and latency.

View original SVG, values and source annotations

All Sourced (descriptions in each paper or technical report).

Our calculation: resolution and token count, and audio bitrate

5. Diffusion models and flow matching — making data from noise

(1) Diffusion models: learning the "reverse" of adding noise

A diffusion model is a generative model that fixes a process which gradually adds noise to data until only noise remains, and learns the reverse (the process that returns from noise to the original data) with a neural network. The DDPM of Ho et al. showed high-quality image generation in this framework (reported FID 3.17 and Inception score 9.46 for unconditional CIFAR-10 generation)Sourced [Source 9]. Song et al. formulated diffusion as a stochastic differential equation (SDE) and explained earlier score-based generative modeling and diffusion probabilistic modeling in one framework. The reverse-time process depends only on the score (the gradient field of the data distribution), and an equivalent ordinary differential equation (ODE) that samples from the same distribution can also be derivedSourced [Source 10].

Plain diffusion models need many iterative steps, which makes generation slow. Methods proposed to ease this include DDIM (same training procedure, reported to generate 10 to 50 times faster in wall-clock time) [Source 12] and consistency models, which map noise directly to data in a single evaluation (one-step generation is supported by design, and multistep sampling can trade compute against quality) [Source 13]Sourced.

A widely used way to make a condition such as text take effect is classifier-free guidance. A conditional and an unconditional score estimate are trained together and the two are combined to adjust the trade-off between sample quality and diversitySourced [Source 11].

(2) Latent diffusion and Transformers (DiT)

As in the previous section, latent diffusion models run diffusion in the latent space of an autoencoder to cut computation and supply conditions such as text through cross-attention [Source 8]. The DiT of Peebles and Xie replaces the backbone of a diffusion model, from a U-Net to a Transformer that handles latent patches, and shows that the larger the compute (Gflops), the lower the FID tends to be. The largest, DiT-XL/2, reports an FID of 2.27 on class-conditional ImageNet at 256 x 256 (a value under those conditions and at the time of the paper)Sourced [Source 14].

(3) Flow matching and rectified flow: near-straight paths

Flow matching by Lipman et al. is a "simulation-free" way to train continuous normalizing flows (CNFs) by regressing the vector fields of fixed probability paths. It includes diffusion paths as special cases and can also use other paths based on optimal transport. The paper reports that on ImageNet it consistently gave better likelihood and sample quality than diffusion-based methodsSourced [Source 15]. Rectified flow by Liu et al. learns an ordinary differential equation that connects points of two distributions as straight as possible, and it is reported that nearly straight paths can give high-quality results even with a single Euler discretization stepSourced [Source 16].

View original SVG, values and source annotations

The Stable Diffusion 3 paper (Esser et al.) reports that training rectified flow with noise sampling biased toward perceptually relevant scales gave better results than established diffusion formulations for high-resolution text-to-image synthesis, and proposes a Transformer architecture (MMDiT) that uses separate weights for image and text with information flowing in both directions between them. It also states that model scale and validation loss follow predictable trendsSourced [Source 17]. The model card of FLUX.1 [dev] from Black Forest Labs describes the model as "a 12 billion parameter rectified flow transformer" (checked 15 October 2026)Sourced [Source 18].

(4) Understanding and generation in one model

Transfusion trains a single Transformer on mixed-modality sequences with the language-modeling next-token loss and diffusion together. It reports models up to 7 billion parameters trained on 2 trillion mixed tokens, producing images and text on a par with diffusion models and language models of similar scale, and says it scales significantly better than quantizing images and training a language model over discrete image tokensSourced [Source 20]. This is one answer to the problem of "information lost in discretization" in Section 4 (a claim of the paper, from a comparison with limited conditions).

6. Generating video and audio, and real-time voice

(1) Video generation

OpenAI's technical report on Sora (15 February 2024) says that it trained text-conditional diffusion models jointly on videos and images of variable durations, resolutions and aspect ratios, using a Transformer that operates on spacetime patches of video and image latent codes, and that the largest model can generate a minute of high-fidelity video. It also offers a suggestion that scaling video generation is a promising path toward general-purpose simulators of the physical world, yet the same report lists many limitations as a simulator, such as not accurately modeling the physics of basic interactions like glass shatteringSourced [Source 21]. The report states that it does not include model and implementation details. "World simulator" is the developer's view, not a demonstrated fact (world models are covered in this series' explainer on world models and embodied AI).

Google DeepMind's model card for Veo 3 (stated as published 23 May 2025 and updated 13 January 2026) describes Veo 3 as a generation system that makes high-resolution video with audio from text or images, uses latent diffusion, and applies diffusion to the temporal audio latents and the spatio-temporal video latents. It says training used Google's TPUs and JAXSourced [Source 22]. The same card states as a known limitation that "maintaining complete consistency throughout complex scenes or those with complex motion, remains a challenge". On evaluation, it uses MovieGenBench, released by Meta (1,003 prompts for video and 527 for video plus audio), among others, and the developer states that in head-to-head comparisons of outputs by human raters against other companies' models, Veo 3 performed best on overall preference and on following prompts. In what we could check, details such as the number and selection of raters are not given on the cardSourced [Source 22].

Google's Gemini API documentation (the page states it was last updated 30 June 2026; checked 15 October 2026) lists Gemini Omni Flash and Veo 3.1 as the video generation models and recommends the former as the defaultSourced [Source 23]. That the names and recommendations of the offered video models changed within little more than a year of Veo 3 in May 2025 is one example of how fast this field moves (this article's commentary).

(2) Real-time voice: what decides latency

Conventional voice dialogue was a cascaded pipeline that runs voice activity detection, speech recognition, text dialogue and speech synthesis in turn. The Moshi paper lists as problems that delays add up from stage to stage so that responses are delayed by several seconds, that emotion and non-speech sounds are lost because text sits in the middle, and that the segmentation into speaker turns cannot handle overlaps and interruptionsSourced [Source 25].

In contrast, end-to-end systems that generate speech directly from speech have been proposed. Moshi starts from a text language model, generates the codes of a neural audio codec, and models its own speech and the user's as two parallel streams. The paper calls it a real-time, full-duplex spoken language model with a theoretical latency of 160 ms and 200 ms in practice. The Mimi codec's frame is 80 ms, and the paper gives a theoretical latency of 160 ms for the setting with the acoustic-token delay set to one frameSourced [Source 25]. The GPT-4o system card states that it can respond to audio inputs in as little as 232 ms, with an average of 320 msSourced [Source 24]. Both the paper and the card compare with human conversational response times by citing a study of turn-taking (Stivers et al., 2009).

7. Deepfakes belong to a separate explainer — the scope of this article

Generating images, video and audio also leads to misuse: making fake material indistinguishable from the real thing. This article does not cover procedures or concrete methods of misuse. A fuller account is in this series' explainer on copyright, provenance and deepfakes. Here we give only the frameworks that could be confirmed in primary documents.

• Classification of risks: NIST AI 600-1 (July 2024) sorts the risks unique to or exacerbated by generative AI into 12 items (CBRN information, confabulation, dangerous, violent or hateful content, data privacy, environmental impacts, harmful bias and homogenization, human-AI configuration, information integrity, information security, intellectual property, obscene, degrading and abusive content, and value chain and component integration). Information integrity refers to a lowered barrier to producing and circulating content that does not distinguish fact from opinion or fiction, and that could be used for large-scale disinformation and misinformation campaignsSourced [Source 46]
• What a developer says about countermeasures: the Veo 3 card acknowledges that the model can produce deepfakes, and states that their quality was lower than those made by dedicated deepfake tools and that SynthID watermarking can mitigate them in partSourced [Source 22]. This is the developer's self-assessment, not the result of independent verification
• What this article does not write: how to imitate a particular person or work, how to evade detection, or how to bypass any company's safety measures. The thinking behind countermeasures (provenance information, watermarks, the limits of detection) is set out in the separate explainer

8. Limits, and what could not be confirmed

(1) Performance of current models

The models covered here (GPT-4o, Veo 3, Moshi, FLUX.1 [dev] and others) and the evaluation values in each paper are for the time of publication and the settings stated. We have not covered every company's latest model as of 15 October 2026 or newer evaluation results. For evaluations such as BlindTest and MMMU, whose results can change as models are updated after a paper appears, numbers are not given.

(2) Developers' own evaluations

Evaluations the developers published themselves, such as Veo 3's human comparison and GPT-4o's response time, have not been checked for independent third-party reproduction. Items whose details are not given, such as the number of raters, how they were chosen and how prompts were chosen, are stated as not confirmed.

(3) Retrieving official pages

OpenAI's page for the Sora technical report refuses automated retrieval, so the full text was checked in a copy of the same page saved by the Internet Archive. ISO's standard pages cannot be retrieved automatically for the same reason, so titles, editions, publication dates and scope were checked from the official-page entries shown in search results; the texts of the standards were not checked.

(4) Internal structure and scale

Many commercial models do not disclose parameter counts, training data or required compute. This article does not estimate them (they are treated as "not disclosed" or "could not be confirmed"). The 12 billion parameters of FLUX.1 [dev] is a statement in the developer's model card.

(5) What is not covered

Image editing, image-to-3D generation, music generation, voice identity (voice cloning), copyright and training data, and deepfake detection technology are not covered in this article (some are covered in other explainers in this series). Specialist generative models for microscope images and materials data are also not covered.

9. How to read evaluations and standards

Evaluating multimodal AI has five sides: (1) statistical closeness of generated output (FID and others), (2) consistency with the input (CLIPScore, number and position of objects), (3) human preference (head-to-head comparison, scoring by raters), (4) accuracy of understanding (question answering, hallucination), and (5) speed (latency). Each is measured with a different tool, and results change with settings, versions and evaluator. Standards and guidance mostly provide frameworks for AI management, risk, terminology and quality models rather than the performance of the model itself.

The organization of the table is this article's own (commentary), not our calculation. The texts of ISO/IEC standards are paid and were not checked. Checked 15 October 2026.

Three cautions for materials engineers

1. Benchmark values change with settings and evaluator. Even for the same model, results change with the choice of prompts, competitors, raters and version. A human comparison run by the developer, such as Veo 3's, is not an independent evaluation. Third-party leaderboards stay at reference values after checking the method and date
2. Each metric measures one thing. FID measures a distribution, CLIPScore the match with text, GenEval the count, position and color of objects, and latency measures time. Check image quality, correctness and speed separately. It is hard for a user to confirm that questions or images of a public benchmark have not leaked into the training data (this article's commentary); saturation and contamination are covered in this series' explainer on benchmarks and how to read AI evaluations
3. Some standards and guidance are binding and some are not. NIST AI RMF 1.0 and AI 600-1 are frameworks for voluntary use (AI 600-1 notes that the AI RMF is intended for voluntary use). ISO/IEC 42001 is a standard that sets an organization's management system, and meeting it does not mean that an individual model's output is correct. Legally binding regulation such as the EU AI Act is covered in this series' explainer on the EU AI Act

10. Glossary

11. References (primary sources)

1. Dosovitskiy A. et al. "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale", arXiv:2010.11929 (2020) — arxiv.org/abs/2010.11929
2. Radford A. et al. "Learning Transferable Visual Models From Natural Language Supervision", arXiv:2103.00020 (2021) — arxiv.org/abs/2103.00020
3. Liu H. et al. "Visual Instruction Tuning", NeurIPS 2023, arXiv:2304.08485 — arxiv.org/abs/2304.08485
4. Alayrac J.-B. et al. "Flamingo: a Visual Language Model for Few-Shot Learning", arXiv:2204.14198 (2022) — arxiv.org/abs/2204.14198
5. Li J. et al. "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models", arXiv:2301.12597 (2023) — arxiv.org/abs/2301.12597
6. van den Oord A. et al. "Neural Discrete Representation Learning", arXiv:1711.00937 (2017) — arxiv.org/abs/1711.00937
7. Esser P. et al. "Taming Transformers for High-Resolution Image Synthesis", arXiv:2012.09841 (2020) — arxiv.org/abs/2012.09841
8. Rombach R. et al. "High-Resolution Image Synthesis with Latent Diffusion Models", arXiv:2112.10752 (2021) — arxiv.org/abs/2112.10752
9. Ho J. et al. "Denoising Diffusion Probabilistic Models", arXiv:2006.11239 (2020) — arxiv.org/abs/2006.11239
10. Song Y. et al. "Score-Based Generative Modeling through Stochastic Differential Equations", arXiv:2011.13456 (2020) — arxiv.org/abs/2011.13456
11. Ho J., Salimans T. "Classifier-Free Diffusion Guidance", arXiv:2207.12598 (2022) — arxiv.org/abs/2207.12598
12. Song J. et al. "Denoising Diffusion Implicit Models", arXiv:2010.02502 (2020) — arxiv.org/abs/2010.02502
13. Song Y. et al. "Consistency Models", arXiv:2303.01469 (2023) — arxiv.org/abs/2303.01469
14. Peebles W., Xie S. "Scalable Diffusion Models with Transformers", arXiv:2212.09748 (2022) — arxiv.org/abs/2212.09748
15. Lipman Y. et al. "Flow Matching for Generative Modeling", arXiv:2210.02747 (2022) — arxiv.org/abs/2210.02747
16. Liu X. et al. "Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow", arXiv:2209.03003 (2022) — arxiv.org/abs/2209.03003
17. Esser P. et al. "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis", arXiv:2403.03206 (2024) — arxiv.org/abs/2403.03206
18. Black Forest Labs FLUX.1 [dev] model card (Hugging Face). Checked 15 October 2026 — huggingface.co/black-forest-labs/FLUX.1-dev
19. Chameleon Team "Chameleon: Mixed-Modal Early-Fusion Foundation Models", arXiv:2405.09818 (2024) — arxiv.org/abs/2405.09818
20. Zhou C. et al. "Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model", arXiv:2408.11039 (2024) — arxiv.org/abs/2408.11039
21. OpenAI "Video generation models as world simulators", technical report (15 February 2024). The official page blocks automated fetching, so the full text was checked in an Internet Archive copy — openai.com/index/video-generation-models-as-world-simulators
22. Google DeepMind "Veo 3 Model Card" (published 23 May 2025, updated 13 January 2026, as stated on the card). Checked 15 October 2026 — storage.googleapis.com/deepmind-media/Model-Cards/Veo-3-Model-Card.pdf
23. Google "Video generation in the Gemini API" (page states last updated 30 June 2026). Checked 15 October 2026 — ai.google.dev/gemini-api/docs/video
24. OpenAI "GPT-4o System Card", arXiv:2410.21276; the system card itself is dated 8 August 2024 (cdn.openai.com/gpt-4o-system-card.pdf) — arxiv.org/abs/2410.21276
25. Défossez A. et al. (Kyutai) "Moshi: a speech-text foundation model for real-time dialogue", arXiv:2410.00037 (2024) — arxiv.org/abs/2410.00037
26. Défossez A. et al. "High Fidelity Neural Audio Compression", arXiv:2210.13438 (2022) — arxiv.org/abs/2210.13438
27. Zeghidour N. et al. "SoundStream: An End-to-End Neural Audio Codec", arXiv:2107.03312 (2021) — arxiv.org/abs/2107.03312
28. Heusel M. et al. "GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium", NeurIPS 2017, arXiv:1706.08500 (the paper that introduced FID) — arxiv.org/abs/1706.08500
29. Salimans T. et al. "Improved Techniques for Training GANs", arXiv:1606.03498 (2016) — arxiv.org/abs/1606.03498
30. Jayasumana S. et al. "Rethinking FID: Towards a Better Evaluation Metric for Image Generation", arXiv:2401.09603 (2023) — arxiv.org/abs/2401.09603
31. Hessel J. et al. "CLIPScore: A Reference-free Evaluation Metric for Image Captioning", EMNLP 2021, arXiv:2104.08718 — arxiv.org/abs/2104.08718
32. Ghosh D. et al. "GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment", arXiv:2310.11513 (2023) — arxiv.org/abs/2310.11513
33. Lee T. et al. "Holistic Evaluation of Text-To-Image Models", arXiv:2311.04287 (2023) — arxiv.org/abs/2311.04287
34. Unterthiner T. et al. "Towards Accurate Generative Models of Video: A New Metric & Challenges", arXiv:1812.01717 (2018) — arxiv.org/abs/1812.01717
35. Huang Z. et al. "VBench: Comprehensive Benchmark Suite for Video Generative Models", arXiv:2311.17982 (2023) — arxiv.org/abs/2311.17982
36. Yue X. et al. "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI", arXiv:2311.16502 (2023) — arxiv.org/abs/2311.16502
37. Li Y. et al. "Evaluating Object Hallucination in Large Vision-Language Models", arXiv:2305.10355 (2023) — arxiv.org/abs/2305.10355
38. Rahmanzadehgervi P. et al. "Vision language models are blind: Failing to translate detailed visual features into words", arXiv:2407.06581 (2024; v6 of March 2025) — arxiv.org/abs/2407.06581
39. Chiang W.-L. et al. "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference", arXiv:2403.04132 (2024) — arxiv.org/abs/2403.04132
40. ITU-R Recommendation ITU-R BS.1534-3 (10/2015), "Method for the subjective assessment of intermediate quality level of audio systems" (MUSHRA) — itu.int (R-REC-BS.1534-3-201510)
41. ISO/IEC ISO/IEC 42001:2023, "Information technology — Artificial intelligence — Management system" (Edition 1, December 2023) — iso.org/standard/81230.html
42. ISO/IEC ISO/IEC 23894:2023, "Information technology — Artificial intelligence — Guidance on risk management" (Edition 1, February 2023) — iso.org/standard/77304.html
43. ISO/IEC ISO/IEC 22989:2022, "Information technology — Artificial intelligence — Artificial intelligence concepts and terminology" (Edition 1, July 2022) — iso.org/standard/74296.html
44. ISO/IEC ISO/IEC TS 4213:2022, "Information technology — Artificial intelligence — Assessment of machine learning classification performance" — iso.org/standard/79799.html
45. ISO/IEC ISO/IEC 25059:2023, "Software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — Quality model for AI systems" (Edition 1, June 2023) — iso.org/standard/80655.html
46. NIST NIST AI 600-1, "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile" (July 2024) — doi.org/10.6028/NIST.AI.600-1
47. NIST NIST AI 100-1, "Artificial Intelligence Risk Management Framework (AI RMF 1.0)" (January 2023) — doi.org/10.6028/NIST.AI.100-1

12. Claim-to-source audit

Last updated 15 October 2026. Sources are limited to primary material (arXiv papers, developers' model cards, system cards and technical reports, and official pages of NIST, ISO and ITU). The texts of ISO/IEC standards are paid and could not be checked. Because the article includes readings and organization of its own, these are marked as Commentary and kept separate from sourced fact. Each company's latest models and latest evaluation values, independent third-party reproductions, the texts of the standards, and the internal scale of video-generation models could not be confirmed and are not stated here. Evaluation values are for the time of publication and the settings given, and do not show which model is better. Figs. 2 to 6 are vector drawings; Figs. 1 and 7 are AI-generated image slots (none of them shows a real measurement result or a physical product).