Multimodal
A model that handles more than text — images, audio, video, screen pixels.
A model that accepts and/or produces multiple modalities. A vision-language model can read a screenshot, a speech model can transcribe audio, and a generative model can produce an image.
Most modern frontier models are natively multimodal: images go in as tokens alongside text. This is what makes screen-reading agents and document understanding practical.
Multimodal input changes what an agent can perceive, which widens the class of tasks it can attempt: reading a chart, operating software with no API, transcribing a meeting. It also widens the attack surface, because every pixel and every audio sample is untrusted input that can carry instructions.
Multimodal as the unlock for computer use
Automating a legacy internal tool with no API and no export.
Screenshots plus mouse and keyboard are enough to drive it, because the model can read the rendered screen. The trade-off is speed and reliability: DOM-level automation is faster and steadier where it exists, so reserve vision for the gaps.
- paperGemini: a family of highly capable multimodal models ↗
A natively multimodal model family — the technical counterpart to "multimodal model" as a product category.
- docsComputer use tool ↗
Screenshot, mouse and keyboard as a tool loop, with the safeguards that belong around it: isolated environments, minimal privileges, network limits, human confirmation.
- paperWhy language models hallucinate ↗
The statistical argument that training and grading regimes reward guessing over admitting uncertainty — so hallucination is an incentive problem, not a gap in knowledge.
LLM · Large Language Model
A very large model whose only job, at heart, is predicting the next token.
Computer Use · Screen control
An agent that drives a real UI with clicks, typing and screenshots.
Embedding · Vector
A list of numbers representing meaning, so similar things land close together.
Diffusion Model
An image generator that learns to reverse a process of adding noise.