Since 2023, all major AI systems have become multimodal — able to process not just text but images, audio, video, and documents. Here is what this actually enables and where the limitations still are.
Image Understanding
Modern vision-language models (GPT-4o, Claude 3.5 Sonnet and above, Gemini) can: describe and analyse images in natural language, answer questions about image content, read text within images (OCR-equivalent), interpret charts, diagrams, and graphs, compare multiple images, and identify objects, people, scenes, and context. Practical applications: screenshot analysis (describe UI issues, read error messages from screenshots), document processing (extract data from scanned invoices, contracts, forms), medical image interpretation (chest X-rays, pathology slides — at performance approaching specialist radiologist accuracy in some benchmarks), accessibility (describe images for visually impaired users). Limitations: Claude cannot identify specific individuals by face (intentional safety decision); very fine-grained counting (how many objects in a complex scene) can be inaccurate; handwriting recognition varies significantly with writing quality; very small text in images (below approximately 8pt equivalent) may not be readable.
Document Understanding
Beyond basic image understanding, multimodal models excel at document understanding — treating a PDF, scanned document, or image of a form as something to be read and reasoned about holistically, not just pixel-described. Use cases: extracting structured data from invoices, converting scanned documents to structured data, comparing versions of a contract, answering questions about a multi-page financial report. The key advantage over purely text-based approaches: maintaining layout context — the model understands that a number is in a “Total” row versus a “Subtotal” row from the visual layout, not just the text alone. Current best models for document intelligence: GPT-4o, Claude (with the vision API), and specialist document AI products like AWS Textract and Azure Form Recogniser provide structured data extraction.
Audio and Speech
Speech-to-text (transcription): OpenAI’s Whisper (open-source, very accurate) and Google’s Speech-to-Text are the leading models for converting spoken audio to text. Integration into LLM workflows: transcribe audio, then process the transcript with an LLM. Real-time voice: GPT-4o’s voice mode and similar features in Gemini Live process audio in real time and respond in voice — the conversational AI phone call pattern. The latency challenge: real-time voice requires response latency under 500ms to feel natural; achieving this at LLM quality requires specialised infrastructure.
Video Understanding
Video understanding is the frontier as of 2025-2026: Gemini 1.5 Pro and Gemini 2 can process multi-hour videos natively in their context window, answering questions about content at specific timestamps, summarising video content, and identifying key moments. GPT-4o handles short video clips. The use cases emerging: interview analysis (watching a recorded meeting and summarising action items), video content moderation, sports analysis (tracking play patterns), and educational content (answering questions about a lecture video). The limitation: video is expensive to process (tokens per second of video is large), which constrains practical context lengths for most applications. Real-time video understanding at full resolution remains outside affordable commercial reach for most use cases.
