Multimodal stopped being a research line
In 2024 OpenAI released GPT-4o as a natively multimodal model that processes text, image, and audio in a unified embedding space. Anthropic shipped Claude 3.5 Sonnet with first-class image reasoning the same year. Google's Gemini 1.5 followed, paired with documented benchmarks on long-context multimodal reasoning. By the end of 2024 the question for enterprise teams was no longer whether to support multimodal inputs – it was how fast their annotation pipelines could keep up.
Meta's Segment Anything Model (SAM and SAM 2) created a parallel shift on the visual side: high-quality dense segmentation became cheap enough that the annotation bottleneck moved from "label every pixel" to "decide which masks matter". The downstream effect across vision, document AI, and embodied robotics has been to push annotation toward higher-level semantic and relational labels rather than primitive ones.
Why per-modality pipelines stop scaling
Most annotation programmes still run image, video, audio, and text on separate stacks – often separate vendors. That structure was reasonable when models were modality-specific. It breaks under three pressures once frontier multimodal models are in the picture:
- Cross-modal grounding. Tasks like visual question answering, document extraction with figures, or audio transcription with speaker-faces require labels that link a span of text to a region of an image to a window of audio. A pipeline that treats each modality separately cannot encode the link.
- Schema drift. Per-modality teams develop incompatible taxonomies. Labels from the image team disagree with labels from the document team on the same scanned page, eroding training signal.
- Cost duplication. Reviewing a multimodal sample requires the reviewer to load three tools, three schemas, and three audit trails. The cost of context-switching dwarfs the labelling itself.
What unified pipelines look like in practice
A multimodal pipeline that holds up in production usually shares four properties:
- A single schema that explicitly models cross-references – a transcript span linked to a video frame range linked to a speaker entity – rather than three parallel taxonomies that have to be reconciled later.
- Tooling that lets one reviewer see all modalities for the same example simultaneously, with playback synchronised across audio and video and bounding-box overlays anchored to specific transcript spans.
- Pre-labelling using SAM, Whisper, and an LLM as candidate generators – with humans in the loop for adjudication. Meta's SAM 2 reference paper describes the same loop in their own annotation work: model proposes, human refines.
- Evaluation that respects the cross-modal task. Single-modality F1 hides the failure case where the image label is right, the text label is right, but the link between them is wrong. Multimodal benchmarks from MMMU, MMBench, and the LMMs-Eval project all measure cross-modal grounding directly; production pipelines should mirror that.
The pre-labelling tradeoff
Frontier models are good enough to draft labels for many multimodal tasks. They are not yet reliable enough to ship without human review. The honest framing in our experience: pre-labelling cuts per-task time roughly 40-70%, but the remaining human pass is what separates a usable dataset from a noisy one.
The risk is anchoring bias. Once a reviewer sees a model-suggested label, they tend to accept it unless something is obviously wrong. The countermeasure is structural – sampled blind passes, second-reviewer adjudication on a stratified slice, and an inter-model disagreement signal that surfaces ambiguous examples for deeper review. Anthropic and OpenAI have both written publicly about how they manage this tradeoff inside RLHF and annotation pipelines for their own model training.
APAC-specific signal
For teams labelling content that includes APAC languages, the dynamics shift further. Multimodal foundation models still degrade noticeably on low-resource scripts and on culturally specific imagery – Khmer text-in-images, Thai handwritten OCR, Vietnamese diacritics in dense layouts, traditional Chinese vs. simplified-Chinese signage. The pre-labelling lift is real, but the human-review share has to be larger, and the reviewers have to be in-language. We see consistent quality gains when reviewers are co-located in the markets the data was captured in.
Where DataX Power fits
Our annotation practice runs unified multimodal pipelines for clients in autonomous driving, document AI, and content platforms – schema-first, pre-labelled with SAM-class models where it pays off, human-adjudicated where it has to, and instrumented with cross-modal IAA so the team can spot drift before it ships. If your pipeline is currently three vendors and three taxonomies, that is the conversation we have most often.
