An Apple research team introduces AIMV2, a family of vision encoders that is designed to predict both image patches and text tokens within a unified sequence. This combined objective enables the model to excel in a range of tasks, such as image recognition, visual grounding, and multimodal understanding.

The landscape of vision model pre-training has undergone significant evolution, especially with the rise of Large Language Models (LLMs). Traditionally, vision models operated within fixed, predefined paradigms, but LLMs have introduced a more flexible approach, unlocking new ways to leverage pre-trained vision encoders. This shift has prompted a reevaluation of pre-training methodologies for vision models to better align with multimodal applications.

In a new paper Multimodal Autoregressive Pre-training of Large Vision Encoders, an Apple research team introduces AIMV2, a family of vision encoders that employs a multimodal autoregressive pre-training strategy. Unlike conventional methods, AIMV2 is designed to predict both image patches and text tokens within a unified sequence. This combined objective enables the model to excel in a range of tasks, such as image recognition, visual grounding, and multimodal understanding.

The key innovation of AIMV2 lies in its ability to generalize the unimodal autoregressive framework to a multimodal setting. By treating image patches and text tokens as a single sequence, AIMV2 unifies the prediction process for both modalities. This approach enhances its capacity to understand complex visual and textual relationships.

The pre-training process of AIMV2 involves a causal multimodal decoder that first predicts image patches, followed by the generation of text tokens in an autoregressive manner. This simple yet effective design offers multiple advantages:

The architecture of AIMV2 is centered on the Vision Transformer (ViT), a well-established model for vision tasks. However, the AIMV2 team introduces key modifications to enhance its performance:

Empirical evaluations reveal the impressive capabilities of AIMV2. The AIMV2-3B encoder achieves 89.5% accuracy on ImageNet-1k using a frozen trunk, demonstrating its potential for high-performance image recognition. Moreover, AIMV2 consistently surpasses state-of-the-art contrastive models, such as CLIP and SigLIP, in multimodal image understanding across diverse benchmarks.

One of the key contributors to this success is AIMV2's ability to fully utilize the learning signals from all input tokens and image patches. This dense supervision approach allows for more effective training with fewer samples compared to other self-supervised or vision-language pre-trained models.

AIMV2 represents a significant step forward in the development of vision encoders. By unifying image and text prediction under a single multimodal autoregressive framework, AIMV2 achieves superior performance across a broad range of tasks. Its straightforward pre-training process, combined with architectural improvements like SwiGLU and RMSNorm, ensures scalability and adaptability. As vision models continue to scale, AIMV2 offers a blueprint for more efficient, versatile, and unified multimodal learning systems.