Hugging Face's logo

OpenGVLab
/
InternViT-6B-448px-V2_5

Image Feature Extraction
Safetensors
intern_vit_6b
custom_code
Model card Files
xet
Community
1
InternViT-6B-448px-V2_5 / README.md
zwgao's picture
zwgao
Update README.md
828dc2a verified
preview code
|
raw
history blame
10.8 kB
metadata
license: mit
datasets:
  - laion/laion2B-en
  - laion/laion-coco
  - laion/laion2B-multi
  - kakaobrain/coyo-700m
  - conceptual_captions
  - wanng/wukong100m
pipeline_tag: image-feature-extraction

InternViT-6B-448px-V2_5

[๐Ÿ“‚ GitHub] [๐Ÿ†• Blog] [๐Ÿ“œ InternVL 1.0 Paper] [๐Ÿ“œ InternVL 1.5 Report] [๐Ÿ“œ Mini-InternVL Report]

[๐Ÿ—จ๏ธ Chat Demo] [๐Ÿค— HF Demo] [๐Ÿš€ Quick Start] [๐Ÿ“– Documents]

InternViT-6B has been updated. We develop InternViT-6B-448px-V2_5 based on the pre-training of the strong foundation of InternViT-6B-448px-V1-5. Through ViT Incremental Learning with NTP loss, the vision encoder has gained a stronger ability to extract visual features, allowing it to capture more comprehensive informationโ€”especially for domains that are relatively underrepresented in web-scale datasets (e.g., LAION-5B), such as multilingual OCR data and mathematical charts, among others.

Model Name HF Link
InternViT-300M-448px-V2_5 ๐Ÿค— link
InternViT-6B-448px-V2_5 ๐Ÿค— link

Model Details

  • Model Type: vision foundation model, feature backbone
  • Model Stats:
    • Params (M): 5540 (the last 3 blocks are discarded)
    • Image size: 448 x 448, training with 1 - 12 tiles
  • Pretrain Dataset: LAION-en, LAION-zh, COYO, GRIT, COCO, TextCaps, Objects365, OpenImages, All-Seeing, Wukong-OCR, LaionCOCO-OCR, and other OCR-related datasets. To enhance the OCR capability of the model, we have incorporated additional OCR data alongside the general caption datasets. Specifically, we utilized PaddleOCR to perform Chinese OCR on images from Wukong and English OCR on images from LAION-COCO.
  • Note: InternViT-6B originally had 48 blocks, and we found that using the output after the fourth-to-last block worked best for MLLM. For ease of use and to save GPU memory, we simply discarded the last 3 blocks. Now, the model has only 45 blocks and the number of parameters has been reduced from 5.9B to 5.5B. Therefore, if you want to build a MLLM based on this model, please make use of the features from the last layer.

Performance

Image Classification

Model Name res. Linear Probing Attention Pooling Probing ฮ”
IN-1K IN-ReaL IN-V2 IN-A IN-R IN-Ske avg. IN-1K IN-ReaL IN-V2 IN-A IN-R IN-Ske avg.
InternViT-6B-224px 224 88.2 90.4 79.9 77.5 89.8 69.1 82.5 89.2 91.1 82.3 84.7 93.1 72.7 85.5 3.0
InternViT-6B-224px 448 87.8 90.2 79.8 77.2 87.1 65.8 81.3 88.8 91.0 82.0 85.4 91.3 70.5 84.8 3.5
InternViT-6B-448px-V1.0 448 87.0 90.0 78.8 77.2 85.5 65.1 80.6 88.7 91.0 82.0 88.7 92.8 72.0 85.9 5.3
InternViT-6B-448px-V1.2 448 87.0 89.9 78.5 77.1 83.9 59.7 79.4 88.6 91.1 82.0 88.7 92.7 71.6 85.8 6.4
InternViT-6B-448px-V1.5 448 86.5 89.9 78.1 69.8 82.9 60.1 77.9 88.4 91.2 81.6 86.0 92.2 70.9 85.1 7.2
InternViT-6B-448px-V2.5 448 86.6 90.1 77.8 73.7 82.7 60.0 78.5 88.3 91.2 81.3 86.9 92.4 70.8 85.2 6.7

Note: โˆ† represents the performance gap between attention pooling probing and linear probing, where a larger โˆ† suggests a shift from learning simple linear features to capture more complex, nonlinear semantic representations

Semantic Segmentation

Model Name Linear Probing Head Tuning (UperNet) Full Tuning (UperNet) ฮ”1 ฮ”2
ADE20K COCO avg. ADE20K COCO avg. ADE20K COCO avg.
InternViT-6B-224px 47.2 42.8 45.0 54.9 48.9 51.9 58.9 51.6 55.3 6.9 10.2
InternViT-6B-448px-V1.0 43.6 38.5 41.0 55.4 49.4 52.4 58.1 51.7 54.9 11.3 13.9
InternViT-6B-448px-V1.2 40.7 36.1 38.4 55.2 48.8 52.0 58.8 51.7 55.2 13.6 16.8
InternViT-6B-448px-V1.5 40.9 36.3 38.6 55.0 49.1 52.0 58.8 51.5 55.2 13.4 16.6
InternViT-6B-448px-V2.5 39.4 35.6 37.5 55.4 49.7 52.6 58.6 51.8 55.2 15.1 17.7

Note: ฮ”1 represents the gap between head tuning and linear probing, while ฮ”2 shows the gap between full tuning and linear probing. A larger ฮ” value indicates a shift from simple linear features to more complex, nonlinear representations.

Model Usage (Image Embeddings) 

import torch
from PIL import Image
from transformers import AutoModel, CLIPImageProcessor

model = AutoModel.from_pretrained(
    'OpenGVLab/InternViT-6B-448px-V2_5',
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=True,
    trust_remote_code=True).cuda().eval()

image = Image.open('./examples/image1.jpg').convert('RGB')

image_processor = CLIPImageProcessor.from_pretrained('OpenGVLab/InternViT-6B-448px-V2_5')

pixel_values = image_processor(images=image, return_tensors='pt').pixel_values
pixel_values = pixel_values.to(torch.bfloat16).cuda()

outputs = model(pixel_values)

Citation

If you find this project useful in your research, please consider citing:


@article{chen2023internvl,
  title={InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks},
  author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and Li, Bin and Luo, Ping and Lu, Tong and Qiao, Yu and Dai, Jifeng},
  journal={arXiv preprint arXiv:2312.14238},
  year={2023}
}
@article{chen2024far,
  title={How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites},
  author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others},
  journal={arXiv preprint arXiv:2404.16821},
  year={2024}
}