  Feature representation learning is the key recipe for learning-based
Multi-View Stereo (MVS). As the common feature extractor of learning-based MVS,
vanilla Feature Pyramid Networks (FPNs) suffer from discouraged feature
representations for reflection and texture-less areas, which limits the
generalization of MVS. Even FPNs worked with pre-trained Convolutional Neural
Networks (CNNs) fail to tackle these issues. On the other hand, Vision
Transformers (ViTs) have achieved prominent success in many 2D vision tasks.
Thus we ask whether ViTs can facilitate feature learning in MVS? In this paper,
we propose a pre-trained ViT enhanced MVS network called MVSFormer, which can
learn more reliable feature representations benefited by informative priors
from ViT. The finetuned MVSFormer with hierarchical ViTs of efficient attention
mechanisms can achieve prominent improvement based on FPNs. Besides, the
alternative MVSFormer with frozen ViT weights is further proposed. This largely
alleviates the training cost with competitive performance strengthened by the
attention map from the self-distillation pre-training. MVSFormer can be
generalized to various input resolutions with efficient multi-scale training
strengthened by gradient accumulation. Moreover, we discuss the merits and
drawbacks of classification and regression-based MVS methods, and further
propose to unify them with a temperature-based strategy. MVSFormer achieves
state-of-the-art performance on the DTU dataset. Particularly, MVSFormer ranks
as Top-1 on both intermediate and advanced sets of the highly competitive
Tanks-and-Temples leaderboard.
