  Phishing attacks have inflicted substantial losses on individuals and
businesses alike, necessitating the development of robust and efficient
automated phishing detection approaches. Reference-based phishing detectors
(RBPDs), which compare the logos on a target webpage to a known set of logos,
have emerged as the state-of-the-art approach. However, a major limitation of
existing RBPDs is that they rely on a manually constructed brand knowledge
base, making it infeasible to scale to a large number of brands, which results
in false negative errors due to the insufficient brand coverage of the
knowledge base. To address this issue, we propose an automated knowledge
collection pipeline, using which we collect a large-scale multimodal brand
knowledge base, KnowPhish, containing 20k brands with rich information about
each brand. KnowPhish can be used to boost the performance of existing RBPDs in
a plug-and-play manner. A second limitation of existing RBPDs is that they
solely rely on the image modality, ignoring useful textual information present
in the webpage HTML. To utilize this textual information, we propose a Large
Language Model (LLM)-based approach to extract brand information of webpages
from text. Our resulting multimodal phishing detection approach, KnowPhish
Detector (KPD), can detect phishing webpages with or without logos. We evaluate
KnowPhish and KPD on a manually validated dataset, and a field study under
Singapore's local context, showing substantial improvements in effectiveness
and efficiency compared to state-of-the-art baselines.
