Abstract

Phishing attacks pose a significant threat to Internet users, with cybercriminals elaborately replicating the visual appearance of legitimate websites to deceive victims. Visual similarity-based detection systems have emerged as an effective countermeasure, but their effectiveness and robustness in real-world scenarios have been underexplored. In this paper, we comprehensively scrutinize and evaluate the effectiveness and robustness of popular visual similarity-based anti-phishing models using a large-scale dataset of 451k real-world phishing websites. Our analyses of the effectiveness reveal that while certain visual similarity-based models achieve high accuracy on curated datasets in the experimental settings, they exhibit notably low performance on real-world datasets, highlighting the importance of real-world evaluation. Furthermore, we find that the attackers evade the detectors mainly in three ways: (1) directly attacking the model pipelines, (2) mimicking benign logos, and (3) employing relatively simple strategies such as eliminating logos from screenshots. To statistically assess the resilience and robustness of existing models against adversarial attacks, we categorize the strategies attackers employ into visible and perturbation-based manipulations and apply them to website logos. We then evaluate the models' robustness using these adversarial samples. Our findings reveal potential vulnerabilities in several models, emphasizing the need for more robust visual similarity techniques capable of withstanding sophisticated evasion attempts. We provide actionable insights for enhancing the security of phishing defense systems, encouraging proactive actions.

---

Dataset and Source Code

Source code is publicly available in PhishingEval. The re-trained models are saved in the OneDrive. Need to put the trained models into their appropriate positions.
To facilitate reproducibility and accelerate scientific progress (i.e., strengthening collective efforts in combating phishing attacks), we also share our collected datasets. The collected datasets contain 8 types:

1. The APWG total dataset for 25 months from July 2021 to July 2023 with domains, screenshots, and HTML (apwg451514). If you want to download this dataset, please contact us through this Google form.
2. A sampled subset of the APWG dataset (phishing4190)
3. A selected failed 6000 examples from APWG dataset (failed_examples_csv)
4. A general benign dataset covering 100 domains (archive100)
5. A benign dataset for 110 common brands, also used for ablation studies and manipulations (crawl_benign)
6. Reference lists (expand277, expand277_new, merge277, merge277_new)
7. A visible manipulation dataset (visible_dataset2)
8. A perturbed dataset (perturbated_dataset)

---