Which model architecture is the best in adversarial defense?

Introduction

The upcoming topic in the AI security @ CVPR ’23 series is adversarial defense: scientific work that doesn’t present new attacks, but rather focuses on bolstering the defenses against existing ones. To kickstart the shift from attacks to defense in the series, this week I thought I’d create a small tool to compare adversarial defense of various computer vision model architectures based on an interesting CVPR ’23 paper:

A. Liu et al.: Exploring the Relationship Between Architectural Design and Adversarially Robust Generalization (CVPR ’23)

A. Liu et al. have evaluated the defenses of 20 computer vision model architectures against adversarial attacks based on various \(l\)-norms. Recall that the \(l\)-norm used in the attack shapes how the attack looks. \(l_2\)– and \(l_{\infty}\)-norm attacks are the classic perturbations across the entire image. By contrast, sparse attacks arbitrarily modifying only a few select pixels employ \(l_0\)-norm restrictions. The paper brings many interesting insights into the robustness to adversarial attacks. If you’re interested in this topic, I recommend you read the paper. You might also want to check out http://robust.art, a robustness benchmark created by the authors.

Comparing architectural robustness

In my opinion, the real gem is Table 1, which features accuracy results of 20 model architectures under various attacks on CIFAR-10. The table reports vanilla accuracy (standard accuracy in the classic, non-adversarial setting), clean accuracy (accuracy on clean images in the adversarial setting), robust accuracy on images perturbed by adversarial attacks, and worst case accuracy, the lower bound on robustness under multiple adversarial attacks. The authors evaluate two types of attacks, AutoAttack (AA) and projected gradient descent (PGD) attacks. The attacks on CIFAR-10 employ \(l_1\) (\(\epsilon = 40.0\)), \(l_2\) (\(\epsilon = 8.0\)), and \(l_{\infty}\) (\(\epsilon = \frac{8}{255}\)) norms.

So, without further ado: the small tool I advertised in the first paragraph is a sortable version of accuracy under adversarial attacks collected by A. Liu et al. You can sort the table by multiple columns by holding Shift and then click on the column sorting arrows in the order you want. I hope this provides useful comparisons and enables even more insight into the adversarial defense of various computer vision model architectures.

Arch	Params (M)	Vanilla	Clean	PGD-\(l_{\infty}\)	AA-\(l_{\infty}\)	PGD-\(l_2\)	PGD-\(l_1\)	Worst case
PvTv2	12.40	88.34	75.99	46.48	38.18	35.77	46.14	33.54
CoAtNet	16.99	90.73	77.73	48.27	39.85	33.80	42.30	32.17
ViT	9.78	86.73	78.76	46.02	38.00	30.86	39.27	29.24
CPVT	9.49	90.34	78.57	45.02	36.73	30.15	39.22	28.47
ViTAE	23.18	88.24	75.42	40.53	33.22	29.67	40.02	28.13
MLP-Mixer	0.68	83.43	62.86	38.93	31.81	29.27	36.50	27.42
PoolFormer	11.39	89.26	73.66	46.33	38.93	28.84	34.32	27.36
CCT	3.76	92.27	81.23	49.21	40.97	28.29	34.59	26.82
VGG	14.72	94.01	84.30	50.87	41.66	26.78	31.48	25.32
Swin Trans.	27.42	91.58	80.44	48.61	41.31	26.58	30.47	25.04
LeViT	6.67	89.01	77.10	47.16	39.87	26.28	29.58	25.04
MobileViT	5.00	91.47	77.52	49.51	41.50	26.96	29.35	24.41
BoTNet	18.82	94.16	80.76	51.29	42.95	25.84	27.38	23.15
WideResNet	55.85	96.47	89.54	55.17	44.13	22.55	23.68	20.88
DenseNet	1.12	94.42	83.23	53.06	44.02	22.55	21.87	19.48
PreActResNet	23.50	95.86	87.96	54.85	45.81	18.60	16.46	15.11
CeiT	5.56	85.24	71.55	36.20	28.02	15.31	16.77	14.35
ResNet	23.52	95.60	87.92	54.18	45.40	17.52	15.90	14.32
ResNeXt	9.12	95.64	87.12	51.51	42.66	15.07	13.64	12.18
CvT	19.54	87.81	73.76	41.36	33.67	12.75	9.25	8.76

Subscribe

Enjoying the blog? Subscribe to receive blog updates, post notifications, and monthly post summaries by e-mail.