Skip to content

Latest commit

 

History

History
146 lines (113 loc) · 6.8 KB

cifar10_witches_brew_results.md

File metadata and controls

146 lines (113 loc) · 6.8 KB
Raw

Cifar10 Witches' Brew Baseline Evaluation

Results obtained using Armory 0.15.X (July 2022)

Note: the baseline CIFAR resnet was updated in Armory 0.16.1.

Undefended

Accuracy on Non-trigger Images

Poison Ratio Run 1 Run 2 Run 3 Mean Std
0% 0.746446 0.747848 0.748448 0.7475806667 0.0008388886829
10% 0.743744 0.743544 0.739139 0.7421423333 0.002125246391
20% 0.735035 0.731532 0.74004 0.7355356667 0.003491371873
30% 0.750651 0.741842 0.736537 0.74301 0.740463

Accuracy on Trigger Images

Poison Ratio Run 1 Run 2 Run 3 Mean Std
0% 0.7 0.7 0.8 0.7333333333 0.04714045208
10% 0.2 0.3 0.4 0.3 0.08164965809
20% 0.1 0.1 0.1 0.1 0
30% 0.3 0.4 0.3 0.3333333333 0.04714045208

Attack Success Rate

Poison Ratio Run 1 Run 2 Run 3 Mean Std
0% 0 0 0 0 0
10% 0.7 0.5 0.6 0.6 0.08164965809
20% 0.7 0.8 0.6 0.7 0.08164965809
30% 0.7 0.6 0.4 0.5666666667 0.1247219129

Perfect Filter

Accuracy on Non-trigger Images

Poison Ratio Run 1 Run 2 Run 3 Mean Std
0% 0.743944 0.738238 0.733934 0.7387053333 0.004099904498
10% 0.741041 0.741842 0.739339 0.7407406667 0.001043680134
20% 0.732232 0.733534 0.729229 0.731665 0.001802659702
30% 0.727928 0.720921 0.726527 0.7251253333 0.003027431731

Accuracy on Trigger Images

Poison Ratio Run 1 Run 2 Run 3 Mean Std
0% 0.5 0.5 0.5 0.5 0
10% 0.7 0.6 0.5 0.6 0.08164965809
20% 0.6 0.6 0.6 0.6 0
30% 0.5 0.6 0.5 0.5333333333 0.04714045208

Attack Success Rate

Poison Ratio Run 1 Run 2 Run 3 Mean Std
0% 0 0 0.1 0.03333333333 0.04714045208
10% 0.1 0 0 0.03333333333 0.04714045208
20% 0 0 0 0 0
30% 0 0 0 0 0

Random Filter

Accuracy on Non-trigger Images

Poison Ratio Run 1 Run 2 Run 3 Mean Std
0% 0.728128 0.728128 0.708609 0.722523 0.009201344841
10% 0.715716 0.714214 0.716717 0.715549 0.001028646036
20% 0.702503 0.717417 0.700901 0.7069403333 0.007436935271
30% 0.703203 0.688488 0.696597 0.696096 0.006017810067

Accuracy on Trigger Images

Poison Ratio Run 1 Run 2 Run 3 Mean Std
0% 0.6 0.6 0.6 0.6 0
10% 0.1 0.2 0.2 0.1666666667 0.04714045208
20% 0 0.1 0 0.03333333333 0.04714045208
30% 0.1 0.1 0.1 0.1 0

Attack Success Rate

Poison Ratio Run 1 Run 2 Run 3 Mean Std
0% 0.1 0.1 0.1 0.1 0
10% 0.9 0.7 0.6 0.7333333333 0.1247219129
20% 0.8 0.5 0.8 0.7 0.1414213562
30% 0.7 0.7 0.9 0.7666666667 0.09428090416

Activation Clustering

Accuracy on Non-trigger Images

Poison Ratio Run 1 Run 2 Run 3 Mean Std
0% 0.664665 0.658559 0.657257 0.6601603333 0.003229325764
10% 0.65966 0.660561 0.651451 0.657224 0.004098666206
20% 0.667067 0.64004 0.643544 0.650217 0.01200031608
30% 0.637037 0.630731 0.645846 0.6378713333 0.006198811194

Accuracy on Trigger Images

Poison Ratio Run 1 Run 2 Run 3 Mean Std
0% 0.3 0.4 0.3 0.3333333333 0.04714045208
10% 0.4 0.2 0.4 0.3333333333 0.09428090416
20% 0.3 0.1 0 0.1333333333 0.1247219129
30% 0.1 0.2 0.2 0.1666666667 0.04714045208

Attack Success Rate

Poison Ratio Run 1 Run 2 Run 3 Mean Std
0% 0.1 0 0 0.03333333333 0.04714045208
10% 0.6 0.3 0.5 0.4666666667 0.1247219129
20% 0.6 0.7 0.6 0.6333333333 0.04714045208
30% 0.8 0.8 0.5 0.7 0.1414213562

Spectral Signatures

Accuracy on Non-trigger Images

Poison Ratio Run 1 Run 2 Run 3 Mean Std
0% 0.687487 0.683483 0.67958 0.6835166667 0.003228107013
10% 0.664765 0.667067 0.667568 0.6664666667 0.001220519653
20% 0.673373 0.655656 0.664064 0.6643643333 0.007236051978
30% 0.63994 0.657257 0.664064 0.6537536667 0.01015535499

Accuracy on Trigger Images

Poison Ratio Run 1 Run 2 Run 3 Mean Std
0% 0.7 0.6 0.6 0.6333333333 0.04714045208
10% 0.2 0.2 0.2 0.2 0
20% 0 0.1 0.1 0.06666666667 0.04714045208
30% 0 0 0.1 0.03333333333 0.04714045208

Attack Success Rate

Poison Ratio Run 1 Run 2 Run 3 Mean Std
0% 0 0.1 0 0.03333333333 0.04714045208
10% 0.7 0.5 0.8 0.6666666667 0.1247219129
20% 0.6 0.6 0.6 0.6 0
30% 0.6 0.8 0.3 0.5666666667 0.2054804668