Skip to content

Latest commit

 

History

History
296 lines (236 loc) · 10.4 KB

gtsrb_clbd_results.md

File metadata and controls

296 lines (236 loc) · 10.4 KB
Raw

GTSRB Clean-label Backdoor Baseline Evaluation

Results obtained using Armory ~0.14.X (March 2022)

Bullet Hole Trigger

Undefended

Accuracy on Benign Test Data Source Class

Poison Ratio run1 run2 run3 mean std
0 0.967 0.961 0.963 0.963 0.002
20 0.971 0.965 0.975 0.970 0.004
50 0.967 0.964 0.961 0.964 0.002
80 0.967 0.971 0.958 0.965 0.005

Accuracy on Benign Test Data All Classes

Poison Ratio run1 run2 run3 mean std
0 0.928 0.930 0.925 0.928 0.002
20 0.930 0.926 0.932 0.929 0.002
50 0.925 0.927 0.928 0.927 0.001
80 0.927 0.925 0.924 0.925 0.001

Attack Success Rate

Poison Ratio run1 run2 run3 mean std
0 - - - - -
20 0.194 0.222 0.257 0.225 0.026
50 0.237 0.237 0.233 0.236 0.002
80 0.267 0.276 0.261 0.268 0.006

Accuracy on Poisoned Test Data All Classes

Poison Ratio run1 run2 run3 mean std
0 - - - - -
20 0.920 0.914 0.917 0.917 0.002
50 0.913 0.915 0.916 0.914 0.001
80 0.913 0.909 0.910 0.911 0.001

Random Filter

Accuracy on Benign Test Data Source Class

Poison Ratio run1 run2 run3 mean std
0 0.961 0.950 0.947 0.953 0.006
20 0.968 0.961 0.967 0.965 0.003
50 0.969 0.971 0.971 0.970 0.001
80 0.981 0.954 0.972 0.969 0.011

Accuracy on Benign Test Data All Classes

Poison Ratio run1 run2 run3 mean std
0 0.925 0.923 0.923 0.924 0.001
20 0.925 0.924 0.924 0.924 0.000
50 0.926 0.929 0.925 0.927 0.002
80 0.928 0.924 0.929 0.927 0.002

Attack Success Rate

Poison Ratio run1 run2 run3 mean std
0 - - - - -
20 0.236 0.207 0.237 0.227 0.014
50 0.253 0.250 0.239 0.247 0.006
80 0.246 0.279 0.275 0.267 0.015

Accuracy on Poisoned Test Data All Classes

Poison Ratio run1 run2 run3 mean std
0 - - - - -
20 0.912 0.913 0.911 0.912 0.001
50 0.912 0.916 0.912 0.913 0.002
80 0.914 0.910 0.914 0.913 0.002

Activation Clustering

Accuracy on Benign Test Data Source Class

Poison Ratio run1 run2 run3 mean std
0 0.915 0.886 0.842 0.881 0.030
20 0.906 0.897 0.912 0.905 0.006
50 0.894 0.849 0.861 0.868 0.019
80 0.904 0.910 0.907 0.907 0.002

Accuracy on Benign Test Data All Classes

Poison Ratio run1 run2 run3 mean std
0 0.871 0.866 0.861 0.866 0.004
20 0.873 0.879 0.874 0.875 0.003
50 0.854 0.858 0.865 0.859 0.004
80 0.866 0.861 0.854 0.860 0.005

Attack Success Rate

Poison Ratio run1 run2 run3 mean std
0 - - - - -
20 0.197 0.292 0.256 0.248 0.039
50 0.303 0.265 0.282 0.283 0.015
80 0.250 0.260 0.272 0.261 0.009

Accuracy on Poisoned Test Data All Classes

Poison Ratio run1 run2 run3 mean std
0 - - - - -
20 0.866 0.867 0.862 0.865 0.002
50 0.841 0.849 0.855 0.848 0.005
80 0.854 0.849 0.842 0.848 0.005

Perfect Filter

Accuracy on Benign Test Data Source Class

Poison Ratio run1 run2 run3 mean std
0 0.965 0.963 0.963 0.963 0.001
20 0.968 0.964 0.958 0.963 0.004
50 0.963 0.969 0.961 0.964 0.004
80 0.971 0.971 0.969 0.970 0.001

Accuracy on Benign Test Data All Classes

Poison Ratio run1 run2 run3 mean std
0 0.928 0.929 0.925 0.928 0.002
20 0.927 0.929 0.926 0.927 0.001
50 0.926 0.929 0.927 0.927 0.001
80 0.929 0.929 0.927 0.928 0.001

Attack Success Rate

Poison Ratio run1 run2 run3 mean std
0 - - - - -
20 0.126 0.097 0.133 0.119 0.016
50 0.092 0.082 0.147 0.107 0.029
80 0.126 0.118 0.100 0.115 0.011

Accuracy on Poisoned Test Data All Classes

Poison Ratio run1 run2 run3 mean std
0 - - - - -
20 0.920 0.923 0.918 0.920 0.002
50 0.919 0.921 0.919 0.920 0.001
80 0.922 0.919 0.920 0.920 0.001

Peace Sign Trigger

Undefended

Accuracy on Benign Test Data Source Class

Poison Ratio run1 run2 run3 mean std
0 0.968 0.964 0.972 0.968 0.003
20 0.971 0.964 0.975 0.970 0.005
50 0.964 0.956 0.969 0.963 0.006
80 0.964 0.972 0.965 0.967 0.004

Accuracy on Benign Test Data All Classes

Poison Ratio run1 run2 run3 mean std
0 0.930 0.927 0.926 0.928 0.002
20 0.932 0.934 0.932 0.933 0.001
50 0.926 0.926 0.929 0.927 0.002
80 0.929 0.928 0.928 0.928 0.000

Attack Success Rate

Poison Ratio run1 run2 run3 mean std
0 - - - - -
20 0.676 0.560 0.597 0.611 0.049
50 0.547 0.533 0.660 0.580 0.057
80 0.643 0.649 0.679 0.657 0.016

Accuracy on Poisoned Test Data All Classes

Poison Ratio run1 run2 run3 mean std
0 - - - - -
20 0.893 0.900 0.898 0.897 0.003
50 0.895 0.896 0.892 0.895 0.002
80 0.893 0.891 0.888 0.891 0.002

Random Filter

Accuracy on Benign Test Data Source Class

Poison Ratio run1 run2 run3 mean std
0 0.964 0.958 0.956 0.959 0.003
20 0.960 0.963 0.965 0.963 0.002
50 0.957 0.958 0.967 0.961 0.004
80 0.969 0.961 0.965 0.965 0.003

Accuracy on Benign Test Data All Classes

Poison Ratio run1 run2 run3 mean std
0 0.919 0.924 0.924 0.922 0.002
20 0.924 0.926 0.924 0.925 0.001
50 0.924 0.926 0.922 0.924 0.002
80 0.927 0.922 0.924 0.924 0.002

Attack Success Rate

Poison Ratio run1 run2 run3 mean std
0 - - - - -
20 0.637 0.653 0.624 0.638 0.012
50 0.611 0.581 0.656 0.616 0.031
80 0.622 0.646 0.710 0.659 0.037

Accuracy on Poisoned Test Data All Classes

Poison Ratio run1 run2 run3 mean std
0 - - - - -
20 0.885 0.887 0.887 0.886 0.001
50 0.890 0.894 0.885 0.889 0.004
80 0.891 0.885 0.885 0.887 0.003

Activation Clustering

Accuracy on Benign Test Data Source Class

Poison Ratio run1 run2 run3 mean std
0 0.907 0.929 0.914 0.917 0.009
20 0.858 0.897 0.889 0.881 0.017
50 0.914 0.910 0.929 0.918 0.008
80 0.858 0.926 0.892 0.892 0.028

Accuracy on Benign Test Data All Classes

Poison Ratio run1 run2 run3 mean std
0 0.861 0.871 0.872 0.868 0.005
20 0.865 0.852 0.870 0.862 0.008
50 0.871 0.865 0.871 0.869 0.003
80 0.856 0.861 0.860 0.859 0.002

Attack Success Rate

Poison Ratio run1 run2 run3 mean std
0 - - - - -
20 0.579 0.685 0.621 0.628 0.043
50 0.660 0.607 0.675 0.647 0.029
80 0.671 0.722 0.700 0.698 0.021

Accuracy on Poisoned Test Data All Classes

Poison Ratio run1 run2 run3 mean std
0 - - - - -
20 0.838 0.816 0.838 0.831 0.011
50 0.838 0.833 0.835 0.836 0.002
80 0.823 0.823 0.825 0.824 0.001

Perfect Filter

Accuracy on Benign Test Data Source Class

Poison Ratio run1 run2 run3 mean std
0 0.965 0.965 0.965 0.965 0.000
20 0.958 0.960 0.958 0.959 0.001
50 0.974 0.972 0.963 0.969 0.005
80 0.965 0.963 0.972 0.967 0.004

Accuracy on Benign Test Data All Classes

Poison Ratio run1 run2 run3 mean std
0 0.929 0.928 0.929 0.929 0.000
20 0.926 0.925 0.929 0.927 0.002
50 0.926 0.926 0.928 0.927 0.001
80 0.926 0.927 0.927 0.927 0.000

Attack Success Rate

Poison Ratio run1 run2 run3 mean std
0 - - - - -
20 0.222 0.214 0.151 0.196 0.032
50 0.047 0.064 0.057 0.056 0.007
80 0.253 0.272 0.118 0.214 0.069

Accuracy on Poisoned Test Data All Classes

Poison Ratio run1 run2 run3 mean std
0 - - - - -
20 0.895 0.899 0.903 0.899 0.003
50 0.902 0.905 0.910 0.905 0.003
80 0.900 0.899 0.903 0.901 0.001