-
Notifications
You must be signed in to change notification settings - Fork 0
/
introduction.tex
51 lines (48 loc) · 4.12 KB
/
introduction.tex
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
Neural networks have been an indispensable tool when it comes to artificial intelligence. They have
made possible the automation of many tasks that were previously impossible~\cite{redmon2016look,
ILSVRC15}. Due to the Universal Approximation Theorem~\cite{HORNIK1989359}, neural networks are
theoretically able to perform any task, and thus we find them effective at solving previously
unsolved problems. Mainstream training algorithms for neural networks, such as Stochastic Gradient
Descent, require lots of data for the neural network to learn an effective function. However, the
% Donald Williamson caught[66d095] that the "necessarily" in this line was originally
% "necessary", so he pointed out that this needed to be changed
definition of ``effective'' does not necessarily mean that the learned function truly represents the
data on which it is trained~\cite{szegedy2014intriguing}. While some work (for example,
\cite{yosinski2015understanding}) has shown that this seems to be the case, other
evidence~\cite{szegedy2014intriguing} suggests the exact opposite. Clearly, more work into which
training for proper representations is examined is necessary.
In this thesis, we explore some interesting points regarding representation. The first subject we
discuss is \textit{subitization}~\cite{10.2307/1418556, subitizingyoutube}, a counting method that
humans possess in which the process of keeping track of an internal counter of each instance of an
object (or ``each object instance''; the difference is discussed/debated later) is short-circuited,
the count being directly determined by what the view containing the object looks like. The question
we pose is: can we train a neural network to do the same, and does the learning process translate
into the network actually understanding what the object looks like? Further, how can we even
determine that a network has such properties in the first place? If a proper representation exists,
can we localize each instance of an object?
% The reason for the last comma existing in the next line is because I included the section of text
% in the comment below this one.
In the second part, we directly address the issue brought to light by \cite{szegedy2014intriguing},
% David Crandall, in [2dfbd9], asked me to add a part explaning this issue, so this
% text was added to do so (less the period)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% %
, \textit{adversarial examples}. Adversarial examples occur when an attacker adds noise to an image
that people would not recognize as being anything meaningful. However, a neural network's prediction
for the image's class is something unrelated to that of the original image.
% %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
We take a look at multiple types of defenses to prevent this issue, one of which modifies the loss
function, a commonly used strategy~\cite{goodfellow2015explaining, kannan2018adversarial,
madry2019deep}, and three structural techniques, one of which modifies the former non-structural
% "assess" was missing an s at the end, and [2dfbd9] caught this
defense. While not successful, at the very least, we analytically assess the defenses to provide some
kind of justification for them that structural changes can produce some sort of defense against
adversaries. As a result, it may make sense for future research to consider different types of
networks as an alternative.
% David Crandall asked[2dfbd9] me to change the "it" that was previously here to "this thesis"
These two subjects probably do not do much more than scratch the surface of the topic of
representation learning, but hopefully this thesis injects some amount of motivation into the field to
further explore the topic. The author believes that not enough focus has been on the core concepts
that underpin representation, which is backed up by the general consensus that neural networks are a
mathematical black box.