Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support normal and derivative network error functions #78

Open
levithomason opened this issue Nov 11, 2015 · 1 comment
Open

Support normal and derivative network error functions #78

levithomason opened this issue Nov 11, 2015 · 1 comment

Comments

@levithomason
Copy link
Owner

Currently the Neuron.train() method uses the Neuron's error to calculate the delta (used to calculate the gradient and update the weights). This works OK for toy networks with a single output Nueuron. However, the back propagated delta may in fact need to be the derivative of the Neuron's input with respect to the total network error for that training sample, not with respect to the Neuron's particular error. This will represent the weight change needed to affect the total network error, not just that Neuron's error.

Bottom of this page explains it well (now that I know how to work with derivatives, thanks Khan academy!).

@levithomason
Copy link
Owner Author

The rule we were using was the Delta Rule, which calculated deltas differently than regular backpropagation. We've switched to basic backpropagation in #92. We are now calculating the delta as output - targetOuput. The issue was as noted in the linked article in this issue. output - targetOutput is not the error, it is the derivative of the network's error function (cost function) with respect to the input vector.

In order to use various network error functions, we need to also have their derivatives. Just like our activation functions are pairs of normal/prime. Then in training, the Neuron needs to use the derivative of the error function when calculating its delta.

Changing this issue to support normal and prime error functions.

@levithomason levithomason changed the title Neuron train error may be incorrect Support normal and derivative network error functions Dec 28, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant