Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implementation - Find and Fill Method for Dropout Layer #3684

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Implementation - Find and Fill Method for Dropout Layer #3684

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
f2660a7
feat: updated Dropout Forward with find and fill
MarkFischinger Apr 10, 2024
cf6aa30
Merge branch 'mlpack:master' into feat/fill_and_fill_implemenation
MarkFischinger Apr 14, 2024
65e9914
struct: implemented both versions of the algorithm
MarkFischinger Apr 14, 2024
7511561
fixed structure
MarkFischinger Apr 14, 2024
20cf02e
refactor dropout layer implementation
MarkFischinger Apr 22, 2024
862c05e
fix
MarkFischinger Apr 22, 2024
52b6e29
remove line break
MarkFischinger Apr 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
14 changes: 14 additions & 0 deletions src/mlpack/methods/ann/layer/dropout.hpp
View file
Open in desktop
Expand Up @@ -76,6 +76,20 @@ class DropoutType : public Layer<MatType>
* @param output Resulting output activation.
*/
void Forward(const MatType& input, MatType& output);


/**
* Implementation of the forward pass of the dropout layer.
*
* @param input Input data used for evaluating the specified function.
* @param output Resulting output activation.
*/
void ForwardImpl(const MatType& input, MatType& output);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll have to use SFINAE here, e.g. add an argument like const typename std::enable_if_t<arma::is_arma_type<MatType>::value>::type* = 0 (I just did that from memory, it may not be exactly right).


#ifdef MLPACK_HAS_COOT
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we actually need this guard here: we want to use the optimized implementation when we have an Armadillo matrix, and the general implementation otherwise. So long as the second implementation of ForwardImpl() doesn't use the name coot:: inside of it (and it is possible to avoid that), we should be able to avoid the need for MLPACK_HAS_COOT, and just use SFINAE here too with a negated condition (i.e. std::enable_if_t<!arma::is_arma_type< ...).

void ForwardImpl(const MatType& input, MatType& output);
#endif


/**
* Ordinary feed backward pass of the dropout layer.
Expand Down
30 changes: 30 additions & 0 deletions src/mlpack/methods/ann/layer/dropout_impl.hpp
View file
Open in desktop
Expand Up @@ -74,10 +74,18 @@ DropoutType<MatType>::operator=(DropoutType&& other)
return *this;
}


Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

No need for an extra line 👍

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, this one is still missing.

template<typename MatType>
void DropoutType<MatType>::Forward(const MatType& input, MatType& output)
{
// The dropout mask will not be multiplied in testing mode.
ForwardImpl(input, output);
}

template<typename MatType>
void DropoutType<MatType>::ForwardImpl(const MatType& input,
MatType& output)
{
if (!this->training)
{
output = input;
Expand All @@ -92,6 +100,28 @@ void DropoutType<MatType>::Forward(const MatType& input, MatType& output)
}
}

#ifdef MLPACK_HAS_COOT

template<typename MatType>
void DropoutType<MatType>::ForwardImpl(const MatType& input,
MatType& output)
{
if (!this->training)
{
output = input;
}
else
{
mask.randu(input.n_rows, input.n_cols);
arma::uvec indices = arma::find(mask > ratio);
mask.zeros();
mask.elem(indices).fill(1);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
mask.elem(indices).fill(1);
mask.elem(find(mask > ratio)).fill(1);

Would this work? It avoids the use of arma:: here, and would make this function fully general. But I think then we would need to find a way to get rid of the .zeros() call... maybe can we just do mask = (mask > ratio)? Or something along those lines? (It would be interesting to see the speed of that approach.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rcurtin You can take a look at my implementation in the latest commit. It should be a bit faster and is fully functional. I avoided using arma:: to keep it more general.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, it looks good to me! Thanks for doing that.

output = input % mask * scale;
}
}

#endif

template<typename MatType>
void DropoutType<MatType>::Backward(
const MatType& /* input */,
Expand Down