Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add sizespeed for RFP, drop vestigial paragraph #25

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Conversation

dmbates
Copy link
Contributor

@dmbates dmbates commented Oct 12, 2022

  • Re-ran the movie ratings models with rectangular full packed (RFP) storage of L[2,2].
  • Added figures and discussion of the impact of RFP
  • Deleted a vestigial paragraph at the end of largescaleobserved.qmd
  • This version is more for discussion and to see where we should go than it is a final version.
  • Should we incorporate the movie classifications (Comedy, Drama, etc.) as fixed-effects?

@dmbates dmbates marked this pull request as draft October 12, 2022 19:32
@dmbates dmbates requested a review from palday October 12, 2022 19:37
largescaleobserved.qmd Outdated Show resolved Hide resolved
using MixedModelsMakie
using ProgressMeter
using SparseArrays
#using ProgressMeter
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you want all this commented out?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to trim packages that were included in using statements but not actually used. That is a task that is probably best left until after the content and text have stabilized. For the time being I plan to comment out the using statements for packages because that is an action that can easily be reversed.

largescaleobserved.qmd Outdated Show resolved Hide resolved
The time to evaluate the log-likelihood once goes from about 5 minutes for the square storage to a little over 6 minutes for the rectangular storage on the `mc == 1` models.
Because the number of evaluations to convergence stays the same on this set of models the time to fit the model goes from roughly 2 hours (square storage) to about 2.5 hours (rectangular storage).

As the threshold on the number of ratings for inclusion of a movie increases the relative change in storage sizes decreases as does the relative change in time to fit the model.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

?

Is this because the [2,1] block is a smaller fraction of the total memory use?

@palday
Copy link
Member

palday commented Oct 13, 2022

@dmbates I think it makes sense to add some fixed effects. It might even be a nice way to show that the random effects are by far the biggest computational burden -- the fixed effects aren't free, but they're comparatively cheap.

dmbates and others added 2 commits October 13, 2022 10:22
Co-authored-by: Phillip Alday <palday@users.noreply.github.com>
Co-authored-by: Phillip Alday <palday@users.noreply.github.com>
@dmbates
Copy link
Contributor Author

dmbates commented Oct 13, 2022

Can we leave this as a draft for a couple of days? I would like to make one more attempt at speeding up rankUpdate! with RFP and sparse.

@palday
Copy link
Member

palday commented Oct 13, 2022

Can we leave this as a draft for a couple of days?

@dmbates fine with me -- no rush from my side

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants