Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Keep preprocessor variables, drop unneeded columns in chooser table #792

Open
jpn-- opened this issue Feb 6, 2024 · 1 comment
Open
Labels
Feature New feature or request Performance Changes that improve performance

Comments

@jpn--
Copy link
Member

jpn-- commented Feb 6, 2024

In components where the chooser table is copied and/or merged with other data tables (e.g. interaction-simulate and interaction-sample-simulate), we would like the option to copy ONLY the required columns/variables. Doing so should greatly reduce memory requirements when there are many un-used columns.

There are two potential approaches to this:

  • Manual. Allow the user to manually specify columns to keep or columns to drop. This has already been implemented in a handful of particularly problematic components, but having a more generic / widely applicable interface for this capability would be better.
  • Automatic. Scan the specification file and have the program decide in advance of copy/merge what columns will be needed.

Some concerns and complications:

  • Tracing. It may be desirable for an analyst to have access to all variables in a trace file, not just the retained variables.
  • Non-static variable names. It might be difficult to extract all variable name references, especially if some variable names are constructed programmatically ("on the fly") inside the model spec file, instead of appearing as a literal string.
  • Estimation mode. It is desirable for an analyst to have access to all variables in estimation mode.
@jpn--
Copy link
Member Author

jpn-- commented Feb 6, 2024

Possible complications:

  • Need all the columns in estimation mode
  • What if variables names are created "on the fly"
  • What about tracing?

@dhensle dhensle mentioned this issue Feb 20, 2024
2 tasks
@dhensle dhensle added Feature New feature or request Performance Changes that improve performance labels Feb 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Feature New feature or request Performance Changes that improve performance
Projects
Status: Done
Development

No branches or pull requests

2 participants