Main factor identifiers

[lukedex]

Hello,

I was wondering if it's possible for a GAM model to be told on which factor(s) to make tree(s) on first.

The idea behind this is that if I know a factor is very predictive that I want most of my model output to be affected by this factor. By extracting information from this factor (by fitting the first tree on this factor) I assume the model will be most influenced by this factor.

Is my reasoning correct and if so is it possible to do this? I believe it currently chooses a factor by random as I've tried to repeat my results and never managed to replicate the models perfectly- even with the same dataset fed in.

[interpret-ml]

Hi @lukedex,

Good question! In general, the final EBM models should not change much based on the order of the features due to the very low default learning rate (learning_rate = 0.01). If you do wish to influence the order of training per round, our code simply loops through the columns of your training dataset (i.e. X[:, 0] -> X[:, 1] -> ... -> X[:, n] ), so reordering your training set columns will change the order of learning per round.

Your larger question about forcing higher importances on a single feature due to prior knowledge is a trickier one. One option is to train EBM models in stages. For example, if the first feature is the feature of interest:

from interpret.glassbox import ExplainableBoostingRegressor

ebm = ExplainableBoostingRegressor()
ebm.fit(X[:, 0], y)  # Fit first EBM on only the important feature

# Generate new targets from residual left in problem
y_new = y - ebm.predict(X[:, 0])

# Fit second stage EBM on residuals
ebm_secondary = ExplainableBoostingRegressor()
ebm_secondary.fit(X[:, 1:], y_new)

The idea here is to use the public interface to force the boosting process to learn exclusively on the first feature, and only then allow boosting on the subsequent features. You can then simply interpret the global explanations from both objects. However, to generate new predictions, you'll need to treat both of these models like a pipeline and sum their predictions -- unfortunately we don't have a utility today for combining two independent EBM models into one object.

Hope this helps answer your question!
-InterpretML Team

[flysky555]

Does the stage training using the public interface work for classification problem? Let’s say I need to apply the post-hoc treatment on main effects. Then it will be ideal if I can fix the main effects then retrain tne interaction. Appreciate your feedback on this.

[xiaohk]

Hey @flysky555, what kind of post-hoc treatments do you want to apply on the main effects? I think this two-stage training should help you fix the main effect and then learn other features (e.g., interactions).

By the way, I am Jay Wang, a research intern at the InterpretML team. We are developing a new visualization tool for EBM editing and recruiting participants for a user study (see #283 for more details).

We think you are a good fit for this paid user study! If you are interested, you can sign up with the link in #283. Let me know if you have any question. Thank you!

[flysky555]

I am trying the impose post hoc monotonic constraints on the main effect. Then the model needs to be recalibrated for the intercept and interactions. I understand the two-staged training can be straight forward for the regression by training on residual y-y_hat. But for classification, how to define residual? Can you provide more insights on this? Thanks for the links on case study. Let me check it out. Thanks.

…

On Mon, Oct 4, 2021 at 1:30 PM Jay Wang ***@***.***> wrote: Hey @flysky555 <https://github.com/flysky555>, what kind of post-hoc treatments do you want to apply on the main effects? I think this two-stage training should help you fix the main effect and then learn other features (e.g., interactions). By the way, I am Jay Wang, a research intern at the InterpretML team. We are developing a new visualization tool for EBM editing and recruiting participants for a user study (see #283 <#283> for more details). We think you are a good fit for this paid user study! If you are interested, you can sign up with the link in #283 <#283>. Let me know if you have any question. Thank you! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#273 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVR4F4FIK6AQVCKLBEGBAQLUFHQCTANCNFSM5CIBLBHQ> .

[xiaohk]

@flysky555 I see. Yes, I guess it would be harder to do the two-staged classifier training with the public interface. Maybe you can play around with the internal training code. There you can store the internal residuals as described in this paper.

If you want to add monotonicity constraints during training, you might want to check out #184. There is some discussion on why monotonicity through post-processing is preferred.

Spoiler alert, for the user study, we will also discuss about model monotonicity :)

[richcaruana]

Usually it is a mistake to train or retrain parts of the model such as interactions after imposing constraints such as monotonicity on main effect terms because learning will do it's best to undo the constraint. The easiest way to see this is if you have a feature such as age as a main effect, but age is also in one or more of the interaction terms. If you apply a constraint to make the main effect for age monotone when the data doesn't want age to be monotone, then learning will undo the monotonicity you imposed on age when it fits any interactions that include age. This can happen even if age is not included in any interactions because of correlation, but it is easiest to understand and most likely to happen when the same feature you apply the constraint to also is part of the model that you want to train after imposing monotonicity. Happy to chat more if that would make this clearer. From: Jay Wang ***@***.***> Sent: Monday, October 4, 2021 12:36 PM To: interpretml/interpret ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [interpretml/interpret] Main factor identifiers (#273) I see. Yes, I guess it would be harder to do the two-staged classifier training with the public interface. Maybe you can play around with the internal training code<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Finterpretml%2Finterpret%2Fblob%2Fdevelop%2Fshared%2Febm_native%2FGenerateModelUpdate.cpp&data=04%7C01%7Crcaruana%40microsoft.com%7Cfb2c8345a12e409b57f308d9876e409e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637689731056176277%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=MAKrMgnhYo4tRiaNTku39nGeUPhQlau1pONVP8Bz1wI%3D&reserved=0>. There you can store the internal residuals as described in this paper<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.cs.cornell.edu%2F~yinlou%2Fpapers%2Flou-kdd13.pdf&data=04%7C01%7Crcaruana%40microsoft.com%7Cfb2c8345a12e409b57f308d9876e409e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637689731056186272%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Kg%2BlJT%2FaSZ9s4LpM3vRH0CaKsdAkjEMh95GPq9mhme8%3D&reserved=0>. If you want to add monotonicity constraints during training, you might want to check out #184<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Finterpretml%2Finterpret%2Fissues%2F184&data=04%7C01%7Crcaruana%40microsoft.com%7Cfb2c8345a12e409b57f308d9876e409e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637689731056186272%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7g1YcYgazbufaGlU2GHoIDMhyVIbkRLGsI2ZiOyezus%3D&reserved=0>. There is some discussion<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Finterpretml%2Finterpret%2Fissues%2F184%23issuecomment-822702385&data=04%7C01%7Crcaruana%40microsoft.com%7Cfb2c8345a12e409b57f308d9876e409e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637689731056196271%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=pSmquE%2BAzOck8JkfCS6qGx%2B9ouqcDBTdqZ%2BjdJFtcew%3D&reserved=0> on why monotonicity through post-processing is preferred. Spoiler alert, for the user study, we will also discuss about model monotonicity :) - You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Finterpretml%2Finterpret%2Fissues%2F273%23issuecomment-933793520&data=04%7C01%7Crcaruana%40microsoft.com%7Cfb2c8345a12e409b57f308d9876e409e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637689731056196271%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5zG%2FJyGdgxKgAFlnMvcDQDAcRPuvanDco0az6Ge9Tz4%3D&reserved=0>, or unsubscribe<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAIUSSVW7TSHBBEV4I7IQQBLUFH63RANCNFSM5CIBLBHQ&data=04%7C01%7Crcaruana%40microsoft.com%7Cfb2c8345a12e409b57f308d9876e409e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637689731056206261%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=7aiLpC3anmhQGLnC0fcxtwibQE1FgejXmidOSB%2FmS4c%3D&reserved=0>. Triage notifications on the go with GitHub Mobile for iOS<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7Crcaruana%40microsoft.com%7Cfb2c8345a12e409b57f308d9876e409e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637689731056206261%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=gYCa93B9jTnaGbz2dyLuOg9MM4%2FJebzivYaq4jd3hNY%3D&reserved=0> or Android<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7Crcaruana%40microsoft.com%7Cfb2c8345a12e409b57f308d9876e409e%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637689731056216254%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=lJWmMEkeUKdgp9aCeAY2fSlJgKc5eNA6xCByWYv98yc%3D&reserved=0>.