Questions about DML

[benTC74]

Hi All,

I have a couple of questions when I am using the DML package, any help is super much appreciated!!

1. When I am evaluating the performance of the model, how do I know the model is performing well or reliable (e.g. providing trustful treatment estimation) as there is no ground truth to be compared with, and there is not a metrics such as R2 in linear regression that shows how well the model is explaining. Without this, how can I explain to people the model is reliable?

2. With the above question, it brings me to my second question, can the treatment estimation always be trusted as long as the it is significant (pvalue < 0.05)? And how large are the standard error and confidence interval for them to be considered too large? For example, my dependent variable has a range from around -2000 to +8000, and one of the treatment estimation is around -200, standard error is around 60, and confidence interval from around -300 to -90. But another treatment estimation is around -4500, standard error is around 1400, and confidence interval from around -7000 to -1700, They are both significant.

3. The DML runs quite long for even a small dataset of only a few hundred observations and around 40 features, it take around 40 mins to 1.5 hours depending on the treatment variables. I using RandomForest with grid search of 3 different parameters. Is that normal?

4. Can I actually include all the different treatment variables at once in a model instead of iterating one by one? And for the categorical treatment variable, can I put that directly into the model without one-hot encoding?

5. Is there a way in DML for checking whether my data is violating the positivity or overlap assumptions in the propensity score models? For both binary and continuous treatment variables? If not, is there any pointers on how can I be validating those? It will be really really good to know.

6. Just a side question; not DML related, if I have some categorical variables (e.g. number of products - "Many" & "Few") that could be either treatment or control and that are static for the whole dataset in each group (I have 8 groups in the observations), meaning they are always of the same value in each group. Is that actually a problem in modelling? Especially in the case that I have a very small dataset?

Sorry for all of these long questions, but I am super new in this area and am wanting to really understand! I am really appreciating your help here!

[SvenKlaassen]

Hi @benTC74,

just as short comments to your points:

1. This is typical for causal inference. Your conclusions are based on the assumptions of your model (e.g. in the PLR, how good your model approximates reality, does conditional exogeneity hold). You have to argue this based on your usecase and maybe include some sort of Sensitivity Analysis. For each learner you can try to evaluate the cross-fitted performance e.g. directly via DoubleML (see here or on your own. There is no definite answer to this.
2. The p-value and confidence intervals are based on your model assumptions and learner qualities. So first, if your identifying assumptions dont hold then you will estimate a parameter with a different interpretation (see 1.)
   The confidence intervals only relate to statistical uncertainty. The size really depends on the noise in your data and there are no general settings for too large confidence intervals (except if they are out of range for your variables).
3. This can be possible as random forest can be quite time extensive, some other learners might be faster (e.g. Lightgbm)
4. No currently there is no option for that since this can be somewhat complicated.
5. There are no built-in features, but you can access the propensity score yourself and analyze it (via the predictions attribute)
6. I dont know what exactly you are refering to. Are they different than dummy variables? Remark that you shouldn't use other dummy categories as controls for the treatment as this violates the common support assumption (e.g. if you have two categories and want to estimate the effect of the first category, do not use the dummy coded second category as control)

I hope this can help!