-
Notifications
You must be signed in to change notification settings - Fork 2
/
ivlasso.ado
3999 lines (3726 loc) · 137 KB
/
ivlasso.ado
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
*! ivlasso 1.1.02 28july2020
*! pdslasso package 1.3 29july2020
*! authors aa/cbh/ms
* ivlasso: main program
* s_ivparse: Mata basic parse of IV-type lists (endog, dexog, exexog, xctrl)
* _pdsparse: further parsing into partial, notpen, amelioriation set, separately by X and Z vars
* _ivlasso: estimation program
* Updates (release date):
* 1.0.05 (30jan2018)
* First public release.
* Display name for low-dim IVs no longer prefixed with "rho_".
* Display name for endogenous d no longer prefixed with "hat_".
* Display dots and title when estimating cset using user-supplied sup-score grid.
* Display dot at end of a single loop.
* Added ssomitgrid option. Changed default sup-score method to abound.
* Promoted to require version 13 or higher.
* Fixed bug (missing touse) in generation of post-lasso resids with aset option in SelectControls.
* 1.0.06 (10feb2018)
* Fixed bug in SelectControls - was reintroducing nocons in call to rlasso when partial(.) also specified.
* Support for Sergio Correia's FTOOLS FE transform (if installed).
* 1.0.07 (21mar2018)
* Bug fix related to ftools - leaves matastrict set to on after compilation, causing rest of ado
* to fail to load. Fix is to reset matastrict to what it was before calling ftools,compile.
* 1.0.08 (04sept2018)
* Added support for string cluster variable.
* Added support for aweights and pweights.
* Error message if using partial(.) and noconstant together.
* Recoding of sortpreserve so that the temp var is created after unabbreviating user varlists
* 1.0.09 (14oct2018)
* Added error check to catch incompatible use of options with sscset option.
* 1.0.10 (08nov2018)
* Fixed bug in sscset error check added in 1.0.09.
* Added package version number.
* 1.0.11 (11dec2018)
* Now allows partial+nocons.
* Fixed bug in rlasso option + multiple d variables - would save only the last rlasso
* 1.0.12 (14jan2019)
* strrpos() unavailable in Stata 13 so replaced with strpos(strreverse(.)) in s_ivparse.
* Updated ssgamma/gamma/gammad options to be in line with new rlasso usage; gammad retired.
* Bug fix - FE + weights would fail if data were not sorted on xtset panel var.
* Bug fix - FE with full regressor set + weights not weighted correctly unless also partialling-out.
* Bug fix - if no IVs selected, could crash when obtaining PDS coefs.
* 1.0.13 (08mar2019)
* Bug fix - check for rlasso install was clearing previous e(sample) and other stored results.
* Bug fix - was not storing intermediate results if endog regressors were factor variables.
* 1.0.14 (11oct2019)
* Bug fix - for models with exogenous causal variables, no HD controls, a model constant, and no partialling-out,
* CHS estimates were incorrect because the causal variables were not being centered (partial out the constant).
* 1.1.01 (28jul2020)
* Updated to allow psolver option - choice of solver for partialling out.
* Warning issued if collinearities encountered when partialling out.
* Updated to allow full range of VCE options (HAC/AC, 2-way cluster-robust). Now requires ivreg2.
* Fixed bug in idstats - wasn't subtracting #FEs for non-cluster VCEs.
* 1.1.02 (20aug2020)
* Added check for whether ranktest is installed.
program define ivlasso, eclass // sortpreserve handled in _ivlasso
syntax [anything] [if] [in] [aw pw], /// note no "/" after pw
[ ///
cmdname(name) ///
VERsion ///
* ]
version 13
local lversion 1.1.01
local pversion 1.3
if "`version'" != "" { // Report program version number, then exit.
di in gr "ivlasso version `lversion'"
di in gr "pdslasso package version `pversion'"
ereturn clear
ereturn local version `lversion'
ereturn local pkgversion `pversion'
exit
}
if "`cmdname'"=="" {
local cmdname ivlasso // not called by pdslasso so this is an ivlasso estimation
}
if ~replay() { // not replay so estimate
// check for rlasso (eclass-command) only when estimating anew
// preferable to "cap rlasso, version" since no need to use _estimates hold
cap findfile rlasso.ado
if _rc != 0 {
di as err "Error: `cmdname' requires rlasso to run"
di as err "To install, type " _c
di in smcl "{stata ssc install rlasso :ssc install rlasso}"
exit 601
}
// ivreg2 is a necessary component
cap findfile ivreg2.ado
if _rc != 0 {
di as err "Error: `cmdname' requires ivreg2 to run"
di as err "To install, from within Stata type " _c
di in smcl "{stata ssc install ivreg2 :ssc install ivreg2}"
exit 601
}
// as is ranktest
cap findfile ranktest.ado
if _rc != 0 {
di as err "Error: `cmdname' requires ranktest to run"
di as err "To install, from within Stata type " _c
di in smcl "{stata ssc install ranktest :ssc install ranktest}"
exit 601
}
mata: s_ivparse("`anything'")
// Note that these varlists may have abbreviations, wildcards, etc.
local depvar `s(depvar)'
local dendog `s(dendog)'
local dexog `s(dexog)'
local xctrl `s(xctrl)'
local exexog `s(exexog)'
// ... so unabbreviate here, before any temp vars are created
foreach vlist in depvar dendog dexog xctrl exexog {
cap fvunab `vlist' : ``vlist'' // cap needed in case vlist is empty
}
*** Do IV or OLS PDS
_ivlasso `depvar' /// sortpreserve handled here
[`weight'`exp'] `if' `in', /// `exp' has "=" in it (no "/" option above)
dendog(`dendog') ///
dexog(`dexog') ///
xctrl(`xctrl') ///
exexog(`exexog') ///
cmdname(`cmdname') ///
`options'
}
else if "`e(cmd)'"~="ivlasso" & "`e(cmd)'"~="pdslasso" { // replay, so check that ivlasso/pdslasso results exist
di as err "last estimates not found"
exit 301
}
if "`e(firstvar)'" == "" { // display ivlasso/pdslasso main results
DisplayResults // this trap is needed because est replay is used to replay
} // the first stage results
else {
ereturn di
}
end // end wrapper
// parses basic IV varlists into PDS-type varlists
program define _pdsparse, sclass
syntax, /// varlists passed as strings to stop b/n/o operators being inserted
depvar(string) ///
[ ///
dexog(string) ///
dendog(string) ///
xctrl(string) ///
exexog(string) ///
notpen(string) ///
partial(string) ///
aset(string) ///
debugflag(int 0) ///
]
*** Initial syntax checks - overlapping lists
// check that depvar is not in other lists
foreach vlist in dendog dexog notpen partial aset {
local checklist : list depvar & `vlist'
local checknum : word count `checklist'
if `checknum' {
di as err "syntax error - cannot also include dependent variable `checklist' in `vlist'"
exit 198
}
}
// check that dexog vars are not in other lists
foreach vlist in depvar dendog notpen partial aset {
local checklist : list dexog & `vlist'
local checknum : word count `checklist'
if `checknum' {
di as err "syntax error - cannot also include exogenous variable `checklist' in `vlist'"
exit 198
}
}
// check that dendog vars are not in other lists
foreach vlist in depvar dexog notpen partial aset {
local checklist : list dendog & `vlist'
local checknum : word count `checklist'
if `checknum' {
di as err "syntax error - cannot also include endogenous variable `checklist' in `vlist'"
exit 198
}
}
// check that notpen vars are not in other lists
foreach vlist in depvar dendog dexog partial aset {
local checklist : list notpen & `vlist'
local checknum : word count `checklist'
if `checknum' {
di as err "syntax error - cannot also include unpenalized `checklist' in `vlist'"
exit 198
}
}
// check that partial vars are not in other lists
foreach vlist in depvar dendog dexog notpen aset {
local checklist : list partial & `vlist'
local checknum : word count `checklist'
if `checknum' {
di as err "syntax error - cannot also include partialled-out `checklist' in `vlist'"
exit 198
}
}
// check that xctrl vars are not in other lists
foreach vlist in depvar dendog dexog exexog {
local checklist : list xctrl & `vlist'
local checknum : word count `checklist'
if `checknum' {
di as err "syntax error - cannot also include control variable `checklist' in `vlist'"
exit 198
}
}
// check that IVs are not in other lists
foreach vlist in depvar dendog dexog xctrl {
local checklist : list exexog & `vlist'
local checknum : word count `checklist'
if `checknum' {
di as err "syntax error - cannot also include IV `checklist' in `vlist'"
exit 198
}
}
*
*** separate out the X controls and Z IVs
// 2 types: (1) high-dim (penalized); (2) unpenalized or partialled-out
// xhighdim starts as xctrl; zhighdim starts as exexog
local xnotpen : list notpen - exexog // = explicit notpen Xs
local xhighdim : list xctrl - notpen // = Xs not specified as notpen
local znotpen : list notpen - xctrl // = explicit notpen Zs
local zhighdim : list exexog - notpen // = Zs not specified as notpen
// here use xhighdim and zhighdim already started
local xpartial : list partial - exexog // = explicit partial Xs
local xhighdim : list xhighdim - partial // = Xs not specified as partial
local zpartial : list partial - xctrl // = explicit partial Zs
local zhighdim : list zhighdim - partial // = Zs not specified as partial
// checks
if "`notpen'"~="" {
local check : list notpen - xctrl // check that notpen has no extraneous variables
local check : list check - exexog
local checknum : word count `check'
if `checknum' {
di as err "error: variables `check' appear in notpen(.) but not as controls or instruments"
exit 198
}
}
if "`partial'"~="" {
local check : list partial - xctrl // check that partial has no extraneous variables
local check : list check - exexog
local checknum : word count `check'
if `checknum' {
di as err "error: variables `check' appear in partial(.) but not as controls or instruments"
exit 198
}
}
*
*** separate out the amelioration sets
local xaset : list xhighdim - aset // inverse xaset
local xaset : list xhighdim - xaset // reverse back to get xaset
local zaset : list zhighdim - aset // inverse zaset
local zaset : list zhighdim - zaset // reverse back to get zaset
local checklist : list aset - xhighdim
local checklist : list checklist - zhighdim
local checknum : word count `checklist'
if `checknum' {
di as err "syntax error - `checklist' in aset(.) but not in list of high-dim controls or instruments"
exit 198
}
*
if `debugflag' {
di
di as text "_pdsparse parsing:"
local listlist depvar dexog dendog xhighdim xnotpen xpartial xaset zhighdim znotpen zpartial zaset
foreach list of local listlist {
di as res "`list': ``list''"
}
}
// depvar, dendog, dexog are automatically low-dim/model variables
// but return anyway
sreturn local depvar_o `depvar'
sreturn local dendog_o `dendog'
sreturn local dexog_o `dexog'
sreturn local xhighdim_o `xhighdim'
sreturn local xnotpen_o `xnotpen'
sreturn local xaset_o `xaset'
sreturn local xpartial_o `xpartial'
sreturn local zhighdim_o `zhighdim'
sreturn local znotpen_o `znotpen'
sreturn local zaset_o `zaset'
sreturn local zpartial_o `zpartial'
end
// Main program
program define _ivlasso, eclass sortpreserve // sortpreserve is here
syntax varlist(numeric fv ts min=1 max=1) ///
[if] [in] [aw pw/], ///
[ ///
dexog(varlist numeric fv ts) /// low-dim set of exog regressors of interest
xctrl(varlist numeric fv ts) /// all controls incl notpen/partial
exexog(varlist numeric fv ts) /// all IVs incl notpen/partial
dendog(varlist numeric fv ts) /// low-dim set of endog regressors of interest
PNOTPen(varlist numeric fv ts) /// unpenalized variables
partial(string) /// string so that list can contain "_cons"
aset(varlist numeric fv ts) /// amelioration set
///
fe /// requires data to be xtset
dm /// treat data as having zero mean (for debugging use)
///
NOIsily ///
NOCONStant ///
CLuster(varlist max=2) /// shared option with lasso and IV estimation
Robust /// shared option with lasso and IV estimation
bw(int 0) /// shared option with lasso and IV estimation
kernel(string) /// shared option with lasso and IV estimation
sqrt /// shared option with lasso and IV estimation
first /// report first-stage regressions
idstats /// report weak ID stats
post(string) /// which results to ereturn post
///
sscset /// report sup-score CIs
ssgamma(real 0.05) /// gamma (significance) for sup-score test (default 5%)
ssgridmin(numlist missingok) /// gridmin for sup-score test
ssgridmax(numlist missingok) /// gridmax for sup-score test
ssgridpoints(integer 100) /// default gridpoints for sup-score test
ssgridmat(name) /// optional grid for sup-score test
ssomitgrid /// supress display of user-supplied grid
ssmethod(name) /// simulate, abound, select; default = abound
///
rlasso /// store rlasso results?
RLASSO0(name) /// store rlasso results with prefix `name'
LOPTions(string) /// options passed to rlasso
IVOPTions(string) /// options passed to IV or OLS estimation
cmdname(name) /// ivlasso or pdslasso
debug /// triggers debugging output and calcs
fvstrip /// alternative undocumented parsing of factor vars
NOFTOOLS ///
psolver(string) /// optional choice of solver for partialling out
]
*** rlasso-specific
// to distinguish between lasso2 and rlasso treatment of notpen,
// rlasso option is called pnotpen
// rename to notpen here and at end of program save macros as pnotpen
// temporary measure until lasso2 and rlasso code is merge
local notpen `pnotpen'
*
*** debug mode; create flag
local debugflag =("`debug'"~="")
*
*** post option - which results to post
// default is PDS; mixed case allowed, convert to lower case.
local post =strlower("`post'")
if "`post'"=="" {
local post pds
}
else if "`post'"~="pds" & "`post'"~="lasso" & "`post'"~="plasso" {
di as err "error: post(.) must be one of pds, lasso or plasso"
exit 198
}
*
*** FEs. Change flag to 1/0.
// Get panel id
local feflag=("`fe'"~="")
if `feflag' {
cap _xt
if _rc ~= 0 {
di as err "Error: fe option requires data to be xtset"
exit 459
}
else {
local ivar `r(ivar)'
}
// fe transformation may expect data to be sorted on ivar
local sortvar : sortedby
local sortvar : word 1 of `sortvar' // in case sorted on multiple variables
if "`ivar'"~="`sortvar'" {
di as text "(sorting by xtset panelvar `ivar')"
sort `ivar'
}
}
*
// rename dep var
local depvar `varlist'
*
*** Record which observations have non-missing values
marksample touse
markout `touse' `depvar' `dendog' `dexog' `xctrl' `exexog' `cluster' `ivar', strok
sum `touse' if `touse', meanonly // will sum weight var when weights are used
local N = r(N)
*
*** weight flag
local weightflag =("`weight'"~="")
*
*** constant, partial, etc.
// conmodel: constant in original model
// consflag: constant in transformed equation to estimate
// dmflag: treat data as zero-mean
if `weightflag' & ("`noconstant'"~="") {
// incompatible options - with weights, must have a constant to partial out or remove by FE
di as err "incompatible options - weights + noconstant"
exit 198
}
local consmodel =("`noconstant'"=="")
if `feflag' {
local consmodel 0 // if fe, then consmodel=0 always
}
// model with weights must have a constant before partialling out or FE transformation
// weighting implies special treatment of constant - must be partialled-out
// will be automatic if fe
if `weightflag' & ~`feflag' {
local partial `partial' _cons // duplicate _cons not a problem since removed below
}
local partialflag =("`partial'"~="") // =1 even if just cons is being partialled out
// Tell estimation code if cons has been partialled out or there isn't one in the first place
if `feflag' | `partialflag' | (`consmodel'==0) {
local consflag 0
}
else {
local consflag 1
}
if "`dm'"~="" {
// dmflag tells estimation code to treat data as zero-mean; user can force this with dm option
local dmflag 1
}
else if `consmodel' {
// model has an constant that will be partialled out or treated as unpenalized
local dmflag 0
}
else {
// special case - nocons and no FEs means treat vars as if demeaned
local dmflag 1
}
// "_cons" allowed as an argument to partial(.) - remove it
local partial : subinstr local partial "_cons" "", all word count(local pconscount)
local notpen : subinstr local notpen "_cons" "", all word count(local notpenconscount)
if `consflag' {
local consname "_cons" // needed for display purposes
}
// if partial list has factor vars, will need to be replaced with tempvars
cap _fv_check_depvar `partial'
local partialfvflag =(_rc==198)
*
*** Initialize N_clust=0; will be updated to >0 by estimating programs.
local N_clust =0 // Refers to IV/OLS #clusters (not lasso #clusters)
local N_clust1 =0
local N_clust2 =0
*
*** weights ***
if `weightflag' {
// pw = aw + robust ... and more commands allow aw
local wtexp `"[aw=`exp']"'
tempvar wvar
qui gen double `wvar'=`exp'
sum `wvar' if `touse' `wtexp', meanonly
if r(min)<0 {
di as err "error - negative weights encountered"
exit 101
}
// Weight statement
di as text "(sum of wgt is " %14.4e `r(sum_w)' ")"
qui replace `wvar' = `wvar' * `N'/r(sum_w)
}
if "`weight'"=="pweight" {
// pw => robust
local robust robust
}
*
*** Shared options
// robust => ivoptions(robust) + loptions(robust)
if "`robust'" ~= "" {
local ivoptions `ivoptions' robust
local loptions `loptions' robust
local rkoptions robust
}
// cluster
if "`cluster'" ~= "" {
local ivoptions `ivoptions' cluster(`cluster')
local loptions `loptions' cluster(`cluster')
local rkoptions `rkoptions' cluster(`cluster')
}
// HAC/AC
if `bw' > 0 {
local ivoptions `ivoptions' bw(`bw') kernel(`kernel')
local loptions `loptions' bw(`bw') kernel(`kernel')
local rkoptions `rkoptions' bw(`bw') kernel(`kernel')
}
// sqrt lasso
if "`sqrt'" ~= "" {
local loptions `loptions' sqrt
local method sqrt-lasso
}
else {
local method lasso
}
*
if "`noisily'" == "" {
local qui "qui"
}
*
*** define various parameters/locals/varlists
// remove duplicates from varlist
// do this separately, list-by-list, so that factor var bases/omitteds/etc kept separate
// _o list is vars with original names, omitteds dropped, b/n/o stripped off
if "`fvstrip'"~="" {
foreach vlist in depvar dendog dexog xctrl exexog notpen aset partial {
if "``vlist''" ~= "" { // lists can be empty
fvstrip ``vlist'' if `touse', expand dropomit
local `vlist'_o `r(varlist)'
// check for duplicates has to follow expand
local dups : list dups `vlist'_o
if "`dups'"~="" {
di as text "Dropping duplicates: `dups'"
}
local `vlist'_o : list uniq `vlist'_o
}
}
}
else {
foreach vlist in depvar dendog dexog xctrl exexog notpen aset partial {
if "``vlist''" ~= "" { // lists can be empty
fvexpand ``vlist'' if `touse'
local `vlist'_o `r(varlist)'
// check for duplicates has to follow expand
local dups : list dups `vlist'_o
if "`dups'"~="" {
di as text "Dropping duplicates: `dups'"
}
local `vlist'_o : list uniq `vlist'_o
}
}
}
// Complete parsing
_pdsparse, ///
depvar(`depvar_o') ///
dendog(`dendog_o') ///
dexog(`dexog_o') ///
xctrl(`xctrl_o') ///
exexog(`exexog_o') ///
notpen(`notpen_o') ///
aset(`aset_o') ///
partial(`partial_o') /// note that partial gets special treatment - see below
debugflag(`debugflag')
// Y and endog automatically all low-dim so these are aliases
// depvar_o, dendog_o, dexog_o already exist
local varY_o `depvar_o' // same as already-existing depvar_o
// returned by PDS parser
local xhighdim_o `s(xhighdim_o)'
local xnotpen_o `s(xnotpen_o)'
local xpartial_o `s(xpartial_o)'
local xaset_o `s(xaset_o)'
local zhighdim_o `s(zhighdim_o)'
local znotpen_o `s(znotpen_o)'
local zpartial_o `s(zpartial_o)'
local zaset_o `s(zaset_o)'
// and now set xpartial flag
local xpartialflag =("`xpartial_o'"~="") | `pconscount' // =1 even if just cons being partialled out
*
*** constant in lasso estimation
// Tell lassoshooting if cons has been partialled out or there isn't one in the first place.
if `feflag' | `xpartialflag' | (`consflag'==0) {
local lsconsflag 0
}
else {
local lsconsflag 1
}
*
*** Create _t varlists: Y, X, Z, notpen, partial
// _o list is vars with original names, omitteds dropped, b/n/o stripped off
// _t list is temp vars if transform needed, original vars if not
// _f list is _o list with FVs etc. replaced by temps using fvrevar
// FEs or partialling-out FVs => everything has to be transformed
// Partialling out Xs => everything except xpartial has to be transformed
// Otherwise need temp vars only for TS and FV vars.
// note not necessary to list aset vars since they already appear in highdim lists
local allvars_o `varY_o' `dexog_o' `dendog_o' `zhighdim_o' `znotpen_o' `zpartial_o' `xhighdim_o' `xnotpen_o' `xpartial_o'
local dict_o `allvars_o'
local allbutxpartial_o `varY_o' `dexog_o' `dendog_o' `zhighdim_o' `znotpen_o' `zpartial_o' `xhighdim_o' `xnotpen_o'
if `feflag' { // everything needs to be transformed including xpartial
// so create allvars_t and allvars_f
local temp_ct : word count `allvars_o'
mata: s_maketemps(`temp_ct') // uninitialized tempvars
local allvars_t `r(varlist)'
foreach var in `allvars_o' { // we also need an _f list for allvars
fvrevar `var' if `touse' // in case of factor vars or time-series operators
local allvars_f `allvars_f' `r(varlist)' // one-by-one because otherwise omitted base vars are set etc.
} // fvrevar creates temps only when needed
local dict_t `allvars_t' // dictionary is identical to allvars lists
local dict_f `allvars_f'
if `xpartialflag' {
matchnames "`allbutxpartial_o'" "`dict_o'" "`dict_t'"
local allbutxpartial_t `r(names)' // needed in case of FE + partialling-out
}
}
else if `xpartialflag' { // everything except xpartial_o needs to be transformed
// cons if present is partialled out; weights w/o fe enters here
local temp_ct : word count `allbutxpartial_o'
mata: s_maketemps(`temp_ct') // uninitialized tempvars excluding xpartial
local allbutxpartial_t `r(varlist)'
foreach var in `allbutxpartial_o' { // we also need an _f list for allbutxpartial and xpartial
fvrevar `var' if `touse' // in case of factor vars or time-series operators
local allbutxpartial_f `allbutxpartial_f' `r(varlist)' // one-by-one because otherwise omitted base vars are set etc.
} // fvrevar creates temps only when needed
foreach var in `xpartial_o' { // same for xpartial
fvrevar `var' if `touse' //
local xpartial_f `xpartial_f' `r(varlist)' //
} //
local dict_t `allbutxpartial_t' `xpartial_o' // xpartial_t = xpartial_o (untransformed)
local dict_f `allbutxpartial_f' `xpartial_f'
}
else { // no transformation needed but still need temps
foreach var in `allvars_o' { // in case of factor vars or time-series operators
fvrevar `var' if `touse' // one-by-one because otherwise omitted base vars are set etc.
local allvars_t `allvars_t' `r(varlist)' // fvrevar creates temps only when needed
}
local allvars_f `allvars_t' // and in this case the _f and _t lists are identical
local dict_t `allvars_t' // dictionary is identical to allvars lists
local dict_f `allvars_f'
}
// dictionary is now dict_o / dict_t; dict_f is dict_o but with fv and TS vars replaced with temps
// create individual lists of _t variables with corresponding dictionary
foreach vlist in varY dendog dexog xhighdim xnotpen xpartial xaset zhighdim znotpen zpartial zaset {
matchnames "``vlist'_o'" "`dict_o'" "`dict_t'"
local `vlist'_t `r(names)' // corresponding tempnames of vars
}
*
*** More useful flags
local npxflag =("`xnotpen_o'"~="")
local npzflag =("`znotpen_o'"~="")
local hdzflag =("`zhighdim_o'"~="")
local hdxflag =("`xhighdim_o'"~="")
*** misc IV options. Change flag to 1/0. Ignore if no endog.
local idstatsflag =("`idstats'"~="" & "`dendog_o'"~="")
local firstflag =("`first'"~="" & "`dendog_o'"~="")
local sscsetflag =("`sscset'"~="" & `: word count `dendog_o''==1)
local ssgridmatflag =("`ssgridmat'"~="")
local ssomitgridflag=("`ssomitgrid'"~="")
local rlassoflag =("`rlasso'`rlasso0'"~="")
if `rlassoflag' {
if "`rlasso0'"=="" {
local rlassoprefix _`cmdname'
}
else {
local rlassoprefix `rlasso0'
}
}
*
*** More syntax checks
if `sscsetflag' {
if "`dexog_o'"~="" {
di as err "error - cannot specify sscset option with exogenous causal variables `dexog_o'"
exit 198
}
if "`znotpen_o'`zpartial_o'`zaset_o'"~="" {
di as err "error - cannot specify sscset option with unpenalized/partialled/aset instruments `znotpen_o' `zpartial_o' `zaset_o'"
exit 198
}
}
*
*** Create blank variables for orthogonalized Y and Xs and optimal IVs
// All below have already had X ctrls partialled out
// rho_y depvar, residual after high-dim Xs partialled out
// rho_d low-dim exog X, residual after high-dim Xs partialled out
// rho_e low-dim endog X, residual after high-dim Xs partialled out
// rho_z notpen IV, residual after high-dim Xs partialled out
// iv_e optimal IV for low-dim endog X
// dhat fitted value of endog d on hi-dim Xs and all Zs
// Notes:
// r1 = dendog - dhat
// dhathat = fitted value of dhat on hi-dim Xs
// r2 = dhat - dhathat = iv_e = vhat in CHS paper
// rho_e = dendog - dhathat
// = dendog - (dhat - r2) = dendog - dhat + r2
// = r1 + r2
tempvar rho_y_l
tempvar rho_y_pl
qui gen double `rho_y_l' = .
qui gen double `rho_y_pl' = .
// local rho_y_o rho_`varY_o' // prefix "rho_"
local rho_y_o `varY_o' // no prefix
char `rho_y_l'[vname] `varY_o'
char `rho_y_pl'[vname] `varY_o'
// low-dim (model) exogenous regressors
local numvars : word count `dexog_o'
forvalues i=1/`numvars' { // create only if #vars > 0
local var_o : word `i' of `dexog_o'
tempvar tvar1 tvar2
qui gen double `tvar1' = .
qui gen double `tvar2' = .
local rho_d_l `rho_d_l' `tvar1'
local rho_d_pl `rho_d_pl' `tvar2'
// local rho_d_o `rho_d_o' rho_`var_o' // prefix "rho_"
local rho_d_o `rho_d_o' `var_o' // no prefix
char `tvar1'[vname] `var_o'
char `tvar2'[vname] `var_o'
}
// low-dim (model) IVs
local numvars : word count `znotpen_o'
forvalues i=1/`numvars' { // create only if #vars > 0
local var_o : word `i' of `znotpen_o'
tempvar tvar1 tvar2
qui gen double `tvar1' = .
qui gen double `tvar2' = .
local rho_z_l `rho_z_l' `tvar1'
local rho_z_pl `rho_z_pl' `tvar2'
// local rho_z_o `rho_z_o' rho_`var_o' // prefix "rho_"
local rho_z_o `rho_z_o' `var_o' // no prefix
char `tvar1'[vname] `var_o'
char `tvar2'[vname] `var_o'
}
// low-dim (model) endogenous regressors
local numvars : word count `dendog_o'
forvalues i=1/`numvars' { // create only if #vars > 0
local var_o : word `i' of `dendog_o'
tempvar tvar1 tvar2
qui gen double `tvar1' = .
qui gen double `tvar2' = .
local rho_e_l `rho_e_l' `tvar1'
local rho_e_pl `rho_e_pl' `tvar2'
// local rho_e_o `rho_e_o' rho_`var_o' // prefix "rho_"
local rho_e_o `rho_e_o' `var_o' // no prefix
char `tvar1'[vname] `var_o'
char `tvar2'[vname] `var_o'
}
// fitted values for endogenous regressors d
local numvars : word count `dendog_o'
forvalues i=1/`numvars' { // create only if #vars > 0
local var_o : word `i' of `dendog_o'
tempvar tvar1 tvar2
qui gen double `tvar1' = .
qui gen double `tvar2' = .
local dhat_l `dhat_l' `tvar1'
local dhat_pl `dhat_pl' `tvar2'
// local dhat_o `dhat_o' hat_`var_o' // prefix "hat_"
local dhat_o `dhat_o' `var_o' // no prefix
char `tvar1'[vname] `var_o'
char `tvar2'[vname] `var_o'
}
// optimal IVs
local numvars : word count `dendog_o'
local optivflag =`numvars' // awkward but useful
forvalues i=1/`numvars' { // create only if #vars > 0
local var_o : word `i' of `dendog_o'
tempvar tvar1 tvar2
qui gen double `tvar1' = .
qui gen double `tvar2' = .
local iv_e_l `iv_e_l' `tvar1'
local iv_e_pl `iv_e_pl' `tvar2'
local iv_e_o `iv_e_o' iv_`var_o' // prefix "iv_"
// local iv_e_o `iv_e_o' `var_o' // no prefix
char `tvar1'[vname] `var_o'
char `tvar2'[vname] `var_o'
}
// create dictionary for new variables
// (don't add to dict_o/dict_t since lasso and post-lasso entries are different)
local newvars_o `rho_y_o' `rho_d_o' `rho_z_o' `rho_e_o' `dhat_o' `iv_e_o'
local newvars_l `rho_y_l' `rho_d_l' `rho_z_l' `rho_e_l' `dhat_l' `iv_e_l'
local newvars_pl `rho_y_pl' `rho_d_pl' `rho_z_pl' `rho_e_pl' `dhat_pl' `iv_e_pl'
*
******************* Partialling out ***********************************************
// If FE: partial-out FEs from temp variables, then preserve,
// then partial-out low-dim ctrls from temp variables
// restore will restore all temp vars with only FEs partialled-out
// If no FE: leave original variables unchanged.
// partial-out low-dim ctrls from temp variables.
// if no FE/low-dim ctrls, no transform needed
// Weighting takes place after FE or partialling out.
// Since constant if present is partialled out, all weighting is handled here.
// Initialize count of FEs to default of 0.
local N_g = 0
if `feflag' {
di as text "Fixed effects transformation..."
// transform everything
_fe `allvars_f', /// _f list is _o list but with FV and TS vars replaced with temps
touse(`touse') ///
tvarlist(`allvars_t') /// overwrite/initialize these
wvar(`wvar') ///
`noftools' /// provisional
fe(`ivar') //
local N_g =r(N_g) // N_g will be 0 if no FEs
local noftools `r(noftools)' // either not installed or user option
local dofopt "dofminus(`N_g')" // to pass to post-lasso estimation; empty if no FEs
preserve // preserve the original values of tempvars
// then partial out any additional X vars
if `xpartialflag' {
_partial `allbutxpartial_t', /// transform the tempvars
touse(`touse') ///
partial(`xpartial_t') /// partial out X vars only; xpartial_t vars are FE-transformed
wvar(`wvar') ///
psolver(`psolver') /// optional choice of solver
dmflag(1) // FE => data are already demeaned
}
// then transform variables by weighting if requested
if `weightflag' {
_wt `allvars_t', /// need to transform any partialling vars too since they are
touse(`touse') /// needed for the unpartialling after restore
tvarlist(`allvars_t') /// overwrite these
wvar(`wvar') //
}
}
else if `xpartialflag' { // Just partial out; includes partialling out of constant
di as text "Partialling out unpenalized controls..."
// transform everything except xpartial vars
_partial `allbutxpartial_f', /// _f list is _o list but with FV and TS vars replaced with temps
touse(`touse') ///
partial(`xpartial_f') /// partial out X vars only; xpartial_f have FV and TS vars replaced with temps
tvarlist(`allbutxpartial_t') /// overwrite/initialize these
wvar(`wvar') ///
psolver(`psolver') /// optional choice of solver
dmflag(`dmflag') // treat data as not yet demeaned unless nocons
// then transform variables by weighting if requested
if `weightflag' {
_wt `allbutxpartial_t', /// don't need to transform the partialling vars since
touse(`touse') /// since unpartialling uses the originals
tvarlist(`allbutxpartial_t') /// overwrite these
wvar(`wvar') //
}
}
*
************* partialling out END ***********************************************
************* Selection of vars ************************************************
// SelectControls selects HD vars and also creates either immunized LD vars (resids)
// or creates optimal IVs (xb option => fitted values).
local startcol =11 // used for display of messages
*** Step 1: orthogonalize depvar w.r.t. HD Xs
// All estimations but only if there are HD X vars; if no HD Xs, selected=empty and rho=y.
// Corresponds to CHS paper steps of obtaining theta^ (coefs on X) and rho^_y (resids).
// Also used in PDS method.
local msg "1. (PDS/CHS) Selecting HD controls for dep var"
SelectControls, ///
resid /// partial out hi-dim Xs from depvar
touse(`touse') ///
modelvars_o(`varY_o') ///
modelvars_t(`varY_t') ///
genvars_l(`rho_y_l') /// updates to resid or to y if nothing selected
genvars_pl(`rho_y_pl') /// updates to resid or to y if nothing selected
highdim_o(`xhighdim_o') ///
highdim_t(`xhighdim_t') ///
dict_o(`dict_o') /// dictionary
dict_t(`dict_t') /// dictionary
aset_o(`xaset_o') ///
aset_t(`xaset_t') ///
notpen_o(`xnotpen_o') ///
notpen_t(`xnotpen_t') ///
loptions(`loptions') ///
msg("`msg'") ///
startcol(`startcol') ///
debugflag(`debugflag') ///
lsconsflag(`lsconsflag') ///
dmflag(`dmflag') ///
`qui' ///
step(1) ///
rlassoflag(`rlassoflag') ///
rlassoprefix(`rlassoprefix')
local xselected1_o `r(allselected0_o)'
if `rlassoflag' {
local rlassolist `rlassolist' `r(rlassolist)'
}
*
*** Step 2: orthogonalize exog regressors w.r.t. HD Xs
// Only if there are exog regressors d; if no HD Xs, selected=empty and rho=d.
if "`dexog_o'" ~= "" {
local msg "2. (PDS/CHS) Selecting HD controls for exog regressor"
SelectControls, ///
resid /// partial out hi-dim Xs from exog regressors d
touse(`touse') ///
modelvars_o(`dexog_o') ///
modelvars_t(`dexog_t') ///
genvars_l(`rho_d_l') /// updates to resid or to d if nothing selected
genvars_pl(`rho_d_pl') /// updates to resid or to d if nothing selected
highdim_o(`xhighdim_o') ///
highdim_t(`xhighdim_t') ///
dict_o(`dict_o') /// dictionary
dict_t(`dict_t') /// dictionary
aset_o(`xaset_o') ///
aset_t(`xaset_t') ///
notpen_o(`xnotpen_o') ///
notpen_t(`xnotpen_t') ///
loptions(`loptions') ///
startcol(`startcol') ///
msg("`msg'") ///
debugflag(`debugflag') ///
lsconsflag(`lsconsflag') ///
dmflag(`dmflag') ///
`qui' ///
step(2) ///
rlassoflag(`rlassoflag') ///
rlassoprefix(`rlassoprefix')
local xselected2_o `r(allselected0_o)'
if `rlassoflag' {
local rlassolist `rlassolist' `r(rlassolist)'
}
}
*
*** Step 3: (PDS only) endog Xs need HD Xs
// PDS only; selection only, no variables generated
if "`dendog_o'" ~= "" {
local msg "3. (PDS) Selecting HD controls for endog regressor"
SelectControls, /// variable selection only, no vars generated
/* resid */ /// irrelevant since nothing generated
touse(`touse') ///
modelvars_o(`dendog_o') ///
modelvars_t(`dendog_t') ///
highdim_o(`xhighdim_o') ///
highdim_t(`xhighdim_t') ///
dict_o(`dict_o') /// dictionary
dict_t(`dict_t') /// dictionary
aset_o(`xaset_o') ///
aset_t(`xaset_t') ///
notpen_o(`xnotpen_o') ///
notpen_t(`xnotpen_t') ///
loptions(`loptions') ///
msg("`msg'") ///
startcol(`startcol') ///
debugflag(`debugflag') ///
lsconsflag(`lsconsflag') ///
dmflag(`dmflag') ///
`qui' ///
step(3) ///
rlassoflag(`rlassoflag') ///
rlassoprefix(`rlassoprefix')
local xselected3_o `r(allselected0_o)'
if `rlassoflag' {
local rlassolist `rlassolist' `r(rlassolist)'
}
}
*
*** Step 4: (PDS only) If no HD Zs, LD Zs need HD Xs
// PDS only; selection only, no variables generated
// If no HD Zs, lasso of LD Zs on HD Xs
// NB: if HD Zs exist, do with opt IV step - lasso of endog X on HD Xs, LD and HD Zs with LD Zs unpenalized.
if "`dendog_o'" ~= "" & (~`hdzflag') { // no HD IVs
local msg "4. (PDS) Selecting HD controls for IV"
SelectControls, /// variable selection only, no vars generated
/* resid */ /// irrelevant since nothing generated
touse(`touse') ///
modelvars_o(`znotpen_o' `zpartial_o') /// notpen Zs are LD Zs and need HD controls
modelvars_t(`znotpen_t' `zpartial_t') ///
highdim_o(`xhighdim_o') ///
highdim_t(`xhighdim_t') ///
dict_o(`dict_o') /// dictionary
dict_t(`dict_t') /// dictionary
aset_o(`xaset_o') ///
aset_t(`xaset_t') ///
notpen_o(`xnotpen_o') ///
notpen_t(`xnotpen_t') ///
loptions(`loptions') ///
msg("`msg'") ///
startcol(`startcol') ///
debugflag(`debugflag') ///
lsconsflag(`lsconsflag') ///
dmflag(`dmflag') ///
`qui' ///
step(4) ///
rlassoflag(`rlassoflag') ///
rlassoprefix(`rlassoprefix')
local xselected4_o `r(allselected0_o)'
if `rlassoflag' {
local rlassolist `rlassolist' `r(rlassolist)'
}
}
*
*** Step 5: endog d on hi-dim Xs and all Zs
// only if there are endogenous regressors d; get fitted values
// if no IVs selected, fitted values = 0 and model is unidentified