Bring back dprime as a legitimate goal parameter?

[hneth]

When creating FFTs, FFTrees determines cue thresholds and selects and combines cues based on the goal.threshold, goal.chase and goal parameters.

Although this worked in an earlier version and dprime is still noted as a goal_valid when creating FFTs, actually using it (as any of the goal parameters) currently fails.
As some colleagues favor it as the measure to maximize, a first question is:

1. Should we bring back dprime as a legitimate measure to maximize?

Reflecting on this, I see two related questions:

2. As I'm not sure whether the $z$-transformation required to compute dprime makes sense for typical datasets, we could simply maximize the difference of hit rate minus false alarm rate (aka. "column power" or
   $\Delta P_c = \text{sens} - (1 - \text{spec})$, see Table 3 of 10.3389/fpsyg.2020.567817 for details).

This immediately raises the question:

4. Is there an analog measure in the orthogonal "prediction direction" that maximizes
   the difference between $\text{ppv}$ and $1 - \text{npv}$? The answer is yes, of course:
   $\Delta P_r = \text{ppv} - (1 - \text{npv})$?

Hence, I wonder:

• Has anyone systematically compared the results of those measures with each other?

[hneth]

An update on this issue

The latest commits (PR #130 and #131) enabled using dprime as the goal of cue thresholds (goal.threshold), FFT construction (goal.chase), and FFT selection (goal). Additionally, dprime has been added to stored statistics of thresholds and results of competing algorithms (for train and test data).

Although the new options of optimizing for dprime have yet to be tested more thoroughly, initial checks suggest that gains in dprime often coincide with substantial losses on accuracy measures. Thus, optimizing FFTs for dprime may not be as important or helpful as some articles suggest.

PS: I'm still waiting for an answer on the analog measure in the orthogonal "prediction direction" / $\Delta P_r$ (see above).

[hneth]

P.S.

The latest additions to FFTrees() (see PR #145) enable users to maximize a custom my.goal (defined by my.goal.fun). This allows optimizing FFTs for a wide range of measures, including adopting a row-based perspective (e.g., PPV, NPV, $\Delta P_r$).