Contra-Analysis

The official repository for contra-analysis, a holistic method of data analysis for controlled experiments.

This data analysis process includes:

1. Visualization of effect size from broadly related experiments.
2. Determination of the thresholds for meaningful and negligible effect size based on related experiments and real world considerations.
3. Decision-making procedure to determine which experiments show evidence of meaningful or negligible effect size.
4. Statistical hypothesis test performed within a single study to determine if the results have evidence of meaningful or negligible effect size.

This code used to generate all results and figures from the follow papers:
[1] Contra-Analysis: Prioritizing Meaningful Effect Size in Scientific Research (https://arxiv.org/abs/2210.04867).
[2] Contra-Analysis for Determining Negligible Effect Size in Scientific Research (https://arxiv.org/abs/2303.09428).
[3] The Least Difference in Means: A Statistic for Effect Size Strength and Practical Significance (https://arxiv.org/abs/2205.12958).
[4] The Most Difference in Means: A Statistic for the Strength of Null and Near-Zero Results (https://arxiv.org/abs/2201.01239).

This repository contains code necessary to compute the following statistics

1. Most difference in means (δ_M): calculated with mdm_credint() in R/contra.r, with relative=FALSE
2. Relative most difference in means (rδ_M): calculated with mdm_credint() in R/contra.r, with relative=TRUE
3. Least difference in means (δ_L): calculated with ldm_credint() in R/contra.r, with relative=FALSE
4. Relative Least difference in means (rδ_L): calculated with ldm_credint() in R/contra.r, with relative=TRUE

Note: run "set_wd_here.R" in base directoy first to set working directory to the base folder for the repository. All code assumes that is the location for the working directory.

Folder Structure

1. R/: general r code for calculating statistics, integrated risk analysis, and correlation tests

   1. candidate_stats_from_xlsx.R: functions to import data from excel files for applied examples that are used in [1-4].
   2. contra.r: functions to calculate most difference in means and least difference in means statistics that are used in contra-analysis.
   3. contra_experimental.r: experimental functions (in-progress) for alternative statistics for contra-analysis (frenquitist approaches etc.).
   4. contra_plot: visualization used for contra-analysis. Displays interval estimates of relative effect size for a collection of braodly related experiments (measure the same output variable using different model systems and disease systems)
   5. coverage_error_toolbox.R: simulation toolbox to test coverage error of mdm and rmdm (coverage error is the complement of coverage probability used for confidence intervals, code tests how often the mdm and rmdm is wrongly less than the population difference in means and rel. difference in means).
   6. credibility rate_toolbox.R: simulation toolbox to test credibility of mdm and rmdm.
   7. illustrative_plot.R: helper plot functions for simple figures in each manuscript that illustrates basic concepts.
   8. mirrored_transforms.R: helper functions for transforms required for the contra_plot
   9. parallel_utils.R: helper functions for parallel processing.
   10. row_stats_toolbox.R: helper functions for parallel processing.
   11. strength_risk_assessment.R: simulation toolbox to test candidate measures in identifying experiments with higher null strength or effect strength (see [3-4]).

2. contra_analysis_meaningful/: scripts used to generate figures for manuscript [1].

3. contra_analysis_negligible/: scripts used to generate figures for manuscript [2].

4. ldm_t/: scripts used to generate figures for manuscript [3].

5. mdm_t/: scripts used to generate figures for manuscript [4].