bugs in adonis2() or pairwise.adonis2()

[holly2021FL]

I got a p-value>0.05 based on adonis2() , but when I tried to make comparisons between every two groups using code pairwise.adonis2(), some of the p-value <0.05. I believe there are bugs in either adonis2() or pairwise.adonis2(). Hope it can be fixed.

[jarioksa]

Details, please. I think you do something wrong, and it may be useful to see what you did. Please details. In particular, we need a reproducible example. If you have concerns about pairwise.adonis2, I cannot help you: there is no such function in vegan. (Do you have your '<' and '>' correctly? Usually these are other way round in this question, that is adonis2 has P < 0.05 and pairwise.adonis does not.)

[holly2021FL]

Here is the code I used to analyze weighted UniFrac Distance:
disc.wuf<-UniFrac(physeqD, weighted=TRUE, normalized=TRUE, parallel=FALSE)
adonis2(disc.wuf ~ sample_data(physeqD)$group, perm = 999)
library(devtools)
install_github ("pmartinezarbizu/pairwiseAdonis/pairwiseAdonis")
metadata <- as( sample_data ( physeqD ), "data.frame")
pairwise.adonis2 (disc.wuf ~ group, data=metadata)
pairwiseAdonis::pairwise.adonis2(disc.wuf ~ group, data= metadata) # if the previous one doesn't work
I believe either some of my codes were wrong or one of the functions(adonis2() or pairwise.adonis2() ) has a bug.
Thanks!

[holly2021FL]

Details, please. I think you do something wrong, and it may be useful to see what you did. Please details. In particular, we need a reproducible example. If you have concerns about pairwise.adonis2, I cannot help you: there is no such function in vegan. (Do you have your '<' and '>' correctly? Usually these are other way round in this question, that is adonis2 has P < 0.05 and pairwise.adonis does not.)

Hi, I added my code

[eduardszoecs]

I believe there are bugs in either adonis2() or pairwise.adonis2().

Can you elaborate how you come to this conclusion?

Two possibilities how that could be explained come to my mind:

1. (Note, I don't know the pairwise.adonis2() function): By conducting multiple pairwise tests you increase the probability of a type I error, especially if there are many groups. That's what multiplicity adjustments like bonferroni are for. Are the pairwise p-values adjusted?
2. There are different dispersions between groups. Have you checked the dispersion (e.g. with betadisper)?

And please provide us with an reproducible example?

[jarioksa]

In addition to the reasons that @Edild gave, a very common reason for apparent discrepancy between adonis2 and similar pairwise methods (that we do not have in vegan) is that when you take a subset of data for pairwise comparison, sample sizes and the number of possible permutations are much smaller than in full data. In these cases, the situation is normally opposite to the one described here: adonis2 is "significant" (permutation p < 0.05), but all subsets of two levels of a factor are so small that permutation p < 0.05 is impossible (there are not so many different permutations). Also, unbalanced factors where paired subsets are of very different sizes may be discrepant.

This is all wild speculation, as we do not yet have a reproducible example. Not even the output of the analysis, or information on data sizes and factors.

[jarioksa]

I think there really is no need to post detailed information or reproducible example (against my earlier wish). The methods implemented in vegan::adonis2 and pairwiseAdonis::pairwise.adonis2 are so different that there is no reason to expect consistent results. There is no bug in either, but the analyses just are too different. The results are related, though, as they are based on the same kind of ideas, but they are by no means identical, and therefore you can expect different, inconsistent or even conflicting results.

Both methods are based on permutation of the data, but these are implemented in different ways. vegan::adonis2 has an overall test and permutes data freely among all factor levels for an overall analysis. pairwiseAdonis::pairwise.adonis2 in contrast takes a subsample of data for each pair of factor levels. It does not perform a full permutation, but permutes observations only within these two factor levels. It also takes a subset of response data, and therefore each pairwise comparison is based on different response data (different subset of a pair of factor levels). If the factor has an effect, two factor levels define a biased subset of data. For instance, for two similar factor levels, they define a more homogeneous subset than full data, and this influences the p-values estimated within this homogeneous set. This is most clearly seen when you analyse ordered factors.

Vegan implements pairwise permutation tests only for betadisper. These are implemented differently than in pairwise.adonis2: they are based on the same permutations as the overall test, and the observations are freely permuted over all factor levels. The pairwise statistics are evaluated together with the overall statistic in the same permutations. These pairwise statistics are consistent with the overall test.

There is also minor issue of number permutations when some classes have only a few elements. When you take only two factor levels, many of the permutations define identical classifications. For instance, in the dune.env data of 20 sampling units, factor Management has 3 cases of BF and 5 cases of HF, or total of 8 cases in this subset. You can have only choose(5+3, 3) = 56 different samples of these, and you should opt for complete enumeration. The permute package does not know for which purpose you are using the permutation indices and it does not know that only 56 cases should be produced: formally there are 8! = 40320 different permutation vectors, although most of these produce duplicates of 56 distinct combinations of factor levels. Naturally there is a similar problem in adonis2 or betadisper, but as we we work with full data matrices, the risk is essentially smaller. Management has four factor levels with counts 3, 5, 6, 6 and these give 6518191680 distinct permutations. Subsetting data and having only two levels very rapidly reduces the number of distinct permutations.

[holly2021FL]

Thanks for your explanation! And the data I used to compare have 40 samples in total, and 5 groups, there are 8 samples in each group. Hope it helps for figuring it out!