In Step 3 of the algorithm, adjusted p-values are derived from the raw p-values and the permutation matrix. The raw p-values are denoted byp∗j, and assume without loss of generality that p∗1 ≤p∗2 ≤ ⋯ ≤p∗
K(2)
i−1
. Otherwise, rearrange the order of the Ki(−21) statistics such that it matches the order of the raw p-values. Next, three matrices will be needed to be dened. First, a matrix of permuted distance covariance test statistics
DCiπ = (DCπ
i(j, b)),
a matrix of raw p-values
P = (pj,b), and a matrix of minima of raw p-values
Q= (qj,b),
whereqj,b= min
l=j,...,qi
pl,b and thebth column of these matrices corresponds to each permu-
tationπb. With these denitions the minP algorithm is as follow:
0.- Compute raw p-values for each hypothesis. Assume p∗1 ≤ p∗2 ≤ ⋯ ≤ p∗
K(2)
i−1
without loss of generality. Otherwise sort the Ki(−21) test statistics and corresponding raw p-values according to the ordered p∗j.
InitializeqK(2)
i−1+1,b
=1 for b=1, .., B. Initializej =Ki(−21).
1.- For the jth test statistic nTi(j) use the quick sort algorithm to get the B raw
2.- Update the successive minimaqj,b
qj,b=min(qj+1,b, pj,b), b=1, ..., B.
3.- Compute the adjusted p-values for jth test statistic nTi(j)
˜
p∗j =
#{b∶qj,b≤p∗j}
B .
4.- Do j ←j−1. If j =0, go to step 5, otherwise go to step 1. 5.- Enforce monotonicity ofp∗j: ˜ p∗1 ←p˜∗1, p˜∗j ←max(p˜∗j−1,p˜∗j) for j=2, ..., Ki(−21). 6.- SetAdjPi = {p∗1, p∗2,⋯, p∗ K(2) i−1 }.
Remember that the order of the Ki(−21) test statistics has been made to match the order of the raw p-values.
5.4 Simulation Results
Simulation studies were performed to evaluate if our permutation strategy actually preserves the FWER. The simulation were performed in the following two scenarios with varying sample size. A matrix Xn×p was generated as n sample vectors of size p from
a multivariate normal distribution, with mean vector 0p, and covariance matrix Σp×p.
The covariance matrix is block diagonal, with each block corresponding to a subset of correlated variables out of thepvariables. The block diagonal nature of the covariance
matrix creates clusters of variables corresponding to each block. The rst scenario is for the casep=20, andΣp×p is made from 2 blocks each of size10×10. In this scenario there exist 2 groups of clusters of variables. The second scenario, is for the case where p =40, and Σp×p is made from 3 blocks, one of size 20×20, and two of size 10×10. Thus, this scenario incorporates three clusters of variables with dierent cluster sizes. The sample size will take as values n =15,30,45. Note that this set up incorporates both true nulls and true alternative hypotheses.
To assess if our proposed minP permutation procedure preserves the FWER, 10,000 simulations were created. The FWER was calculated as the proportion of times one or more null hypothesis were incorrectly rejected out of all the 10,000 simulations. Each simulation iteration will correspond to one level of the tree. If the procedure controls correctly the FWER for each level, then it should control it correctly for the whole tree. This is because we control the FWER of the tree by controlling it at each level. Each row of the table 5.1 represents a simulation scenario with dierent samples sizes. From table 5.1 it can be seen that for the case p=20the method seems
Table 5.1: Simulation Results FWER at the 0.05 level
n p q Sign. n p q Sign.
15 20 190 0.030 15 40 780 0.040 30 20 190 0.031 30 40 780 0.043 45 20 190 0.035 45 40 780 0.051 n is sample size, p is the number of variables,
and q is the number of tests.
conservative with a calculated FWER of 0.030 when the sample size is 15. The method becomes less conservative as the sample size increases, reaching 0.035 when sample size is 45. For the case p =40, the calculated FWER is closer to its prespecied value of
sample size increases, the calculated FWER is very similar to the requested FWER of 0.05: in the case of n =45 the calculated FWER is 0.051. The reason the calculated FWER is closer to the requested FWER, when the number of tests increases, is because our proposed procedure takes advantage of the correlation among tests by preserving the order of the samples among some of the tests while permuting them, as described in section 5.3.5. Whenp=40, our simulation scenario generates a much larger number of pairs of variables that happen to be correlated. Hence, the amount of correlation that our procedure can use to become less conservative increases, and this makes the true FWER be closer to its specied value of 0.05.
Now, we wouldn't want to use our proposed permutation approach if it did not provide an improvement with respect to the permutation approach of Westfall and Young (1993) which permutes the data to create a null distribution of a given test, independent of other tests. Thus, we also created a simulation study where p = 15 and we vary the sample size as n = 15,30,45,100,150. In this numerical study the data generated is still multivariate normal and the covariance matrix is block diagonal with one block of size 10 and another one of size 5, which generates two clusters of dependent variables. We present in table 5.2 the results of this simulation study. The Complete Null left side of the table corresponds to the results using the method in Westfall and Young (1993) and the right side corresponds to our proposed permutation method. Across all sample sizes, our method is closer to the desired FWER of 0.05. Consequently, our method has more power than the permutation minP of Westfall and Young (1993) for correlation coecients. The reason of this is that our method incorporates some of the correlation among tests.
Table 5.2: FWER at the 0.05 Level
Complete Null Proposed Permutation
n p Actual FWER n p Actual FWER
15 15 0.014 15 15 0.032
30 15 0.024 30 15 0.033
45 15 0.026 45 15 0.035
100 15 0.035 100 15 0.039
150 15 0.033 150 15 0.042