|
nif:isString
|
-
This study was conducted according to the principles expressed in the Declaration of Helsinki. The study was approved by the local ethics committee (Kantonale Ethikkomission: EK-80/2008). All participants provided written informed consent for the collection of samples and subsequent analysis.
Example 1 - real data: For the first illustration of the problem associated with this approach we used EEG data from a previous study [19]. Seventy-four healthy male students (mean/standard deviation: 25,5/4.86 years) participated in the study. After recording seven minutes of spontaneous EEG at rest, subjects conducted the Raven Advanced Progressive Matrices (RAPM) [20], which is a widely used measure of psychometric intelligence. In contrast to the previous study [19], we performed a median-split based on the performance in the RAPM. This resulted in a high IQ group (n = 25) and a low IQ group (n = 34). The median raw score was 23 correctly solved items. Subjects who scored at the median level of the RAPM were excluded. Spontaneous EEG at rest was used to analyze connectivity parameters of intracortical sources of brain oscillations in the upper alpha band (10,5–12 Hz). The coherence between 84 anatomical regions of interest in both hemispheres was computed (for the details of the analyses see [19]). This resulted in an 84×84 correlation matrix (84 ROIs) for each subject. The connectivity matrices of all subjects from the low IQ group to the high IQ group were averaged separately, resulting in a mean connectivity matrix for the low IQ and high IQ groups. The connectivity matrices were then thresholded at different coherence values. This multiple-threshold approach resulted in as many networks per group as the number of thresholds applied to the connectivity matrix. Network parameters (clustering coefficient, characteristic path length and number of edges) were then calculated for each connectivity matrix by using the tnet software [21]. In order to draw statistical inferences regarding group differences in network parameters, such as, the clustering coefficient and the characteristic path length, we used a classical parametric statistical test (t-test for independent samples). As mentioned above, this multiple-threshold-approach is problematic because both the sample size and the statistical power depend on the number of thresholds used. In addition, the key assumption of independency between samples in t-tests is violated when using differently thresholded correlation matrices. We demonstrated this by using three different numbers of thresholds while keeping the ranges constant (range: 0.65–0.99). The sparsest network (threshold r = 0.99) was omitted, because the networks became no longer consistent. In the first trial we thresholded the connectivity matrix 10 times (increments: 0.034) resulting in 10 networks per group, in the second trial we thresholded the connectivity matrix 15 times (increments: 0.0227), and in the third trial we thresholded the connectivity matrix 35 times (increments 0.01). In a second step, the small-world parameters were calculated for each threshold per group. The different thresholded networks served as the different measurements units within each group. Thus, in the first trial we obtained 10 measurements for each small-world parameter, in the second trial we obtained 15 measurements for each small-world parameter, and in the third trial we obtained 35 measurements for each small-world parameter. Afterwards, we separately compared these small-world parameters between the low IQ and the high IQ groups for each trial by using a t-test for independent samples (p<0.05). Since we have to consider the fact that p-values depend on sample size, we also calculated effect sizes according to Cohen [22]. All statistical analyses in the present study were performed with MATLAB [23].
For the first trial (thresholding the matrix 10 times), there were no significant differences between the low and the high IQ groups regarding small-world parameters (clustering coefficient: t(8) = 1.87, p = 0.078, Cohen’s d = 0.42; path length: t(8) = −1.30, p = 0.21, Cohen’s d = 0.31; number of edges: t(8) = 1.85, p = 0.08, Cohen’s d = 0.42). For the second trial (thresholding the matrix 15 times), we found significantly more edges (t(13) = 2.40, p = 0.02, Cohen’s d = 0.38), a higher cluster coefficient (t(13) = 3.07, p = 0.004, Cohen’s d = 0.46), and no differences regarding characteristic path length (t(13) = −1.51, p = 0.14, Cohen’s d = 0.25) for the high IQ group compared to the low IQ group. For the third trial (thresholding the matrix 35 times), t-tests revealed highly significant differences between the high and the low IQ groups. There was a significantly increased number of edges (t(33) = 3.52, p = 7.76*10−4, Cohen’s d = 0.39), and a higher clustering coefficient (t(33) = 4.44, p = 3.33*10−5, Cohen’s d = 0.47) in the high IQ group. In contrast, we found a significantly decreased characteristic path length (t(33) = −2.24, p = 0.02, Cohen’s d = 0.26). An overview of this data is presented in Figure 1.
Figure data removed from full text. Figure identifier and caption: 10.1371/journal.pone.0053199.g001 Results of the multiple-thresholds-approach of the example with the real data.Mean values for the small-world parameters clustering coefficient, path length, and number of edges. We thresholded the correlation matrix 10, 15, and 35 times; this resulted in different statistical results. For the version with 10 increments, t-tests revealed no statistical differences. For the version with 15 increments, the clustering coefficient and number of edges was significantly increased in the high IQ group compared to the low IQ group. In the version with 35 different thresholds, the comparison between the high and low IQ groups revealed significant effects for all small-world parameters. The high IQ group showed a significantly enhanced small-world topology. For an optimized display, the numbers of edges were scaled (number of edges divided by 1000).
In our second example, we use a simulation to illustrate how the commonly used multiple-threshold-approach may lead to false positive results. An illustration of the method is displayed in Figure 2. We set up our simulation to mimic the multiple-threshold-approach with data obtained by structural MRI or FA-DTI data. We simulated a study with 60 subjects, who comprised two experimental groups of equal size (30 subjects per group). This is a commonly used sample size for studies conducted in this field [2], [24], [25]. As in the first example, we used 84 brain regions (e.g. 84 Brodmann Areas). A randomly created value of a z-distribution was allocated for each of the 84 brain regions. This was done separately for each subject. Since we only have one value per node and sample, there is no possibility of calculating a correlation matrix for a single subject. Therefore, in order to calculate the strength of the association between nodes, we needed to calculate correlations between the nodes (84 brain regions) of each group. This results in two association matrices with 84 rows and columns. Each entry of the row and column represents the correlation coefficient (connectivity strength) between the two simulated brain regions. Since there is now only one network per group, the groups cannot be statistically compared at this stage. We followed the common multiple-threshold-approach to “deal” with this problem by thresholding the two networks over a set of thresholds (range: 0.01–0.91; increments: 0.001, total: 900); this resulted in 900 networks per group. For each thresholded network we then obtained the small-world parameters, namely, the number of edges, the clustering coefficient, and the characteristic path length by using the tnet software [21]. To compare the small-world network parameters, t-tests for independent samples (p<0.05) were used, which is common practice. We calculated three examples (three different threshold ranges) with the simulation data because we aimed to replicate the analysis and to demonstrate that in addition to the number of thresholds, the threshold limits (upper and lower threshold of the threshold range) might influence the results. In the first step, we extracted three different threshold ranges between 0.01 and 0.91. A low threshold range (0.01–0.06), a middle threshold range (0.50–0.54), and a high threshold range (0.86–0.91) were chosen. This resulted in 50 differently thresholded connectivity matrices per group within the threshold range. Analog to the example of the real data, we compared the networks of the two simulated groups over different numbers of thresholds. The different thresholded connectivity matrices served as the different measurement units within each group. In the first trial, we took 10 differently thresholded connectivity matrices (increments: 0.005) for the group comparison using independent t-tests. In the second trial, we calculated with 25 connectivity matrices (increments: 0.002) per group. In the third trial, we calculated with 50 connectivity matrices (increments: 0.001) per group. This was done for the three threshold ranges (0.01–0.06; 0.50–0.54; 0.86–0.91). Because the networks were randomly generated, we hypothesized that there would be no differences between the networks of the two groups in any small-world parameter.
Figure data removed from full text. Figure identifier and caption: 10.1371/journal.pone.0053199.g002 Procedure of the multiple-thresholds-approach with artificial data.Networks of two groups based on artificial data. The networks were thresholded over a set of thresholds.
Comparing the random networks of the two simulated groups for the first trial (10 thresholded connectivity matrices) within the threshold range of 0.86–0.91 revealed no significant difference in any of the small-world parameters. In the second trial (25 thresholded connectivity matrices), we found significantly more edges (t(23) = 2.18, p, = 0.03, Cohen’s d = 0.29) and a lower characteristic path length (t(23) = −2.09, p, = 0.04, Cohen’s d = 0.28) for the first group. For the third trial (50 thresholded connectivity matrices), the t-tests revealed highly significant differences between the two simulated groups. There was also a significant increase in the number of edges (t(48) = 3.19, p, = 0.002, Cohen’s d = 0.30) and in the clustering coefficient (t(48) = 2.20, p, = 0.03, Cohen’s d = 0.21). In contrast, we found a significant decrease in the characteristic path length (t(48) = −3.05. p = 0.003, Cohen’s d = 0.29). Within the middle threshold range (0.50–0.54), there were no significant differences between the random networks of the two simulated groups in the first trial (10 thresholded connectivity matrices). However, for the second trial (25 thresholded connectivity matrices) there were only significant differences in the clustering coefficient (t(23) = −2.19, p, = 0.03, Cohen’s d = 0.29) between the two simulated groups. The analysis of the number of edges displayed a trend to decreased number of edges in group one (t(23) = −1.83, p, = 0.07, Cohen’s d = 0.23). In the third trial (50 thresholded connectivity matrices), the random network of the first group showed a decreased number of edges (t(48) = −2.61, p = 0.03, Cohen’s d = 0.25) and a decreased clustering coefficient (t(48) = −2.97, p = 0.02, Cohen’s d = 0.28) compared to the random network of the second group. The path length of the first group was significantly higher (t(48) = 2.24, p = 0.03, Cohen’s d = 0.22). For the lower threshold range (0.001–0.06), the first and second trials revealed no significant differences, but the third trial showed (50 thresholded connectivity matrices) a lower number of edges (t(48) = −2.41, p = 0.02, Cohen’s d = 0.23) and a lower clustering coefficient (t(48) = −2.21, p = 0.03, Cohen’s d = 0.22) for the first group’s random network. All the results are presented in Figure 3.
Figure data removed from full text. Figure identifier and caption: 10.1371/journal.pone.0053199.g003 Results of the multiple-thresholds-approach of the example with the artificial data.Displayed are the results of the second example, which used artificial data. The comparison of the two networks, based on artificial data, revealed several significant differences. Depending on the number of thresholds (defining the different measurement units within each group) and the threshold range used for the comparison, completely distinct results could be obtained. For an optimized display, the numbers of edges were scaled (number of edges divided by 1000).
Example 1 - real data: The same data set was used as in the first example, which made use of multiple-thresholds-approach (see above). In line with the first example using the multiple-thresholds-approach, we created a mean connectivity matrix (averaged across all subjects), which was then thresholded with a set of different thresholds (range r = 0.55–0.95, increments: 0.05). In the second step, small-world network parameters (clustering coefficient, path length) were calculated for the different thresholded mean coherence matrices. Here we present the results for the particular chosen threshold that best corresponds to a small-world topology (r = 0.85). This threshold was applied to the mean connectivity matrices of the low and high IQ groups. This is only one of several possible approaches to choosing a threshold. In the upcoming discussion section we delineate the other possibilities. For more information regarding the results of the other thresholds please consider Table S1 and Figure S1. As in the first example of the multiple-thresholds-approach, the subjects were allocated to a high or to a low IQ group based on a median-split, as previously described. The small-world network parameters were then calculated for the equally thresholded (threshold r = 0.85) connectivity matrices of the low and the high IQ groups. The small-world network parameters of the high IQ group were then subtracted from the parameters of the low IQ group. In order to statistically test these differences, we used permutation statistics. Permutation tests are a sub-group of non-parametric statistics. The basic principle has originally been described by Fisher [26] and has been extended by others [27], [28], [29], [30]. The principle assumption is that within a test group all subjects are equivalent and that every subject is the same before sampling started [31]. From this point, one can compute a statistic and then observe the amount to which this statistic is distinctive by comparing the test statistics under rearrangements of the treatment assignments [26]. In contrast to classical parametric tests, which rely on theoretical probability distributions, permutation tests can be applied when the assumptions of parametric tests are untenable [31]. In situations where it is not feasible to compute the statistics for all the rearrangements, as is required in the Fisher’s exact test, a subsample can be used [27], [28]. Such a test is sometimes known as an approximate permutation test, because the permutation distribution is approximated by a subsample, also known as Monte-Carlo permutation tests or random permutation tests [31]. In the present study, we used the Edgington approach. To this end, we allocated the subjects randomly to one of two groups and created 1000 randomly assigned pairs of groups. For each random group pair, we calculated the mean correlation matrix and then the small-world parameters of the networks. In the second step of analysis, the differences in small-world network parameters between the pairs were obtained. To statistically prove the real differences between the high and the low IQ groups, we tested the real differences within the distribution of the randomly generated differences and a global level of significance was set at p<0.05. When setting the error probability to p<0.05, the real difference must exceed the extreme of 5% of the difference distribution, in order to reach statistical significance.
The permutation analysis revealed that the high IQ group demonstrate significantly more edges than the low IQ group (p<0.001). Moreover, we found an increased clustering coefficient (p<0.001) and a decreased characteristic path length (p = 0.004) for the high IQ group compared to the low IQ group. Thus, the high IQ group exhibits significantly more small-world topology. All results are summarized in Figure 4.
Figure data removed from full text. Figure identifier and caption: 10.1371/journal.pone.0053199.g004 Results of the group-level-permutation-statistics-approach of the example with the real data.Displayed are the distributions of the randomly generated group pair differences. The red arrow indicates where the differences of the real data ( = empirical difference between high and low IQ groups) are located within the distribution. The results show that the high IQ group revealed increased small-world network parameters.
Example 2 - simulated data: In the present example, we used the same data set as in the second example of the multiple-thresholds-approach with artificial random networks (connectivity matrices). Again we assume to have two different groups with 30 subjects per group, but there is only one value per node and subject (i.e. cortical thickness or FA value in this specific region). We again have 84 simulated brain regions per subject, where we again allocated random values to each simulated brain region for each single subject. These data were used to construct the correlation matrix between all pairs of nodes, resulting in an 84×84 association matrix (network) for each group. They served as representation for the networks of two different groups. However, instead of using the multiple-thresholds-approach, we now use no particular threshold, calculate the network parameters on the basis of the unthresholded data set, and subject these parameters to randomization tests, in order to conduct between-group comparisons. Since we did not threshold the connectivity matrices in this particular analysis, all connectivity matrices have an equal number of edges. Therefore, the between-group comparison of the number of edges is obsolete. Using unthresholded networks is only reasonable in the case of weighted networks (if every node is connected to every other node). Other alternative and valid approaches are discussed in the discussion section. However, the same procedure could also be applied to thresholded connectivity matrices. In the second step of analysis, we only computed the difference between the small-world parameters of the two groups. To statistically bolster this difference, we performed between-groups randomization tests by calculating different small-world parameters on the basis of 1000 randomized assignments of the subjects to the groups. We then computed a correlation matrix and small-world parameters for each randomization. This resulted in 1000 random group pairs. As for the originally assigned group, we again calculated the difference between the small-world parameters for each of the 1000 random group pairs, which resulted in 1000 difference values. These randomly achieved difference values now form the test-distribution and the difference of our originally assigned group of interest can now be tested using this distribution. A global level of significance was set at p = 0.05. Since all groups were randomly generated, we assumed that there would be no significant differences regarding small-world topology.
The permutation analysis revealed no significant differences regarding the clustering coefficient (p = 0.46, p>0.05) or the characteristic path length (p = 0.88, p>0.05). The results are illustrated in Figure 5.
Figure data removed from full text. Figure identifier and caption: 10.1371/journal.pone.0053199.g005 Results of the group-level-permutation-statistics-approach of the example with the artificial data.Displayed are the distributions of the randomly generated group pair differences. The red arrow indicates where the differences of the original data are located within the distribution. The results show, that there are no significant differences regarding the clustering coefficient or the characteristic path length.
The same data set was used as in Example 1 of the multiple-thresholds-approach and the group-level-permutation-statistics-approach (See above). In contrast to the two previous methods, we now used the correlation matrix of each subject instead of averaging the connectivity matrices over the entire group. The correlation matrices were thresholded by applying a set of different thresholds (r = 0.65–0.95, increments: 0.05). The particular threshold, which identified the best small-world topology was chosen (r = 0.85) and applied to the correlation matrices of each individual subject. For the results of the other thresholds, please refer to (Table S2). Obtaining single subject correlation matrices is only available for times series data (e.g. fMRI, EEG, MEG) or DTI with fibre tractography. Subsequently the correlation matrix of each subject was subjected to tnet software [21], [32], [33], which calculated the small-world indices for each individual subject (for further details see [19]). For statistical comparisons of the small-world networks, we compared the subjects of the low IQ group with those of the high IQ group (based on median-split in the RAPM performance) by calculating a t-test for independent samples. The global level of significance was set at p<0.05. Another possibility would be to calculate a regression analysis between the performance in the intelligence task and the small-world parameters, as was done in our previous study [19].
The t-test for independent samples comparing the high IQ group vs. the low IQ group revealed a significantly increased number of edges (t(57) = 2.83, p = 0.006), a significantly increased clustering coefficient (t(57) = 3.54, p = 0.001), and a significantly decreased characteristic path length (t(57) = −2.70, p = 0.009) (See [19], for the results of the regression analysis).
In this example we used a similar data set as in the second example of the multiple-thresholds-approach and the group-level-permutation-statistics-approach with artificial random networks. Again, we assume to have two different groups with 30 subjects per group, but in this example we assume that each subject has an individual network, as is the case for MEG, EEG, resting fMRI, and DTI data when using tractography. We artificially created 60 networks with 84 nodes per network; representing for each subject a particular network. Subsequently the unthresholded weighted correlation matrix of each subject was subjected to tnet software, which calculated the small-world indices (clustering coefficient and characteristic path length) for each individual subject. The two groups were then compared with a t-test for independent samples (threshold was set p<0.05).
The t-test for independent samples comparing the two groups did not reveal significant effects for the clustering coefficient (t(58) = 0.24, p = 0.81) or for the characteristic path length (t(58) = 0.56, p = 0.58).
|
![[This page as RDF]](http://data.gesis.org/somesci/static/rdf-icon.gif){kind=icon}