Review: Comparison of Brain Networks with Unknown Correspondences

Summary of reviews

Review 1 (Daniel Moyer)

  • Reviewer's confidence: 2 (Knowledgeable)
  • Overall recommendation: 1 (Probably Accept)

SUMMARY
This paper should be accepted. There are portions of both the theoretical content and presentation as well as empirical results that could be improved upon, but I believe this paper introduces interesting and relatively novel concepts to the connectomics community. Furthermore, I believe this paper could stimulate important discussions about comparisons between subject-level defined networks, an issue that may become more common in future research.

STRENGTHS
This paper is one of the few connectomics papers to consider graph metrics without node correspondence. This is induced by their use of the method introduced in [17], which produces a data driven parcellation for each individual subject; the induced graphs do not have node correspondences, and perhaps may not even have the same number nodes. This paper provides a metric that accounts for this. Graph similarity is well studied in sub-fields of computer science and mathematics, and the author nicely adapt a more mature literature to connectomics.

Since the use of individual parcellations may become more frequent, this problem is important, and the workshop would benefit from a discussion thereof. This paper introduces this problem well in a succinct manner.

SHORTCOMINGS
This paper omits specific details from the theoretical description of the metric, making it difficult to understand exactly how distances are being measured. Occasionally choices in the method seem ad hoc (the use of a log transformation in equation (1)), and/or mysterious (the nearest neighbor Hungarian algorithm, which was not fully presented).

The data violate one or more of the assumptions of the statistical tests performed. (Independence, possibly normality assumptions), and p-values are presented for only some of the results (notably, Figure 3 p-values are missing).

CONSTRUCTIVE FEEDBACK
It would be exciting and interesting to include graphical explorations of the induced latent space of brain networks. While understandably this is possibly unjust to the dimension of the space (...and adds several more layers of hyper parameter choices), displaying the top eigenvectors of the similarity matrix or its distribution of eigenvalues could be very interesting, especially as the number of subjects increases.

Due to the application of reference [17], each ROI is (hopefully) a set of voxels. However, in 2.3, the spatial edit distance is characterized by distances between representative voxels. Is this the centroid of the particular voxel set?

Is the equation for d_euclidean normalized by the inflated cortical sphere, or the mesh of the (MNI transformed) surface? I would assume the latter, but the use of the phrase "Cortical Sphere" casts doubt on this.

What are P and Q in the definition of $d_canberra$? If there are varying numbers of nodes, this is not computable. On the other hand, if these are simply the edge weights, they need no summation. If nodes are to be added, which indices should they take?

While I understand the number of tests is still rather small (approx. 50), it would still be appropriate to perform multiple test correction on the results presented in Figure 4. Furthermore, for each of these tests the data are the distances between all pairs of a fixed group of points; this means the distances are not independent. A permutation test is possibly more appropriate (...and might provide more power?). I'm assuming forms of the t-test were used for all of the presented results.

$alpha=0.5$ is found to produce the best within-group/between-group distances, having tested two other values. By what criterion is it best? Total difference? Squared difference? Statistical separation (p-values? other measures?)?

Review 2 (Olga Tymofiyeva)

  • Reviewer's confidence: 2 (Knowledgeable)
  • Overall recommendation: 2 (accept)

SUMMARY
The paper is well-written, possesses many strengths (please see below) and is timely, since it addresses the problem of comparison of networks with unknown correspondences between network edges and nodes due to different underlying subdivisions of the brain.

STRENGTHS
The method that the author suggest (a version of graph edit distance method) is tailored to brain networks and takes into account both spatial constrains as well as local network information as a node’s “signature”. It reduces the computational time by enforcing spatial constraints and can even be applied to networks with different number of nodes.

The authors tested their method on 30 unrelated subjects as well as 40 twin pairs from the Human Connectome Project, which allowed them to demonstrate that node distances obtained with their method are much smaller for twin pairs than for unrelated subjects.

The paper is timely because the conventional anatomy-based group parcellations are suffering from the underlying intersubject variability and individual parcellations are being sought, which however, give rise to unknown correspondences.

SHORTCOMINGS

The introduction does not present the need for the proposed method in a clear way. Currently, the vast majority of MRI connectome studies use anatomical alignment of brains and group-based parcellations, which don't lead to any unknown correspondences between network edges and nodes. However, due to the underlying intersubject variability of folding patters vs. histological/functional patterns and variability present in developmental and brain atrophy cases, individual-based parcellations are being sought – such as random parcellations, functional task-based or connectivity-based parcellations in individual subject’s space – which, however, give rise to unknown correspondences across subjects. This is a problem for edge-based or node-based comparison of networks (but not for comparison using global/summary network measures such as global efficiency) and leads to a need for network alignment.

A clearer introduction of the problem and some other clarifications (see comments below) would strengthen the paper.

CONSTRUCTIVE FEEDBACK
p.1, Abstract: I would rephrase the following sentence “… show that this method can accurately reflect the similarities between two networks…” as “… show that this method can accurately reflect the higher similarity between two networks in the case of twins compared to unrelated subjects…”. The paper only provides evidence for the latter.

p.1, Abstract: “these methods” should be “this method”.

p.1 I assume the sentence “Nodes in brain networks represent regions obtained with a certain parcellation technique, which often accounts for variations in anatomy and function of individual brains” should read “… which often does not account …”? After this sentence a logical link is missing to the case with unknown correspondences, maybe something like: “Individual-based parcellations are therefore being sought – such as random parcellations, functional task-based or connectivity-based parcellations in individual subject’s space – which, however, give rise to unknown correspondences”.

p.2 What does “more diverse applications” mean?

p.2 Why does (local) node centrality measure “discard a huge amount of local information”?

Difference between local and global (summary) network measures need to be presented more clearly and approaches to standard global vs local or edge-wise comparison of networks (such as the network-based statistic) may be mentioned.

Meskaldji, D.E., Fischi-Gomez, E., Griffa, A., Hagmann, P., Morgenthaler, S., Thiran, J.-P., 2013. Comparing connectomes across subjects and populations at different scales. NeuroImage, Mapping the Connectome 80, 416–425. doi:10.1016/j.neuroimage.2013.04.084

Machine learning techniques are still far less common in MRI connectomics than the more standard statistical methods described in the paper above.

p.2 Isn’t graph “modality” supposed to be graph “modularity”?

p.3 The authors performed connectivity-driven parcellation based on the following reference:

Parisot, S., Rajchl, M., Passerat-Palmbach, J., Rueckert, D., 2015. A Continuous Flow-Maximisation Approach to Connectivity-Driven Cortical Parcellation, in: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (Eds.), Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Lecture Notes in Computer Science. Springer International Publishing, pp. 165–172.

It is not clear whether both structural and functional connectivity profiles were used for that. In the cited paper only diffusion MRI data were used to build the parcellation, however, it is stated that “The method is generalisable and can be applied to both fMRI and dMRI data.” If the authors included functional connectivity as well, more details need to be provided.

p.3 I assume that all parcellations were performed in each individual’s brain space separately, not for all individuals simultaneously, in spite of the fact that the individuals were all registered to a common MNI space (p.2). (It might be obvious from the authors’ point of view, but for the reader used to the standard MRI connectome pipelines it might be helpful to state it explicitly). Was the registration to the MNI space required for pre-alignment, which allowed the authors to a) calculate the spatial cost component and b) consider “only the n spatially nearest neighbors of graph G2 for substitution by a node in G1” (p.6)?

p.3 In Figure 1 the tractography result looks predominantly red, which usually represents left-right orientation of the streamline. This might indicate erroneous reconstruction. What software was used for visualization?

p.7 “Two different sets of networks were generated for 30 healthy unrelated individuals…”: do the two sets correspond to structural and functional networks?

p.7 How was the number of q=50 parcels for the left hemisphere chosen? Should it be specified in the Methods section?

p.9 I wonder how the authors’ approach compares to the previously published study:

Tymofiyeva, O., Ziv, E., Barkovich, A.J., Hess, C.P., Xu, D., 2014. Brain without Anatomy: Construction and Comparison of Fully Network-Driven Structural MRI Connectomes. PLOS ONE 9, e96196. doi:10.1371/journal.pone.0096196

p.9 Is it an expected result that for the twin pairs nodes in the occipital and frontal lobes were more consistent compared to the parietal and temporal lobes? A reference to previous literature would be helpful.

Rebuttal

Response to Review 1

  • Graphical explorations sound reasonable and could possibly be chosen to investigate the optimal choice of parameters for the spatial and feature weights. However, this analysis is not pursued in this paper due to lack of space.
  • Each ROI is indeed a set of contiguous voxels. The representative voxel is, then, defined as the voxel with the highest correlation (in terms of connectivity profile) to the rest of the ROI. Its coordinates are therefore used as node coordinates for the spatial edit distance.
  • Equation d_euclidean is normalised by the mesh of the MNI transformed surface, ensuring correspondence between all subjects.
  • After changing the notation to make it more clear, P and Q refer to the d-dimensional feature vectors of nodes v_P and v_Q, respectively. These feature vectors have the same dimensions for all nodes compared (and include the egonet-based network features described), hence the distance is computable.
  • The comment about the statistical tests is valid and has been corrected. Initially, a paired t-test was performed by assuming independence (which is not the case here). The tests have now been replaced with non-parametric permutation tests which do not make any assumptions about normality or independence of the samples. Additionally, the statistical tests have been calculated and provided for Figure 3 in the camera-ready version.
  • $alpha=0.5$ is found to produce the best within-group/between-group distances by means of statistical separation (p-values).

Response to Reviewer 2

  • Comments on the abstract and introduction have been incorporated in the camera ready version. Edge-wise comparisons and methods like the network-based statistic are not presented in the literature review, because they cannot be used with machine learning techniques for classification or regression.
  • The newest version of the connectivity-driven parcellations used is described in detail in [17] of the final manuscript. The method is applied on both fMRI (in the case of functional networks) and dMRI data (in the case of structural networks).
  • All parcellations were performed in each individual’s brain space separately in spite of the fact that the individuals were all registered to a common MNI space. Registration to the MNI space was required for pre-alignment and this allowed us to calculate the spatial cost component and restrict node substitutions to the n-spatially-nearest neighbors.
  • The probabilistic tractography image in Figure 1 has now been updated with the tractography output for one of the subjects. Visualisation is done with FSL view.
  • The sets of networks for the unrelated individuals (presented in section 3.1) correspond to structural networks and this has been updated in the paper.
  • In comparison to the previously published study (Tymofiyeva, O., Ziv, E., Barkovich, A.J., Hess, C.P., Xu, D., 2014. Brain without Anatomy: Construction and Comparison of Fully Network-Driven Structural MRI Connectomes. PLOS ONE 9, e96196. doi:10.1371/journal.pone.0096196) which proposes a very interesting framework that accounts for inter-subject variability, the following are the main differences. First of all, connectivity matrices need to have the same number of nodes, hence the method is not applicable to networks with different numbers of nodes (which is the case for some of the single-subject parcellations using [17]). In contrast, graph edit distance takes into account node substitutions/deletions to deal with this issue. Secondly, it is not clear whether the framework could be applied on functional networks (or even networks derived from cortical thickness) in a straightforward manner. Last but not least, the main difference between the two methods is that semantic information is not used for the alignment of networks. Comparison is only performed at an edge level and it is not very clear how anatomical or task-related information about the nodes could be incorporated in that framework. Graph edit distance can deal with that since node labels are d-dimensional vectors.