In this brief note, I wish to bring attention to a problem that has been discussed many times before, but whose implications are still not widely appreciated. As a result, many researchers (present author included) keep making distorted inferences about the relative size and importance of certain effects, by directly comparing the proportions of variance they account for. Using the proportion of explained variance as an index of effect size does not just distort the interpretation of individual effects, but also exaggerates the differences between effects, which may lead to strikingly incorrect judgements of relative importance. Luckily, a meaningful and interpretable “effect ratio” can be easily calculated as the square root of the ratio between proportions of explained variance. In several real-world examples, effect ratios tell a different story than variance components, and might prompt one to rethink the interpretation of certain canonical results (e.g., regarding the role of the shared environment in the development of psychological traits). This simple but consequential point should be understood more widely; with no pretense of originality, I hope that this note will contribute to raise awareness and prevent fallacious interpretations of research findings.

Are we comparing apples or squared apples? The proportion of explained variance exaggerates differences between effects

Del Giudice, Marco
2021-01-01

Abstract

In this brief note, I wish to bring attention to a problem that has been discussed many times before, but whose implications are still not widely appreciated. As a result, many researchers (present author included) keep making distorted inferences about the relative size and importance of certain effects, by directly comparing the proportions of variance they account for. Using the proportion of explained variance as an index of effect size does not just distort the interpretation of individual effects, but also exaggerates the differences between effects, which may lead to strikingly incorrect judgements of relative importance. Luckily, a meaningful and interpretable “effect ratio” can be easily calculated as the square root of the ratio between proportions of explained variance. In several real-world examples, effect ratios tell a different story than variance components, and might prompt one to rethink the interpretation of certain canonical results (e.g., regarding the role of the shared environment in the development of psychological traits). This simple but consequential point should be understood more widely; with no pretense of originality, I hope that this note will contribute to raise awareness and prevent fallacious interpretations of research findings.
2021
Pubblicato
https://openpsych.net/paper/61/
File in questo prodotto:
File Dimensione Formato  
Del Giudice_2021a.pdf

accesso aperto

Tipologia: Documento in Versione Editoriale
Licenza: Creative commons
Dimensione 140.17 kB
Formato Adobe PDF
140.17 kB Adobe PDF Visualizza/Apri
Pubblicazioni consigliate

I documenti in IRIS sono protetti da copyright e tutti i diritti sono riservati, salvo diversa indicazione.

Utilizza questo identificativo per citare o creare un link a questo documento: https://hdl.handle.net/11368/3066292
Citazioni
  • ???jsp.display-item.citation.pmc??? ND
  • Scopus ND
  • ???jsp.display-item.citation.isi??? ND
social impact