What Could the Statistical Synthesis of Research Findings Contribute to the Disciplined Inquiry for Education?

<p>  This paper discusses issues related to the contribution of statistical synthesis of research findings, such as meta-analysis and synthesizing a number of existing meta-analysis results by means of the same procedure as meta-analysis (i.e., super-synthesis), which could play a dual role of disciplined inquiry for education in understanding the chosen phenomenon and conceptualizing theories, and providing information wanted by decision-makers. We describe the brief history that Japanese educational psychologists have accepted meta-analysis and its recent research trends, and indicate the significance that the statistical synthesis of research findings could have. Specifically, it has affected attitudes to drawing conclusions from the accumulated literature through considering primary research as an individual sample. It is also pointed out that the widespread adoption of statistical research synthesis might have led both researchers and decision-makers to an overly simplistic view, which focuses only on the effect size and has encouraged the rough use of research findings, with the background of a growing debate on promoting evidence-based education. We then discuss the following: The statistical synthesis of research findings makes reasonable contributions to understanding phenomena and establishing theories. However, meta-analysis based on individual data is required to draw causal inferences, and in order to do so, it will be necessary to promote open science. Also, there will be some potential risks that statistical research synthesis can narrow classroom practitioners' choices, and therefore weaken their autonomy, because it can be considered as the high-level of evidence and its results tend to be presented in terms of simple indices such as mean effect size. Throughout this discussion, the following conclusions are drawn. For understanding phenomena and establishing theories, the statistical synthesis of research findings can be an effective methodology to improve the external validity of the existing research literature, and evaluate the relevance or validity of psychological constructs. Thereby, it is supposed to improve the accuracy and certainty of theories of educational psychology. These findings can provide information on "what works" and clear guiding principles that decision-makers demand. The issues pointed out in this discussion call into question the impact of educational psychological research findings on areas outside of their discipline, as well as some problems related to the statistical research synthesis. Finally, we argue that in order to promote both roles of disciplined inquiry for education with statistical research syntheses, educational psychologists and experts from other fields should be conscious of the need for communicating warranted findings with each other.</p>