A Systematic Review of Research Designs and Tests Used for Quantification of Treatment Effects in <i>ARELE </i>13–28

DOI Open Access

Abstract

<p>  The present study reviewed the quality of quantitative research in Annual Review of English Language Education in Japan (ARELE) volumes 13 to 28 from two perspectives: research designs and test traits. In articles on foreign language pedagogy including ARELE, quantitative data have been used as scientific evidence to determine if a teaching method is effective. This requires us to revisit the types of research designs and tests that have been employed to quantify the efficacy of instructions used in English education in Japan. In this systematic review, we targeted research that claimed a causal relationship between a pedagogical intervention and its outcome and coded 398 ARELE articles according to research designs and test traits for a meta-analysis. Results showed that 59% of primary studies adopted less appropriate designs involving threats to internal validity. Similarly, research that used less appropriate tests was more likely to overestimate and/or underestimate the treatment effects. Although our research community has continuously provided information about statistical reforms, such as statistical power and statistical indices to be reported, findings further suggest that we should revisit methods of designing our quantitative research.</p>

Journal

Related Projects

See more

Details

  • CRID
    1390001288141927168
  • NII Article ID
    130007622690
  • DOI
    10.20581/arele.29.0_129
  • ISSN
    24320412
    13448560
  • Text Lang
    en
  • Data Source
    • JaLC
    • CiNii Articles
    • KAKEN
  • Abstract License Flag
    Disallowed

Report a problem

Back to top